Electronic device and method for processing user utterance

Abstract

An electronic device and a method of processing a user utterance are disclosed. An operation method of an electronic device (101, 201, 501) according to an embodiment may include, in response to obtaining a portion of a user utterance, outputting an operation of a first response (63) to the portion. The method may include obtaining an operation of a user input (67) for the first response. The method may include an operation of outputting a second response (65) to the user utterance based on the user input (67) in response to termination of the user utterance. Other embodiments are also possible.

Description

Technical Field

[0001] The following disclosure relates to an electronic device and a user speech processing method. Background Technology

[0002] Applications related to speech recognition and / or natural language processing (such as voice assistants) can utilize large language models to process user speech.

[0003] The above information is presented as relevant technical information to aid in understanding this disclosure. No claim or judgment is made regarding whether anything described above can be considered prior art in connection with this disclosure.

[0004] To improve user convenience, technologies are needed that can shorten the time interval between user input and system response and / or reduce query-related costs.

[0005] The technical objectives to be achieved are not limited to those mentioned above, and other unmentioned technical objectives can be clearly understood by those skilled in the art through the following description. Summary of the Invention

[0006] Technical solution

[0007] An electronic device 101, 201, or 501 according to an embodiment includes at least one processor 120, 203, or 620 and a memory 130, 207, or 630 storing instructions. When the instructions are executed individually or jointly by the at least one processor 120, 203, or 620, the electronic device 101, 201, or 501 can perform a plurality of operations. The plurality of operations may include outputting a first response 63 for said portion in response to receiving a portion of a user utterance. The plurality of operations may include receiving user input 67 for said first response. The plurality of operations may include outputting a second response 65 for said user utterance based on said user input 67 in response to termination of said user utterance. Outputting the first response 63 may include retrieving data from the memory 130, 207, or 630 associated with an entity or a combination of entities contained in said portion of said user utterance. Outputting the first response 63 may include outputting a list of items associated with said entity or combination of entities based on the retrieved data.

[0008] An electronic device 101, 201, or 501 according to an embodiment includes at least one processor 120, 203, or 620 and a memory 130, 207, or 630 storing instructions. When the instructions are executed individually or jointly by the at least one processor 120, 203, or 620, the electronic device 101, 201, or 501 can perform a plurality of operations. The plurality of operations may include: identifying an entity from a portion of the user's speech before the user's speech is terminated; the plurality of operations may include obtaining data related to the entity from a data source in response to the identification of the entity; and the plurality of operations may include outputting information about the entity based on the data before the user's speech is terminated.

[0009] A method for operating an electronic device 101, 201, or 501 according to an embodiment includes: outputting a first response 63 for a portion of a user's speech in response to acquiring that portion; the method includes acquiring user input 67 for the first response; and the method includes outputting a second response 65 for the user's speech based on the user input 67 in response to termination of the user's speech.

[0010] According to an embodiment, a computer-readable storage medium storing one or more computer programs may include instructions that cause a processor to perform the methods described above. Attached Figure Description

[0011] Figure 1 This is a block diagram of an electronic device in a network environment according to an embodiment.

[0012] Figure 2 This is a block diagram of an integrated intelligent system according to an embodiment.

[0013] Figure 3 This is a schematic diagram illustrating the storage format of information relating concepts and actions in a database (DB) according to an embodiment.

[0014] Figure 4 This is a schematic diagram of a screen showing an electronic device according to an embodiment processing received voice input via a smart application.

[0015] Figure 5 This is a schematic diagram of the speech processing flow of an electronic device according to an embodiment.

[0016] Figure 6 This is a schematic block diagram of an electronic device according to an embodiment.

[0017] Figure 7 This is an example schematic diagram of the speech processing flow according to an embodiment.

[0018] Figure 8 This is an example schematic diagram of the speech processing flow according to an embodiment.

[0019] Figure 9 This is an example schematic diagram of the speech processing flow according to an embodiment.

[0020] Figure 10 This is an example schematic diagram of the speech processing flow according to an embodiment.

[0021] Figure 11 This is an example schematic diagram of the speech processing flow according to an embodiment.

[0022] Figure 12 This is an example schematic diagram of the speech processing flow according to an embodiment.

[0023] Figure 13 This is an example schematic diagram of the speech processing flow according to an embodiment.

[0024] Figure 14 This is an operation flowchart of an electronic device according to an embodiment.

[0025] Figure 15 This is an operation flowchart of an electronic device according to an embodiment. Detailed Implementation

[0026] Hereinafter, embodiments are described in detail with reference to the accompanying drawings. When describing embodiments with reference to the accompanying drawings, the same reference numerals refer to the same components, and related repeated descriptions will be omitted.

[0027] Figure 1 This is a block diagram of an electronic device 101 in a network environment 100 according to an embodiment. (See reference...) Figure 1 In network environment 100, electronic device 101 can communicate with electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or with at least one of electronic device 104 or server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, electronic device 101 can communicate with electronic device 104 via server 108. According to an embodiment, electronic device 101 may include a processor 120, memory 130, input module 150, sound output module 155, display module 160, audio module 170, sensor module 176, interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, user identification module (SIM) 196, or antenna module 197. In some embodiments, electronic device 101 may omit at least one component (e.g., connection terminal 178), or add one or more other components. In some embodiments, some components (e.g., sensor module 176, camera module 180, or antenna module 197) may be implemented as a single component (e.g., display module 160).

[0028] Processor 120 may, for example, execute software (e.g., program 140) to control at least one other component (e.g., hardware or software component) of electronic device 101 coupled to processor 120, and may perform various data processing or calculations. According to embodiments, as at least part of data processing or calculation, processor 120 may store instructions or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132, process the instructions or data stored in volatile memory 132, and store the resulting data in non-volatile memory 134. According to embodiments, processor 120 may include a main processor 121 (e.g., central processing unit (CPU) or application processor (AP)), or an auxiliary processor 123 (e.g., graphics processing unit (GPU), neural processing unit (NPU), image signal processor (ISP), sensor central processor, or communication processor (CP)) that may operate independently of or in conjunction with main processor 121. For example, when electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121 or as part of the main processor 121.

[0029] When the main processor 121 is inactive (e.g., in a sleep state), the auxiliary processor 123 may replace the main processor 121 in controlling at least some functions or states of the components of the electronic device 101 associated with at least one component (e.g., display module 160, sensor module 176, or communication module 190), or may cooperate with the main processor 121 in co-controlling when the main processor 121 is active (e.g., executing an application). According to embodiments, the auxiliary processor 123 (e.g., ISP or CP) may be implemented as part of another component (e.g., camera module 180 or communication module 190) associated with the functionality of the auxiliary processor 123. According to embodiments, the auxiliary processor 123 (e.g., NPU) may include hardware architecture dedicated to artificial intelligence (AI) model processing. The AI ​​model may be generated through machine learning. Such learning may be performed, for example, on the electronic device 101 performing the AI ​​or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The AI ​​model may include multiple layers of artificial neural networks. Artificial neural networks can be deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), deep Q-networks, or combinations of two or more thereof, but are not limited to these. AI models may additionally or alternatively include software structures in addition to hardware structures.

[0030] Memory 130 may store various data used by at least one component of electronic device 101 (e.g., processor 120 or sensor module 176). The various data may include, for example, input or output data of software (e.g., program 140) and associated instructions. Memory 130 may include volatile memory 132 or non-volatile memory 134. According to embodiments, when instructions stored in memory 130 are executed individually or jointly by at least one processor (e.g., main processor 121 and / or auxiliary processor 123), the instructions may cause electronic device 101 to perform one or more operations. For example, instructions stored in memory 130 may be executed by a single processor (e.g., main processor 121 or auxiliary processor 123 such as CP) or by multiple processors operating collaboratively (e.g., main processor 121 and auxiliary processor 123).

[0031] Program 140 may be stored as software in memory 130 and may include, for example, an operating system (OS) 142, middleware 144, or application 146.

[0032] The input module 150 can receive instructions or data from outside the electronic device 101 (e.g., a user) that will be used by another component of the electronic device 101 (e.g., a processor 120). The input module 150 may include, for example, a microphone, a mouse, a keyboard, buttons (e.g., keys), or a digital pen (e.g., a stylus).

[0033] The audio output module 155 can output audio signals to the outside of the electronic device 101. The audio output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as playing multimedia or recording. The receiver can be used to answer incoming calls. According to an embodiment, the receiver may be implemented separately from the speaker or as part of the speaker.

[0034] Display module 160 can provide information visually to the outside of electronic device 101 (e.g., to a user). Display module 160 may include, for example, a display, a holographic device, or a projector, and control circuitry for controlling a respective one of the display, holographic device, and projector. According to an embodiment, display module 160 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the intensity of the force generated by the touch.

[0035] The audio module 170 can convert sound into electrical signals and vice versa. According to an embodiment, the audio module 170 can acquire sound through the input module 150, or output sound through the sound output module 155 or through an external electronic device (e.g., electronic device 102) (e.g., a speaker or headphones) directly or wirelessly connected to the electronic device 101.

[0036] Sensor module 176 can detect the operating state of electronic device 101 (e.g., power or temperature) or the environmental state outside electronic device 101 (e.g., user state), and then generate an electrical signal or data value corresponding to the detected state. According to embodiments, sensor module 176 may include, for example, a gesture sensor, gyroscope sensor, barometric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biometric sensor, temperature sensor, humidity sensor, or illuminance sensor.

[0037] Interface 177 may support one or more specified protocols for direct or wireless coupling of electronic device 101 to external electronic device (e.g., electronic device 102). According to embodiments, interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital card (SD) card interface, or an audio interface.

[0038] The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (e.g., electronic device 102). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0039] The haptic module 179 can convert electrical signals into mechanical stimuli (e.g., vibration or motion) or electrical stimuli that a user can perceive through touch or kinesthesia. According to embodiments, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0040] Camera module 180 can capture still images and moving images. According to an embodiment, camera module 180 may include one or more lenses, an image sensor, an ISP, or a flash.

[0041] The power management module 188 manages the power supplied to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0042] Battery 189 can supply power to at least one component of electronic device 101. According to an embodiment, battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0043] Communication module 190 can support the establishment of a direct (e.g., wired) or wireless communication channel between electronic device 101 and external electronic devices (e.g., electronic device 102, electronic device 104, or server 108), and perform communication through the established communication channel. Communication module 190 may include one or more CPs that can operate independently of processor 120 (e.g., AP) and support direct (e.g., wired) or wireless communication. According to embodiments, communication module 190 may include wireless communication module 192 (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module) or wired communication module 194 (e.g., local area network (LAN) communication module or power line communication (PLC) module). One of these communication modules can communicate with an external electronic device 104 via a first network 198 (e.g., a short-range communication network such as Bluetooth™, Wi-Fi Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, 5G network, next-generation communication network, the Internet, or a computer network such as a LAN or wide area network (WAN)). These various types of communication modules can be implemented as a single component (e.g., a single chip) or as multiple components separate from each other (e.g., multiple chips). The wireless communication module 192 can use user information (e.g., International Mobile Subscriber Identity (IMSI)) stored in the SIM 196 to identify and authenticate the electronic device 101 in a communication network such as the first network 198 or the second network 199.

[0044] Wireless communication module 192 supports 5G networks beyond 4G and next-generation communication technologies such as New Radio (NR) access technology. NR access technology supports enhanced mobile broadband (eMBB), massive machine-type communications (mMTC), or ultra-reliable low-latency communications (URLLC). Wireless communication module 192 supports high-frequency bands (e.g., millimeter-wave bands) to achieve, for example, high data transmission rates. Wireless communication module 192 supports various technologies used to ensure high-frequency band performance, such as beamforming, massive MIMO, full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive MIMO. Wireless communication module 192 supports various requirements specified in electronic device 101, external electronic device (e.g., electronic device 104), or network system (e.g., second network 199). According to an embodiment, the wireless communication module 192 may support peak data rates (e.g., 20 Gbps or higher) for implementing eMBB, loss coverage (e.g., 164 dB or lower) for implementing mMTC, or user plane latency (e.g., 0.5 ms or lower for both downlink (DL) and uplink (UL), or 1 ms or lower for round trip) for implementing URLLC.

[0045] Antenna module 197 can transmit or receive signals or power to or from the exterior of electronic device 101 (e.g., external electronic device). According to an embodiment, antenna module 197 may include an antenna comprising a radiating element made of conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, antenna module 197 may include multiple antennas (e.g., an array antenna). In this case, communication module 190 may select at least one antenna from the multiple antennas, for example, an antenna suitable for a communication scheme used in a communication network such as a first network 198 or a second network 199. Signals or power can then be transmitted or received between communication module 190 and external electronic device via the selected at least one antenna. According to an embodiment, another component besides the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of antenna module 197.

[0046] According to an embodiment, antenna module 197 may form a millimeter-wave antenna module. According to an embodiment, the millimeter-wave antenna module may include a PCB, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the PCB and capable of supporting a specified high frequency band (e.g., millimeter-wave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top or side surface) of the PCB and capable of transmitting or receiving signals of a specified high frequency band.

[0047] At least some of the aforementioned components may be coupled to each other and communicate signals (e.g., instructions or data) therebetween via peripheral communication schemes (e.g., bus, general purpose input / output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).

[0048] According to an embodiment, instructions or data can be sent or received between electronic device 101 and external electronic device 104 via server 108 coupled to a second network 199. Each of electronic devices 102 or 104 can be a device of the same or different type as electronic device 101. According to an embodiment, all or some operations to be performed at electronic device 101 can be performed at one or more of external electronic devices 102, 104, or 108. For example, if electronic device 101 is to automatically or in response to a request from a user or another device to perform a function or service, electronic device 101 may request one or more external electronic devices to perform at least a portion of the function or service, and not or in addition to performing that function or service. One or more external electronic devices receiving the request may perform at least a portion of the requested function or service, or additional functions or services related to the request, and transmit the execution result to electronic device 101. Electronic device 101 may provide the result (whether or not the result is further processed) as at least part of a response to the request. For this purpose, technologies such as cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing can be used. Electronic device 101 may use, for example, distributed computing or MEC to provide ultra-low latency services. In another embodiment, external electronic device 104 may include Internet of Things (IoT) devices. Server 108 may be an intelligent server using machine learning and / or neural networks. According to embodiments, external electronic device 104 or server 108 may be included in a second network 199. Electronic device 101 can be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology or IoT-related technologies.

[0049] refer to Figure 2 The integrated intelligent system 20 according to the embodiment may include an electronic device 201 (e.g., Figure 1 Electronic devices 101), intelligent servers 200 (e.g.) Figure 1 Server 108) and server 300 (e.g. Figure 1 Server 108).

[0050] The electronic device 201 according to the embodiment may be an internet-connected terminal device (or electronic device), such as a mobile phone, smartphone, personal digital assistant (PDA), laptop, television, white goods, wearable device, head-mounted display (HMD), or smart speaker.

[0051] According to the illustrated embodiment, the electronic device 201 may include a communication interface 202 (e.g., Figure 1 Interface 177), microphone 206 (e.g.) Figure 1 Input module 150), speaker 205 (e.g.) Figure 1(e.g., sound output module 155), display module 204) Figure 1 The display module 160), memory 207 (e.g., the display module 160), and the memory Figure 1 The memory 130) or processor 203 (e.g.) Figure 1 (Processor 120). The components listed above are operably or electrically connected to each other.

[0052] The communication interface 202 according to an embodiment can be connected to an external device and configured to send and receive data with the external device. The microphone 206 according to an embodiment can receive sound (e.g., user speech) and convert the sound into electrical signals. The speaker 205 according to an embodiment can output electrical signals as sound (e.g., voice).

[0053] The display module 204 according to an embodiment can be configured to display images or videos. The display module 204 according to an embodiment can also display the graphical user interface (GUI) of a running application (or application program). The display module 204 of the embodiment can receive touch input via a touch sensor. For example, the display module 204 can receive text input via a touch sensor in the on-screen keyboard area displayed in the display module 204.

[0054] The memory 207 can store the client module 209, the software development kit (SDK) 208, and multiple applications 211. The client module 209 and the SDK 208 can be configured to perform a framework (or solution program) for common functions. In addition, the client module 209 or the SDK 208 can be configured to handle user input (such as voice input, text input, or touch input).

[0055] The plurality of applications 211 stored in memory 207 may be programs for performing specified functions. According to an embodiment, the plurality of applications 211 may include a first application 211_1 and a second application 211_2. According to an embodiment, each of the plurality of applications 211 may include multiple actions for performing the specified function. For example, the applications may include an alarm clock application, a messaging application, and / or a calendar application. According to an embodiment, the plurality of applications 211 may be executed by processor 203 to sequentially perform at least a portion of the plurality of actions.

[0056] According to the embodiment, the processor 203 can control the overall operation of the electronic device 201. For example, the processor 203 can be electrically connected to the communication interface 202, microphone 206, speaker 205, and display module 204 to perform specified actions.

[0057] According to an embodiment, the processor 203 can also perform specified functions by executing a program stored in the memory 207. For example, the processor 203 can execute at least one of the client module 209 or the SDK 208 to perform the following operations for processing user input. The processor 203 can, for example, control the actions of multiple applications 211 via the SDK 208. The following operations of the client module 209 or the SDK 208 can be performed by the processor 203.

[0058] According to an embodiment, the client module 209 can receive user input. For example, the client module 209 can receive voice signals corresponding to user speech sensed by the microphone 206. Alternatively, the client module 209 can receive touch input sensed by the display module 204. Alternatively, the client module 209 can receive text input sensed by a keyboard or on-screen keyboard. Furthermore, the client module 209 can receive various types of user input sensed by input modules included in or connected to the electronic device 201. The client module 209 can send the received user input to the intelligent server 200. The client module 209 can also send the status information of the electronic device 201 along with the received user input to the intelligent server 200. The status information may, for example, be application execution status information.

[0059] According to the embodiment, the client module 209 can receive a result corresponding to the received user input. For example, when the intelligent server 200 is able to calculate a result corresponding to the received user input, the client module 209 can receive the result corresponding to the received user input. The client module 209 can display the received result on the display module 204. In addition, the client module 209 can output the received result in audio form through the speaker 205.

[0060] According to an embodiment, the client module 209 can receive a plan corresponding to the received user input. The client module 209 can display the results of executing multiple actions of the application according to the plan on the display module 204. For example, the client module 209 can sequentially display the results of executing multiple actions on the display module 204 and output the results in audio form via the speaker 205. In another example, the electronic device 201 can display only a portion of the results of executing multiple actions (e.g., the result of the last action) on the display module 204 and output that portion of the result in audio form via the speaker 205.

[0061] According to an embodiment, the client module 209 may receive a request from the intelligent server 200 to obtain information needed to calculate the result corresponding to the user input. According to an embodiment, the client module 209 may send the required information to the intelligent server 200 in response to the request.

[0062] According to an embodiment, the client module 209 can send information about the results of performing multiple actions according to a plan to the intelligent server 200. The intelligent server 200 can use the result information to confirm that the received user input has been processed correctly.

[0063] According to an embodiment, client module 209 may include a speech recognition module. According to an embodiment, client module 209 may use the speech recognition module to recognize voice input for performing limited functions. For example, client module 209 may execute an intelligent application for processing voice input to perform organic operations upon specifying input (e.g., "Wake up!").

[0064] According to an embodiment, the intelligent server 200 can receive information related to user voice input from the electronic device 201 via a communication network. According to an embodiment, the intelligent server 200 can convert the data related to the received voice input into text data. According to an embodiment, the intelligent server 200 can generate a plan for performing tasks corresponding to the user's voice input based on the text data.

[0065] According to embodiments, the plan can be generated by an AI system. This AI system can be a rule-based system or a neural network-based system (e.g., a feedforward neural network (FNN) or a recurrent neural network (RNN)). Alternatively, the AI ​​system can be a combination of the above systems or other AI systems. According to embodiments, the plan can be selected from a predefined plan set or can be generated in real time in response to a user request. For example, the AI ​​system can select at least one plan from predefined plans.

[0066] According to an embodiment, the intelligent server 200 can send the results of the generated plan to the electronic device 201 or send the generated plan to the electronic device 201. According to an embodiment, the electronic device 201 can display the results of the plan on the display module 204. According to an embodiment, the electronic device 201 can display the results of the actions performed according to the plan on the display module 204.

[0067] The intelligent server 200 may include a front-end 215, a natural language platform 220, a capsule database (DB) 230, an execution engine 240, a terminal UI 250, a management platform 260, a big data platform 270, or an analysis platform 280.

[0068] According to an embodiment, front-end 215 can receive user input from electronic device 201. Front-end 215 can send a response corresponding to the user input.

[0069] According to an embodiment, the natural language platform 220 may include an automatic speech recognition (ASR) module 221, a natural language understanding (NLU) module 223, a planner module 225, a natural language generator (NLG) module 227, or a text-to-speech (TTS) module 229.

[0070] According to an embodiment, the ASR module 221 can convert data related to voice input received from the electronic device 201 into text data. According to an embodiment, the NLU module 223 can determine a domain (and / or intent information) corresponding to the voice input (e.g., user speech) based on the text data of the voice input. The domain may correspond to a category (or service) associated with an action (or function) the user intends to perform using the device. The domain may be categorized based on text-related services (e.g., applications). For example, the Gracenote domain may correspond to a music search service (e.g., the Gracenote service). The Melon domain may correspond to a music streaming service (e.g., the Melon service). The domain may be associated with intent information corresponding to the text. According to an embodiment, the NLU module 223 can use the text data of the voice input to discern the user's intent. For example, the NLU module 223 can discern the user's intent by performing syntactic or semantic analysis on the user input in text data form. According to an embodiment, the NLU module 223 can use linguistic features (e.g., grammatical elements) of morphemes or phrases to discern the meaning of words extracted from the user input and determine the user's intent by matching the discerned meaning of the words with the intent. NLU module 223 can acquire intent information corresponding to a user's utterance. The intent information may be information indicating a user's intent determined by analyzing text data. The intent information may include information indicating that the user intends to perform an action (or function) using the device. A slot may be detailed information associated with the intent information. The slot may be acquired based on the domain corresponding to the utterance. The slot may be variable information required to perform the action. In an embodiment, the variable information contained in the slot may include named entities.

[0071] According to an embodiment, planner module 225 can generate a plan using parameters and intents determined by NLU module 223. According to an embodiment, planner module 225 can determine multiple domains required to perform a task based on the determined intent. Planner module 225 can determine multiple actions contained in each of the multiple domains based on the intent. According to an embodiment, planner module 225 can determine parameters required to perform the determined multiple actions or result values ​​output from performing the multiple actions. Parameters and result values ​​can be defined as concepts of a specified form (or class). Therefore, a plan can include multiple actions and multiple concepts determined by user intent. Planner module 225 can determine the relationships between the multiple actions and multiple concepts step-by-step (or hierarchically). For example, based on multiple concepts, planner module 225 can determine the execution order of the multiple actions determined based on user intent. In other words, planner module 225 can determine the execution order of the multiple actions based on the parameters required to perform the multiple actions and the result output from performing the multiple actions. Therefore, planner module 225 can generate a plan that includes connection information (e.g., ontology) regarding the connections between the multiple actions and multiple concepts. The planner module 225 can generate plans using information stored in capsule DB 230, which stores a set of relationships between concepts and actions.

[0072] According to an embodiment, NLG module 227 can convert specified information into text form. The information converted into text form can be in the form of natural language speech. According to an embodiment, TTS module 229 can convert information in text form into information in speech form.

[0073] According to an embodiment, some or all of the functions of the natural language platform 220 may also be implemented in the electronic device 201.

[0074] Capsule DB 230 can store information about relationships between multiple concepts and actions corresponding to multiple domains. According to embodiments, capsules may include multiple action objects (or action information) and concept objects (or concept information) contained in a plan. According to embodiments, capsule DB 230 may store multiple capsules in the form of a Concept Action Network (CAN). According to embodiments, multiple capsules may be stored in a function registry included in capsule DB 230.

[0075] Capsule DB 230 may include a strategy registry storing strategy information required to determine a plan corresponding to a voice input. The strategy information may include reference information for determining a plan when multiple plans corresponding to user input exist. According to an embodiment, Capsule DB 230 may include a follow-up registry storing information about follow-up actions for suggesting subsequent actions to the user in a given situation. Follow-up actions may include, for example, follow-up utterances. According to an embodiment, Capsule DB 230 may include a layout registry storing layout information of information output through electronic device 201. According to an embodiment, Capsule DB 230 may include a vocabulary registry storing vocabulary information contained in capsule information. According to an embodiment, Capsule DB 230 may include a dialogue registry storing information about a conversation (or interaction) with the user. Capsule DB 230 may be updated with stored objects via developer tools. Developer tools may include, for example, a function editor for updating action objects or concept objects. Developer tools may include a vocabulary editor for updating vocabulary. Developer tools may include a strategy editor for generating and registering strategies for determining plans. Developer tools may include a dialogue editor for generating conversations with the user. Developer tools may include a follow-up editor capable of activating follow-up goals and editing follow-up utterances that provide prompts. Subsequent objectives can be determined based on currently set objectives, user preferences, or environmental conditions. In this embodiment, the capsule DB 230 can also be implemented in the electronic device 201.

[0076] According to an embodiment, the execution engine 240 can use the generated plan calculation results. The terminal UI 250 can send the calculation results to the electronic device 201. Therefore, the electronic device 201 can receive the results and provide the received results to the user. According to an embodiment, the management platform 260 can manage the information used by the intelligent server 200. According to an embodiment, the big data platform 270 can collect user data. According to an embodiment, the analysis platform 280 can manage the quality of service (QoS) of the intelligent server 200. For example, the analysis platform 280 can manage the components and processing speed (or efficiency) of the intelligent server 200.

[0077] According to an embodiment, the service server 300 can provide a specified service (e.g., food ordering or hotel reservation) to the electronic device 201. According to an embodiment, the service server 300 can be a server operated by a library server. According to an embodiment, the service server 300 can provide the intelligent server 200 with information for generating a plan corresponding to received user input. The provided information can be stored in the capsule DB 230. Furthermore, the service server 300 can provide the intelligent server 200 with result information based on the plan.

[0078] In the aforementioned integrated intelligent system 20, the electronic device 201 can provide various intelligent services to the user in response to user input. User input may include, for example, input via physical buttons, touch input, or voice input.

[0079] In this embodiment, the electronic device 201 may provide voice recognition services through a smart application (or voice recognition application) stored therein. For example, the electronic device 201 may recognize user speech or voice input received through a microphone and provide the user with services corresponding to the recognized voice input.

[0080] In this embodiment, the electronic device 201 may perform a specified action based on the received voice input, either alone or in conjunction with a smart server and / or a service server. For example, the electronic device 201 may execute an application corresponding to the received voice input and perform the specified action through the executed application.

[0081] In this embodiment, when the electronic device 201 provides services together with the intelligent server 200 and / or the service server 300, the electronic device 201 can use the microphone 206 to detect user speech and generate a signal (or voice data) corresponding to the detected user speech. The electronic device 201 can use the communication interface 202 to send the voice data to the intelligent server 200.

[0082] According to an embodiment, the intelligent server 200 can generate a plan for performing a task corresponding to the voice input, or the result of performing actions according to the plan, as a response to voice input received from the electronic device 201. The plan may include, for example, multiple actions for performing the task corresponding to the user's voice input, and multiple concepts associated with the multiple actions. A concept may be defined as a parameter input for performing the multiple actions or a result value output from performing the multiple actions. The plan may include connection information regarding the connections between the multiple actions and the multiple concepts.

[0083] Electronic device 201 can receive responses using communication interface 202. Electronic device 201 can output voice signals generated internally to the outside using speaker 205, or output images generated internally to the outside using display module 204.

[0084] Figure 3 This is a schematic diagram illustrating the storage format of information relating concepts and actions in a database based on various embodiments.

[0085] Intelligent servers (e.g.) Figure 2 Capsule DB (e.g., intelligent server 200) Figure 2The capsule DB 230 can store capsules in CAN 400 format. The capsule DB can store actions for processing tasks corresponding to the user's voice input, as well as the parameters required to perform those actions, in CAN format.

[0086] The capsule database can store multiple capsules (capsule A 401 and capsule B 404) corresponding to multiple domains respectively. According to an embodiment, a capsule (e.g., capsule A 401) may correspond to a domain (e.g., location (geography) or application). Furthermore, a capsule may correspond to at least one service provider (e.g., CP 1 402 or CP 2 403) for performing functions related to the domain associated with that capsule. According to an embodiment, a capsule may include at least one action 410 and at least one concept 420 for performing a specified function.

[0087] Natural language platforms (e.g.) Figure 2 The natural language platform 220 can use capsules stored in the capsule DB to generate plans for performing tasks corresponding to received speech input. For example, the planner module of the natural language platform (e.g., Figure 2 The planner module 225 can generate plans using capsules stored in the capsule DB. For example, plans 407 can be generated using actions 4011 and 4013 and concepts 4012 and 4014 of capsule A 401, and actions 4041 and concepts 4042 of capsule B 404.

[0088] Figure 4 These are schematic diagrams of screens showing how electronic devices, according to various embodiments, process received voice input via smart applications.

[0089] Electronic device 201 can execute intelligent applications to access intelligent servers (e.g., Figure 2 The intelligent server 200 processes user input.

[0090] According to an embodiment, on screen 310, when a specified voice input (e.g., "Wake up!") is recognized or input is received via a hardware key (e.g., a dedicated hardware key), electronic device 201 can execute a smart application for processing voice input. For example, electronic device 201 can execute a smart application while executing a schedule application. According to an embodiment, electronic device 201 can be displayed on display module 204 (e.g., ...). Figure 1 Display module 160 and Figure 2The display module 204 displays an object (e.g., an icon) 311 corresponding to the smart application. According to an embodiment, the electronic device 201 can receive voice input via user speech. For example, the electronic device 201 can receive voice input such as "Tell me my schedule for this week." According to an embodiment, the electronic device 201 can display the UI 313 (e.g., an input window) of the smart application on the display module 204, where the text data of the received voice input is displayed.

[0091] According to an embodiment, on screen 320, electronic device 201 can display the result corresponding to the received voice input on display module 204. For example, electronic device 201 can receive a schedule corresponding to the received user input and display "This Week's Schedule" on display module 204 according to the schedule.

[0092] Figure 5 This is a schematic diagram of the speech processing flow of an electronic device according to an embodiment.

[0093] refer to Figure 5 According to an embodiment, electronic device 501 may include reference Figure 1 The described electronic device 101 and reference Figure 2 The described electronic device 201 includes at least some components. The intelligent server 503 may include references... Figure 2 At least some components of the intelligent server 200 are described. References to electronic device 501 and intelligent server 503 are omitted. Figures 1 to 4 The provided description is duplicated.

[0094] According to an embodiment, electronic device 501 (e.g.) Figure 1 Electronic devices 101 or Figure 2 Electronic device 201) can be connected to intelligent server 503 (e.g., via LAN, WAN, value-added network (VAN), mobile wireless communication network, satellite communication network, or any combination thereof) through LAN, WAN, VAN, mobile wireless communication network, satellite communication network, or any combination thereof. Figure 2 The intelligent server 200. Electronic device 501 and intelligent server 503 can communicate with each other via wired or wireless communication methods (e.g., Wireless LAN (Wi-Fi), Bluetooth, Bluetooth Low Energy, ZigBee, Wi-Fi Direct (WFD), Ultra Wideband (UWB), Infrared Data Association (IrDA), and Near Field Communication (NFC)). Figure 1 Electronic device 102 or Figure 1 Electronic devices 104) communication.

[0095] According to an embodiment, the electronic device 501 may be at least one of a smartphone, tablet computer (PC), mobile phone, speaker (e.g., AI speaker), video phone, e-book reader, desktop PC, laptop PC, netbook computer, workstation, server, personal digital assistant (PDA), portable multimedia player (PMP), MP3 player, mobile medical device, camera, or wearable device.

[0096] According to an embodiment, electronic device 501 can acquire a voice signal corresponding to the utterance (e.g., voice utterance) of user 50 and send the voice signal to intelligent server 503. Intelligent server 503 can acquire text data corresponding to the utterance of user 50 based on the voice signal. The text data can be obtained by performing an automatic speech recognition (ASR) process on the voice signal to convert the speech portion into computer-readable text. However, when the user's utterance is text utterance rather than voice utterance, the process of converting the voice signal into text data can be omitted. Intelligent server 503 can analyze the user's utterance using the text data. Intelligent server 503 can use the analysis results (e.g., intent information, entities, and / or capsules) to perform desired functions or provide responses (e.g., questions and answers) to be provided to the user to the device (e.g., electronic device 501). Intelligent server 503 can be implemented as software. Part or all of intelligent server 503 can be implemented in electronic device 501 and / or intelligent server 503 (e.g., Figure 2 This is implemented in the intelligent server 200. A device-side AI for processing user speech without communicating with the intelligent server 503 can be installed in the electronic device 501. (Reference) Figures 2 to 4 The described components (e.g., natural language platform 220) can be implemented in electronic device 501.

[0097] According to an embodiment, the ASR module included in the natural language platform 220 (e.g.) Figure 2 The ASR module 221 can convert user speech into text data. The NLU module included in the Natural Language Platform 220 (e.g., Figure 2 The NLU module 223 can use text data corresponding to user speech to determine domain and / or intent information corresponding to voice input (e.g., user speech).

[0098] According to embodiments, a domain may correspond to a category (or service) associated with an action (or function) that a user intends to perform using the device. Domains may be categorized based on services (e.g., applications) related to the user's utterance. For example, the Grammynot domain may correspond to a music search service (e.g., the Grammynot service). For example, the Melon domain may correspond to a music playback service (e.g., the Melon service).

[0099] According to embodiments, multiple user utterances can be processed based on their respective domains. Tasks corresponding to user utterances can be processed within capsules (e.g., applications). A capsule may correspond to a domain. A capsule may include at least one action and at least one concept for performing a specified function. A capsule (or domain) may include or process multiple intent messages. Capsules can process tasks corresponding to user utterances based on intent messages. Intent messages may be determined by the capsule or an NLU module.

[0100] According to an embodiment, intent information may be information indicating a user's intent, determined by interpreting text data. Intent information may include information indicating that the user intends to perform an action (or function) using the device.

[0101] According to an embodiment, a slot can be a placeholder (or variable) associated with a specific type of information extracted from user input (e.g., user utterance) required to generate a response corresponding to the user's utterance. An entity can be a value extracted from user input and assigned to the corresponding slot. For example, when the text converted from the user utterance is "Tell me today's weather," the electronic device 501 can identify a "date slot" from the user input and assign "today" to the "date slot."

[0102] According to an embodiment, electronic device 501 can output a response corresponding to a user's speech. For example, in response to receiving the user's speech "Tell me the weather in Seoul today" from user 50, electronic device 501 can output a response (e.g., "The weather in Seoul is sunny today") by performing a speech recognition process and a natural language process.

[0103] According to embodiments, electronic device 501 can output responses in various ways. For example, electronic device 501 may use a display module (e.g., Figure 1 The display module 160 provides a visual output response. In another example, the electronic device 501 may use an audio module (e.g., Figure 1 The audio module 170 outputs a response in the form of sound.

[0104] According to an embodiment, some or all of the operations performed by electronic device 501 may be performed by electronic device 501 and / or intelligent server 503. The following description is based on the assumption that electronic device 501 performs the operations.

[0105] Figure 6 This is a schematic block diagram of an electronic device according to an embodiment.

[0106] refer to Figure 6 According to an embodiment, electronic device 501 may include processor 620 (e.g., Figure 1 Processor 120 and Figure 2 The processor 203) and memory 630 (e.g.) Figure 1 The memory 130 and Figure 2 (Memory 207). Processor 620 can access memory 630 and execute one or more instructions. Processor 620 can cause electronic device 501 to perform one or more operations to generate (or output) a response to user speech based on one or more modules (e.g., software modules).

[0107] According to an embodiment, processor 620 may include ASR module 641 (e.g., Figure 2 The system includes: ASR module 221, entity extraction module 643, first retrieval unit 645, data crawler 647, data processor 649, second retrieval unit 651, first sorting module 653, second sorting module 655, reader 657, and capsule 659 (e.g., ASR module 221), ... Figure 3 (capsule) and / or response module 661.

[0108] According to an embodiment, the ASR module 641 can convert the speech signal corresponding to the user's speech (e.g., voice speech) into text data in real time. When the user's speech is text speech rather than voice speech, the process performed by the ASR module 641 can be omitted. Hereinafter, it can be assumed that the text data corresponding to the user's speech is input to the entity extraction module 643 and / or the second retrieval unit 651 in real time.

[0109] According to an embodiment, entity extraction module 643 can extract values ​​(e.g., entities or entity combinations) related to information used to determine the intent of the user's utterance from a portion of the user's speech in real time. For example, when the user's utterance is "Michael Jackson's songs, tell me," entity extraction module 643 can extract the entities "Michael Jackson" and / or the entity combination "Michael Jackson's songs" from a portion of the user's speech in real time before the user's speech ends. Entity extraction module 643 can send data related to the entities (or entity combinations) extracted from that portion of the user's speech to a first retrieval unit 645. For example, the value sent to the first retrieval unit 645 could be a token (e.g., a token in an n-gram).

[0110] According to an embodiment, the first retrieval unit 645 may send a request to the data crawler 647 to obtain information about data (e.g., entities or combinations of entities) received from the entity extraction module 643. For example, when the entity "Michael Jackson" is extracted from user utterances, the first retrieval unit 645 may send a request to the data crawler 647 to obtain information (e.g., image or text data) about the entity "Michael Jackson". The first retrieval unit 645 may include rule-based models and / or statistical models.

[0111] According to an embodiment, the first retrieval unit 645 may output a first response 63 (e.g., information about an entity or combination of entities and an information source) to a user based on data received from the data processor 645. In this disclosure, the first response 63 may be a response to a portion of a user's utterance (e.g., an entity or combination of entities). The first retrieval unit 645 may output the first response 63 in various ways. For example, the first retrieval unit 645 may visually output the first response via the screen of the electronic device 501.

[0112] According to an embodiment, the first retrieval unit 645 may send data related to the first response 63 to the first sorting module 653.

[0113] According to an embodiment, data crawler 647 may, in response to an information request from first retrieval unit 645, retrieve information about entities (or combinations of entities) extracted from a portion of a user's utterance from one or more data sources. For example, data crawler 647 may retrieve (or acquire) information (or data) from external data sources such as websites and / or internal data sources stored in memory 630. For example, data crawler 647 may extract text data related to entities (or combinations of entities) from data sources. Data crawler 647 may send information and / or information sources to data processor 649.

[0114] According to an embodiment, the data processor 649 can process data (e.g., information and / or information sources) received from the data crawler 647, adapt it to the first retriever 645, and send the processed data to the first retriever 645. For example, the data processor 649 can perform data preprocessing, such as DB indexing.

[0115] According to an embodiment, the first sorting module 653 can determine a user intent related to an entity (or combination of entities) extracted from a portion of the user's utterance based on user input 67 in response to the first response 63. For example, when the user utterance is "Tell me about Michael," the first retrieval unit 645 can output a list of celebrities with the name "Michael" (e.g., a first name) as the first response 63, and the user can select a person (e.g., singer Michael Jackson) from the list. The first sorting module 653 can determine, based on the user selection, that the user intent related to the entity "Michael" contained in a portion of the user's utterance is "Michael Jackson." The first sorting module 653 can send data related to the user intent to the reader 657 in various formats. For example, the first sorting module 653 can send data to the reader 657 in a format suitable for the generative model (e.g., a prompt format or JavaScript Object Notation (JSON)).

[0116] According to an embodiment, the second retrieval unit 651 may receive text data corresponding to a user utterance (e.g., the entire user utterance) and generate a response corresponding to the user utterance. In this disclosure, the user utterance may be the user utterance of the detected endpoint. The text data input to the second retrieval unit 651 may be text comprising one or more sentences. For example, the text input to the second retrieval unit 651 may be text such as “Tell me the age difference between the director of the movie Avatar and the director of the movie Titanic”. According to an embodiment, the second retrieval unit 651 may include rule-based models and / or statistical models.

[0117] According to an embodiment, the second retrieval unit 651 can generate one or more candidate responses corresponding to the user's utterance. For example, when the user's utterance is "Tell me about sports that suit me," the second retrieval unit 651 can generate baseball, basketball, and soccer as candidate responses based on data (e.g., the user's lifestyle patterns) stored in a data source (e.g., memory 630). The second retrieval unit 651 can send one or more candidate responses to the reader 657. The process performed by the second retrieval unit 651 can be omitted based on the user's utterance 61 and the first user response 63. For example, when the user's utterance is "Tell me about Michael Jackson," and the first response 63 is substantially the same as the second response 65, one or more processes performed by the second retrieval unit 651 can be omitted. In this disclosure, the second response 65 can be a response to the user's utterance at the detected endpoint.

[0118] According to an embodiment, the second sorting module 655 may receive one or more candidate responses from the second retrieval unit 651 and determine the sorting order of the one or more candidate responses. For example, when receiving "baseball," "basketball," and "football" as candidate responses from the second retrieval unit 651, the second sorting module 655 may determine the sorting order of the candidate responses in the order of "baseball," "basketball," and "football" based on user-related information such as user policies (e.g., children's mode), user lifestyle patterns, and / or user electronic device usage patterns (e.g., application usage frequency)). The second sorting module 655 may send data related to the candidate responses (e.g., the highest-ranked candidate response) to the reader 657 in various formats. For example, the second sorting module 655 may send data to the reader 657 in a format suitable for generating the model (e.g., a prompt format or JSON). The second sorting module 655 may generate additional user responses based on user input for a second response 65 generated based on the candidate responses (e.g., the highest-ranked candidate response). For example, although the second response 65 “Baseball is the right sport for you” is output, the user may respond to the second response 65 by saying “I don’t like baseball. Tell me another sport”. The second sorting module 655 may output additional user responses, such as “Basketball is the right sport for you”, based on the sorting of previously generated candidate responses (e.g., baseball (1), basketball (2), football (3)). One or more processes performed by the second sorting module 655 may be omitted when the process performed by the second retriever 651 is omitted and / or when the second retriever 651 generates only one candidate response.

[0119] According to embodiments, reader 657 (e.g., a generative model) can generate various forms of output based on data input from first sorting module 653 and / or second sorting module 653. For example, reader 657 can generate output including one or more pieces of information corresponding to user utterances, application programming interfaces (APIs), and commands.

[0120] According to an embodiment, capsule 659 (e.g., an application) can perform a predetermined action based on data received from reader 657. For example, capsule 659, such as an alarm clock application (e.g., a clock), can receive data (e.g., data related to alarm clock settings) from reader 657, set an alarm, and provide the user with information about the set alarm.

[0121] According to an embodiment, the response module 661 can output the data received from the reader 657 or capsule 659 as is, or post-process the data received from the reader 657 or capsule 659 and output the post-processed data. For example, the response module 661 can convert the data received from the reader 657 or capsule 659 into another format such as an image or video, and output the converted data.

[0122] According to an embodiment, memory 630 may store data used by at least one component of electronic device 501 (e.g., processor 620). For example, memory 630 may store user-related data, such as information about external electronic devices (e.g., Internet of Things (IoT) devices) connected to electronic device 501, the user's electronic device usage patterns (e.g., application usage frequency), and / or the user's lifestyle patterns.

[0123] Figure 7 This is an example schematic diagram of the speech processing flow according to an embodiment.

[0124] refer to Figure 7 According to an embodiment, the electronic device 501 can identify entities (or combinations of entities) contained in a portion of a user's speech in real time and output a response to the identified entities (or combinations of entities) (e.g., Figure 6 The first response 63). The electronic device 501 can output a response for the identified entity (or combination of entities) before the user's utterance ends. For example, when the user's utterance is "Michael Jackson's songs, tell me", the electronic device 501 can identify the entity "Michael Jackson" and / or the entity combination "Michael Jackson's songs" contained in a part of the user's utterance in real time, and output information 71 about the identified entity and / or information 73 about the entity combination before the user's utterance ends.

[0125] Figure 8 This is an example schematic diagram of the speech processing flow according to an embodiment.

[0126] refer to Figure 8 According to an embodiment, electronic device 501 can identify entities (or combinations of entities) contained in a portion of a user's speech in real time and provide a response (e.g., ...) for each identified entity (or combination of entities). Figure 6 The first response 63). The electronic device 501 can provide a response for the identified entity (or combination of entities) before the user's utterance ends. For example, when the user's utterance is "Tell me the difference between the screen size of S23 and the screen size of S22", the electronic device 501 can identify the first entity combination "screen size of S23" and the second entity combination "screen size of S22" from the user's utterance in real time, and output information 81 and 83 about the identified entity combinations before the user's utterance ends.

[0127] According to an embodiment, electronic device 501 can output a response to the user's speech after the user's speech has ended (e.g., Figure 6The second response 65). For example, when the user utters "The screen size of S23 and the screen size of S22, tell me the difference between the two", the electronic device 501 may output information 81 and 83 (e.g., the first response) about the combination of entities contained in a part of the user utterance before the user utterance ends, and output information 85 (e.g., the second response) about the user utterance after the user utterance ends.

[0128] According to an embodiment, electronic device 501 may indicate that user speech processing is in progress from the time user speech is acquired until information 85 (e.g., a second response) about the user speech is output. For example, electronic device 501 may display a status notification message, such as “User speech processing is in progress. The final response to the user speech has not yet been output.” In another example, electronic device 501 may distinguish the output format (e.g., the color and / or shape of speech bubbles) of information 81 and 83 (e.g., a first response) about combinations of entities contained in a portion of the user speech from the output format of information 85 about the user speech.

[0129] According to an embodiment, electronic device 501 may terminate user speech processing in response to user input received before outputting information 85 about the user speech. When the user obtains target information from information 81 and 83 about entities contained in a portion of the user speech, the user may no longer need information 85 about the user speech. Electronic device 501 may provide a UI (e.g., button 560) for terminating user speech processing, and electronic device 501 may terminate user speech processing in response to user input received through this UI.

[0130] According to the embodiment, since information about entities (or combinations of entities) contained in a portion of the user's speech (e.g., information 81 and 83) is output before the user's speech ends, the user can obtain substantially the same information as the information the user expects to obtain through the user's speech (e.g., information 85) in a short period of time.

[0131] Figure 9 This is an example schematic diagram of the speech processing flow according to an embodiment.

[0132] refer to Figure 9 According to an embodiment, electronic device 501 may respond based on an entity (or combination of entities) contained in a portion of a user's speech (e.g., Figure 6 The first response (63) and the response to the user's words (e.g.) Figure 6The electronic device 501 determines whether to cancel the output of a response (e.g., first response 63) for the entity (or combination of entities) based on the relationship between the response for the entity (e.g., first response 63) and the response for the user's utterance (e.g., second response 65). For example, when the relevance between the response for the entity (e.g., first response 63) and the response for the user's utterance (e.g., second response 65) is low, the electronic device 501 may cancel the output of the response for the entity (e.g., first response 63) and only output the response for the user's utterance (e.g., second response 65). In another example, when the response for the entity (e.g., first response 63) contradicts the response for the user's utterance (e.g., second response 65), the electronic device 501 may cancel the output of the response for the entity (e.g., first response 63) and only output the response for the user's utterance (e.g., second response 65). For example, when a user utters the phrase "Tell me which American singers were active during the same period as Michael Jackson," electronic device 501 can identify the entity "Michael Jackson" from a portion of the user's utterance and output information 1001 about the identified entity. Electronic device 501 can output information 1003 about the user's utterance after the user utterance ends. When the relevance between information 1001 and information 1003 is low, electronic device 501 can output UI 1005 or UI 1007 after the user utterance ends. UI 1005 may include both information 1001 and information 1003, while UI 1007 may include only information 1003.

[0133] Figure 10 This is an example schematic diagram of the speech processing flow according to an embodiment.

[0134] refer to Figure 10 According to an embodiment, electronic device 501 may output a response (e.g., to an entity (or combination of entities) contained in a portion of a user's speech) Figure 6 The first response 63), and based on user input for that response (e.g., the first response) Figure 6 User input 67), outputting a response to the user's utterance (e.g., ... Figure 6 The second response (65).

[0135] According to an embodiment, based on parameters associated with the entity (e.g., domain and category), electronic device 501 can output a response (e.g., response 63) in the form of a list of items associated with the entity. For example, electronic device 501 can output a response in list form for an entity associated with a fashion item (e.g., shoes or clothing), which includes fuzzy entities such as color and / or various sub-entities. For example, when the user utters "Find me shoes that go well with red clothing," electronic device 501 can identify the entities "red," "clothing," and "shoes" from a portion of the user's utterance in real time and output responses 1111, 1113, and 1115 in list form for the identified entities. Response 1111 could be a list including colors similar to red (e.g., red, crimson, cherry red, and magenta).

[0136] According to an embodiment, based on user input (e.g., user selection) to responses 1111, 1113, and 1115, electronic device 501 can output a response 1117 to the user's speech after the user's speech has ended. For example, the user can select "dark red," "suit," and "boots" from responses 1111, 1113, and 1115 respectively through various means (e.g., voice input or non-voice input such as touch or gesture). Based on the user selection, electronic device 501 (e.g., reader 657 of electronic device 501) can generate one or more prompts (e.g., "Shoes that match red clothing, help me find them" and "Red is close to dark red, clothing is a suit, shoes are boots"). Electronic device 501 (e.g., reader 657 of electronic device 501) can use a generative model to generate a response 1117 corresponding to one or more prompts, such as "Black boots match well." Electronic device 501 can display response 1117 on a portion of the screen of electronic device 501.

[0137] According to an embodiment, based on the form of a response (e.g., first response 63) to an entity (or combination of entities) contained in a portion of a user's speech, the electronic device 501 can maintain the output state of the response (e.g., first response 63) to that entity (or combination of entities) even after the user's speech has ended. For example, when the response (e.g., first response 63) to that entity (or combination of entities) is output in a user-selectable form (e.g., a list), the electronic device 501 can maintain the output state of the response (e.g., first response 63) to that entity (or combination of entities) even after the user's speech has ended.

[0138] According to an embodiment, even after outputting a response to a user's utterance (e.g., second response 65), the electronic device 501 can acquire additional user input for a response to that entity (or combination of entities) (e.g., first response 63). Based on this additional user input, the electronic device 501 can modify the response to the user's utterance (e.g., second response 65). For example, when the user's utterance is "Find shoes that go well with red clothing," and the user selects "dark red," "suit," and "boots" from responses 1111, 1113, and 1115, the electronic device 501 can output response 1117 "Black boots go well." After outputting response 1117, the user can select an item different from the previously selected item (e.g., boots) (e.g., sneakers) from response 1115 for the entity "shoes." In response to this user selection, the electronic device 501 can change response 1117 to response 1118 "White sneakers go well."

[0139] Figure 11 This is an example schematic diagram of the speech processing flow according to an embodiment.

[0140] refer to Figure 11 According to an embodiment, electronic device 501 may use the memory stored in electronic device 501 (e.g., Figure 6 The data in the memory 630) outputs a response to an entity (or combination of entities) contained in a portion of the user's speech (e.g., Figure 6 The first response 63). For example, when the user's statement is "Set the TV to turn on in 30 minutes", the electronic device 501 can identify the entity "TV" from a portion of the user's statement in real time and retrieve data related to the identified entity from the memory 630. The electronic device 501 can retrieve information about the TVs connected to the electronic device 501 (e.g., living room TV and bedroom TV) from the memory 630 and output a TV list 1201. Based on user input to the list 1201 (e.g., user selection of items in the list by touch and / or gesture), the electronic device 501 can output a response 1203 to the user's statement after the user's statement ends.

[0141] Figure 12 This is an example schematic diagram of the speech processing flow according to an embodiment. Figure 12 It could be assumed that the output was for... Figure 11 The flowchart shown illustrates the process of handling additional user statements 1303 in the case of a response 1203 to user statement 1301.

[0142] refer to Figure 12According to an embodiment, electronic device 501 may process additional user speech 1303 based on intermediate user input for user speech 1301. Additional user speech 1303 may be user speech acquired after user speech 1301 has terminated. Intermediate user input may be user input for a response (e.g., list 1201) related to an entity (or combination of entities) contained in a portion of user speech 1301. For example, in a state where a response 1203 for user speech 1301 is output, electronic device 501 may acquire the additional user speech 1303 “Turn off TV”, and electronic device 501 may output a response 1205 “Living room TV is off” for the additional user speech 1303 based on intermediate user input (e.g., selection of “living room TV”).

[0143] Figure 13 This is an example schematic diagram of the speech processing flow according to an embodiment. Figure 13 It could be assumed that the output was for... Figure 12 The flowchart shown illustrates the process of generating an additional response 1207 in the case of a response 1205 to an additional user statement 1303.

[0144] refer to Figure 13 According to an embodiment, in response to receiving additional user input on an intermediate response, electronic device 501 may generate an additional response based on the final response. The intermediate response may be a response to an entity (or combination of entities) contained in a portion of the user's utterance (e.g., ...). Figure 6 The first response (61), and the final response can be a response to the user's words (e.g., Figure 6 The second response 63). For example, in the state of outputting a response 1203 to user utterance 1301, the user can select an item from list 1201 that is different from the previously selected item (e.g., living room TV) (e.g., bedroom TV). In response to receiving additional user input for list 1201 (e.g., selection of "bedroom TV"), electronic device 501 can output an additional response "Bedroom TV is set to turn on in 30 minutes" based on the final response 1203 to user utterance 1301. In this case, as Figure 14 As shown, when the final response 1203 to user statement 1301 and the final response 1205 to additional user statement 1303 are output, the electronic device 501 can output an additional response 1207 "The bedroom TV has been set to turn on in 30 minutes and the bedroom TV has been turned off" based on the final responses 1203 and 1205 to user statement 1301 and additional user statement 1303.

[0145] Figure 14 This is an operation flowchart of an electronic device according to an embodiment.

[0146] refer to Figure 14According to the embodiment, operations 1410 to 1430 can be executed sequentially, but the embodiment is not limited thereto. For example, two or more operations can be executed in parallel. Operations 1410 to 1430 can be related to reference... Figures 1 to 13 The described electronic device (e.g.) Figure 1 Electronic devices 101 Figure 4 Electronic devices 201 and Figures 5 to 13 The operation of the electronic device 501 is basically the same. Therefore, related redundant descriptions are omitted.

[0147] In operation 1410, in response to acquiring a portion of the user's speech, electronic devices 101, 201, or 501 may output a first response to that portion of the user's speech (e.g., Figure 6 First Response 63).

[0148] In operation 1420, electronic devices 101, 201, or 501 may acquire user input (e.g., for the first response 63) in response to the first response 63. Figure 6 User input 67).

[0149] In operation 1430, in response to the termination of the user's speech, based on user input 67, electronic devices 101, 201, or 501 may output a second response to the user's speech (e.g., Figure 6 The second response (65).

[0150] Figure 15 This is an operation flowchart of an electronic device according to an embodiment.

[0151] refer to Figure 15 According to the embodiment, operations 1510 to 1530 can be executed sequentially, but the embodiment is not limited thereto. For example, two or more operations can be executed in parallel. Operations 1510 to 1530 can be related to reference... Figures 1 to 13 The described electronic device (e.g.) Figure 1 Electronic devices 101 Figure 4 Electronic devices 201 and Figures 5 to 13 The operation of the electronic device 501 is basically the same. Therefore, related redundant descriptions are omitted.

[0152] In operation 1510, electronic devices 101, 201, or 501 can identify entities from a portion of the user's speech before the user's speech ends.

[0153] In operation 1520, in response to the identification of an entity, electronic devices 101, 201, or 501 may obtain data related to the entity from a data source (such as an external data source like a website and / or an internal data source stored in memory 630).

[0154] In operation 1530, before the user's utterance ends, information about the entity can be output based on the acquired data.

[0155] An electronic device 101, 201, or 501 according to an embodiment includes at least one processor 120, 203, or 620 and a memory 130, 207, or 630 storing instructions. When the instructions are executed individually or jointly by the at least one processor 120, 203, or 620, the electronic device 101, 201, or 501 can perform a plurality of operations. The plurality of operations may include: outputting a first response 63 for a portion of a user's speech in response to receiving the portion; acquiring user input 67 in response to the first response; and outputting a second response 65 to the user's speech based on the user input 67 in response to the termination of the user's speech.

[0156] The portion may include one or more entities.

[0157] Outputting the first response 63 may include: outputting information about an entity or combination of entities before the user's utterance ends.

[0158] The output information may include: a list of items related to entities and combinations of entities.

[0159] The output information may include: retrieving data associated with an entity or combination of entities from memory 130, 207, or 630. The output information may also include: outputting information about the entity or combination of entities based on the retrieved data.

[0160] Obtaining user input 67 may include receiving user selections for a list.

[0161] Electronic devices 101, 201, or 501 may further include a display operatively connected to processors 120, 203, or 620. A first response 63 and a second response 65 may be output via the display.

[0162] The output of the second response 65 may include, based on the first response 63, outputting one of the first UI and the second UI through the display.

[0163] The first UI may include a second response. The second UI may include both the first and second responses.

[0164] The output through the display may include: based on the relationship between the first response 63 and the second response 65 or the form of the first response 63, outputting one of the first UI and the second UI through the display.

[0165] The operations may further include: after outputting the second response 65, in response to receiving additional user input for the first response 63, outputting the third response 1207 based on the second response 65.

[0166] An electronic device 101, 201, or 501 according to an embodiment includes at least one processor 120, 203, or 620 and a memory 130, 207, or 630 storing instructions. When the instructions are executed individually or jointly by the at least one processor 120, 203, or 620, the electronic device 101, 201, or 501 can perform a plurality of operations. The plurality of operations may include: identifying an entity from a portion of the user's speech before the user's speech is terminated; obtaining entity-related data from a data source in response to entity identification; and outputting information about the entity based on the data before the user's speech is terminated.

[0167] This acquisition may include: extracting text data related to the entity from the data source.

[0168] An operating method of an electronic device 101, 201, or 501 according to an embodiment includes: outputting a first response 63 for a portion of a user's speech in response to acquiring that portion. The method includes: acquiring user input 67 for the first response. The method includes: outputting a second response 65 to the user's speech based on the user input 67 in response to the termination of the user's speech.

[0169] Outputting the first response 63 may include: outputting information about an entity or combination of entities before the user's utterance ends.

[0170] The output information may include: a list of items related to entities and combinations of entities.

[0171] The output information may include retrieving data associated with an entity or combination of entities from memories 130, 207, or 630 contained in electronic devices 101, 201, or 501. The output information may also include outputting information about the entity or combination of entities based on the retrieved data.

[0172] The first response 63 and the second response 65 can be output through the display of electronic devices 101, 201 or 501.

[0173] Outputting a second response 65 may include: based on the first response 63, outputting one of a first UI and a second UI via a display. The first UI may include the second response. The second UI may include both the first response and the second response.

[0174] The output through the display may include: based on the relationship between the first response 63 and the second response 65 or the form of the first response 63, outputting one of the first UI and the second UI through the display.

[0175] The electronic devices disclosed in the various embodiments herein can be one of a variety of types of electronic devices. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer equipment, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. According to embodiments of this disclosure, the electronic devices are not limited to those described above.

[0176] It should be understood that the various embodiments of this disclosure and the terminology used therein are not intended to limit the technical features set forth herein to the particular embodiments, but rather to include various modifications, equivalents, or substitutions of the corresponding embodiments. In the description of the drawings, similar reference numerals may be used to refer to similar or related elements. It should be understood that, unless the relevant context clearly indicates otherwise, the singular form of a noun corresponding to an item may include one or more of that thing. As used herein, each phrase such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” may include any or all possible combinations of the items listed together in the corresponding phrase. As used herein, terms such as “first” and “second” may be used to simply distinguish one component from another without otherwise limiting the components (e.g., in terms of importance or order). It should be understood that if an element (e.g., a first element) is referred to (whether or not the terms “operably” or “communically” are used) as “coupled to another element (e.g., a second element),” “coupled to another element,” “connected to another element,” or “connected to another element,” it means that the element can be coupled to the other element directly (e.g., wired), wirelessly, or via a third element.

[0177] As used in conjunction with various embodiments of this disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and is used interchangeably with other terms such as "logic," "logic block," "part," or "circuit." A module may be a single integral component or its smallest unit or part adapted to perform one or more functions. For example, according to an embodiment, the module may be implemented as an application-specific integrated circuit (ASIC).

[0178] The various embodiments described herein can be implemented as software (e.g., program 1740), including one or more instructions stored in a machine-readable storage medium (e.g., internal memory 1736 or external memory 1738). For example, a processor (e.g., processor 1720) of the machine (e.g., electronic device 1701) can invoke at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated according to the invoked at least one instruction to perform at least one function. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" simply means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between cases where data is semi-permanently stored in the storage medium and cases where data is temporarily stored in the storage medium.

[0179] According to embodiments, the methods of various embodiments of this disclosure may be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)), or distributed online (e.g., downloaded or uploaded) through an app store (e.g., Play Store™), or directly between two user devices (e.g., smartphones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as the memory of a manufacturer's server, an app store's server, or a relay server.

[0180] According to various embodiments, each of the above components (e.g., a module or program) may include a single entity or multiple entities, and some of the multiple entities may be respectively located in different components. According to various embodiments, one or more of the above components or operations may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, the integrated component may still perform the functions in the same or similar manner as the corresponding one of the multiple components performed one or more functions of each of the multiple components before integration. According to various embodiments, the operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or tentatively, or one or more operations may be performed in a different order or omitted, or one or more other operations may be added.

[0181] The effects to be achieved are not limited to those described above, and those skilled in the art can clearly understand other effects not mentioned in this document.

Claims

1. An electronic device (101, 201, 501), comprising: At least one processor (120, 203, 620); as well as The memory (130, 207, 630) stores instructions. When the instructions are executed individually or jointly by the at least one processor (120, 203, 620), the electronic device (101, 201, 501) performs multiple operations. The multiple operations include: In response to obtaining a portion of the user's speech, output a first response (63) for that portion. Obtain user input (67) for the first response; and In response to the termination of the user utterance, based on the user input (67), a second response (65) to the user utterance is output, and The output of the first response (63) includes: Retrieve data from the memories (130, 207, 630) associated with entities or combinations of entities included in the portion of the user speech; and Based on the retrieved data, output a list of items related to the entity or a combination of the entities.

2. The electronic device (101, 201, 501) according to claim 1, wherein, Obtaining the user input (67) includes receiving user selections for the list.

3. The electronic device (101, 201, 501) according to any one of claims 1 and 2, further comprising: The display is operatively connected to the processors (120, 203, 620). The first response (63) and the second response (65) are output through the display.

4. The electronic device (101, 201, 501) according to any one of claims 1 to 3, wherein, Outputting the second response (65) includes, based on the first response (63), outputting one of the first user interface (UI) and the second UI through the display.

5. The electronic device (101, 201, 501) according to any one of claims 1 to 4, wherein the first UI includes the second response, and The second UI includes the first response and the second response.

6. The electronic device (101, 201, 501) according to any one of claims 1 to 5, wherein, The output of the display includes outputting one of the first UI and the second UI through the display based on the relationship between the first response (63) and the second response (65) or in the form of the first response (63).

7. The electronic device (101, 201, 501) according to any one of claims 1 to 6, wherein, The multiple operations further include: After outputting the second response (65), in response to receiving additional user input for the first response (63), a third response (1207) is output based on the second response (65).

8. An electronic device (101, 201, 501), comprising: At least one processor (120, 203, 620); as well as The memory (130, 207, 630) stores instructions. Wherein, when the instructions are executed individually or jointly by the at least one processor (120, 203, 620), the electronic device (101, 201, 501) performs multiple operations, and The multiple operations include: Identify entities from a portion of the user's speech before the user's speech ends; In response to the identification of the entity, data related to the entity is obtained from the data source; and Before the user's utterance ends, information about the entity is output based on the data.

9. The electronic device (101, 201, 501) according to claim 8, wherein, The acquisition includes extracting text data related to the entity from the data source.

10. A method for operating an electronic device (101, 201, 501), the method comprising: In response to obtaining a portion of the user's speech, output a first response to the portion (63). Obtain user input for the first response (67); as well as In response to the termination of the user's speech, a second response (65) to the user's speech is output based on the user input (67).

11. The method according to claim 10, wherein, Outputting the first response (63) includes outputting information about an entity or a combination of entities before the user's utterance is terminated.

12. The method according to any one of claims 10 and 11, wherein, The output of the information includes outputting a list of items related to the entity and combinations thereof.

13. The method according to any one of claims 10 to 12, wherein, The information output includes: Retrieve data associated with the entity or a combination of the entities from the memory (130, 207, 630) contained in the electronic devices (101, 201, 501); and Based on the retrieved data, output information about the entity or a combination of the entities.

14. The method according to any one of claims 10 to 13, wherein, The first response (63) and the second response (65) are output through the display of the electronic device (101, 201, 501).

15. The method according to any one of claims 10 to 14, wherein, Outputting the second response (65) includes, based on the first response (63), outputting one of a first user interface (UI) and a second UI through the display. The first UI includes the second response, and The second UI includes the first response and the second response.

Images