Electronic device for performing an action using a voice recognition function and method thereof

Abstract

An electronic device that performs an action using a voice recognition function and a method thereof are provided. The method includes receiving a specified event related to a second application when an execution screen of a first application is displayed on a display. The method also includes executing an artificial intelligence application in response to the specified event. The method also includes transmitting data related to the specified event to an external server based on the executed artificial intelligence application. In addition, the method includes sensing a user utterance related to the specified event for a specified period of time. The method also includes transmitting the user utterance to the external server. The method also includes receiving an action sequence for executing a function related to the user utterance from the external server. The method also includes executing the second application based on at least the received action sequence. The method also includes outputting a result of executing the function by using the second application.

Description

TECHNICAL FIELD

[0001] Various embodiments relate to an electronic device configured to perform an artificial intelligence action using a voice recognition function, and a method for providing a notification related to an action using the electronic device. BACKGROUND

[0002] According to recent developments in voice recognition technology, a voice recognition function can be implemented in various electronic devices having a voice input device (e.g., a microphone). For example, an electronic device can recognize a user utterance through a voice recognition function, determine an intent of the user utterance, and perform an action corresponding to the intent.

[0003] The above information is presented as background information only to assist with an understanding of the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above information could be applicable as prior art with regard to the present disclosure. SUMMARY

[0004] In the case of an electronic device having a voice recognition function according to the related art, if an event occurs, and if the corresponding event is to be recognized, or if an action related to the event is to be performed by an intelligent application, a separate input for executing the intelligent application can be required. For example, as a method for activating an intelligent application, an additional user input such as a wake-up utterance or a press of a physical button can be required.

[0005] To address the above-discussed problems, various embodiments can provide an electronic device and method in which, if an event occurs, an artificial intelligence action using a voice recognition function can be more easily performed.

[0006] An electronic device according to an embodiment can include a communication circuit, a display, a microphone, a processor operatively connected to the communication circuit, the display, and the microphone, and a memory operatively connected to the processor. The memory can be configured to store instructions that, when executed, cause the processor to: receive a specified event related to a second application while displaying an execution screen of a first application on the display; execute an artificial intelligence application in response to the specified event; transmit data related to the specified event to an external server through the communication circuit based on the executed artificial intelligence application; sense a user utterance related to the specified event through the microphone for a specified period of time; transmit the user utterance to the external server through the communication circuit if the user utterance is sensed; receive an action sequence for executing a function related to the user utterance from the external server through the communication circuit; execute the second application based on at least the received action sequence; and output a result of executing the function by using the second application.

[0007] An electronic device according to an embodiment can include a communication circuit; a display; a microphone; a processor operatively connected to the communication circuit, the display, and the microphone; and a memory operatively connected to the processor. The memory can be configured to store instructions that, when executed, cause the processor to: receive a specified event related to a second application when an execution screen of a first application is displayed on the display; execute an artificial intelligence application in response to the specified event; pre-analyze data related to the specified event based on the executed artificial intelligence application; sense a user utterance related to the specified event through the microphone for a specified period of time; determine an action sequence for executing a function related to the user utterance based on a result of the pre-analyzing of the data if the user utterance is sensed; execute the second application based on at least the determined action sequence; and output a result of executing the function by using the second application.

[0008] A method for performing an action by an electronic device using a voice recognition function according to an embodiment can include the following operations: receiving a specified event related to a second application when an execution screen of a first application is displayed on a display; executing an artificial intelligence application in response to the specified event; transmitting data related to the specified event to an external server through a communication circuit based on the executed artificial intelligence application; sensing a user utterance related to the specified event through a microphone for a specified period of time; transmitting the user utterance to the external server through the communication circuit if the user utterance is sensed; receiving an action sequence for executing a function related to the user utterance from the external server through the communication circuit; executing the second application based on at least the received action sequence; and outputting a result of executing the function by using the second application.

[0009] A method for performing an action by an electronic device using a voice recognition function according to an embodiment can include the following operations: receiving a specified event related to a second application when an execution screen of a first application is displayed on a display; executing an artificial intelligence application in response to the specified event; pre-analyzing data related to the specified event based on the executed artificial intelligence application; sensing a user utterance related to the specified event through a microphone for a specified period of time; determining an action sequence for executing a function related to the user utterance based on a result of the pre-analyzing of the data if the user utterance is sensed; executing the second application based on at least the determined action sequence; and outputting a result of executing the function by using the second application.

[0010] An electronic device configured to perform an artificial intelligence action using a voice recognition function and a method for providing a notification related to the action using the electronic device according to various embodiments can more easily perform the artificial intelligence action using the voice recognition function if an event occurs.

[0011] Before the specific embodiments are described, it can be advantageous to set forth definitions of certain words and phrases that have been used throughout this patent document: the terms “include” and “comprise,” as well as derivatives thereof, mean inclusion without limitation; the term “or,” is inclusive, meaning and / or; the phrases “associated with” and “associated therewith,” as well as derivatives thereof, can mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have a property of, have, have a property or be proper gy of, or the like; and the term “controller” means any device, system or part thereof that controls at least one operation, such a device can be implemented in hardware, firmware or software, or some combination of at least two of the same. It should be noted that the functionality associated with any particular controller can be centralized or distributed, whether locally or remotely.

[0012] In addition, various functions described below can be implemented or supported by one or more computer programs, each of which is formed from computer readable program code and embodied in a computer readable medium. The terms “application” and “program” refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof, that are adapted for implementation in a suitable computer readable program code. The phrase “computer readable program code” includes any type of computer code, including source code, object code, and executable code. The phrase “computer readable medium” includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory. A “non-transitory” computer readable medium excludes wired, wireless, optical, or other communication links. The non-transitory computer readable medium includes media where data is permanently stored and media where data is stored and later overwritten, such as a rewritable optical disc or an erasable memory device.

[0013] Definitions for certain words and phrases are provided throughout this patent document, and include the definitions below. The definition of such terms, however, can vary depending on the context in which they are used. Accordingly, the definitions provided herein should not be construed to be limiting. BRIEF DESCRIPTION OF DRAWINGS

[0014] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings in which like parts are marked with like numerals:

[0015] Figure 1 A block diagram of an electronic device inside a network according to an embodiment is illustrated;

[0016] Figure 2A A block diagram of an integrated intelligence system according to an embodiment is illustrated;

[0017] Figure 2B A diagram illustrating a storage type of information about relationships between concepts and actions in a database according to an embodiment is illustrated;

[0018] Figure 3 A diagram illustrating a screen used by an electronic device according to an embodiment for processing a received voice input through an intelligent application is illustrated;

[0019] Figure 4 A diagram illustrating a screen used by an electronic device according to a comparative example for processing a received event through an intelligent application is illustrated;

[0020] Figure 5 A diagram illustrating a screen used by an electronic device according to an embodiment for processing a received event through an intelligent application is illustrated;

[0021] Figure 6 A diagram illustrating an example operation of an electronic device according to an embodiment outputting a user interface for selecting an application in response to receiving an event is illustrated;

[0022] Figure 7 A block diagram of an operating system architecture of an electronic device according to an embodiment is illustrated;

[0023] Figure 8 A block diagram of an example artificial intelligence (AI) platform embedded in an electronic device according to an embodiment is illustrated;

[0024] Figure 9 A diagram illustrating an example screen used by an electronic device according to an embodiment for configuring whether to perform an AI action based on voice recognition related to an event is illustrated;

[0025] Figure 10 A flowchart for describing an operation of an electronic device according to an embodiment is illustrated;

[0026] Figure 11 A flowchart for describing an operation of an electronic device having an embedded AI platform according to an embodiment is illustrated;

[0027] Figure 12 A flowchart for describing an operation of an electronic device according to an embodiment processing a received event when no intelligent application is executed is illustrated;

[0028] Figure 13a flowchart for describing operations of an electronic device according to an embodiment to process a received event when a smart application is executed is shown;

[0029] Figure 14 a flowchart for describing operations of an electronic device according to an embodiment to perform a voice recognition function based on a user interface for selecting an application is shown; and

[0030] Figure 15 an exemplary user interface used by an electronic device according to an embodiment to configure a specific smart application as a default AI service is shown. DETAILED DESCRIPTION

[0031] The following discussion is presented to enable a Figures 1 to 15 The principles of the disclosure discussed below are for the purpose of illustration only and should not be construed in any way as limiting the scope of the disclosure. Those skilled in the art will understand that the principles of the disclosure can be implemented in any suitably arranged system or device.

[0032] Figure 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments. Referring to Figure 1 , the electronic device 101 in the network environment 100 can communicate with an electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or an electronic device 104 or a server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 can include a processor 120, a memory 130, an input device 150, a sound output device 155, a display 160, an audio module 170, a sensor module 176, an interface 177, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a subscriber identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one (e.g., the display 160 or the camera module 180) of the components can be omitted from the electronic device 101, or one or more other components can be added in the electronic device 101. In some embodiments, the components can be implemented as single integrated circuitry. For example, the sensor module 176 (e.g., a fingerprint sensor, an iris sensor, or an illuminance sensor) can be implemented as embedded in the display 160 (e.g., a display).

[0033] The processor 120 can execute, for example, software (e.g., a program 140) to control at least one other component (e.g., a hardware or software component) of the electronic device 101 coupled with the processor 120 and can perform various data processing or computation. According to one embodiment, as at least part of the data processing or computation, the processor 120 can load a command or data received from another component (e.g., the sensor module 176 or the communication module 190) to a volatile memory 132, process the command or the data stored in the volatile memory 132, and store resulting data in a non-volatile memory 134. According to an embodiment, the processor 120 can include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)), and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor 121. Additionally, or alternatively, the auxiliary processor 123 can be adapted to consume less power than the main processor 121, or to be specific to a specified function. The auxiliary processor 123 can be implemented as separate from, or as part of the main processor 121.

[0034] The auxiliary processor 123, rather than the main processor 121, can control at least some of the functions or states related to at least one component of the electronic device 101, e.g., the display 160, the sensor module 176, or the communication module 190, while the main processor 121 is in an inactive (e.g., sleep) state, or the auxiliary processor 123 can cooperate with the main processor 121 to control at least some of the functions or states related to at least one component of the electronic device 101, e.g., the display 160, the sensor module 176, or the communication module 190, while the main processor 121 is in an active state (e.g., executing an application). According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) can be implemented as part of another component functionally related to the auxiliary processor 123 (e.g., the camera module 180 or the communication module 190).

[0035] The memory 130 can store various data used by at least one component (e.g., the processor 120 or the sensor module 176) of the electronic device 101. The various data can include, for example, software (e.g., a program 140) and input data or output data for a command related thereto. The memory 130 can include the volatile memory 132 or the non-volatile memory 134.

[0036] The program 140 can be stored in the memory 130 as software, and can include, for example, an operating system (OS) 142, middleware 144, or an application 146.

[0037] The input device 150 can receive a command or data to be used by other component (e.g., the processor 120) of the electronic device 101, from the outside (e.g., a user) of the electronic device 101. The input device 150 can include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus pen).

[0038] The sound output device 155 can output sound signals to the outside of the electronic device 101. The sound output device 155 can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as playing multimedia or playing record, and the receiver can be used for incoming calls. According to an embodiment, the receiver can be implemented as separate from, or as part of, the speaker.

[0039] The display 160 can visually provide information to the outside (e.g., a user) of the electronic device 101. The display 160 can include, for example, a display, a hologram device, or a projector and a control circuit for controlling a corresponding one of the display, the hologram device, and the projector. According to an embodiment, the display 160 can include a touch circuit adapted to detect a touch, or a sensor circuit (e.g., a pressure sensor) adapted to measure the intensity of force incurred by the touch.

[0040] The audio module 170 can convert a sound into an electrical signal and vice versa. According to an embodiment, the audio module 170 can obtain the sound via the input device 150, or output the sound via the sound output device 155 or a headphone of an external electronic device (e.g., an electronic device 102) directly (e.g., wirelessly) connected with the electronic device 101.

[0041] The sensor module 176 can detect an operational state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) external to the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 can include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0042] The interface 177 can support one or more specified protocols to be used for the electronic device 101 to be coupled with the external electronic device (e.g., the electronic device 102) directly (e.g., wiredly) or wirelessly. According to an embodiment, the interface 177 can include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0043] The connection terminal 178 can include a connector via which the electronic device 101 can be physically connected with the external electronic device (e.g., the electronic device 102). According to an embodiment, the connection terminal 178 can include, for example, a HDMI connector, a USB connector, a SD card connector, or an audio connector (e.g., a headphone connector).

[0044] The haptic module 179 can convert electrical signal into a mechanical stimulus (e.g., a vibration or a movement) or electrical stimulus that can be recognized by users through their tactile sensation or kinesthetic sensation. According to an embodiment, the haptic module 179 can include, for example, a motor, a piezoelectric element, or an electrical stimuluser.

[0045] The camera module 180 can capture still images or moving images. According to an embodiment, the camera module 180 can include one or more lenses, image sensors, image signal processors, or flashes.

[0046] The power management module 188 can manage power supplied to the electronic device 101. According to an embodiment, the power management module 188 can be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0047] The battery 189 can supply power to at least one component of the electronic device 101. According to an embodiment, the battery 189 can include, for example, a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.

[0048] The communication module 190 can support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., the electronic device 102, the electronic device 104, or the server 108) and performing communication between the electronic devices via the established communication channel. The communication module 190 can include one or more communication processors that are operable independently from the processor 120 (e.g., an application processor (AP)) and supports a direct (e.g., wired) communication or a wireless communication. According to an embodiment, the communication module 190 can include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with the external electronic device via the first network 198 (e.g., a short-range communication network, such as Bluetooth, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or the second network 199 (e.g., a long-range communication network, such as a cellular network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)). These various types of communication modules can be implemented as a single component (e.g., a single chip) or can be implemented as separate components (e.g., separate chips) from each other. The wireless communication module 192 can identify and authenticate the electronic device 101 in a communication network, such as the first network 198 or the second network 199, using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module 196.

[0049] The antenna module 197 can transmit or receive a signal or power to or from an external electronic device (e.g., the external electronic device) of the electronic device 101. According to an embodiment, the antenna module 197 can include an antenna including a radiating element composed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a PCB). According to an embodiment, the antenna module 197 can include a plurality of antennas. In this case, at least one antenna appropriate for a communication scheme used in a communication network, such as the first network 198 or the second network 199, can be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). Then, the signal or the power can be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiating element can be additionally formed as part of the antenna module 197.

[0050] At least some of the above-described components can be coupled mutually, and communicate signals (e.g., commands or data) therebetween via an inter-peripheral communication scheme (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).

[0051] According to an embodiment, commands or data can be transmitted or received between the electronic device 101 and the external electronic device 104 via the server 108 coupled with the second network 199. Each of the electronic devices 102 and 104 can be a device of a same type as or different from the electronic device 101. According to an embodiment, all or some of operations to be executed at the electronic device 101 can be executed at one or more of the external electronic devices 102, 104, or server 108. For example, if the electronic device 101 is to automatically perform a function or a service or is to perform a function or a service in response to a request from a user or another device, the electronic device 101, instead of executing the function or the service itself, or in addition to executing the function or the service itself, can request the one or more external electronic devices to execute at least part of the function or the service. The one or more external electronic devices receiving the request can execute the at least part of the function or the service requested, or perform another function or another service related to the request, and transfer an execution result to the electronic device 101. The electronic device 101 can provide the execution result, with or without further processing, as at least part of a reply to the request. To this end, a cloud computing, distributed computing, or client-server computing technology can be used, for example.

[0052] The electronic device (e.g., 101) according to an embodiment can include a communication circuit (e.g., a communication module 190) in Figure 1 The electronic device (e.g., 101) according to an embodiment can include a communication circuit (e.g., a communication module 190) in Figure 1 The electronic device (e.g., 101) according to an embodiment can include a communication circuit (e.g., a communication module 190) in Figure 1 The electronic device (e.g., 101) according to an embodiment can include a communication circuit (e.g., a communication module 190) in Figure 2A The electronic device (e.g., 101) according to an embodiment can include a communication circuit (e.g., a communication module 190) in Figure 1 The electronic device (e.g., 101) according to an embodiment can include a communication circuit (e.g., a communication module 190) in Figure 1 The electronic device (e.g., 101) according to an embodiment can include a communication circuit (e.g., a communication module 190) in Figure 1 The electronic device (e.g., 101) according to an embodiment can include a communication circuit (e.g., a communication module 190) in Figure 2A The electronic device (e.g., 101) according to an embodiment can include a communication circuit (e.g., a communication module 190) in Figure 1Memory 130 (of which 130) is operatively connected to processor 120. Memory 130 can store instructions that, when executed, cause processor 120 to perform the following operations: when a first application is displayed on display 160 (e.g., ... Figure 2A When executing the screen of 135_1), receive the message from the second application (e.g., Figure 2A The specified event (135_2) in the specified event; the execution of an artificial intelligence application in response to the specified event (e.g., Figure 7 752); Based on the executed artificial intelligence application 752, data related to a specified event is sent to an external server (e.g., via communication module 190) through communication module 190. Figure 2Athe user utterance to the intelligent server 201 through the communication module 190; receiving an action sequence for executing a function related to the user utterance from the intelligent server 201 through the communication module 190; executing the second application 135_2 based on at least the received action sequence; and outputting a result of executing the function by using the second application 135_2. The instructions can be configured to cause the processor 120 to execute the following operations: if the artificial intelligence application 752 is executed in response to the designated event, activating the microphone 173; and delivering a user utterance acquired through the activated microphone 173 to the artificial intelligence application 752. The data related to the designated event can include at least one of identification information of the second application 135_2, a time at which the designated event occurs, or a type of the designated event. The instructions can be configured to cause the processor 120 to execute the following operations: in response to receiving the designated event, displaying a user interface for selecting an application on the display 160; and if a user input selecting the second application 135_2 is received through the user interface, starting to sense a user utterance for a designated period of time. The instructions can be configured to cause the processor 120 to execute the following operations: outputting a result of executing the function by using the second application 135_2 in a voice type, a graphic type, or a text type. The instructions can be configured to cause the processor 120 to execute the following operations: in response to receiving the designated event related to the second application 135_2 while the display 160 is deactivated, executing the artificial intelligence application 752; based on the executed artificial intelligence application 752, transmitting data related to the designated event to the intelligent server 201 through the communication module 190; sensing a user utterance related to the designated event through the microphone 173 for a designated period of time; if the user utterance is sensed, transmitting the user utterance to the intelligent server 201 through the communication module 190; receiving an action sequence for executing a function related to the user utterance from the intelligent server 201 through the communication module 190; executing the second application 135_2 based on at least the received action sequence; and outputting a result of executing the function by using the second application 135_2. The instructions can be configured to cause the processor 120 to execute the following operations: if the designated event is received while an execution screen of the first application 135_1 is displayed on the display 160, identifying a user configuration regarding the first application 135_1; and determining whether to execute the artificial intelligence application 752 based on the user configuration.

[0053] The electronic device 101 according to an embodiment can include a communication module 190, a display 160, a microphone 173, a processor 120 operatively connected to the communication module 190, the display 160, and the microphone 173, and a memory 130 operatively connected to the processor 120. The memory 130 can store instructions that, when executed, cause the processor 120 to display an execution screen of a first application 135_1 on the display 160, receive a designated event related to a second application 135_2, execute an artificial intelligence application 752 in response to the designated event, pre-analyze data related to the designated event based on the executed artificial intelligence application 752, sense a user utterance related to the designated event through the microphone 173 for a designated period of time, if the user utterance is sensed, determine an action sequence for executing a function related to the user utterance based on a result of the pre-analyzing of the data, execute the second application 135_2 based on at least the determined action sequence, and output a result of executing the function by using the second application 135_2. The instructions can be configured to cause the processor 120 to activate the microphone 173 if the artificial intelligence application 752 is executed in response to the designated event, and deliver a user utterance acquired through the activated microphone 173 to the artificial intelligence application 752. The data related to the designated event can include at least one of identification information of the second application 135_2, a time at which the designated event occurs, or a type of the designated event. The instructions can be configured to cause the processor 120 to display a user interface for selecting an application on the display 160 in response to receiving the designated event, and if a user input selecting the second application 135_2 is received through the user interface, start sensing the user utterance for the designated period of time. The instructions can be configured to cause the processor 120 to execute the artificial intelligence application 752 in response to receiving the designated event related to the second application 135_2 while the display 160 is deactivated, pre-analyze data related to the designated event based on the executed artificial intelligence application 752, sense a user utterance related to the designated event through the microphone 173 for a designated period of time, if the user utterance is sensed, determine an action sequence for executing a function related to the user utterance based on a result of the pre-analyzing of the data, execute the second application 135_2 based on at least the determined action sequence, and output a result of executing the function by using the second application 135_2. The instructions can be configured to cause the processor 120 to identify a user configuration regarding the first application 135_1 if the designated event is received while the execution screen of the first application 135_1 is displayed on the display 160, and determine whether to execute the artificial intelligence application 752 based on the user configuration.

[0054] Figure 2A A block diagram 200 of an integrated intelligence system according to an embodiment is illustrated.

[0055] Referring to Figure 2A , the integrated intelligent system of the block diagram 200 according to an embodiment can include the electronic device 101, the intelligent server 201, and the service server 260.

[0056] In an embodiment, the electronic device 101 can be a terminal device capable of connecting to the Internet, and can be, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a TV, a home appliance, an HMD, or a smart speaker.

[0057] According to an embodiment, the electronic device 101 can include an interface (e.g., the interface 177 in the electronic device 101), a microphone 173, a speaker 171, a display (e.g., the display 160 in the electronic device 101), a memory (e.g., the memory 130 in the electronic device 101), or a processor (e.g., the processor 120 in the electronic device 101). The listed elements can be operatively or electrically connected to each other. Figure 1 Figure 1 According to an embodiment, the interface 177 can be connected to an external device (e.g., the electronic device 102 or the electronic device 104 or the server 108 in the electronic device 101) and configured to transmit / receive data. According to an embodiment, the microphone 173 can receive a sound (e.g., a user utterance) and convert the received sound into an electrical signal. According to an embodiment, the speaker 171 can output an electrical signal as a sound (e.g., a voice). According to an embodiment, the display 160 can be configured to display an image or a video. According to an embodiment, the display 160 can display a graphical user interface (GUI) of an executed application (or an application program). Figure 1 Figure 1 According to an embodiment, the memory 130 can store a client module 131, a software development kit (SDK) 133, and a plurality of applications 135. The client module 131 and the SDK 133 can constitute a framework (or a solution) for performing a multi-function. In addition, the memory 130 and the client module 131 or the SDK 133 can constitute a framework for processing a voice input.

[0058] According to an embodiment, the plurality of applications 135 stored in the memory 130 can be programs for performing a designated function. According to an embodiment, the plurality of applications 135 can include a first application 135_1, a second application 135_2, or an event manager 137. According to an embodiment, each of the plurality of applications 135 can include a plurality of actions for performing a designated function. For example, the plurality of applications 135 can include at least one of an alarm application, a message application, and a schedule application. According to an embodiment, the plurality of applications 135 can be executed by the processor 120 to continuously perform at least some of the plurality of actions. Figure 1

[0059]

[0060] According to an embodiment, the plurality of applications 135 stored in the memory 130 can be programs for performing a designated function. According to an embodiment, the plurality of applications 135 can include a first application 135_1, a second application 135_2, or an event manager 137. According to an embodiment, each of the plurality of applications 135 can include a plurality of actions for performing a designated function. For example, the plurality of applications 135 can include at least one of an alarm application, a message application, and a schedule application. According to an embodiment, the plurality of applications 135 can be executed by the processor 120 to continuously perform at least some of the plurality of actions. ​​​​

[0061] According to an embodiment, the event manager 137 can determine an application (or a domain) for processing an event occurring in the electronic device 101 based on a configuration value input by a user. For example, if a specified event occurs in the electronic device 101, the event manager 137 according to an embodiment can identify a configuration value regarding an application currently being executed and can determine whether to perform a subsequent action related to the specified event based on the configuration value. According to an embodiment, the event manager 137 can ignore the specified event based on a configuration value corresponding to the application currently being executed, or can transmit information (or data) related to the specified event to an artificial intelligence (AI) system (e.g., an event proactive module (EPM) 265) through the client module 131.

[0062] According to an embodiment, the memory 130 of the electronic device 101 can store a plurality of intelligent applications. According to an embodiment, the plurality of intelligent applications can be based on voice recognition, and the electronic device 101 can pre-configure a default application (or a root application) from the plurality of intelligent applications based on a user input. For example, the electronic device 101 can have a plurality of intelligent applications installed therein, such as BIXBY TM or GOOGLE ASSISTANT TM , and can configure the BIXBY TM application as a default application based on a user input. According to an embodiment, the event manager 137 can schedule an event specified based on a configuration value of an application currently being executed so that the event is processed by the intelligent application configured as the default application. Accordingly, even if a plurality of intelligent applications are installed in the electronic device 101, the electronic device 101 can flexibly process a user utterance. For example, if a specified event (e.g., message reception) occurs while the GOOGLE ASSISTANT TM application is being executed, and if a user utterance is received after the specified event occurs, the electronic device 101 according to an embodiment can process the user utterance in the BIXBY TM application configured as the default application.

[0063] According to an embodiment, the processor 120 can control all actions of the electronic device 101. For example, the processor 120 can be electrically connected to the interface 177, the microphone 173, the speaker 171, the display 160, and the memory 130 to perform a specified action.

[0064] According to an embodiment, the processor 120 can further execute a program stored in the memory 130, thereby performing a designated function. For example, the processor 120 can execute at least one of the client module 131 or the SDK 133, thereby performing a subsequent action for processing a voice input. For example, the processor 120 can control an action of a plurality of applications 135 through the SDK 133. The subsequent action described as an action of the client module 131 or the SDK 133 can be an action resulting from the execution of the processor 120.

[0065] According to an embodiment, the client module 131 can receive a voice input. For example, the client module 131 can receive a voice signal corresponding to a user utterance sensed through the microphone 173. The client module 131 can transmit the received voice input to the intelligent server 201. According to an embodiment, the client module 131 can transmit state information of the electronic device 101 to the intelligent server 201 together with the received voice input. For example, the state information can be information about a state of executing an application.

[0066] According to an embodiment, the client module 131 can receive a result corresponding to the received voice input. For example, the client module 131 can receive a result corresponding to the voice input from the intelligent server 201. The client module 131 can display the received result on the display 160.

[0067] According to an embodiment, the client module 131 can receive a plan corresponding to the received voice input. The client module 131 can display a result of performing a plurality of actions of an application according to the plan on the display 160. For example, the client module 131 can continuously display the result of performing the plurality of actions on the display 160. As another example, the electronic device 101 can display only some of the results (e.g., a result of a last action) of performing the plurality of actions on the display 160.

[0068] According to an embodiment, the client module 131 can receive a request for acquiring information for obtaining a result corresponding to a voice input from the intelligent server 201. The information for obtaining the result can be, for example, state information of the electronic device 101. According to an embodiment, the client module 131 can transmit necessary information to the intelligent server 201 in response to the request.

[0069] According to an embodiment, the client module 131 can transmit information about a result of performing a plurality of actions according to a plan to the intelligent server 201. The intelligent server 201 can confirm that the received voice input has been processed correctly based on the result information.

[0070] According to an embodiment, the client module 131 can include a voice recognition module (not shown). According to an embodiment, the client module 131 can receive a voice input for performing a limited function through the voice recognition module (not shown). For example, the client module 131 can execute an intelligent application (or AI application) (e.g., BIXBY TM ) for processing a voice input for performing a designated input (e.g., wake up!) to perform an organic action.

[0071] According to an embodiment, the intelligent server 201 can receive information related to a user voice input from the electronic device 101 through a communication network. According to an embodiment, the intelligent server 201 can convert data related to the received user voice input into text data. According to an embodiment, the intelligent server 201 can generate a plan for performing a task (e.g., a function) corresponding to the user voice input based on the text data.

[0072] According to an embodiment, the plan (e.g., an action sequence) can be generated by an artificial intelligence (AI) system. The AI system can be a rule-based system, or can be a neural network-based system (e.g., a feed-forward neural network (FNN) or a recurrent neural network (RNN)). Alternatively, the AI system can be a combination of the above-described systems, or can be an AI system different from the above-described systems. According to an embodiment, the plan can be selected from a pre-defined plan set, or can be generated in real time in response to a user request. For example, the AI system can select at least one plan from a plurality of pre-defined plans.

[0073] According to an embodiment, the intelligent server 201 can transmit a result following the generated plan to the electronic device 101, or can transmit the generated plan to the electronic device 101. According to an embodiment, the electronic device 101 can display a result obtained according to the plan on the display 160. According to an embodiment, the electronic device 101 can display a result of performing an action according to the plan on the display 160. According to an embodiment, the electronic device 101 can output a result of performing an action according to the plan in a voice type, a graphic type, or a text type.

[0074] The intelligent server 201 according to an embodiment can include a front end 210, a natural language platform 220, a capsule DB 230, an execution engine 235, an end user interface 240, a management platform 245, a big data platform 250, an analytics platform 255, or an event proactive module (EPM) 265.

[0075] According to an embodiment, the front end 210 can receive a voice input from the electronic device 101. The front end 210 can transmit a response corresponding to the voice input.

[0076] According to an embodiment, the natural language platform 220 can include an automatic speech recognition (ASR) module 221, a natural language understanding (NLU) module 223, a planner module 225, a natural language generator (NLG) module 227, or a text-to-speech (TTS) module 229.

[0077] According to an embodiment, the ASR module 221 can convert a voice input received from the electronic device 101 into text data. According to an embodiment, the NLU module 223 can identify an intention of a user by using the text data of the voice input. For example, the NLU module 223 can identify the intention of the user by performing syntactic analysis or semantic analysis. According to an embodiment, the NLU module 223 can identify a meaning of a word extracted from the voice input by using a linguistic feature (e.g., a syntactic element) of a morpheme or a phrase, and can match the identified meaning of the word with an intention, thereby determining the intention of the user.

[0078] According to an embodiment, the planner module 225 can generate a plan by using the intention and parameters determined by the NLU module 223. According to an embodiment, the planner module 225 can determine a plurality of domains required to perform a task based on the determined intention. The planner module 225 can determine a plurality of actions included in each of the plurality of domains determined based on the intention. According to an embodiment, the planner module 225 can determine parameters required to perform the determined plurality of actions, or can determine a result value output by performing the plurality of actions. The parameters and the result value can be defined as concepts related to a specified format (or class). Accordingly, the plan can include the plurality of actions determined by the intention of the user and the plurality of concepts. The planner module 225 can determine a relationship between the plurality of actions and the plurality of concepts in a step-by-step (or hierarchical) manner. For example, the planner module 225 can determine an order of performing the plurality of actions based on the plurality of concepts, based on the intention of the user. In other words, the planner module 225 can determine an order of performing the plurality of actions based on parameters required to perform the plurality of actions and a result output by performing the plurality of actions. Accordingly, the planner module 225 can generate a plan including association information (e.g., an ontology) between the plurality of actions and the plurality of concepts. The planner module 225 can generate the plan by using information stored in the capsule DB 230, which stores a set of relationships between concepts and actions.

[0079] According to an embodiment, the NLG module 227 can convert specified information into a text type. The information converted into the text type can be a natural language utterance type. The TTS module 229 in an embodiment can convert the text type information into a voice type information.

[0080] According to an embodiment, the package DB 230 can store information about relationships between a plurality of concepts and a plurality of actions corresponding to a plurality of domains. For example, the package DB 230 can include a plurality of action objects (or action information) of a plan and a concept object (or concept information) of the plurality of action objects. According to an embodiment, the package DB 230 can store a plurality of packages in a concept-action network (CAN) type. According to an embodiment, the plurality of packages can be stored in a function registry included in the package DB 230.

[0081] According to an embodiment, the package DB 230 can include a policy registry that stores policy information required to determine a plan corresponding to a voice input. If there are a plurality of plans corresponding to a voice input, the policy information can include reference information for determining one plan. According to an embodiment, the package DB 230 can include a follow-up registry that stores follow-up action information for proposing a follow-up action to a user in a specified situation. The follow-up action can include, for example, a follow-up utterance. According to an embodiment, the package DB 230 can include a layout registry that stores layout information of information output by the electronic device 101. According to an embodiment, the package DB 230 can include a vocabulary registry that stores vocabulary information included in package information. According to an embodiment, the package DB 230 can include a dialogue registry that stores information about a dialogue (or interaction) with a user.

[0082] According to an embodiment, the package DB 230 can update the stored objects through a developer tool. The developer tool can include, for example, a function editor for updating an action object or a concept object. The developer tool can include a vocabulary editor for updating a vocabulary. The developer tool can include a policy editor for generating and registering a policy for determining a plan. The developer tool can include a dialogue editor for generating a dialogue with a user. The developer tool can include a follow-up editor capable of activating a follow-up target and editing a follow-up utterance that provides a prompt. The follow-up target can be determined based on a currently configured target, a user's preference, or an environmental condition.

[0083] According to an embodiment, the package DB 230 can also be implemented inside the electronic device 101. In other words, the electronic device 101 can include the package DB 230 that stores information for determining an action corresponding to a voice input.

[0084] According to an embodiment, the execution engine 235 can obtain a result by using the generated plan. According to an embodiment, the end user interface 240 can transmit the obtained result to the electronic device 101. Accordingly, the electronic device 101 can receive the result and provide the received result to the user. According to an embodiment, the management platform 245 can manage information used by the intelligent server 201. According to an embodiment, the big data platform 250 can collect data of the user.

[0085] According to an embodiment, the analysis platform 255 can manage a quality of service (QoS) of the intelligent server 201. For example, the analysis platform 255 can manage elements of the intelligent server 201 and a processing rate (or efficiency).

[0086] According to an embodiment, the service server 260 can provide a designated service (e.g., food ordering or hotel reservation) to the electronic device 101. According to an embodiment, the service server 260 can be operated by a third party. For example, the service server 260 can include a first service server 261, a second service server 262, and a third service server 263 operated by different third parties. According to an embodiment, the service server 260 can provide information for generating a plan corresponding to a received voice input to the intelligent server 201. The provided information can be stored in, for example, the capsule DB 230. In addition, the service server 260 can provide information about a result of following the plan to the intelligent server 201.

[0087] According to an embodiment, if a designated event (e.g., message reception) occurs in the electronic device 101, the EPM 265 can pre-analyze data related to the event, and can determine an application (or domain) for processing a user utterance received by the electronic device 101 after the event occurs. For example, the EPM 265 can determine whether to process a user utterance received after the event occurs by an application currently executed in the electronic device 101 or to determine the user utterance as a root utterance so that the user utterance is processed by a new application (or domain) according to the data.

[0088] According to an embodiment, the data related to the event can include at least one of identification information of a first application currently executed in the electronic device 101, identification information of a second application related to the event, an event occurrence time, or an event type. The EPM 265 can receive the data from the electronic device 101 and can pre-analyze the received data before a user utterance. According to an embodiment, if the electronic device 101 senses a user utterance, the EPM 265 can determine an application (or domain) for processing the user utterance based on a result of the pre-analysis. For example, if a first application is currently executed in the electronic device 101 and if a task corresponding to the user utterance can be performed by a second application, the EPM 265 according to an embodiment can determine that the task is to be performed by the second application.

[0089] According to an embodiment, if the electronic device 101 has an embedded intelligent platform, the EPM 265 can be included in the electronic device 101. In this case, the electronic device 101 can pre-analyze data related to an event independently of the intelligent server 201, and can determine a domain (e.g., an application) for processing a user utterance received by the electronic device 101 after the event occurs (see Figure 8 ).

[0090] In connection with the integrated intelligent system of the above-described block diagram 200, the electronic device 101 can provide various intelligent services to a user in response to a user input. The user input can include, for example, an input through a physical button, a touch input, or a voice input.

[0091] According to an embodiment, the electronic device 101 can provide a voice recognition service through a smart application (or a voice recognition application) stored in the electronic device 101. In this case, for example, the electronic device 101 can recognize a user utterance or a voice input received through the microphone 173, and can provide a service corresponding to the recognized voice input to the user.

[0092] According to an embodiment, based on a received voice input, the electronic device 101 can individually perform a designated action or perform the designated action together with the intelligent server 201 and / or the service server 260. For example, the electronic device 101 can execute an application corresponding to the received voice input, and can perform a designated action through the executed application.

[0093] According to an embodiment, when the electronic device 101 provides a service together with the intelligent server 201 and / or the service server 260, the electronic device 101 can sense a user utterance by using the microphone 173, and can generate a signal (or voice data) corresponding to the sensed user utterance. The electronic device 101 can transmit the voice data to the intelligent server 201 by using the interface 177.

[0094] According to an embodiment, in response to a voice input received from the electronic device 101, the intelligent server 201 can generate a plan for performing a task corresponding to the voice input, or generate a result of performing an action according to the plan. The plan can include, for example, a plurality of actions for performing a task corresponding to a user's voice input and a plurality of concepts related to the plurality of actions. The concept can correspond to a definition of a parameter input to perform the plurality of actions or a result value output by performing the plurality of actions. The plan can include information about an association between the plurality of actions and the plurality of concepts.

[0095] The electronic device 101 in the embodiment can receive a response by using the interface 177. The electronic device 101 can output a voice signal generated inside the electronic device 101 to the outside using the speaker 171, or can output an image generated inside the electronic device 101 to the outside using the display 160.

[0096] Figure 2B A diagram 270 showing a storage type of information about a relationship between a concept and an action in a database.

[0097] Referring to Figure 2B , a capsule database (e.g., the capsule database 230 in Figure 2A ) of a smart server (e.g., the smart server 201 in Figure 2A ) can store a capsule in a concept-action network (CAN) type. The capsule database can store an action for processing a task corresponding to a voice input of a user and a parameter required for the action in the CAN type. The CAN can represent an organic relationship between an action and a concept, in which the concept defines a parameter required to perform the action.

[0098] The capsule database can store a plurality of capsules (e.g., a capsule A 271 and a capsule B 274) respectively corresponding to a plurality of domains (e.g., applications). According to an embodiment, one capsule (e.g., the capsule A 271) can correspond to one domain (e.g., an application). Also, one capsule can correspond to at least one service provider (e.g., CP1 272, CP2 273, CP3 276, or CP4 275) for performing a function of a domain related to the capsule. According to an embodiment, one capsule can include at least one action 280 for performing a designated function and at least one concept 290.

[0099] According to an embodiment, a natural language platform (e.g., the natural language platform 220 in Figure 2A ) can generate a plan for performing a task corresponding to a received voice input by using a capsule stored in a capsule database. For example, a planner module (e.g., the planner module 225 in Figure 2A ) of the natural language platform can generate a plan by using a capsule stored in the capsule database. For example, a plan 277 can be generated by using the actions 2711 and 2713 and the concepts 2712 and 2714 of the capsule A 271 and the action 2741 and the concept 2742 of the capsule B 274.

[0100] Figure 3 A diagram 300 showing a screen for processing a received voice input by a smart application, used by an electronic device.

[0101] The electronic device 101 can execute a smart application in order to process a user input through a smart server 201.

[0102] According to an embodiment, on the screen 310, the electronic device 101 can execute a smart application for processing a voice input when a designated voice input (e.g., Wake up!) is received or when an input is received through a hardware key (e.g., a dedicated hardware key or a designated physical key). For example, the electronic device 101 can execute a smart application while executing a schedule application. According to an embodiment, the electronic device 101 can display an object (e.g., an icon) 311 corresponding to the smart application on the display 160. According to an embodiment, the electronic device 101 can receive a voice input generated by a user utterance. For example, the electronic device 101 can receive a voice input "Let me know the schedule for this week!". According to an embodiment, the electronic device 101 can display a user interface (UI) 313 (e.g., an input window) on which text data of the received voice input is displayed on the display 160.

[0103] According to an embodiment, on the screen 320, the electronic device 101 can display a result corresponding to the received voice input on the display 160. For example, the electronic device 101 can receive a plan corresponding to the received user input, and can display "This week schedule" on the display 160 according to the plan.

[0104] Figure 4 A diagram illustrating a screen for processing a received event through a smart application by an electronic device according to a comparative example is shown. Figure 5 A diagram illustrating a screen for processing a received event through a smart application by an electronic device according to an embodiment is shown.

[0105] Referring to Figure 4 According to a comparative example, on the screen 410, the electronic device can execute a smart application in response to receiving a wake-up utterance (e.g., Hi Bixby!) or receiving an input made by pressing a physical button, and can sense a user utterance by activating a voice recognition function after the smart application is executed. For example, on the screen 410 according to the comparative example, if the electronic device receives a user utterance such as "Hi Bixby, play music 1 in the music application", the electronic device can execute a music application and play music 1.

[0106] According to a comparative example, on the screen 420, the electronic device can receive a further user utterance while the music application is executed, and if the corresponding user utterance is related to the music application, the electronic device can process a function related to the user utterance through the music application. For example, on the screen 420 according to the comparative example, if the electronic device receives a user utterance such as "play the next song", the electronic device can play the next song (e.g., music 2) through the music application.

[0107] At operation 425, the electronic device according to the comparative example can receive a designated event, for example, a message, while executing the music application. For example, according to the comparative example, the designated event can be related to another application (for example, a message application) that cannot be processed by the application (for example, the music application) currently executed in the electronic device.

[0108] According to the comparative example, on screen 430, the electronic device can receive a user utterance related to the designated event after receiving the designated event. For example, on screen 430, the electronic device can receive a user utterance such as "read the message that just arrived." Such a user utterance can be related to the designated event that occurred previously, but the designated event and the user utterance can not be processed by the music application (or the music domain) currently executed in the electronic device.

[0109] According to the comparative example, on screen 440, upon confirming that the designated event and the user utterance cannot be processed by the music application (or the music domain) currently executed in the electronic device, the electronic device can notify that the corresponding user utterance cannot be processed. For example, the electronic device according to the comparative example can output an error message such as "not found, please say again."

[0110] As such, the electronic device according to the comparative example can have an inconvenience in that, if an event occurs, and if the user wants to check the corresponding event or wants to perform an action related to the event through a smart application by using a voice recognition function, the user has to re-input a wake-up utterance or a physical key.

[0111] Referring to Figure 5 , the electronic device (for example, the electronic device 101 in Figure 2A ) according to an embodiment can solve the inconvenience of the electronic device according to the comparative example in Figure 4 . For example, based on an event such as a notification or an incoming call, the electronic device 101 according to an embodiment can guarantee that a user's utterance is more clearly processed according to an actual intention, even if 1) the terminal is idle (for example, the display is deactivated), 2) if the terminal enters a specific application and settles down (for example, a state other than "punch out"), or 3) if a specific application has an embedded voice recognition service and thus occupies a voice processing module (for example, the microphone 173 in Figure 2A ). Accordingly, a user who recognizes that an event has occurred can input an utterance related to the event, regardless of a current state of an application in the electronic device 101, or regardless of any state in which a mixture of a plurality of voice services exists, thereby receiving a result of a task related to the utterance from the electronic device 101. For example, according to an embodiment, if an event occurs, and if the user wants to check the corresponding event or wants to perform an action related to the event through a smart application (for example, a message application), the user can input an utterance such as "read the message that just arrived," and the electronic device 101 according to an embodiment can output a result of the corresponding message application (for example, a message that just arrived) to the user.Figure 7 BIXBY in TM The application 752) performs an action related to an event by using a voice recognition function, the electronic device 101 can provide a voice recognition function through the intelligent application 752 without requesting the user to re-input a wake-up utterance or a physical key. It should be noted that, as used herein, BIXBY TM The application 752 and the intelligent application 752 can be used interchangeably.

[0112] According to an embodiment, on the screen 510, the electronic device 101 can execute the intelligent application 752 in response to receiving a wake-up utterance (e.g., Hi Bixby!!) or receiving an input made by pressing a physical button, and can sense a user utterance by activating a voice recognition function after the intelligent application 752 is executed. For example, on the screen 510, if the electronic device 101 receives a user utterance such as "Hi Bixby, play music 1 in the music application," the electronic device 101 can execute the music application (e.g., the first application 135_1 in Figure 2A

[0113] According to an embodiment, on the screen 520, the electronic device 101 can receive a further user utterance while the music application 135_1 is executed, and if the corresponding user utterance is related to the music application 135_1, the electronic device 101 can process a function related to the user utterance through the music application 135_1. For example, on the screen 520, if the electronic device 101 receives a user utterance such as "play the next song," the electronic device 101 can play the next song (e.g., music 2) through the music application 135_1.

[0114] In operation 525, the electronic device 101 according to an embodiment can receive a designated event, e.g., a message, while the music application 135_1 is executed. For example, the designated event can be related to another application (e.g., the second application 135_2 in Figure 2A

[0115] ​​According to an embodiment, on the screen 530, the electronic device 101 can receive a user utterance related to the specified event after receiving the specified event. For example, on the screen 530, the electronic device 101 can receive a user utterance such as "read the message just arrived." Such a user utterance can be related to the specified event that occurred previously, but the specified event and the user utterance can not be processed by the music application 135_1 (or the music domain) that is currently executed in the electronic device 101. According to an embodiment, on the screen 530, the electronic device 101 can sense the user utterance within a specified period after receiving the specified event. For example, the electronic device 101 can sense the user utterance within a specified period (e.g., ten seconds) after receiving the message. According to an embodiment, on the screen 530, if the user utterance is not sensed within the specified period after receiving the specified event, the electronic device 101 can maintain the execution screen of the application (e.g., the music application 135_1) before receiving the event. According to an embodiment, the ten seconds as the specified period is only an example, and various modifications or configurations can be made to the specified period. According to an embodiment, the specified period can be substantially the same as the period between the activation of the display (e.g., the display 160) and the deactivation of the display in the display 160. Figure 1 For example, the electronic device 101 can activate the display 160 for the specified period in response to receiving the event, and if the display 160 is deactivated after the specified period elapses, the voice reception action can be stopped.

[0116] According to an embodiment, on the screen 540, the electronic device 101 can temporarily execute another application (or a second application) (e.g., a message application 135_2) capable of processing the designated event and the user utterance, upon confirming that the designated event and the user utterance can not be processed by an application (or a first application) (e.g., a music application 135_1) currently executed in the electronic device 101. The electronic device 101 according to an embodiment can perform a task corresponding to the user utterance and can output a result of performing the task through another application (e.g., the message application 135_2). For example, on the screen 540, the electronic device 101 according to an embodiment can output the content of the message received in operation 525, such as "the sender is Jane, the appointment at 9 o'clock tomorrow is …", in a voice type, a graphic type, or a text type. According to an embodiment, on the screen 540, if a designated period of time elapses after performing the task corresponding to the user utterance and outputting the result of performing the task, the electronic device 101 can return to the execution screen of the application (e.g., the music application 135_1) before the event is received. According to an embodiment, on the screen 540, the electronic device 101 can display a user interface 541 for returning to the execution screen of the application (e.g., the music application 135_1) before the event is received, and if a user input is sensed through the user interface 541, can display the execution screen of the previous application (e.g., the music application 135_1).

[0117] According to an embodiment, in order to return to the execution screen of the previous application (e.g., the music application 135_1), the electronic device 101 can store a time configuration (e.g., a timer configuration) for connecting (e.g., a session) with the intelligent server (e.g., the intelligent server 201 in Figure 2A

[0118] Figure 6 A diagram illustrating an example operation of an electronic device according to an embodiment showing outputting a user interface for selecting an application in response to receiving an event is illustrated.

[0119] Referring to Figure 6 When an event occurs, the electronic device (e.g., the electronic device 101 in Figure 2A may determine an application (or a domain) for processing a user utterance related to the event based on a user input through the user interface 631.

[0120] According to an embodiment, on the screen 610, the electronic device 101 can execute a smart application (e.g., BIXBY in Figure 7 TM ​​application 752), and can sense a user utterance by activating a voice recognition function after the intelligent application 752 is executed. For example, on screen 610, if the electronic device 101 receives a user utterance such as "Hi Bixby, play music 1 in the music application," the electronic device 101 can execute the music application (e.g., the first application 135_1 in the music application 752) and play music 1. Figure 2A

[0121] According to an embodiment, on screen 620, the electronic device 101 can receive a further user utterance while the music application 135_1 is executed, and if the corresponding user utterance is related to the music application 135_1, can process a function related to the user utterance through the music application 135_1. For example, on screen 620, if the electronic device 101 receives a user utterance such as "play the next song," the electronic device 101 can play the next song (e.g., music 2) through the music application 135_1.

[0122] In operation 625, the electronic device 101 according to an embodiment can receive a designated event, for example, a message, while the music application 135_1 is executed. For example, the designated event can be related to another application (e.g., the second application 135_2 in the message application) that cannot be processed by the application (e.g., the music application 135_1) currently executed in the electronic device 101. Figure 2A

[0123] According to an embodiment, on screen 630, the electronic device 101 can display a user interface 631 for replacing the currently executed application (e.g., the first application 135_1 or the current domain) with another application (e.g., the second application 135_2 or another domain) related to the designated event in response to receiving the designated event. For example, if the event is received, the electronic device 101 can determine an application (e.g., the second application 135_2 or another domain) capable of processing a task related to the event based on an artificial intelligence (AI) system (e.g., the event proactive module (EPM) 265 in the AI system 750), and can display a user interface 631 capable of selecting the determined application 135_2 for replacement. For example, the user interface 631 can include an object 6311 capable of selecting the message application 135_2 or the music application 135_1 as on screen 630. Figure 2A

[0124] ​​​According to an embodiment, on screen 630, upon receiving a user input for selection of change to another application (e.g., second application 135_2 or another domain) through user interface 631, the electronic device 101 can sense a user utterance for a designated period of time. For example, on screen 630, the electronic device 101 can receive a user utterance such as "read the message just arrived." For example, the electronic device 101 can sense a user utterance for a designated period of time (e.g., ten seconds) after receiving a user input through the user interface 631. According to an embodiment, on screen 630, if a user utterance is not sensed for the designated period of time, the electronic device 101 can maintain the execution screen of the application (e.g., music application 135_1) before the event is received. According to an embodiment, the ten seconds as the designated period of time is merely an example, and various modifications or configurations can be made to the designated period of time.

[0125] According to an embodiment, on screen 640, the electronic device 101 can execute another application (or second application 135_2) (e.g., message application 135_2) capable of processing the designated event and the user utterance. The electronic device 101 according to an embodiment can perform a task corresponding to the user utterance through the other application 135_2, and can output a result of performing the task. For example, on screen 640, the electronic device 101 according to an embodiment can output the content of the message received in operation 625, such as "the sender is Jane, the appointment at 9 o'clock tomorrow is...", in a voice type, a graphic type, or a text type. According to an embodiment, on screen 640, if a designated period of time elapses after performing a task corresponding to the user utterance and outputting a result of performing the task, the electronic device 101 can return to the execution screen of the application (e.g., music application 135_1) before the event is received. According to an embodiment, on screen 640, the electronic device 101 can display a user interface 641 for returning to the execution screen of the application (e.g., music application 135_1) before the event is received, and if a user input is sensed through the user interface 641, can display the execution screen of the previous application (e.g., music application 135_1).

[0126] Figure 7 A block diagram illustrating an operating system architecture of an electronic device according to an embodiment is shown.

[0127] Referring to Figure 7 The operating system architecture 700 of the electronic device 101 according to an embodiment can include hardware 710, a Linux kernel 720, a hardware abstraction layer (HAL) 730, an application framework 740, or a plurality of applications 750. The operating system architecture 700 of the electronic device 101 according to an embodiment can be, for example, an ANDROID architecture.

[0128] According to an embodiment, the hardware 710 can include physical components constituting the electronic device 101. For example, the hardware 710 can include a microphone 173 (e.g., 173 in FIG. 1), a display (not shown) (e.g., 160 in FIG. 1), a speaker (not shown) (e.g., 171 in FIG. 1), or a camera module (not shown) (e.g., 180 in FIG. 1). Figure 2A Figure 2A Figure 2A Figure 1

[0129] According to an embodiment, the Linux kernel 720 (i.e., the drivers 721, 722, and 723) can include a set of instructions required to control the hardware 710. For example, the Linux kernel 720 can include a microphone driver 721 for controlling the microphone 173, a display driver 722 for controlling the display 160, or an audio driver 723 for controlling the speaker 171.

[0130] According to an embodiment, the HAL 730 can be a logical code segment that plays a role of an abstraction layer between the hardware 710 and the software. The HAL 730 can include a general instruction set regarding the hardware 710, and can provide a library module and a device driver interface of the hardware 710, so that a program can communicate with the hardware.

[0131] According to an embodiment, the application framework 740 can be an aggregation of a standard structure for operating an application of the operating system in a class or a library type. According to an embodiment, the application framework 740 can be a JAVA TM API framework. According to an embodiment, the application framework 740 can include a notification manager 741, a resource manager 742, or a package manager 743.

[0132] According to an embodiment, the plurality of applications 750 (e.g., 135 in FIG. 1) can be programs for performing a specified function. For example, each of the plurality of applications 750 can include a plurality of actions for performing a specified function. For example, the plurality of applications 750 can include at least one of an alarm application 754, a message application 753, or a map application 751. According to an embodiment, the plurality of applications 750 can be executed by a processor (e.g., 120 in FIG. 1) to continuously perform at least some of the plurality of actions. Figure 2A Figure 1 Figure 2A

[0133] According to an embodiment, the plurality of applications 750 can store a plurality of intelligent applications different from each other. According to an embodiment, the plurality of applications can be intelligent applications based on voice recognition (e.g., BIXBY TM ​​​​​​​application 752), and the electronic device 101 can pre-configure a default application (or root application) among the plurality of intelligent applications based on a user input. For example, the electronic device 101 can have a plurality of intelligent applications installed therein, such as BIXBY TM application 752 or GOOGLE ASSISTANT TM application (not shown), and can configure BIXBY TM application 752 as a default application based on a user input. According to an embodiment, the intelligent application 752 (e.g., BIXBY TM application) configured as a default application can include an event manager (e.g., 137 in Figure 2A

[0134] According to an embodiment, if a specified event occurs in the electronic device 101, the event manager 137 can identify a configuration value related to an application currently being executed, and can determine whether to perform a subsequent action related to the specified event based on the configuration value. According to an embodiment, the event manager 137 can ignore the specified event based on the configuration value corresponding to the application currently being executed, or can transmit information (or data) about the specified event to an artificial intelligence (AI) system. According to an embodiment, the AI system can be the intelligent server 201 including the event proactive module 265. According to an embodiment, the electronic device 101 can have an embedded AI system (e.g., 810 in Figure 8 Figure 8 Figure 8

[0135] Hereinafter, operations of the electronic device 101 according to an embodiment will be described with reference to Figure 7

[0136] In operation 761, an event can occur in a specific application (e.g., an alarm application 754) among the plurality of applications 750, and the electronic device 101 according to an embodiment can deliver corresponding event occurrence information to the notification manager 741 of the application framework 740.

[0137] In operation 762, the electronic device 101 according to an embodiment can share information about the event occurring in the specific application (e.g., the alarm application 754) with the intelligent application 752 as a root service (deliver information about the event occurring in the specific application (e.g., the alarm application 754) to the intelligent application). For example, the electronic device 101 can store a plurality of intelligent applications (e.g., BIXBY TM application or GOOGLE ASSISTANT TM ​​​​​application), and a specific smart application (e.g., BIXBY TM application 752) can be configured as a default application (or a root service) through user configuration. According to an embodiment, when an event occurs in a specific application (e.g., an alarm application 754), the electronic device 101 can deliver event occurrence information to the smart application (e.g., BIXBY TM application 752) configured as the default application. For example, an alarm event occurring in the alarm application 754 can be delivered to BIXBY TM application 752) configured as the default application.

[0138] In operation 763, the electronic device 101 according to an embodiment can change the authority to use the microphone 173 so that the smart application 752 (e.g., BIXBY TM ) receiving the event occurrence information performs a voice recognition function. For example, in response to receiving the event occurrence information, the smart application 752 can request the application framework 740 to allocate the authority to use the microphone 173.

[0139] In operation 764, the electronic device 101 according to an embodiment can cause the application framework 740 receiving the request to use the microphone 173 from the smart application 752 to allocate the authority to occupy the microphone 173 to the smart application 752. For example, if an application (e.g., a map application 751) that has been executed before the event occurrence has held the authority to occupy the microphone 173, the application framework 740 can transfer the authority to occupy the microphone 173 from the map application 751 to the smart application 752. Accordingly, information about a user utterance acquired through the microphone 173 can be delivered to the smart application 752.

[0140] In operation 765, the electronic device 101 according to an embodiment can determine whether the event manager 137 of the smart application 752 should perform a follow-up action related to a specified event based on a configuration value (or a default configured value) that a user has input in advance. According to an embodiment, the event manager 137 can ignore the specified event based on a configuration value corresponding to a currently executed application (e.g., the map application 751), or can transmit information (or data) related to the specified event to an event proactive module 265 of a smart server (e.g., 201) in operation 765. Figure 2A According to an embodiment, the information transmitted by the smart application 752 of the electronic device 101 to the smart server 201 can include at least one of identification information of an application in which the event occurs (e.g., the alarm application 754), a time at which the event occurs, or a type of the event.

[0141] According to an embodiment, upon acquiring a user utterance through the microphone 173, the electronic device 101 can transmit data corresponding to the acquired user utterance to the smart server 201.

[0142] According to an embodiment, the electronic device 101 can have an embedded artificial intelligence (AI) platform, and can incorporate at least some of the elements embedded in the smart server 201. In this case, the electronic device 101 can transmit information related to an event that occurs to the AI platform embedded in the electronic device 101 (e.g., 810 in Figure 8 ), instead of the smart server 201, and the AI platform 810 can include an event proactive module (e.g., 840 in Figure 8 ). The embodiment in which the electronic device 101 has an embedded AI platform will be described in greater detail later in this document with reference to Figure 8 .

[0143] According to an embodiment, the event proactive module 265 of the smart server 201 can pre-analyze information received from the electronic device 101, and can determine an application (or domain) for processing a user utterance received from the electronic device 101 after an event occurs.

[0144] According to an embodiment, if a user utterance is received from the electronic device 101 after the event proactive module 265 determines an application (or domain) for processing the user utterance, the smart server 201 can determine a plan for performing a task corresponding to the voice input, and can transmit the determined plan to the electronic device 101. According to an embodiment, the electronic device 101 can have an embedded AI platform having the same or similar functions and elements as those of the smart server 201, and in this case, the AI platform embedded in the electronic device 101 can determine an application (or domain) for processing a user utterance and can determine a plan for performing a task corresponding to a received voice input, independently.

[0145] Figure 8 A block diagram illustrating an exemplary AI platform embedded in an electronic device according to an embodiment is shown.

[0146] Referring to Figure 8 , the smart server 201 according to an embodiment can include a front end 210, a natural language platform 220, an execution engine 235, or an end user interface 240. According to an embodiment, the natural language platform 220 can include an automatic speech recognition module 221, a natural language understanding module 223, a planner module 225, or a text-to-speech module 229. Figure 8 The illustrated smart server 201 can be substantially the same as or similar to the smart server 201 illustrated in Figure 2A . Accordingly, a description of elements in the smart server 201 embedded in Figure 8 that are the same as those of the smart server 201 in Figure 7 will be omitted here.

[0147] According to an embodiment, the electronic device 101 can have an embedded AI platform 810. According to an embodiment, the electronic device 101 can include the AI platform 810, an execution engine 820, an end user interface 830, an event proactive module 840, at least one intelligent application 850 (e.g., BIXBY TM application), an event manager (e.g., 137 in Figure 2A ), or a plurality of applications (e.g., a first application 135_1 or a second application 135_2 in Figure 8 ). According to an embodiment, the AI platform 810 can include an automatic speech recognition module 811, a natural language understanding module 812, a planner module 813, or a text-to-speech module 814, and these elements can perform substantially the same functions as those of the elements 221, 223, 225, and 227 embedded in the intelligent server 201, respectively. According to an embodiment, the execution engine 820 or the end user interface 830 embedded in the electronic device 101 can perform the same functions as those of the execution engine 235 or the end user interface 240 embedded in the intelligent server 201.

[0148] According to an embodiment, at least some of the event proactive module 840 or the event manager 137 can be embedded in the intelligent application 850.

[0149] Hereinafter, operations of the electronic device 101 according to an embodiment will be described with reference to Figure 5 .

[0150] At operation 861, the electronic device 101 according to an embodiment can execute the intelligent application 850 in response to receiving a wake-up utterance (e.g., Hi Bixby!!) or receiving an input made by pressing a physical button, and can sense a user utterance by activating a voice recognition function after the intelligent application 850 is executed. According to an embodiment, as on the screen 510 in Figure 7 , the electronic device 101 can receive a first user utterance such as "Hi Bixby, play music 1 in the music application," and can transmit data (or information) corresponding to the received first user utterance to the intelligent server 201.

[0151] At operation 862 and operation 863, the electronic device 101 according to an embodiment can receive a plan for performing a task related to the first user utterance from the smart server 201, and can execute the first application 135_1 (e.g., a music application) based on at least the received plan. According to an embodiment, the electronic device 101 can output a result of performing a task by using the first application 135_1 based on at least the received plan. For example, the electronic device 101 can execute the music application, and can play "music 1" through the music application. According to an embodiment, the electronic device 101 can end execution of the smart application 752 after performing all tasks related to the first user utterance.

[0152] At operation 864, the electronic device 101 according to an embodiment can receive a designated event related to a second application 135_2 (e.g., a message application) while displaying an execution screen of the first application 135_1, and the received event can be shared with the smart application 850 as a root service (the received event can be delivered to the smart application 850). According to an embodiment, the smart application 850 can transmit event-related information to the event manager 137. The event manager 137 according to an embodiment can identify a configuration value regarding an application (e.g., the first application 135_1) that is currently executed, and can determine whether to perform a subsequent action related to the designated event based on the configuration value. According to an embodiment, the event manager 137 can deliver information about the event to the event proactive module 840 embedded in the electronic device 101 based on a user configuration value related to the application (e.g., the first application 135_1) that is currently executed.

[0153] At operation 865, the electronic device 101 according to an embodiment can cause the event proactive module 840 to pre-analyze information related to the event received from the event manager 137, and determine an application (or a domain) for processing a user utterance received from the electronic device 101 after the event occurs. According to an embodiment, the event proactive module 840 can execute the smart application 850 based on a result of pre-analyzing the event-related information, and can assign a right to occupy a microphone (e.g., 173) of the electronic device 101 to the smart application 850. Figure 7

[0154] At operation 866, the electronic device 101 according to an embodiment can cause the smart application 850 that has acquired the right to occupy the microphone (e.g., 173) to receive a second user utterance and deliver the received second user utterance to the AI platform 810. According to an embodiment, as on screen 530 in Figure 5 Figure 1 At operation 866, the electronic device 101 according to an embodiment can cause the smart application 850 that has acquired the right to occupy the microphone (e.g., 173) to receive a second user utterance and deliver the received second user utterance to the AI platform 810. According to an embodiment, as on screen 530 in ​​

[0155] In operation 867, according to an embodiment, electronic device 101 can enable AI platform 810 to determine a plan for performing a task corresponding to a second user utterance. According to an embodiment, electronic device 101 can execute a second application 135_2 based on the determined plan and can output the results of performing the task using the second application 135_2. According to an embodiment, electronic device 101 can display on a screen (e.g., Figure 2A The results of performing the task corresponding to the second user's speech are displayed on the display 160 (in the middle), or transmitted through a speaker (e.g., Figure 5 171) Outputs a voice message. According to an embodiment, as in... Figure 9 On the screen 540, the electronic device 101 can output the content of a message in voice, graphic or text type, such as "The sender is Jane, the appointment is at 9 o'clock tomorrow...".

[0156] According to an embodiment, electronic device 101 can terminate the execution of smart application 850 after performing all tasks related to the second user's speech, and can return to the state before receiving the event related to the second application 135_2. For example, electronic device 101 can switch the first application 135_1 from a background state to a foreground state, and can display the execution screen of the first application 135_1 on display 160.

[0157] Figure 9 An illustration shows an exemplary configuration screen 910 used by an electronic device according to an embodiment for configuring whether to perform AI actions based on event-related speech recognition.

[0158] Reference Figure 2A When an event occurs, the electronic device according to the embodiment (e.g., Figure 2A The electronic device 101 may provide a configuration screen 910 (e.g., a user interface) for configuring whether to perform AI actions based on event-related voice recognition. According to an embodiment, the configuration screen 910 may include multiple status menus 911, 912, and 913 indicating user status. For example, the multiple status menus 911, 912, and 913 may include user statuses that can be pre-configured by the user, such as "Do Not Disturb" 911, Driving 912, or Listening to Music 913. According to an embodiment, the multiple status menus 911, 912, and 913 may include a quick configuration icon 921 for making on / off configurations regarding whether to perform AI actions based on voice recognition. According to an embodiment, the quick configuration icon 921 can be used to quickly configure whether to perform AI actions based on voice recognition pre-configured by the user for a corresponding status menu (e.g., Driving 912). For example, the multiple status menus 911, 912, and 913 may include submenus 931, making it possible to configure multiple applications stored in the electronic device 101 (e.g., ...). Figure 2AEach application in the first application 135_1 or the second application 135_2 is configured to perform AI actions based on speech recognition. Users can select at least some of the multiple applications 9311, 9312, and 9313 (e.g., ...) via submenu 931. Figure 9 The user can select whether to perform AI actions based on voice recognition, using either the first application 135_1 or the second application 135_2. For example, in the case of driving 912 among multiple status menus 911, 912, and 913, the user can configure the device to perform AI actions based on voice recognition related to events concerning the map application 9311 among the multiple applications 9311, 9312, and 9313 installed on the electronic device 101, but not based on voice recognition related to events concerning the calendar application 9312. In this case, if an event related to another application (e.g., the music application 9313) occurs while the map application 9311 is being executed, the electronic device 101 can provide AI services based on voice recognition via the smart application 752, which is a root service. On the other hand, if an event related to another application (e.g., the music application 9313) occurs while the calendar application 9312, which is configured to be deactivated on the configuration screen, is being executed, the electronic device 101 may not perform AI actions based on voice recognition.

[0159] According to an embodiment, if based on, as Figure 2A When an event occurs due to user input on the configuration screen 910 shown, the electronic device (e.g., Figure 2A The event manager of the electronic device 101 in the middle (e.g., Figure 1 The event manager (137) in the system can determine whether to perform AI actions based on speech recognition.

[0160] For use by an electronic device according to an embodiment (e.g., Figure 1 101) The method of performing an action using voice recognition may include the following operations: on a display (e.g., Figure 2A The first application (e.g., 160) is displayed on the screen. Figure 2A When executing the screen of 135_1), receive the message from the second application (e.g., Figure 7 The specified event (135_2) in the specified event; the execution of an artificial intelligence application in response to the specified event (e.g., Figure 1 752); based on the execution of artificial intelligence applications 752, via communication circuits (e.g., Figure 2A The communication module 190 in the middle) sends data related to the specified event to an external server (e.g., Figure 2A 201 in the middle); within a specified time period, through the microphone (e.g., Figure 10sensing a user utterance related to the designated event; if the user utterance is sensed, transmitting the user utterance to the intelligent server 201 through the communication module 190; receiving an action sequence for performing a function related to the user utterance from the intelligent server 201 through the communication module 190; executing the second application 135_2 based on at least the received action sequence; and outputting a result of performing the function by using the second application 135_2. The method can further include the following operations: if the artificial intelligence application 752 is executed in response to the designated event, activating the microphone 173; and delivering information acquired through the activated microphone 173 to the artificial intelligence application 752. The method can further include the following operations: in response to receiving the designated event, displaying a user interface for selecting an application on the display 160; and if a user input selecting the second application 135_2 is received through the user interface, starting to sense a user utterance for a designated period of time. The method can further include the following operations: in response to receiving the designated event related to the second application 135_2 while the display 160 is deactivated, executing the artificial intelligence application 752; transmitting data related to the designated event to the intelligent server 201 through the communication module 190 based on the executed artificial intelligence application 752; sensing a user utterance related to the designated event through the microphone 173 for a designated period of time; if the user utterance is sensed, transmitting the user utterance to the intelligent server 201 through the communication module 190; receiving an action sequence for performing a function related to the user utterance from the intelligent server 201 through the communication module 190; executing the second application 135_2 based on at least the received action sequence; and outputting a result of performing the function by using the second application 135_2.

[0161] A method for performing an action using a voice recognition function by an electronic device 101 according to an embodiment may include the following operations: receiving a specified event related to a second application 135_2 while displaying an execution screen of a first application 135_1 on a display 160; executing an artificial intelligence application 752 in response to the specified event; pre-analyzing data related to the specified event based on the executed artificial intelligence application 752; sensing user speech related to the specified event via a microphone 173 within a specified time period; determining an action sequence for performing a function related to the user speech based on the result of the pre-analysis of the data if user speech is sensed; executing the second application 135_2 at least based on the determined action sequence; and outputting the result of performing the function using the second application 135_2. The method may further include the following operations: activating the microphone 173 if the artificial intelligence application 752 is executed in response to the specified event; and transmitting user speech acquired via the activated microphone 173 to the artificial intelligence application 752. The method may also include the following operations: in response to receiving a specified event, displaying a user interface for selecting an application on a display 160; and if user input for selecting a second application 135_2 is received through the user interface, initiating sensing of user speech within a specified time period.

[0162] Figure 2A A flowchart is shown to describe the operation of an electronic device according to an embodiment.

[0163] In operation 1010, the electronic device according to the embodiment (e.g., Figure 2A The electronic device 101 in the middle can be used to display the first application (e.g., Figure 2A When the first application (135_1) executes the screen, it receives data from the second application (e.g., ...). Figure 1 The second application 135_2) is related to a specified event. For example, a message receiving event may occur when the music application execution screen is displayed on the electronic device 101.

[0164] In operation 1020, the electronic device 101 according to the embodiment can execute an AI application (hereinafter referred to as smart application 752) in response to a specified event. According to the embodiment, the electronic device 101 can execute multiple smart applications (e.g., BIXBY) based on the occurrence of the specified event. TM Applications, Google Assistant TM Smart apps 752 that are designated as the default app in the application (e.g., BIXBY) TM application).

[0165] In operation 1030, the electronic device 101 according to the embodiment can communicate via a communication circuit (e.g., based on the executed smart application 752) Figure 2AThe communication module 190 in the middle sends data related to the specified event to an external server, such as an intelligent server (e.g., Figure 7 (Intelligent server 201 in the middle). According to an embodiment, electronic device 101 can enable the event manager of intelligent application 752 (e.g., Figure 7 The event manager 137 in the context determines whether to execute subsequent actions related to a specified event based on configuration values ​​pre-entered by the user (or default configuration values). According to an embodiment, the event manager 137 may be based on the currently executing application (e.g., Figure 2A The map application 751) has the corresponding configuration value that ignores the specified event, or information (or data) related to the specified event can be sent to the smart server (e.g., Figure 2A The event-active module (e.g., 201) in the middle. Figure 2A (265 in the original text). For example, if a message receiving event occurs while the screen of a music application is being displayed, the electronic device 101 can recognize the user configuration associated with the music application and can send information related to the music receiving event to a smart server (e.g., based on the user configuration). Figure 1 The event active module 265 of (201) in the middle.

[0166] According to an embodiment, the information sent from the smart application 752 of the electronic device 101 to the smart server 201 may include at least one of the following: identification information of the application where the event occurred (e.g., alarm application 754), the time when the event occurred, or the type of the event.

[0167] According to an embodiment, the event-initiated module 265 of the intelligent server 201 can pre-analyze the information received from the electronic device 101 and determine the application (or domain) for processing user speech received from the electronic device 101 after an event occurs.

[0168] In operation 1040, the electronic device 101 according to the embodiment can sense user speech related to a specified event within a specified time period, and if user speech is sensed, it can communicate via a communication circuit (e.g., Figure 7 The communication module 190 in the device sends user speech to an external server (e.g., a smart server 201). According to an embodiment, the electronic device 101 can enable a smart application 752 (e.g., BIXBY) to receive event occurrence information. TM ) Change the permissions for using microphone 173 in order to perform voice recognition functions. For example, in response to receiving event information, smart application 752 may request the application framework (e.g., Figure 7 740) in the middle) allocated the use of microphones (e.g., Figure 1The electronic device 101 can send data corresponding to the acquired user speech to the intelligent server 201 when acquiring user speech via microphone 173, according to an embodiment.

[0169] In operation 1050, the electronic device 101 according to the embodiment can communicate via a communication circuit (e.g., Figure 5 The communication module 190 receives a plan (e.g., sequence of actions) from an external server (e.g., a smart server) for performing tasks (e.g., functions) related to the user's speech. According to an embodiment, if a user speech is received from electronic device 101 after event initiation module 265 determines an application (or domain) for processing the user speech, smart server 201 can determine a plan (e.g., sequence of actions) for performing tasks (e.g., functions) corresponding to the voice input and can send the determined plan (e.g., sequence of actions) to electronic device 101. For example, as in... Figure 1 On the screen 530, the electronic device 101 can receive user speech such as “read the message that just arrived” and can transmit the data (or information) corresponding to the received user speech to the intelligent server 201.

[0170] During operations 1060 and 1070, the electronic device 101 according to the embodiment can execute the second application 135_2 at least based on a received plan (e.g., sequence of actions), and can output the results of performing a task (e.g., a function) using the second application 135_2. According to the embodiment, the electronic device 101 can display (e.g., on a display screen). Figure 2A The results of performing the task (e.g., function) corresponding to the second user's speech are displayed on the display 160, or can be displayed through a speaker (e.g., Figure 5 171) Outputs a voice message. For example, as in Figure 11 On the screen 540, the electronic device 101 can output the content of a message in voice, graphic or text type, such as "The sender is Jane, the appointment is at 9 o'clock tomorrow...".

[0171] Figure 2A A flowchart is shown to describe the operation of an electronic device having an embedded artificial intelligence (AI) platform according to an embodiment.

[0172] In operation 1110, the electronic device according to the embodiment (e.g., Figure 2A The electronic device 101 in the middle can be used to display the first application (e.g., Figure 2A When the first application (135_1) executes the screen, it receives data from the second application (e.g., ...). Figure 7 The second application 135_2) is related to a specified event. For example, a message receiving event may occur when the music application execution screen is displayed on the electronic device 101.

[0173] In operation 1120, the electronic device 101 according to an embodiment can execute an AI application (e.g., the intelligent application 752 in the BIXBY Figure 7 application) in response to the designated event. According to an embodiment, the electronic device 101 can execute a plurality of intelligent applications (e.g., the BIXBY TM application, the GOOGLE ASSISTANT TM application) based on the occurrence of the event. According to an embodiment, the electronic device 101 can execute the intelligent application 752 (e.g., the BIXBY TM application) designated as a default application among the plurality of intelligent applications.

[0174] In operation 1130, the electronic device 101 according to an embodiment can pre-analyze data related to the designated event based on the executed intelligent application 752. According to an embodiment, the event manager 137 of the electronic device 101 can ignore the designated event based on a configuration value corresponding to the currently executed application (e.g., the map application 751 in the BIXBY Figure 8 ), or can transmit information (or data) related to the designated event to the event proactive module (e.g., 840 in the BIXBY Figure 7 ) embedded in the electronic device 101. According to an embodiment, the event proactive module 840 embedded in the electronic device 101 can pre-analyze information related to the received event, and can determine an application (or domain) for processing a user utterance received by the electronic device 101 after the event occurs.

[0175] In operation 1140, the electronic device 101 according to an embodiment can sense a user utterance related to the designated event for a designated period of time, and if the user utterance is sensed, can determine a plan (e.g., an action sequence) for executing a task (e.g., a function) related to the user utterance based on a result of the pre-analysis. According to an embodiment, the electronic device 101 can execute the intelligent application 752, and can assign a right to occupy a microphone (e.g., 173 in the BIXBY Figure 7 ) of the electronic device 101 to the intelligent application 752. According to an embodiment, the electronic device 101 can cause the intelligent application 752 that has acquired the right to occupy the microphone (e.g., 173 in the BIXBY Figure 8 ) to receive a user utterance and transfer the received user utterance to an AI platform (e.g., 810 in the BIXBY Figure 12 ) embedded in the electronic device 101. According to an embodiment, the AI platform 810 can determine a plan (e.g., an action sequence) for executing a task (e.g., a function) corresponding to the second user utterance.

[0176] The electronic device 101 according to an embodiment can execute the second application 135_2 based on at least the determined schedule (e.g., action sequence) and output a result of performing a task (e.g., function) by using the second application 135_2 at operation 1150 and operation 1160.

[0177] Figure 7 A flowchart for describing an operation of an electronic device according to an embodiment to process a received event without executing a smart application is illustrated.

[0178] The electronic device 101 according to an embodiment can receive a designated event without executing an AI service (e.g., a smart application 752) in operation 1210 and operation 1220. Figure 13 The electronic device 101 according to an embodiment can receive a designated event without executing an AI service (e.g., a smart application 752) in operation 1210 and operation 1220. TM The message reception event can occur without the electronic device 101 executing the smart application 752.

[0179] The electronic device 101 according to an embodiment can execute an AI service (e.g., a smart application 752) in response to a designated event and can transfer information related to the designated event to the AI service (e.g., the smart application 752) at operation 1230.

[0180] The electronic device 101 according to an embodiment can cause the AI service (e.g., the smart application 752) to identify a state configuration value regarding a current application at operation 1240. According to an embodiment, the electronic device 101 can identify a configuration state of a user rather than a state configuration value of a current application. For example, the AI service (e.g., the smart application 752) of the electronic device 101 can identify a user state that a user can pre-configure, such as "Do not disturb," driving, or listening to music.

[0181] The electronic device 101 according to an embodiment can determine whether to execute an AI service related to a designated event based on a configuration value corresponding to a currently executed application (e.g., a map application 751) at operation 1250. According to an embodiment, the electronic device 101 can identify a configuration state of a user rather than a state configuration value of a current application. For example, the AI service (e.g., the smart application 752) can identify a user state that a user can pre-configure, such as "Do not disturb," driving, or listening to music, and can determine whether to execute an AI action related to a designated event based on the user state.

[0182] In operation 1260, if it is determined to execute the AI service related to the designated event (for example, the result of operation 1250 is "Yes"), the electronic device 101 according to an embodiment can receive a user utterance for a designated period of time. For example, the electronic device 101 can sense a user utterance related to the designated event for a designated period of time (for example, ten seconds). If it is determined not to execute the AI service related to the designated event (for example, the result of operation 1250 is "No"), the electronic device 101 according to an embodiment can proceed to operation 1210.

[0183] Figure 2A A flowchart for describing an operation of the electronic device 101 according to an embodiment to process a received event while the smart application 752 is executed is illustrated.

[0184] In operation 1310, the electronic device (for example, the electronic device 101) according to an embodiment can execute a smart application (for example, 752 in Figure 2A Figure 7 in response to receiving a wake-up utterance (for example, Hi Bixby!!) or receiving an input made by pressing a physical button, and can receive a first user utterance by activating a voice recognition function after executing the smart application 752.

[0185] In operation 1320, the electronic device 101 according to an embodiment can receive a plan (for example, an action sequence) for executing a task (for example, a function) related to the first user utterance from a smart server (for example, the smart server 201 in Figure 2A Figure 2A , and can execute a first application (for example, 135_1 in Figure 2A Figure 2A ) (for example, a music application) based on at least the received plan (for example, an action sequence). According to an embodiment, the electronic device 101 can output a result of executing a task (for example, a function) by using the first application 135_1 based on at least the received plan (for example, an action sequence).

[0186] In operations 1330 and 1340, the electronic device 101 according to an embodiment can receive a designated event related to a second application (for example, 135_2 in Figure 7 Figure 7 ) (for example, a message application) while displaying an execution screen of the first application 135_1, and the received event can be shared with the smart application 752 as a root service (the received event can be delivered to the smart application 752). According to an embodiment, the smart application 752 can transmit event-related information to an event manager (for example, the event manager 210 in Figure 8 Figure 8 ​​The event manager 137 according to an embodiment can identify a configuration value regarding the currently executed application (e.g., the first application 135_1), and can determine whether to perform a subsequent action related to a specified event based on the configuration value. According to an embodiment, the event manager 137 can deliver information related to the event to an event proactive module (e.g., the event proactive module 840) embedded in the electronic device 101 based on the user configuration value regarding the currently executed application (e.g., the first application 135_1), and can perform a function related to the event based on the information related to the event. Figure 8

[0187] At operation 1350, the electronic device 101 according to an embodiment can receive a second user utterance within a specified time period. For example, the electronic device 101 can sense a user utterance within a specified time period (e.g., ten seconds) after a message reception event occurs. According to an embodiment, if a user utterance is not sensed within the specified time period after the specified event is received, the electronic device 101 can maintain the application execution screen before the event is received. According to an embodiment, ten seconds within the specified time period is merely an example, and various modifications or configurations can be made. According to an embodiment, the electronic device 101 can deliver user utterance information acquired through the microphone 173 to the intelligent server 201. According to an embodiment, the electronic device 101 can deliver user utterance information acquired through the microphone 173 to an AI platform (e.g., the AI platform 810) embedded in the electronic device 101. Figure 14

[0188] At operation 1360, the electronic device 101 (or the intelligent server) according to an embodiment can confirm whether the second user utterance is related to the specified event.

[0189] At operation 1370, when the electronic device 101 (or the intelligent server 201) determines that the second user utterance is related to the specified event (e.g., the result of operation 1360 is "Yes"), the electronic device 101 according to an embodiment can determine a plan (e.g., an action sequence) for executing a task (e.g., a function) corresponding to the second user utterance, can execute the second application 135_2 based on the determined plan (e.g., the action sequence), and can output a result of executing the task (e.g., the function) by using the second application 135_2.

[0190] According to an embodiment, when the electronic device 101 (or the intelligent server 201) determines that the second user utterance is not related to the specified event (e.g., the result of operation 1360 is "No"), the electronic device 101 can proceed to operation 1320 and display the execution screen of the first application 135_1.

[0191] Figure 14 A flowchart for describing operations of an electronic device according to an embodiment performing a voice recognition function based on a user interface for selecting an application is illustrated.​​

[0192] According to an embodiment, Figure 13 Operations 1410 through 1440 shown in FIG. 13 can be the same or similar as Figure 13 Operations 1310 through 1340 shown in FIG. 13. Figure 14 Therefore, a repeated description of operations 1310 through 1340 shown in FIG. 13 will be omitted here. Figure 6 Operations 1410 through 1440 shown in FIG. 13.

[0193] In operation 1450, the electronic device 101 according to an embodiment can display a user interface for selecting an application (e.g., 631 in FIG. 13). For example, as on screen 630 in FIG. 13, the electronic device 101 can display a user interface 631 for replacing a currently executed application (e.g., the first application 135_1 or the current domain) with another application (e.g., the second application 135_2 or another domain) related to the specified event in response to receiving the specified event. Figure 6 Figure 6

[0194] In operation 1460, the electronic device 101 according to an embodiment can confirm whether a user selects the second application 135_2.

[0195] In operation 1470, if a user input of selecting the second application 135_2 is received (e.g., the result of operation 1460 is "Yes"), the electronic device 101 according to an embodiment can sense a second user utterance for a specified period of time. For example, on screen 630 in FIG. 13, if the electronic device 101 receives a user input of selecting a change to another application (e.g., the second application 135_2 or another domain) through the user interface 631, the electronic device 101 can sense a second user utterance for a specified period of time. Figure 15

[0196] If a user input of selecting the second application 135_2 is not received (e.g., the result of operation 1460 is "No"), the electronic device 101 according to an embodiment can proceed to operation 1420 and display an execution screen of the first application 135_1.

[0197] ​​​At operation 1480, the electronic device 101 according to an embodiment can determine a plan (e.g., an action sequence) for performing a task (e.g., a function) corresponding to the second user utterance, can execute the second application 135_2 based on the determined plan (e.g., action sequence), and can output a result of performing the task (e.g., function) by using the second application 135_2. According to an embodiment, the electronic device 101 can receive a plan (e.g., action sequence) for performing a task (e.g., function) corresponding to the second user utterance from the intelligent server 201, can execute the second application 135_2 based on the received plan (e.g., action sequence), and can output a result of performing the task (e.g., function) by using the second application 135_2.

[0198] Figure 15 An exemplary user interface used by an electronic device according to an embodiment to configure a specific smart application as a default AI service is illustrated.

[0199] Referring to Figure 7 , the electronic device 101 according to an embodiment can store a plurality of smart applications (e.g., BIXBY TM or GOOGLE ASSISTANT TM ), and can configure one smart application (e.g., Figure 15 BIXBY TM application 752) as a default application. According to an embodiment, the electronic device 101 can provide a configuration screen 1510 for configuring a specific smart application 752 as a default application. For example, when initially executing a smart application 752 as illustrated in ​ , the electronic device 101 can display a user interface 1511 for configuring a corresponding smart application (such as BIXBY TM application 752) as a default application.

[0200] An electronic device according to various embodiments can be one of various types of electronic devices. The electronic devices can include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. According to an embodiment of the disclosure, the electronic devices are not limited to those described above.

[0201] It should be understood that various embodiments of the present disclosure and the terms used therein are not intended to limit technically described features to particular embodiments and include various changes, equivalents, or replacements for a corresponding embodiment. With regard to the description of the drawings, like reference numerals can be used to refer to like or similar elements. It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. As used herein, each of the phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C," can include any one of the items enumerated together with all possible combinations of the items. As used herein, the terms "first," "second," "third," "fourth," "fifth," "sixth," and the like, do not limit the number or order of the constituent elements. Rather, these terms are used to identify particular elements from one another. It will be understood that the terms "on," "under," "right," "left," "rear," "front," "upper," "lower," and the like in the description and in the claims used in connection with an element can be used herein with respect to the element's orientation to another element as shown in the drawings and should not be construed as limiting the position of the element. It will be understood that if an element (e.g., first element) is referred to as being connected to or with another element (e.g., second element), it can be directly connected to the other element or connected to the other element via a third element.

[0202] As used herein, the term "module" can include a unit implemented in hardware, software, or firmware, and can interchangeably be used with other terms, e.g., "logic," "logic block," "part," or "circuitry." The module can be a single integral component, or a minimum unit or part thereof, adapted to perform one or more functions. For example, according to an embodiment, the module can be implemented in a form of an application-specific integrated circuit (ASIC).

[0203] Various embodiments as set forth herein can be implemented as software (e.g., the program 140) including one or more instructions that are stored in a storage medium (e.g., internal memory 136 or external memory 138) that are readable by a machine (e.g., electronic device 101). For example, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) can invoke at least one of the one or more instructions stored in the storage medium, and execute it, with or without using one or more other components under the control of the processor. This allows the machine to be operated as a special purpose machine to perform at least one function. The one or more instructions can include a code generated by a compiler or a code that forms at least a part of a language as provided in a high-level programming language. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. The term "non-transitory" simply means that the storage medium is tangible, but does not include a signal (e.g., an electromagnetic wave). The term "non-transitory" does not distinguish between where data is semi-permanently stored in the storage medium and where the data is temporarily stored in the storage medium.

[0204] According to an embodiment, a method according to various embodiments of the disclosure can be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or be distributed online via an application store (e.g., PlayStore®). If distributed online, at least part of the computer program product can be temporarily stored in a storage medium such as a memory of a manufacturer's server or a server of an online store, and then be installed in a user device (e.g., a smart phone). In addition, each component shown in the various embodiments of the disclosure can comprise a single component or a plurality of components. TM ) online (e.g., downloaded or uploaded), or can be directly distributed (e.g., downloaded or uploaded) among two user devices (e.g., smart phones). If distributed online, at least part of the computer program product can be temporarily generated or at least temporarily stored in a server of a manufacturer, a server of an online store, or a memory of a relay server.

[0205] According to various embodiments, each component (e.g., a module or a program) of the above-described components can include a single entity or multiple entities. According to various embodiments, one or more of the above-described components can be omitted, or one or more other components can be added. Alternatively or additionally, a plurality of components (e.g., modules or programs) can be integrated into a single component. In such a case, according to various embodiments, the integrated component can still perform one or more functions of each of the plurality of components in the same or similar manner as they are performed by a corresponding one of the plurality of components before the integration. According to various embodiments, operations performed by the module, the program, or another component can be carried out sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations can be executed in a different order or omitted, or one or more other operations can be added.

Claims

1. An electronic device comprising: a communication circuit; a display; a memory storing a computer program including a smart application, wherein the smart application is configured to process a voice input associated with at least one of a first application and a second application; and a processor operatively connected to the communication circuit, the display, and the memory; wherein the processor is configured to execute one or more of the computer programs stored in the memory to cause the electronic device to: connect the electronic device to an external electronic device via the communication circuit; receive a first voice input including a wake-up utterance from the external electronic device via the communication circuit; in response to receiving the first voice input, execute a first application via the smart application based on a portion of the first voice input other than the wake-up utterance; while keeping the first application executing, receive a specified event corresponding to a second application; after receiving the specified event and before a time associated with the specified event elapses, receive a second voice input related to the specified event from the external electronic device while keeping the first application executing via the communication circuit, wherein the second voice input does not include the wake-up utterance; and in response to receiving the second voice input after receiving the specified event and before the time associated with the specified event elapses, control the smart application to cause a task of the second application corresponding to the second voice input to be executed. 2.The electronic device of claim 1, wherein, The processor is further configured to analyze data related to the specified event. 3.The electronic device of claim 2, wherein, The data related to the specified event includes at least one of identification information of the second application, a time at which the specified event occurs, or a type of the specified event. 4.The electronic device of claim 1, wherein The processor is further configured to output a result of executing the task of the second application using the second application, wherein the result is at least one of a voice type, a graphic type, or a text type. 5.A method of an electronic device, the method comprising: connecting the electronic device to an external electronic device via a communication circuit of the electronic device; receiving a first voice input including a wake-up utterance from the external electronic device via the communication circuit; in response to receiving the first voice input, executing a first application via a smart application stored in a memory of the electronic device based on a portion of the first voice input other than the wake-up utterance; while keeping the first application executing, receiving a specified event corresponding to a second application; after receiving the specified event and before a time associated with the specified event elapses, receiving a second voice input related to the specified event from the external electronic device while keeping the first application executing via the communication circuit, wherein the second voice input does not include the wake-up utterance; and in response to receiving the second voice input after receiving the specified event and before the time associated with the specified event elapses, controlling the smart application to cause a task of the second application corresponding to the second voice input to be executed. analyzing data related to the specified event.

6. The method of claim 5, further comprising: ​ 7. The method of claim 6, wherein, The data related to the designated event includes at least one of identification information of a second application, a time at which the designated event occurs, or a type of the designated event. 8.A non-transitory recording medium storing a computer program readable by an electronic device, wherein, One or more of the computer programs cause the electronic device to perform the following operations: connecting the electronic device to an external electronic device via a communication circuit of the electronic device; receiving, from the external electronic device via the communication circuit, a first voice input including a wake-up utterance; in response to receiving the first voice input, executing a first application via a smart application stored in a memory of the electronic device based on a portion of the first voice input other than the wake-up utterance; while maintaining execution of the first application, receiving a designated event corresponding to a second application; after receiving the designated event and before a time associated with the designated event elapses, receiving, from the external electronic device via the communication circuit, a second voice input related to the designated event while maintaining execution of the first application, wherein the second voice input does not include the wake-up utterance; and in response to receiving the second voice input after receiving the designated event and before the time associated with the designated event elapses, controlling the smart application to cause execution of a task of the second application corresponding to the second voice input. ​

Images