Electronic device for controlling voice command execution and method thereof

Abstract

An electronic device may include communication circuitry, a microphone, a speaker, a memory, and a processor. The processor may identify a first audio signal received through the microphone. The processor may perform at least one first function of the plurality of functions based on identifying a first audio signal including sound production for sequentially performing the plurality of functions. The processor may perform at least one second function of the plurality of functions by using an external electronic device connected via the communication circuit based on identifying the first audio signal including the utterance. The at least one second function may be different from the at least one first function. The processor may sequentially output, through a speaker of the electronic device, one or more second audio signals representing one or more results obtained by performing the at least one first function and the at least one second function according to an order related to sound emission.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and method for controlling execution of a voice command. Background Art

[0002] The interface between the electronic device and the user may include a keyboard and / or a mouse. The interface type between the electronic device and the user may be expanded to more intuitively control the electronic device. For example, the electronic device may use a microphone to recognize the user's speech to control the electronic device. Summary of the invention

[0003] According to an embodiment, the electronic device may include a communication circuit, a microphone, a speaker, a memory storing at least one instruction, and at least one processor. The at least one processor may be configured to execute at least one instruction to identify a first audio signal received through the microphone. The at least one processor may be configured to perform at least one first function among multiple functions based on recognizing that the first audio signal may include speech for sequentially performing multiple functions. The at least one processor may be configured to perform at least one second function among multiple functions on an external electronic device through the communication circuit based on recognizing that the first audio signal may include speech. The at least one second function may be different from the at least one first function. The at least one processor may be configured to sequentially output at least one second audio signal representing at least one result of performing at least one first function and at least one second function through the speaker based on the order associated with the speech.

[0004] According to an embodiment, a method of an electronic device may include: based on receiving a first audio signal through a microphone of the electronic device, recognizing speech in the first audio signal. The method may include: based on recognizing multiple voice commands from the speech, obtaining the state of a first voice engine executed on the electronic device and the state of a second voice engine executed on an external electronic device connected through a communication circuit of the electronic device. The method may include: based on the state of the first voice engine and the state of the second voice engine, obtaining information for dividing multiple voice commands into at least one first voice command executed by the first voice engine and at least one second voice command executed by the second voice engine. The method may include: using the first voice engine to execute a first function corresponding to at least one first voice command, and requesting the external electronic device to execute a second function corresponding to at least one second voice command. The method may include: based on the order of multiple voice commands indicated by the speech, outputting at least one second audio signal representing at least one result of executing the first function and the second function through a speaker of the electronic device.

[0005] According to an embodiment, a method of an electronic device may include: identifying a first audio signal received through a microphone of the electronic device. The method may include: executing at least one first function among a plurality of functions based on identifying that the first audio signal may include speech for sequentially executing the plurality of functions. The method may include: executing at least one second function among a plurality of functions different from at least one first function on an external electronic device connected through a communication circuit of the electronic device based on identifying that the first audio signal may include speech. The method may include: sequentially outputting at least one second audio signal representing at least one result of executing at least one first function and at least one second function through a speaker of the electronic device based on a sequence associated with the speech.

[0006] According to an embodiment, the electronic device may include a communication circuit, a microphone, a speaker, and a processor. The processor may be configured to recognize a first audio signal received through the microphone. The processor may be configured to perform at least one first function among the multiple functions based on recognizing speech for sequentially performing the multiple functions from the first audio signal. The processor may be configured to perform at least one second function among the multiple functions that is different from the at least one first function by using an external electronic device connected via the communication circuit based on the recognition of speech. The processor may be configured to sequentially output a second audio signal representing the result of performing at least one first function and at least one second function through the speaker according to the order associated with the speech.

[0007] According to an embodiment, a method of an electronic device may include: recognizing a first audio signal received through a microphone of the electronic device. The method may include: executing at least one first function among a plurality of functions based on recognizing speech for sequentially executing the plurality of functions from the first audio signal. The method may include: executing at least one second function among a plurality of functions different from at least one first function by using an external electronic device connected via a communication circuit of the electronic device based on the recognition of speech. The method may include: sequentially outputting a second audio signal representing a result of executing at least one first function and at least one second function through a speaker of the electronic device according to a sequence associated with the speech.

[0008] According to an embodiment, the electronic device may include a communication circuit, a microphone, a speaker, and a processor. The processor may be configured to: based on receiving an audio signal through a microphone, recognize a speech included in a first audio signal. The processor may be configured to: based on recognizing multiple voice commands from the speech, obtain the state of a first voice engine executed by the processor to process the voice command and a second voice engine executed by an external electronic device connected through the communication circuit. The processor may be configured to: based on the state, obtain information for dividing multiple voice commands into a first voice command corresponding to the first voice engine and a second voice corresponding to the second voice engine. The processor may be configured to: based on the information, execute a first function corresponding to the first voice command by using the first voice engine, and request the external electronic device to execute a second function corresponding to the second voice command. The processor may be configured to: based on the order of multiple voice commands indicated by the speech, output a second audio signal representing the result of executing the first function and the second function through the speaker.

[0009] According to an embodiment, a method of an electronic device may include: based on receiving a first audio signal through a microphone of the electronic device, recognizing speech included in the first audio signal. The method may include: based on recognizing multiple voice commands from the speech, obtaining the state of a first voice engine executed by a processor to process voice commands, and the state of a second voice engine executed by an external electronic device connected through a communication circuit of the electronic device. The method may include: based on the state, obtaining information for dividing multiple voice commands into a first voice command corresponding to the first voice engine and a second voice corresponding to the second voice engine. The method may include: based on the information, executing a first function corresponding to the first voice command by using the first voice engine, and requesting the external electronic device to execute a second function corresponding to the second voice command. The method may include: based on the order of multiple voice commands indicated by the speech, outputting a second audio signal representing the result of executing the first function and the second function through a speaker of the electronic device. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a block diagram of an electronic device in a network environment according to an embodiment.

[0011] Figure 2 An operation of an electronic device according to an embodiment to perform one or more functions based on speech is illustrated.

[0012] Figure 3 is a block diagram of an electronic device and external electronic devices according to an embodiment.

[0013] Figure 4 is a block diagram of a program executed by an electronic device according to an embodiment.

[0014] Figure 5 is a block diagram of a voice command database included in an electronic device according to an embodiment.

[0015] Figure 6 is a block diagram of a voice command database included in an electronic device according to an embodiment.

[0016] Figure 7 is a block diagram of a program executed by an electronic device according to an embodiment.

[0017] Fig. 8A and Figure 8B An example state in which an electronic device displays a user interface (UI) according to an embodiment is illustrated.

[0018] Fig. 9 An operation of an electronic device according to an embodiment to perform a plurality of functions based on speech is illustrated.

[0019] Fig.10 is a flowchart of operations performed by an electronic device according to an embodiment.

[0020] Fig.11 is a flowchart of operations performed by an electronic device according to an embodiment.

[0021] Fig.12 is a flowchart of operations performed by an electronic device according to an embodiment.

[0022] Fig.13 is a block diagram illustrating an artificial intelligence (AI) system according to an embodiment.

[0023] Fig.14 is a diagram illustrating a scheme for storing relationship information between concepts and actions in a database according to an embodiment.

[0024] Fig.15 is a diagram illustrating a user terminal displaying a screen to process a voice input received through a smart application according to an embodiment. DETAILED DESCRIPTION

[0025] Hereinafter, example embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0026] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or alternative forms for the corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one or all possible combinations of the items listed together with the corresponding one of the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish the corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “coupled with another element (e.g., the second element)”, “coupled to another element (e.g., the second element)”, “connected with another element (e.g., the second element)”, or “connected to another element (e.g., the second element)” with or without the terms “operably” or “communicatively” being used, it means that the element may be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0027] As used in connection with various embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0028] Figure 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to an embodiment.

[0029] Reference Figure 1, the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with at least one of the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the above components (e.g., the connection terminal 178) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the above-described components (eg, sensor module 176, camera module 180, or antenna module 197) may be implemented as a single integrated component (eg, display module 160).

[0030] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware component or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 120 may store a command or data received from another component (e.g., sensor module 176 or communication module 190) in the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) or an auxiliary processor 123 (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. For example, when the electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121, or as part of the main processor 121.

[0031] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 (rather than the main processor 121) may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component among the components of the electronic device 101 (e.g., the display module 160, the sensor module 176, or the communication module 190) together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., a camera module 180 or a communication module 190) that is functionally related to the auxiliary processor 123. According to an embodiment, the auxiliary processor 123 (e.g., a neural processing unit) may include a hardware structure dedicated to artificial intelligence model processing. The artificial intelligence model may be generated through machine learning. For example, such learning may be performed by the electronic device 101 where the artificial intelligence is executed or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, for example. The artificial intelligence model may include multiple artificial neural network layers. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q network or a combination of two or more thereof, but is not limited thereto. Additionally or optionally, the artificial intelligence model may include a software structure in addition to a hardware structure.

[0032] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0033] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0034] The input module 150 may receive commands or data to be used by other components (e.g., the processor 120) of the electronic device 101 from outside (e.g., a user) of the electronic device 101. The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus).

[0035] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive incoming calls. Depending on the embodiment, the receiver can be implemented as a separate part from the speaker, or as part of the speaker.

[0036] The display module 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display module 160 may include a touch sensor adapted to detect a touch or a pressure sensor adapted to measure the strength of a force caused by a touch.

[0037] The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 may obtain sound via the input module 150, or output sound via the sound output module 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0038] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) outside the electronic device 101, and then generate an electrical signal or a data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0039] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly with an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0040] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0041] The haptic module 179 may convert the electric signal into mechanical stimulation (eg, vibration or motion) or electric stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electric stimulator.

[0042] The camera module 180 may capture still images or moving images. According to an embodiment, the camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0043] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0044] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0045] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multiple components (e.g., multiple chips) separated from each other. The wireless communication module 192 may identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0046] The wireless communication module 192 can support 5G networks after 4G networks and next-generation communication technologies (e.g., new radio (NR) access technology). NR access technology can support enhanced mobile broadband (eMBB), massive machine type communication (mMTC), or ultra-reliable low-latency communication (URLLC). The wireless communication module 192 can support high-frequency bands (e.g., millimeter wave bands) to achieve, for example, high data transmission rates. The wireless communication module 192 can support various technologies for ensuring performance on high-frequency bands, such as, for example, beamforming, massive multiple-input multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. The wireless communication module 192 can support various requirements specified in the electronic device 101, an external electronic device (e.g., electronic device 104), or a network system (e.g., a second network 199). According to an embodiment, the wireless communication module 192 may support a peak data rate for implementing eMBB (e.g., 20 Gbps or greater), loss coverage for implementing mMTC (e.g., 164 dB or less), or U-plane delay for implementing URLLC (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less for a round trip).

[0047] The antenna module 197 may transmit or receive a signal or power to or from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a radiation element formed of a conductive material or a conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, the antenna module 197 may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected from the plurality of antennas by, for example, the communication module 190 (e.g., the wireless communication module 192). A signal or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, another component (e.g., a radio frequency integrated circuit (RFIC)) other than the radiation element may be additionally formed as a part of the antenna module 197.

[0048] According to various embodiments, the antenna module 197 may form a millimeter wave antenna module. According to an embodiment, the millimeter wave antenna module may include a printed circuit board, a radio frequency integrated circuit (RFIC), and a plurality of antennas (e.g., array antennas), wherein the RFIC is disposed on a first surface (e.g., bottom surface) of the printed circuit board, or adjacent to the first surface and capable of supporting a designated high frequency band (e.g., millimeter wave band), and the plurality of antennas are disposed on a second surface (e.g., top surface or side surface) of the printed circuit board, or adjacent to the second surface and capable of transmitting or receiving signals of the designated high frequency band.

[0049] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)) and communicatively transfer signals (e.g., commands or data) therebetween.

[0050] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic device 102 or the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be executed in the electronic device 101 may be executed in one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices receiving the request may execute at least part of the requested function or service, or execute another function or another service related to the request, and transmit the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, cloud computing technology, distributed computing technology, mobile edge computing (MEC) technology, or client-server computing technology, for example, may be used. The electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, the external electronic device 104 may include an Internet of Things (IoT) device. The server 108 may be an intelligent server using machine learning and / or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199. The electronic device 101 may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology or IoT-related technologies.

[0051] Figure 2 An operation of the electronic device 101 performing one or more functions based on the speech 220 according to an embodiment is shown.

[0052] Figure 2 The electronic device 101 may include Figure 1 The electronic device 101 may be a terminal owned by a user. The terminal may include, for example, a personal computer (PC) such as a laptop computer and a desktop computer, a smart phone, a smart tablet, and a tablet PC. The terminal may include smart accessories such as a smart watch and / or a head mounted device (HMD).

[0053] According to an embodiment, the electronic device 101 may interact with the user by using sound. The electronic device 101 may interact with the user by using a microphone (e.g., Figure 1 The input module 150 of the electronic device 101) obtains an audio signal, which is an electrical signal depending on the sound of the external space including the electronic device 101. The electronic device 101 can recognize the user's speech 220 from the audio signal. In response to recognizing the speech 220, the electronic device 101 can perform one or more functions related to the speech 220. The electronic device 101 can output information including the result of performing the one or more functions to the user. Figure 2 , the electronic device 101 can generate an audio signal including speech 250. By controlling a speaker (e.g., Figure 1 The electronic device 101 can play the sound depending on the audio signal. Figure 3 An example structure of an electronic device 101 including a microphone and / or a speaker is described.

[0054] According to an embodiment, the electronic device 101 may recognize an input indicating execution of at least one function related to the electronic device 101 from the speech 220. Hereinafter, a voice command may include a command (e.g., a command of a natural language) input to the electronic device 101 based on a sound such as the speech 220. A voice command may be a unit of function and / or task performed by the electronic device 101 and / or an external electronic device 210 connected to the electronic device 101 through a sound recognized by the electronic device 101. Executing a voice command may include an operation in which the electronic device 101 performs at least one function corresponding to the voice command. The electronic device 101 may recognize one or more voice commands included in the speech 220 based on the natural language included in the speech 220. The external electronic device 210 connected to the electronic device 101 may include a server (e.g., a server connected to the electronic device 101). Figure 1 Server 108 ) for processing an audio signal including speech 220 .

[0055] Reference Figure 2, showing an example case where the electronic device 101 recognizes multiple voice commands (e.g., the first voice command 230 and / or the second voice command 240) from an audio signal recognized by a microphone. The electronic device 101 may recognize speech 220 for sequentially executing multiple functions from the audio signal. The speech 220 may include a preset natural language (e.g., a word such as "Hey, Bixby") for triggering the electronic device 101 to recognize the speech 220. The electronic device 101 that recognizes the preset natural language may recognize one or more voice commands included in the speech 220 by recognizing the speech 220. For example, the speech 220 recognized by the electronic device 101 may include a natural language (e.g., a word, a phrase, and / or a sentence) representing each of the multiple functions. For example, the speech 220 recognized by the electronic device 101 may include a name assigned to a plurality of function groups. The name assigned to the group may be referred to as a quick command and / or an abbreviation. Based on recognizing the name from the speech 220, the electronic device 101 may execute a plurality of functions included in the group distinguished by the name. Based on the interaction between the electronic device 101 and the user, a voice command group for performing multiple functions may be registered with the electronic device 101. Fig. 8A An example of a user interface (UI) displayed by the electronic device 101 to recognize an input for registering a group is described.

[0056] According to an embodiment, the electronic device 101 may execute functions corresponding to the first voice command 230 and the second voice command 240 based on the recognition of the speech 220 for sequentially executing the first voice command 230 and the second voice command 240. These functions may be executed based on applications executed by the electronic device 101 and / or the external electronic device 210. Applications executed by the electronic device 101 and / or the external electronic device 210 to execute voice commands may be referred to as speech engines and / or natural language units (NLUs). The electronic device (e.g., the electronic device 101 and / or the external electronic device 210) in which the speech engine is executed may be controlled to perform operations matching the speech commands. The duration and / or speed of performing operations matching the speech commands by the execution of the speech engine may depend on the performance and / or resource status of the electronic device in which the speech engine is executed.

[0057] exist Figure 2In an embodiment, the voice engines are executed in both the electronic device 101 and the external electronic device 210 connected via a network, and functions corresponding to the first voice command 230 and the second voice command 240 can be executed by at least one of the voice engines. The electronic device 101 that recognizes the speech 220 for sequentially executing the first voice command 230 and the second voice command 240 can match each of the first function corresponding to the first voice command 230 and the second function corresponding to the second voice command 240 with the first voice engine executed by the electronic device 101 and the second voice engine executed by the external electronic device 210. For example, the electronic device 101 can control or schedule the execution of the first function and the second function by matching the first voice engine and the second voice engine with the first voice command 230 and the second voice command 240. For example, all the first functions and the second functions can be centrally executed by the first voice engine or the second voice engine. For example, each of the first function and the second function can be deployed and executed by the first voice engine and the second voice engine. Reference will be made to Figures 3 to 7 An example of an application executed by the electronic device 101 to match a plurality of voice engines with a plurality of voice commands included in the utterance 220 is described.

[0058] Reference Figure 2 , according to an embodiment, an example state of the electronic device 101 executing the function of each of the first voice command 230 and the second voice command 240 by using different voice engines is shown. The electronic device 101 may select a voice engine for executing at least one voice command included in the speech 220 based on the states of voice engines executed by different electronic devices including the electronic device 101. For example, at least one of the state of the first voice engine executed by the electronic device 101 or the state of the second voice engine executed by the external electronic device 210 may be monitored. The state of the voice engine may be related to whether the function corresponding to the voice command can be executed based on the execution of the voice engine, the time required to execute the function, and / or the delay before the function is executed. Based on the monitored state, the electronic device 101 may select a voice engine from the first voice engine and the second voice engine to execute each of the first voice command 230 and the second voice command 240 included in the speech 220. In Figure 2 In the example case of , the electronic device 101 may match the first voice engine with the first voice command 230 , and match the second voice engine with the second voice command 240 .

[0059] Based on matching the first voice command 230 with the first voice engine, the electronic device 101 may execute a first function corresponding to the first voice command 230. The electronic device 101 may obtain first information 235 including a result of executing the first function based on the execution of the first voice engine. Based on matching the second voice command 240 with the second voice engine, the electronic device 101 may request the external electronic device 210 executing the second voice engine to execute a second function corresponding to the second voice command 240. The electronic device 101 may receive second information 245 related to the second function as a response to the request from the external electronic device 210. Based on the external electronic device 210 and / or the second voice engine, the second information 245 may include a result of executing the second function. Based on the execution of the second function, the second information 245 may include information for controlling the electronic device 101.

[0060] According to an embodiment, the electronic device 101 may generate an audio signal including speech 250 based on obtaining the first information 235 and / or the second information 245, wherein the speech 250 indicates the result of executing the first function matching the first voice command 230 and the second function matching the second voice command 240. The electronic device 101 may obtain speech 250 indicating the first information 235 including the result of executing the first function and the second information 245 including the result of executing the second function based on one or more natural language sentences. The electronic device 101 may generate an audio signal including speech 250. By controlling a speaker based on the generated audio signal, the electronic device 101 may output a sound dependent on the audio signal through the speaker. The form in which the electronic device 101 outputs speech 250 is not limited to Figure 2 The audio signal and / or sound will be referred to Figure 8B An embodiment is described in which the electronic device 101 visualizes a result of executing at least one voice command for the speech 220 by using hardware such as the display 260 .

[0061] As described above, according to an embodiment, the electronic device 101 can schedule a voice engine to execute multiple functions based on recognizing speech 220 for sequentially executing multiple functions from an audio signal output from a microphone. For example, the electronic device 101 can divide multiple functions into at least one first function executed by a first voice engine and at least one second function executed by a second voice engine based on the state of at least one of the voice engines. The electronic device 101 can execute at least one first function among the multiple functions based on the result of scheduling the voice engine, and execute at least one second function among the multiple functions that is different from at least one first function by using an external electronic device 210. Figure 2 The first information 235 may be an example of a result of performing at least one first function based on the first voice engine executed by the electronic device 101 . Figure 2 The second information 245 may be an example of a result of executing at least one second function based on a second voice engine executed by the external electronic device 210. The electronic device 101 may output a second audio signal indicating the result of executing at least one first function and at least one second function through a speaker. The order in which the electronic device 101 outputs the second audio signal may match the order of the multiple functions related to the speech 220. Since the multiple functions corresponding to the speech 220 are executed by distributed processing based on voice engines executed by different electronic devices, the electronic device 101 may obtain the results of executing the multiple functions more quickly.

[0062] In the following, reference is made to Figure 3 , will refer to Figure 3 An example structure of the electronic device 101 and the external electronic device 210 for processing voice commands including the first voice command 230 and the second voice command 240 is described.

[0063] Figure 3 is a block diagram of an electronic device 101 and an external electronic device 210 according to an embodiment. Figure 3 The electronic device 101 may include Figure 1 to Figure 2 An electronic device 101. Figure 3 The external electronic device 210 may include Figure 2 An external electronic device 210.

[0064] The electronic device 101 may include at least one of a processor 120, a memory 130, a display 260, a speaker 310, a microphone 320, or a communication circuit 330. The processor 120, the memory 130, the display 260, the speaker 310, the microphone 320, and the communication circuit 330 may be electrically and / or operably connected to each other through electronic components such as a communication bus 305. Hereinafter, the operational connection of hardware components may mean establishing a direct or indirect connection between hardware components by wire or wirelessly, so that a second hardware component is controlled by a first hardware component of the hardware components. Although illustrated based on different blocks, the embodiments are not limited thereto, Figure 3 Some of the hardware of the electronic device 101 (for example, the processor 120, the memory 130, and at least a portion of the communication circuit 330) may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of hardware components included in the electronic device 101 are not limited to Figure 3 For example, the electronic device 101 may include only Figure 3 Some of the hardware shown.

[0065] According to an embodiment, the processor 120 of the wearable device 101 may include a hardware component for processing data based on one or more instructions. For example, the hardware component for processing data may include an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU) and / or an application processor (AP). According to an embodiment, the processor 120 may include a plurality of processors. For example, the processor 120 may have a structure of a multi-core processor, such as a dual-core, a quad-core, or a hexa-core. Figure 3 The processor 120 may include Figure 1 The processor 120.

[0066] The memory 130 of the electronic device 101 may include a hardware component for storing data and / or instructions input and / or output to the processor 120. The memory 130 may include, for example, a volatile memory (e.g., a random access memory (RAM)) and / or a non-volatile memory (e.g., a read-only memory (ROM)). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM (PSRAM), and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, an optical disk, a solid-state drive (SSD), and an embedded multimedia card (eMMC). Figure 3 The memory 130 may include Figure 1 Memory 130.

[0067] The display 260 of the electronic device 101 can output visual information (eg, Fig. 8A and Figure 8B ). For example, the display 260 may be controlled by a controller such as a graphics processing unit (GPU) and / or the processor 120 to output visual information to the user. The display 260 may include a flat panel display (FPD) and / or electronic paper. The FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LED may include an organic LED (OLED). Figure 3 The display 260 may include Figure 1 The display module 160 is provided.

[0068] According to an embodiment, the display 260 of the electronic device 101 may include a sensor (e.g., a touch sensor panel (TSP)) for detecting an external object (e.g., a user's finger) on the display 260. For example, based on the TSP, the electronic device 101 may detect an external object in contact with the display 260 or hovering on the display 260. In response to detecting the external object, the electronic device 101 may perform a function related to a specific visual object among the visual objects displayed in the display 260 corresponding to the position of the external object on the display 260.

[0069] The electronic device 101 may include a speaker 310 as an output device for outputting information in a form different from a visual form. The speaker 310 may include an audio signal (eg, including a Figure 2 The number of speakers 310 included in the electronic device 101 is not limited to Figure 3 The electronic device 101 may include one or more speakers. The electronic device 101 may include other output devices for outputting information in forms other than visual and auditory forms. For example, the electronic device 101 may include a motor for providing vibration-based tactile feedback.

[0070] The microphone 320 of the electronic device 101 may output an electrical signal indicating air vibration. For example, the electronic device 101 may recognize the user's speech (eg, Figure 2 The user's speech 220 is an electrical signal output from the microphone 320. Based on a speech recognition model and / or a natural language understanding model as an application and / or process executed by the processor 120, the user's speech included in the audio signal can be converted into information in a format recognizable by the electronic device 101. For example, the electronic device 101 can perform one or more functions of a plurality of functions provided by the electronic device 101 by recognizing the user's speech. Figure 3 The speaker 310 and / or microphone 320 may include Figure 1 The sound output module 155 and / or the audio module 170 are also provided.

[0071] The communication circuit 330 of the electronic device 101 may include hardware that supports the transmission and / or reception of electrical signals between the electronic device 101 and the external electronic device 210. The number of external electronic devices 210 connected to the electronic device 101 through the communication circuit 330 is not limited to Figure 2 and / or Figure 3Embodiment. The communication circuit 330 may include, for example, at least one of a MODEM, an antenna, and an optical / electrical (O / E) converter. The communication circuit 330 may support transmission and / or reception of electrical signals based on various types of protocols, and the various types of protocols may be, for example, Ethernet, local area network (LAN), wide area network (WAN), wireless fidelity (Wi-Fi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), and 5G new radio (NR). Figure 3 The communication circuit 330 may include Figure 1 communication module 190, user identification module 196 and / or antenna module 197.

[0072] One or more instructions (or commands) that indicate calculations and / or operations to be performed on data by the processor may be stored in memory 130. A collection of one or more instructions may be referred to as firmware, an operating system (e.g., Figure 1 operating system 142), programs (e.g., Figure 1 140), processes, routines, subroutines, and / or applications (e.g., Figure 1 For example, when executing a set of multiple instructions deployed in the form of an operating system, firmware, driver and / or application, the electronic device 101 and / or processor 120 may execute Figures 10 to 12 Hereinafter, the fact that an application is installed in the electronic device 101 indicates that one or more instructions provided in the form of an application are stored in the memory 130 of the electronic device 101, and means that one or more applications are stored in a format that can be executed by the processor 120 of the electronic device 101 (for example, a file with an extension specified by the operating system of the electronic device 101).

[0073] exist Figure 3In an embodiment, the external electronic device 210 may be connected to the electronic device 101 by wire or wirelessly to perform functions related to speech recognition. The external electronic device 210 may include a server. The server may include one or more PCs and / or workstations. In an embodiment, the server is a service that performs one or more functions of speech recognized corresponding to an audio signal output from a microphone 320 of the electronic device 101, and may include, for example, a speech recognition service. The external electronic device 210 may include at least one of a processor 120, a memory 130, and a communication circuit 330. The processor 120, the memory 130, and the communication circuit 330 may be electrically connected and / or operably connected via a communication bus 305 in the external electronic device 210. The processor 120, the memory 130, and the communication circuit 330 included in the external electronic device 210 may include hardware components and / or circuits corresponding to the processor 120, the memory 130, and the communication circuit 330 of the electronic device 101. Hereinafter, in order to reduce repetition, description of the processor 120 , the memory 130 , and the communication circuit 330 included in the external electronic device 210 may be omitted within the scope of overlapping with the processor 120 , the memory 130 , and the communication circuit 330 in the electronic device 101 .

[0074] Reference Figure 3 , the input receiver 340, the fast command processor 350, the speech processor 360 and / or the first voice engine 370 are illustrated as programs executed by the processor 120 of the electronic device 101. For example, a set of multiple instructions stored in the memory 130 and / or multiple processes executed by the processor 120 can be divided into the input receiver 340, the fast command processor 350, the speech processor 360 and / or the first voice engine 370. The input receiver 340, the fast command processor 350, the speech processor 360 and / or the first voice engine 370 may be included in an application installed in the electronic device 101. The application may be executed by the processor 120 of the electronic device 101 to control the electronic device 101 based on speech including natural language sentences.

[0075] Reference Figure 3, the second voice engine 380 is illustrated as a program executed by the processor 120 of the external electronic device 210. The second voice engine 380 may include a collection of multiple applications (e.g., one application) stored in the memory 130. The second voice engine 380 may be executed by the processor 120 of the external electronic device 210 to execute a voice command requested from another electronic device (e.g., the electronic device 101) connected to the external electronic device 210. The electronic device 101 may establish a communication link with the external electronic device 210 by using the communication circuit 330 to execute at least one voice command recognized by the electronic device 101 by using the second voice engine 380 executed by the external electronic device 210. For example, the electronic device 101 may recognize a preset natural language based on an audio signal output from the microphone 320 to guide the speaking of a natural language sentence corresponding to at least one voice command to establish a communication link.

[0076] According to an embodiment, the electronic device 101 may recognize speech from an audio signal output from the microphone 320 based on execution of the input receiver 340. In a state where the input receiver 340 is executed, the electronic device 101 may perform an operation of recognizing a voice included in an audio signal based on receiving the audio signal through the microphone 320. The audio signal processed by the input receiver 340 may be sent by the processor 120 of the electronic device 101 to the second voice engine 380 of the external electronic device 210. The input receiver 340 may recognize text corresponding to the speech of the user from one or more characters input through a software keyboard displayed by the display 260 and / or a hardware keyboard connected to the electronic device 101 and an audio signal output from the microphone 320.

[0077] The electronic device 101 may obtain text representing speech included in the audio signal from the external electronic device 210 executing the second voice engine 380. Based on the execution of the quick command processor 350, the electronic device 101 may identify whether the text obtained from the external electronic device 210 includes a name (e.g., a quick command) assigned to one or more voice command groups. Based on the execution of the quick command processor 350, the electronic device 101 may obtain a name and one or more voice command groups corresponding to the name from the user. Based on the execution of the quick command processor 350, the electronic device 101 may process whether the user's speech includes a name assigned to the group or a user input for registering the group. Figure 6 An example structure of the fast command processor 350 is described.

[0078] When the speech included in the audio signal obtained by the microphone 320 includes a name assigned to one or more voice command groups, the electronic device 101 may control the execution of one or more voice commands included in the group based on the execution of the speech processor 360. Based on the execution of the speech processor 360, the electronic device 101 may select a voice engine to which one or more voice commands will be input from the first voice engine 370 or the second voice engine 380. For example, based on the execution of the speech processor 360, the electronic device 101 may schedule the execution of one or more functions corresponding to the one or more voice commands. The electronic device 101 may obtain the state of one or more voice engines (e.g., the first voice engine 370 and / or the second voice engine 380) executed to process the voice commands based on recognizing a plurality of voice commands from the speech. The electronic device 101 may schedule the execution of one or more functions based on the state. Based on the execution of the speech processor 360, the electronic device 101 may input one or more voice commands in the group to the first voice engine 370, or send one or more voice commands to the external electronic device 210 through the communication circuit 330 of the electronic device 101.

[0079] Based on the execution of the speech processor 360, the electronic device 101 may obtain information including the result of executing one or more functions corresponding to one or more voice commands from the first voice engine 370 and / or the second voice engine 380 (for example, Figure 2 The electronic device 101 may output the information obtained using the speech processor 360 through the speaker 310 and / or the display 260. For example, the electronic device 101 may send an audio signal representing the information to the speaker 310. For example, the electronic device 101 may display a visual object and / or a screen including the information by controlling the display 260. Figures 4 to 7 An example structure of the speech processor 360 executed by the electronic device 101 to obtain a result of dispatching a voice command and / or performing a function corresponding to the voice command is described.

[0080] Based on the execution of the first voice engine 370, the electronic device 101 can recognize at least one voice command matched with the first voice engine 370 through the speech processor 360. The electronic device 101 can execute at least one function corresponding to the at least one voice command. The at least one function may include at least one of a function for controlling hardware included in the electronic device 101 (e.g., a function for adjusting the volume of the speaker 310), a function supported by an application installed in the memory 130 (e.g., an alarm clock, playing music), and a function for retrieving information from a network through the communication circuit 330 (e.g., weather query). The electronic device 101 can store the result of executing the at least one voice command by using the speech processor 360, or output the result to the user.

[0081] Based on the execution of the second voice engine 380, the processor 120 of the external electronic device 210 may execute at least one function corresponding to at least one voice command requested from the electronic device 101. The electronic device 101 may request the external electronic device 210 to execute at least one voice command based on the speech processor 360. In the state of executing the second voice engine 380, the processor 120 of the external electronic device 210 may execute at least one function corresponding to at least one voice command included in the request. The processor 120 of the external electronic device 210 may send the result of executing at least one function to the electronic device 101 by using the communication circuit 330. The result sent to the electronic device 101 may be output to the user by the processor 120 of the electronic device 101 executing the speech processor 360.

[0082] As described above, according to an embodiment, the electronic device 101 can schedule or control the execution of at least one function performed by multiple voice engines (e.g., the first voice engine 370 and the second voice engine 380) based on an application such as a speech processor 360. Based on the scheduling of multiple voice engines, the electronic device 101 can reduce the resources of the electronic device 101 and / or the external electronic device 210 occupied for executing multiple voice engines, while efficiently using the multiple voice engines.

[0083] According to an embodiment, in a state where a voice command group is registered by interacting with a user, the electronic device 101 may identify the attributes of the voice commands included in the group. The attributes may be used to schedule a voice engine to execute a function corresponding to the voice command included in the group. Figure 4 An example of an operation in which the electronic device 101 stores a plurality of voice command groups is described.

[0084] Figure 4 is a block diagram of a program executed by the electronic device 101 according to an embodiment.

[0085] Figure 4 The electronic device 101 may be Figure 3 For example, by Figure 3 The electronic device 101, the memory 130 and the input receiver 340, the fast command processor 350, the speech processor 360 and the first voice engine 370 executed by the processor 120 may include Figure 4 The electronic device 101, the memory 130, the input receiver 340, the fast command processor 350, the speech processor 360 and the first voice engine 370.

[0086] Reference Figure 4 , the instructions and / or subroutines of the first speech engine 370 for performing functions related to the user's speech (e.g., speech included in the audio signal) may be divided into a speech recognizer 450, a text-to-speech converter 455, a natural language processor 460, a slave manager 470, and / or an event processor 475. The instructions and / or subroutines of the natural language processor 460 may be divided into a domain classifier 462 and / or a target classifier 464.

[0087] Reference Figure 4 , the instructions and / or subroutines of the speech processor 360 for recognizing one or more voice commands from the user's speech (e.g., speech included in the audio signal) can be divided into a speech type classifier 410, a voice command interpreter 420, and / or a voice command controller 440. Based on the execution of the speech processor 360, the electronic device 101 can manage the voice command database 430 in the memory 130. Figure 5 An example of information stored in the voice command database 430 is described. The instructions and / or subroutines in the voice command interpreter 420 may be divided into a metadata analyzer 422 and / or a voice command generator 424. The instructions and / or subroutines in the voice command controller 440 may be divided into a voice command manager 442, a voice engine scheduler 446, and a speech memory 448.

[0088] Based on the execution of the speech processor 360, the electronic device 101 can control the execution of one or more voice commands that match the user's intention included in the speech. In the state of recognizing the input from the user for registering multiple voice command groups, based on the execution of the speech type classifier 410, the electronic device 101 can obtain the attributes of each of the multiple voice commands included in the group. The electronic device 101 can generate metadata of the voice command based on the attribute. The metadata can be stored in the memory 130 of the electronic device 101. Table 1 may include examples of attributes contained in the metadata of the voice command.

[0089] [Table 1]

[0090] The names of Table 1 may be example names assigned to attributes. An attribute with a name of "prompt" may indicate whether additional parameters are required to execute the function of a voice command based on a Boolean data type. An attribute with a name of "time" may indicate whether the result of executing a function varies according to time. An attribute with a name of "location" may indicate whether the result of executing a function varies according to the location of the electronic device 101. An attribute with a name of "device context" may indicate whether the result of executing a function varies according to the state of the electronic device 101 based on a Boolean data type. An attribute with a name of "support on device" may indicate whether a function can be executed by the first speech engine 370 based on a Boolean data type. Attributes included in metadata (e.g., attributes with names of "prompt", "time", "location", "device context", "support on device") may include information managed by the slave manager 470 (e.g., slave attribute mapping).

[0091] Referring to Table 1, an attribute with the name of "Event List" may include a list of one or more functions corresponding to a voice command. For example, the names of one or more functions may be stored in an attribute with the name of "Event List". For example, the fact that the text "Message" is stored in an attribute with the name of "Event List" may indicate that a voice command request corresponding to metadata executes a function of displaying a message. For example, the fact that the "TTS_Response" text is stored in an attribute with the name of "Event List" may indicate that a voice command request corresponding to metadata executes a TTS function. For example, the fact that the "appLaunch" text is stored in an attribute with the name of "Event List" may indicate that a voice command request corresponding to metadata executes at least one application installed in the electronic device 101.

[0092] The electronic device 101 obtains metadata for the voice command based on the first voice engine 370 executed using the speech type classifier 410. Based on the execution of the natural language processor 460 in the first voice engine 370, the electronic device 101 can perform pre-processing on the natural language included in the voice command. For example, based on the execution of the domain classifier 462, the electronic device 101 can identify whether the voice command can be processed by the first voice engine 370. Whether the voice command can be processed by the first voice engine 370 can be stored in the attribute with the name of "support on device" in Table 1. Based on the execution of the target classifier 464, the electronic device 101 can identify the target of the function (or action) performed by the voice command. Based on the execution of the slave manager 470 and / or the event processor 475, the electronic device 101 can perform post-processing on the pre-processed voice command. Based on the execution of the slave manager 470, the electronic device 101 can identify the device dependency of the resource for processing the voice command. The device dependency may include, for example, whether the function corresponding to the voice command can be processed by the first voice engine 370 and / or the electronic device 101. Based on the execution of the event processor 475, the electronic device 101 can obtain a list of functions corresponding to the voice command. The list obtained by the electronic device 101 can be stored in an attribute with a name of "event list" in the metadata.

[0093] Based on the execution of the speech type classifier 410, the electronic device 101 may obtain metadata corresponding to one or more voice commands grouped by the quick command. The obtained metadata may be stored in the memory 130 of the electronic device 101. The electronic device 101 may include an operation of storing the metadata in the memory 130. Registering the quick command by the electronic device 101 may include an operation of storing the metadata in the memory 130.

[0094] The electronic device 101 may process the metadata obtained based on the execution of the speech type classifier 410 based on the voice command interpreter 420. Based on the execution of the voice command interpreter 420, the electronic device 101 may identify the information stored in the voice command database 430 based on the metadata. Based on the execution of the metadata analyzer 422, the electronic device 101 may perform analysis of the metadata stored in the memory 130 based on the speech type classifier 410. Based on the execution of the voice command generator 424, the electronic device 101 may store the information parsed by the metadata analyzer 422 in the voice command database 430. Figure 5 An example of a scheme of the voice command database 430 is described.

[0095] As described above, according to an embodiment, based on an input indicating registration of a quick command, the electronic device 101 may obtain metadata corresponding to one or more voice commands that match the quick command. The electronic device 101 may store information corresponding to the one or more voice commands in the voice command database 430. Based on recognizing speech including the quick command after registering the quick command (e.g., Figure 2 The electronic device 101 can recognize one or more voice commands executed by the quick command and the attributes of one or more voice commands by using the stored metadata. Based on the metadata stored in the memory 130 by the speech type classifier 410 and the information stored in the voice command database 430, the electronic device 101 can more quickly recognize the one or more voice commands and attributes. Figure 6 Operations performed by the electronic device 101 based on recognition of a voice including a quick command are described.

[0096] In the following, reference will be made to Figure 5 An example of information stored in the voice command database 430 by the electronic device 101 in response to an input indicating registration of a quick command is described.

[0097] Figure 5 is a block diagram of a voice command database 430 included in an electronic device according to an embodiment.

[0098] Figure 5 The electronic device can be Figure 3 and Figure 4 For example, Figure 4 The voice command database 430 may include Figure 5 Voice command database 430. Based on Figure 3 to Figure 4 The implementation of the speech processor 360 can manage Figure 5 Voice command database 430.

[0099] According to an embodiment, the electronic device may divide and store the information included in the voice command database 430 into a first table 510 for storing parameters corresponding to the voice command and a second table 520 for storing parameters related to one or more events executed by the voice command. In a state of registering a quick command including one or more voice commands, the electronic device may update the voice command database 430 based on the one or more voice commands.

[0100] In the first table 510 of the voice command database 430, the electronic device may store parameters corresponding to the voice command. Table 2 may include examples of parameters stored in the first table 510 of the voice command database 430.

[0101] [Table 2]

[0102] Referring to Table 2, a parameter with an "ID" name may include an identifier (e.g., a key value) uniquely assigned to a record in the first table 510 of the voice command database 430. A parameter with a "command" name may include a natural language corresponding to the voice command. A parameter with an "engine type" name may include a value for identifying a voice engine capable of performing one or more functions corresponding to the voice command. For example, a parameter with an "engine type" name may include an indication of whether one or more functions can be performed by a first voice engine (e.g., Figure 3 The first speech engine 370) and the external electronic device (for example, Figures 2 to 3 The second speech engine (eg, Figure 3 The parameter with the name of "Static Event List" may indicate whether the function corresponding to the voice command has changed. The parameter with the name of "Event List" may indicate a method of outputting the result of executing one or more functions corresponding to the voice command. In the first table 510, the parameter with the name of "Event List" may include an identifier uniquely assigned to a record of the second table 520. Based on the parameter with the name of "Event List", the electronic device may identify a record in the second table 520 that stores a method of outputting the result of executing one or more functions of a voice command that matches a specific record of the first table 510. For example, the parameter with the name of "Event List" may be used for a combination of the first table 510 and the second table 520 (e.g., a table combination based on a join operation).

[0103] In the second table 520, based on the parameter (or field) with the name of "message (event message)", the electronic device can indicate a method of outputting the result of one or more functions of executing the event voice command. These parameters can indicate a method based on a structured text such as JavaScript Object Notation (JSON). The embodiment is not limited to this, and the parameter with the name of "event message" can indicate a method of outputting the result based on extended markup language (xml). In the parameter with the name of "event list", it can be stored whether the result of executing one or more functions will be displayed through a display or in a speech form based on a natural language.

[0104] As mentioned above Figure 4 and Figure 5As described, according to an embodiment, an electronic device may process input indicating registration of quick commands for executing multiple function groups. For example, a quick command may cause the sequential execution of user-arranged functions (e.g., music playback function, alarm dismissal function, and weather forecast output function) based on simplified natural language such as "good morning". The electronic device may store information for executing functions in a memory of the electronic device in units of voice commands corresponding to each of the functions. The information stored by the electronic device may include parameters for scheduling voice commands (e.g., the type of electronic device and / or voice engine that can execute the voice command).

[0105] After registering a quick command and one or more voice commands corresponding to the quick command, the electronic device can execute one or more functions corresponding to the one or more registered voice commands in response to a voice including the quick command. Figure 4 By executing the voice command controller 440, the electronic device can control the scheduling and / or execution of one or more functions. Figure 6 and Figure 7 An example of an operation performed by an electronic device based on recognizing a quick command from an audio signal output from a microphone according to an embodiment is described.

[0106] Figure 6 is a block diagram of a voice command database included in an electronic device according to an embodiment.

[0107] Figure 6 The electronic device 101 may be Figure 3 and Figure 4 For example, by Figure 3 The electronic device 101, the memory 130 and the input receiver 340, the fast command processor 350, the speech processor 360 and the first voice engine 370 executed by the processor 120 may include Figure 6 The electronic device 101, the memory 130, the input receiver 340, the fast command processor, the speech processor 360 and the first voice engine 370.

[0108] Reference Figure 6, the instructions and / or subroutines of the quick command processor 350 for identifying whether the user's speech includes a quick command may be divided into a comparator 610, an executor 620, and a quick command generator 630. Based on the execution of the quick command processor 350, the electronic device 101 may manage a quick command database 640 including information related to the quick commands in the memory 130. In a state where a quick command is registered from a user, the electronic device 101 may store in the quick command database 640 a natural language corresponding to the quick command, a language type (e.g., English), an identifier uniquely assigned to the quick command, and a time when the quick command is registered. The information stored in the quick command database 640 may be stored in an external electronic device (e.g., Figure 2 and Figure 3 in a memory 130 of an external electronic device 210 ).

[0109] According to an embodiment, the electronic device 101 can detect the presence of a signal from a microphone (eg, Figure 3 The electronic device 101 may recognize the user's speech from the audio signal output by the microphone 320 of the external electronic device. The recognition of speech by the electronic device 101 may include: based on the first speech engine 370 and / or the second speech engine (for example, Figure 2 The text corresponding to the audio signal is obtained by executing the speech recognizer 450 in the second speech engine 380. Based on the execution of the comparator 610, the electronic device 101 can identify whether the quick command is included in the speech by comparing the text with the quick commands stored in the quick command database 640.

[0110] When the quick command is not included in the speech, based on the execution of the speech processor 360, the electronic device 101 can recognize one or more voice commands included in the speech. The electronic device 101 can recognize metadata corresponding to the one or more voice commands from the memory 130 through the speech type classifier 410. When the metadata corresponding to the one or more voice commands exists in the memory 130, the electronic device 101 can recognize the attributes of the one or more voice commands based on the metadata. Based on the recognized attributes, the electronic device 101 can select the first voice engine 370 or the second voice engine (e.g., Figure 3 A voice engine is selected from the second voice engine 380) for executing one or more voice commands.

[0111] When the quick command is not included in the speech and a plurality of voice commands are recognized from the speech, the electronic device 101 may recognize the plurality of voice commands by using the first voice engine 370 or the second voice engine (eg, Figure 3The electronic device 101 may process the plurality of voice commands by selecting at least one of the first voice engine 370 and the second voice engine 380. For example, the electronic device 101 may select an idle voice engine from the first voice engine 370 and the second voice engine. In the example, the electronic device 101 may sequentially execute a plurality of functions corresponding to the plurality of voice commands by using the selected voice engine. In order to select the idle voice engine, the electronic device 101 may communicate with an external electronic device (e.g., Figures 2 to 3 The external electronic device 210 in the communication interface establishes a communication link, or maintains an established communication link.

[0112] Based on identifying the quick command included in the speech, the electronic device 101 can perform operations related to the quick command based on the executor 620. For example, the electronic device 101 can control the execution of the speech processor 360 based on the executor 620. In the state of identifying the quick command included in the speech, the electronic device 101 can identify metadata related to the quick command and parameters stored in the voice command database 430 by executing the speech processor 360. The metadata and parameters can indicate at least one voice command that matches the quick command. When the quick command is registered in the electronic device 101 to execute multiple voice commands, the user can execute the functions corresponding to the multiple voice commands by speaking a quick command with a relatively short length without speaking a sentence corresponding to each of the multiple voice commands. When the voice commands frequently executed by the user are grouped as quick commands, the electronic device 101 can execute multiple voice commands by identifying relatively short quick commands instead of identifying sentences corresponding to the voice commands.

[0113] In the state of identifying the information in the voice command database 430 and the metadata related to the quick command, based on the execution of the voice command controller 440, the electronic device 101 can obtain a list of at least one voice command corresponding to the quick command. The electronic device 101 can process at least one voice command based on the list scheduling voice engine. In the state of executing the voice command controller 440, the electronic device 101 can obtain information related to the first voice engine 370 and the second voice engine executed by the external electronic device. The information of the second voice engine may include the delay time and / or delay caused by the external electronic device that executes the second voice engine that processes the voice command. The information related to the second voice engine may include the state of the second voice engine (for example, whether it is idle). The electronic device 101 can divide at least one voice command corresponding to the quick command based on whether each of the first voice engine 370 and the second voice engine indicated by the state of the first voice engine 370 and the second voice engine can execute at least one function corresponding to the voice command. For example, when multiple voice commands are recognized in the list, based on the states of the first voice engine 370 and the second voice engine, the electronic device 101 can divide the multiple voice commands into a first group executed by the first voice engine 370 and a second group executed by the second voice engine.

[0114] As described above, according to an embodiment, based on recognizing a quick command from the user's speech, the electronic device 101 can recognize a plurality of voice commands corresponding to the quick command. Based on the information stored in the memory 130 when registering the quick command (e.g., metadata and / or voice command database 430), the electronic device 101 can obtain information for scheduling the plurality of voice commands. The electronic device 101 can select a voice engine from different voice engines (e.g., the first voice engine 370 and the second voice engine) to process each of the plurality of voice commands by executing the voice command controller 440 based on the information. Hereinafter, reference will be made to Figure 7 An example of an operation in which the electronic device 101 selects a voice engine to process each of a plurality of voice commands in a state in which the voice command controller 440 is executed is described.

[0115] Figure 7 is a block diagram of a program executed by the electronic device 101 according to an embodiment.

[0116] Figure 7 The electronic device 101 may be Figures 3 to 6 For example, Figures 3 to 6 The first speech engine 370 may include Figure 7 The first speech engine 370. Figure 4 The voice command controller 440 and the voice command database 430 may include Figure 7The voice command controller 440 and the voice command database 430 are configured as follows. Figure 3 The external electronic device 210 may include Figure 7 An external electronic device 210.

[0117] Reference Figure 7 , the instructions and / or subroutines of the voice command controller 440 executed by the electronic device 101 may be divided into a voice command manager 442, a voice engine scheduler 446, and / or a speech memory 448. Based on the quick command registered in the electronic device 101 from voice recognition, the electronic device 101 may execute the voice command controller 440. The electronic device 101 may obtain information required to execute the voice command controller 440 from the voice command database 430. The electronic device 101 may obtain information related to at least one voice command corresponding to the quick command from the voice command database 430.

[0118] Based on the execution of the voice command manager 442, the electronic device 101 may obtain information related to at least one voice command corresponding to the quick command. Based on the information, the electronic device 101 may determine whether to process the at least one voice command based on the voice engine. For example, when the result of executing at least one function corresponding to the voice command is cached in the memory of the electronic device 101 (e.g., Figure 3 When the result of executing at least one function corresponding to the voice command is not cached in the memory or the cached result is invalid, the electronic device 101 may determine to process the voice command based on the voice engine. When multiple voice commands correspond to quick commands, based on the execution of the voice command manager 442, the electronic device 101 may identify the voice command processed by the voice engine among the multiple voice commands.

[0119] Based on the execution of the voice engine scheduler 446, the electronic device 101 may perform scheduling of one or more voice commands processed by the voice engine. Scheduling may include selecting a voice engine for processing one or more voice commands executed by the electronic device 101 from among the first voice engine 370 and the second voice engine 380 executed by the external electronic device 210. In a state of performing scheduling of multiple voice commands, the electronic device 101 may assign multiple voice commands to one of the first queue 710 corresponding to the first voice engine 370 and the second queue 720 corresponding to the second voice engine 380. The first queue 710 is an area formed in the memory of the electronic device 101, and may store information related to one or more voice commands to be sequentially input to the first voice engine 370. The second queue 720 is another area formed in the memory, and may store information related to one or more voice commands to be sequentially sent to the external electronic device 210. The figure shows an embodiment in which the electronic device 101 forms a first queue 710 and a second queue 720 having a first-in-first-out (FIFO) data structure based on the execution of the voice engine scheduler 446, but is not limited thereto.

[0120] In the state of executing the voice engine scheduler 446, based on at least one of the states of the first voice engine 370 and / or the second voice engine 380 and the attributes of the voice command, the electronic device 101 may store the voice command in either the first queue 710 or the second queue 720. For example, based on the attribute with the name of "support on device" in Table 1 among the attributes, the electronic device 101 may obtain whether the voice command should be processed exclusively by a specific voice engine (e.g., the second voice engine 380). In response to identifying a voice command having an attribute that is processed exclusively by the second voice engine 380, the electronic device 101 may store the voice command in the second queue 720 in the first queue 710 or the second queue 720. For example, based on whether each of the first voice engine 370 or the second voice engine 380 is idle, the electronic device 101 may classify the voice command. For example, when the second voice engine 380 is in another state different from the idle state, the electronic device 101 may store at least one voice command to be processed by the voice engine in the first queue 710.

[0121] Based on the execution of the voice engine scheduler 446, the electronic device 101 can calculate the total duration during which the multiple voice commands included in the quick command are processed by using a cost function. In order to minimize the total duration, the electronic device 101 can classify the multiple voice commands into one of the first queue 710 and the second queue 720. The cost function can depend on the state and / or performance of each of the first voice engine 370 and the second voice engine 380.

[0122] Based on the execution of the voice engine scheduler 446, the electronic device 101 can directly or indirectly control the voice engine by using the result of classifying the multiple voice commands corresponding to the quick command into different voice engines. The electronic device 101 can execute the first voice engine 370 based on at least one voice command accumulated in the first queue 710. The electronic device 101 can send at least one signal including at least one voice command accumulated in the second queue 720 to the external electronic device 210 that executes the second voice engine 380. Based on the at least one signal, the electronic device 101 can indirectly control the second voice engine 380 executed by the external electronic device 210.

[0123] Based on the execution of the speech memory 448, the electronic device 101 may store the result of executing the voice command stored in each of the first queue 710 and the second queue 720 in the memory of the electronic device 101. Based on the execution of the speech memory 448, the electronic device 101 may store the result of processing at least one voice command accumulated in the first queue 710 by using the first voice engine 370 in the memory. Based on the execution of the speech memory 448, the electronic device 101 may store the result of processing at least one voice command included in the signal received from the external electronic device 210 and accumulated in the second queue 720 in the memory. The electronic device 101 may store the result of processing the voice command by different voice engines based on the order of processing multiple voice commands set when registering the quick command. For example, the results accumulated in the memory of the electronic device 101 based on the execution of the speech memory 448 may be arranged based on the order of processing multiple voice commands. Based on the execution of the speech memory 448, the electronic device 101 may form a queue, and the results of processing multiple voice commands are stored in the memory in the queue.

[0124] The electronic device 101 can obtain an audio signal corresponding to each of the results from processing the results of multiple voice commands corresponding to the quick command accumulated in the memory of the electronic device 101 based on the speech memory 448. The electronic device 101 can sequentially send the audio signal to the speaker based on the order for the multiple voice commands. Based on the sequential transmission of the audio signal, the electronic device 101 can output a sound representing the result through the speaker. The audio signal may include a speech (e.g., a natural language sentence) representing the result.

[0125] As described above, according to an embodiment, the electronic device 101 can divide and process a plurality of voice commands corresponding to one quick command by using each of the first voice engine 370 and the second voice engine 380. Since a plurality of voice commands are divided and processed, the idle time of each of the first voice engine 370 and the second voice engine 380 can be minimized.

[0126] In the following, according to the embodiments, reference will be made to Fig. 8A and Figure 8B An example of an operation in which the electronic device 101 registers a quick command and processes a plurality of voice commands corresponding to the quick command in response to an utterance including the quick command is described.

[0127] Fig. 8A and Figure 8B An example state 801 and an example state 802 of an electronic device displaying a user interface (UI) according to an embodiment are shown.

[0128] Fig. 8A and Figure 8B The electronic device 101 may be Figures 3 to 7 For example, Figure 3 The electronic device 101 and the display 260 may include Fig. 8A and Figure 8B The electronic device 101 and the display 260. Fig. 8A and Figure 8B , showing different states 801 and 802 of the electronic device 101 displaying a UI provided by an application for recognizing speech from an audio signal. The application may include a Figure 3 One or more instructions of the input receiver 340, the fast command processor 350, the speech processor 360 and / or the first speech engine 370.

[0129] Reference Fig. 8A In state 801, the electronic device 101 may display a screen for registering a quick command in the display 260. In state 801, the electronic device 101 may receive information required to register a quick command from a user through the display 260. The electronic device 101 may display a text box 810 for obtaining a natural language for identifying a quick command in the display 260. Together with the text box 810, the electronic device 101 may display a button 815 for obtaining a natural language based on speech. In response to an input indicating selection of the button 815, the electronic device 101 may obtain a natural language from an audio signal obtained by using a microphone.

[0130] exist Fig. 8A In state 801, the electronic device 101 may display a button 860 for adding one or more voice commands executed by the quick command on the display 260. In response to an input indicating selection of the button 860, the electronic device 101 may obtain the voice command through a text and / or audio signal. Fig. 8AIn the example state 801 in which four voice commands are obtained based on the quick command, the electronic device 101 may display a visual object 820, a visual object 830, a visual object 840, and a visual object 850 corresponding to each of the four voice commands in the display 260. The order in which the visual object 820, the visual object 830, the visual object 840, and the visual object 850 are arranged in the display 260 may indicate the order in which the four voice commands are executed based on the quick command.

[0131] exist Fig. 8A In example state 801, the electronic device 101 can visualize the voice command executed by the quick command by using visual object 820, visual object 830, visual object 840, and visual object 850. Visual object 820 may correspond to a voice command for executing a function related to an alarm clock. In visual object 820, the electronic device 101 may display a natural language sentence representing the voice command (e.g., "Set an alarm"). In visual object 820, the electronic device 101 may display a button 822 for adjusting the order in which the voice commands corresponding to visual object 820 are executed. In visual object 820, the electronic device 101 may display a button 824 for excluding the voice command corresponding to visual object 820 from the voice commands to be executed by the quick command. Similarly, in visual object 830, the electronic device 101 may display a natural language sentence representing the voice command corresponding to visual object 830 (e.g., "Read the last text message"). In visual object 840, electronic device 101 may display a natural language sentence representing a voice command corresponding to visual object 840 (e.g., "Set a reminder in 10 minutes"). In visual object 850, electronic device 101 may display a natural language sentence representing a voice command corresponding to visual object 850 (e.g., "Tell me about the weather"). With reference to visual objects 820, 830, 840, and 850 and text box 810, the speech (e.g., "command") specified for the quick command may be different from the natural language sentence required to execute visual objects 820, 830, 840, and 850. For example, the length of the specified speech may be shorter than the natural language sentence.

[0132] Reference Fig. 8A Based on the information displayed in the display 260, the electronic device 101 may display a button 870 for registering a quick command. Together with the button 870, the electronic device 101 may display a button 875 for bypassing the registration of the quick command. In response to an input indicating selection of the button 870, the electronic device 101 may register a quick command executed by the text input in the text box 810. To register a quick command, the electronic device 101 may perform a reference to Figure 4 and Figure 5Based on the above operations. Figure 3 and Figure 4 By executing the speech processor 360 of the electronic device 101, the electronic device 101 may store information related to the voice commands corresponding to the visual object 820, the visual object 830, the visual object 840, and the visual object 850. The information may include metadata of the voice commands described with reference to Table 1 and / or attributes of the voice commands described with reference to Table 2. For example, as shown in Table 3, the electronic device 101 may obtain the attributes of the voice commands.

[0133] [Table 3]

[0134] The attributes of Table 3 may be stored in metadata of voice commands corresponding to visual object 820, visual object 830, visual object 840, and visual object 850. Referring to Table 3, the first voice command corresponding to visual object 820 may include a natural language sentence related to an alarm (e.g., "set an alarm"). Since the natural language sentence does not include parameters (e.g., time) for adjusting the alarm, additional input of the parameters may be required to process the first voice command. In the metadata corresponding to the first voice command, the electronic device 101 may assign a preset value within an attribute with a "prompt" name, which indicates that additional input is required to process the first voice command.

[0135] Referring to Table 3, the second voice command corresponding to the visual object 830 may include a natural language sentence related to a short message service (SMS) (e.g., "read the last text message"). The second voice command may depend on the state of the electronic device 101 when processing the second voice command (e.g., a state in which at least one text message is stored in the electronic device 101). In the metadata corresponding to the second voice command, the electronic device 101 may assign a preset value in an attribute with a name of "device context", which indicates that the state of the electronic device 101 needs to be recognized to process the second voice command.

[0136] Referring to Table 3, the third voice command corresponding to the visual object 840 may include a natural language sentence related to an alarm clock (e.g., "set a reminder in 10 minutes"). The function executed by the third voice command may depend on the time point at which the third voice command is processed. In the metadata corresponding to the third voice command, the electronic device 101 may indicate that the current time point of the electronic device 101 is required to process the third voice command by assigning a preset value to an attribute with a "time" name.

[0137] Referring to Table 3, the fourth voice command corresponding to the visual object 850 may include a natural language sentence for obtaining information through a network (e.g., "Tell me about the weather conditions"). The function executed by the fourth voice command may require communication with an external electronic device different from the electronic device 101 (e.g., a third-party server for providing weather information). In the metadata corresponding to the fourth voice command, the electronic device 101 may indicate that an external electronic device is required to process the fourth voice command by assigning a preset value to an attribute with a name of "Supported on Device".

[0138] As described above with reference to Table 3, the electronic device 101 can obtain the attributes of four voice commands that match the user's quick command when registering a quick command recognized by the "command" speech. Fig. 8A After state 801, based on recognizing the "command" speech in the audio signal output from the microphone, the electronic device 101 can recognize the quick command registered to execute the four voice commands. Figure 8B , showing a state 802, in which the electronic device 101 identifies a Fig. 8A The state 801 is followed by the execution of the utterance of a quick command ("command").

[0139] exist Figure 8B In state 802, the electronic device 101 may obtain metadata corresponding to the four voice commands described with reference to Table 3 for the quick commands. Based on the metadata, the electronic device 101 may match each of the voice commands with any of the different voice engines. The voice engines may include a first voice engine executed by the electronic device 101 (e.g., Figure 3 The first speech engine 370 of the electronic device 101 and the second speech engine (eg, Figure 3 The electronic device 101 may assign the voice command to the voice engine so as to minimize the total duration of processing the voice command based on the voice engine. The operation of the electronic device 101 directly or indirectly processing the voice command based on the voice engine after assigning the voice command to the voice engine may be based on the above reference. Figure 6 and Figure 7 Described operations are performed.

[0140] According to an embodiment, the electronic device 101 may sequentially output audio signals representing the results of processing the voice commands according to the order of the voice commands. Fig. 8A In state 801 , the electronic device 101 may sequentially output audio signals based on the order of the visual object 820 , the visual object 830 , the visual object 840 , and the visual object 850 .

[0141] According to an embodiment, the electronic device 101 may visualize the result of processing the voice command in the display 260. The electronic device 101 may display the visual object 882, the visual object 884, the visual object 886, and the visual object 888 corresponding to each of the results in parallel through the visual object 880 in the form of a pop-up window. The order and / or layout of the electronic device 101 displaying the visual object 882, the visual object 884, the visual object 886, and the visual object 888 is not limited to Figure 8B Reference Figure 8B , in visual object 882, the electronic device 101 may display information including the result of executing the first voice command. In visual object 884, the electronic device 101 may display information including the result of executing the second voice command. In visual object 886, the electronic device 101 may display information including the result of executing the third voice command. In visual object 888, the electronic device 101 may display information including the result of executing the fourth voice command. The electronic device 101 may separately display the results of executing voice commands in parallel using different voice engines by using visual object 882, visual object 884, visual object 886, and visual object 888. For example, before the processing of the second voice command is completed, the electronic device 101 may use visual object 886 to display the result of processing the third voice command.

[0142] As described above, according to an embodiment, in response to a speech including a quick command, the electronic device 101 can quickly execute multiple voice commands that match the quick command by using different voice engines. The electronic device 101 can output an audio signal representing the result of processing multiple voice commands by using a speaker. The electronic device 101 can display the results in parallel in the display 260. The embodiment is not limited to this, and for example, the electronic device 101 can display the results sequentially in the display 260. Since the electronic device 101 displays the results in parallel through the display 260, the electronic device 101 can display the result of processing another voice command to the user, regardless of the failure of processing a specific voice command among the multiple voice commands.

[0143] Fig. 9 An operation of the electronic device 101 performing a plurality of functions based on speech 910 according to an embodiment is shown.

[0144] Fig. 9 The electronic device 101 may be Figures 3 to 7 For example, Figure 3 The electronic device 101 and the display 260 may include Fig. 9 electronic device 101 and display 260. Fig. 9 The external electronic device 210 may include Figure 2 and / or Figure 7An external electronic device 210.

[0145] According to an embodiment, the electronic device 101 may recognize a speech 910 including a plurality of voice commands. The speech 910 may be recognized from an audio signal output through a microphone included in the electronic device 101. Fig. 9 , the electronic device 101 can recognize a first voice command 920 corresponding to a function for keeping an alarm from a first portion 912 (e.g., "set an alarm for 7 o'clock") in a natural language sentence included in the speech 910. The electronic device 101 can recognize a second voice command 930 corresponding to a function for outputting information collected from the network from a second portion 914 (e.g., "tell me the news") in a natural language sentence included in the speech 910. Based on Figure 4 By executing the speech processor 360 , the electronic device 101 can recognize the voice command 920 and the voice command 930 from the speech 910 .

[0146] Based on recognizing multiple voice commands (e.g., first voice command 920 and second voice command 930) from speech 910, electronic device 101 may input the multiple voice commands into at least one of voice engines executed by different electronic devices. The voice engine may include a first voice engine executed by electronic device 101 (e.g., Figure 3 ) and a second speech engine (eg, a first speech engine 370 in the electronic device 101) executed by an external electronic device 210 connected to the electronic device 101. Figure 3 The second speech engine 380 in the example above).

[0147] Reference Fig. 9 , the electronic device 101 may obtain first information 925 including a result of processing the first voice command 920 by processing the first voice command 920 using the first voice engine. The electronic device 101 may send a signal including a second voice command 930 to the external electronic device 210 to process the second voice command 930 by using the second voice engine. The electronic device 101 may receive another signal including the second information 935 as a response to the signal from the external electronic device 210. The second information 935 included in the another signal may include a result of processing the second voice command 930 by the external electronic device 210 executing the second voice engine.

[0148] According to an embodiment, the electronic device 101 may output to the user a speech 940 including information (e.g., first information 925 and second information 935) including a result of processing a plurality of voice commands (e.g., first voice command 920 and second voice command 930) recognized from the speech 910. The electronic device 101 may output the speech 940 by using an audio signal transmitted to a speaker. Fig. 9, speech 940 may include a natural language sentence 942 based on first information 925 as a result of processing the first voice command 920 (e.g., “The alarm will go off at 7 o'clock”), and a natural language sentence 944 based on second information 935 as a result of processing the second voice command 930 (e.g., “News: Today, S Company is…”).

[0149] As described above, according to an embodiment, based on speech including a quick command (e.g., Figure 2 220) and / or speech representing multiple voice commands (e.g., Fig. 9 In the speech 910 in the electronic device 101, the electronic device 101 can recognize an input for processing multiple voice commands. Based on the input, based on the states of the first voice engine executed by the electronic device 101 and the second voice engine executed by the external electronic device 210, the electronic device 101 can process the multiple voice commands in parallel. When multiple voice commands are processed in parallel in response to a quick command, the electronic device 101 can sequentially display the results of processing the multiple voice commands in parallel according to the order of the multiple voice commands arranged by the quick command.

[0150] Fig.10 is a flowchart of operations performed by an electronic device according to an embodiment.

[0151] Fig.10 The electronic device may include Figures 1 to 9 An electronic device 101. Fig.10 At least one of the operations may be performed by Figure 3 The electronic device 101 and / or Figure 3 The processor 120 in the electronic device 101 is executed. In the following embodiments, each operation can be performed sequentially, but it is not necessary to perform sequentially. For example, the order of each operation can be changed, and at least two operations can be performed in parallel.

[0152] Reference Fig.10 In operation 1010, according to an embodiment, the electronic device may recognize a first audio signal through a microphone. The first audio signal may indicate vibration of an external space including the microphone. The microphone of operation 1010 may include a microphone connected to the electronic device by wire (e.g., Figure 3 microphone 320 and headphones), or through Figure 3 The communication circuit 330 is wirelessly connected to the microphone of the electronic device (e.g., Figure 3 at least one of wireless headphones).

[0153] Reference Fig.10 In operation 1020, according to an embodiment, the electronic device may determine whether speech for sequentially executing a plurality of functions is recognized from the first audio signal. The speech may include, for example, Figure 2 The speech 220 is a natural language registered in the electronic device to execute the quick command. The speech may include such as Fig. 9 The speech 910 is a natural language sentence representing multiple functions.

[0154] According to an embodiment, in a state where speech for sequentially executing a plurality of functions is recognized from a first audio signal (1020-Yes), the electronic device may execute at least one first function among the plurality of functions based on a processor based on operation 1030. In this state, based on operation 1040, by using a communication circuit (e.g., Figure 3 The communication circuit 330 of the electronic device 101) is connected to an external electronic device (for example, Figure 3 In the external electronic device 210, the electronic device may execute at least one second function different from the at least one first function among the plurality of functions. In order to perform operations 1030 and 1040, the electronic device may divide the plurality of functions matched with the speech recognized from the first audio signal into at least one first function and at least one second function executed by a processor of the electronic device.

[0155] The operation of the electronic device dividing the multiple functions into at least one first function and at least one second function can be based on the above reference Figure 6 and Figure 7 The electronic device may perform the operations described in the foregoing. For example, based on the states of the first voice engine executed by the processor and the second voice engine executed by the external electronic device, the electronic device may divide the multiple functions into at least one first function and at least one second function. The electronic device may process at least one first function divided from the multiple functions by using the first voice engine executed by the processor of the electronic device based on operation 1030. The electronic device may transmit at least one second function to the external electronic device based on operation 1040, and obtain the result of executing the at least one second function from the external electronic device.

[0156] Reference Fig.10 In operation 1050, according to an embodiment, the electronic device may output a second audio signal indicating a result of executing at least one first function and at least one second function through a speaker. The electronic device may output the second audio signal as a response to the speech recognized through the first audio signal. Each of the second audio signals may include a natural language sentence indicating a result of processing each of the plurality of operation functions 1020. The order of the natural language sentences output from the electronic device through the second audio signal may correspond to the order of the plurality of functions indicated by the speech.

[0157] When a speech for executing a single function (e.g., a third function) is recognized from the first audio signal (1020-No) based on operation 1060, the electronic device may execute the third function corresponding to the first audio signal by using either the processor of the electronic device or the external electronic device. The electronic device may recognize the state of the first voice engine executed by the processor and / or the state of the second voice engine executed by the external electronic device. For example, the electronic device may execute the third function by using the first voice engine or the second voice engine that is in an idle state. In operation 1060, similar to operation 1050, the electronic device may output a third audio signal representing the result of executing the third function.

[0158] As described above with reference to operations 1050 and 1060, based on the speech included in the first audio signal, the electronic device may output the second audio signal and / or the third audio signal. Figure 8B As described, by using a display in the electronic device, the electronic device can visualize the results of performing one or more functions corresponding to the speech recognized by operation 1020 .

[0159] Fig.11 is a flowchart of operations performed by an electronic device according to an embodiment.

[0160] Fig.11 The electronic device may include Figures 1 to 9 An electronic device 101. Fig.11 At least one operation can be performed by Figure 3 The electronic device 101 and / or Figure 3 Executed by the processor 120 in the electronic device 101. Fig.11 At least one operation can be combined with Fig.10 In the following embodiments, each operation may be performed sequentially, but not necessarily sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0161] Reference Fig.11 In operation 1110, according to an embodiment, the electronic device may recognize speech from an audio signal. The electronic device may recognize speech by using a first speech engine executed by a processor of the electronic device, or may recognize speech by using a second speech engine executed by an external electronic device connected to the electronic device. Recognizing speech by the electronic device may include obtaining text data corresponding to the speech.

[0162] Reference Fig.11 In operation 1120, according to an embodiment, the electronic device may determine whether metadata corresponding to speech is recognized. The speech processor (e.g., Figure 3 The speech processor 360 in the embodiment of the present invention is used to identify metadata.

[0163] In the state of identifying metadata corresponding to speech (1120-Yes), based on operation 1130, the electronic device can select a speech engine to process the speech based on the metadata identified by operation 1120. The electronic device can select a speech engine for performing one or more functions related to speech from the first speech engine or the second speech engine. When multiple voice commands are identified from the speech, the electronic device can independently perform operation 1130 for each of the multiple voice commands. For example, based on operation 1130, a speech engine that processes each of the multiple voice commands can be independently selected from the first speech engine or the second speech engine. Based on the attributes in the metadata, the electronic device can select a speech engine from the first speech engine and the second speech engine to process the speech. For example, when an attribute indicating that speech is selectively performed by a specific speech engine is identified through metadata, the electronic device can select a speech engine to process the speech based on the attribute. When the metadata corresponding to the speech is not identified (1120-No), the selection of the speech engine based on operation 1130 can be bypassed.

[0164] Reference Fig.11 In operation 1140, according to an embodiment, the electronic device may determine whether to recognize parameters required to execute a function corresponding to the speech from the speech. When the parameters are recognized from the speech (1140-No) based on operation 1160, the electronic device may request the user to perform the parameters required for the function. When the parameters are recognized from the speech, or the function does not require any parameters (1140-Yes), based on operation 1150, the electronic device may process the speech by using a voice engine. The electronic device may output the result of processing the speech (e.g., the result of executing one or more voice commands included in the speech) by using a speaker and / or a display.

[0165] Fig.12 is a flowchart of operations performed by an electronic device according to an embodiment.

[0166] Fig.12 The electronic device may include Figures 1 to 9 An electronic device 101. Fig.12 At least one operation can be performed by Figure 3 The electronic device 101 and / or Figure 3 Executed by the processor 120 in the electronic device 101. Fig.12 At least one operation can be combined with Figure 10 to Figure 11 In the following embodiments, each operation may be performed sequentially, but not necessarily sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0167] Reference Fig.12 In operation 1210, according to an embodiment, the electronic device may recognize speech included in a first audio signal received through a microphone. Fig.11 At operation 1110, the electronic device may perform Fig.12 Operation 1210.

[0168] Reference Fig.12 In operation 1220, according to an embodiment, based on recognizing multiple voice commands from speech, the electronic device may obtain the status of a first voice engine executed by a processor and a second voice engine executed by an external electronic device. Based on recognizing a quick command for executing multiple voice command groups within speech, the electronic device may perform operation 1220. The status obtained by the electronic device based on operation 1220 may indicate whether the first voice engine and / or the second voice engine is in an idle state.

[0169] Reference Fig.12 , in operation 1230, according to an embodiment, the electronic device may divide a plurality of voice commands into a first voice command corresponding to a first voice engine and a second voice command corresponding to a second voice engine based on a state. Based on the possibility of executing a function of each of the first voice engine or the second voice engine indicated by the state, the electronic device may divide each of the plurality of functions into at least one first function and at least one second function. Based on the feasibility of the function of each of the first voice engine or the second voice engine indicated by the state, the electronic device may divide each of the plurality of functions into at least one first function and at least one second function. Based on operation 1230, the electronic device may perform scheduling of the plurality of voice commands of operation 1220. The electronic device may store the first voice command and the second voice command in different areas (e.g., Figure 7 The first queue 710 and the second queue 720).

[0170] Reference Fig.12 In operation 1240, according to an embodiment, the electronic device may execute the first function corresponding to the first voice command by using the first voice engine. Since the first voice engine is executed by the processor of the electronic device, the electronic device may selectively process the first voice command among the plurality of voice commands based on operation 1240.

[0171] Reference Fig.12In operation 1250, according to an embodiment, the electronic device may request the external electronic device to execute the second function corresponding to the second voice command. The electronic device may request to execute the second function by sending information related to the second voice command to the external electronic device executing the second voice engine. The electronic device may receive a signal and / or information including a result of executing the second function in response to the request from the external electronic device. The order in which the electronic device executes operations 1240 and 1250 is not limited. Fig.12 For example, the electronic device may perform operation 1240 and operation 1250 in parallel.

[0172] Reference Fig.12 In operation 1260, according to an embodiment, the electronic device may output a second audio signal indicating the execution of the first function and the second function based on the order of the plurality of voice commands indicated by the speech. The sound output by the electronic device through the second audio signal may include a natural language sentence indicating the result of executing the first function and the second function. The order in which the natural language sentences are output may depend on the order of processing the plurality of voice commands set in the state of registering the quick command.

[0173] Fig.13 is a block diagram illustrating an artificial intelligence (AI) system according to an embodiment.

[0174] Reference Fig.13 According to the embodiment, the artificial intelligence system 13 may include a user terminal 1300, an intelligent server 1400 and a service server 1500.

[0175] According to the embodiment, the user terminal 1300 (eg, Figure 1 The electronic device 131) may be a terminal device (or electronic device) that can be connected to the Internet, and for example, it may be a mobile phone, a smartphone, a personal digital assistant (PDA), a laptop computer, a TV, a white goods, a wearable device, an HMD, or a smart speaker.

[0176] According to an embodiment, the user terminal 1300 may include a communication interface 1310, a microphone 1320, a speaker 1330, a display 1340, a memory 1350, and a processor 1360. The above-listed components may be operably connected or electrically connected to each other.

[0177] According to an embodiment, the communication interface 1310 may be configured to be connected to an external device to send and receive data. According to an embodiment, the microphone 1320 may receive sound (e.g., user speech) and convert it into an electrical signal. According to an embodiment, the speaker 1330 may output an electrical signal as sound (e.g., voice). According to an embodiment, the display 1340 may be configured to display an image or video. According to an embodiment, the display 1340 may display a graphical user interface (GUI) of an application (or application program) being executed.

[0178] The display 1340 according to an embodiment may be configured to display an image or video. The display 1340 according to an embodiment may also display a graphical user interface (GUI) of an application (or application program) being executed. The display 1340 according to an embodiment may receive a touch input through a touch sensor. For example, the display 1340 may receive a text input through a touch sensor in an image keyboard area displayed in the display 1340.

[0179] According to an embodiment, the memory 1350 may store a client module 1351, a software development kit (SDK) 1353, and a plurality of applications 1355. The client module 1351 and the SDK 1353 may include a framework (or solution program) for performing general functions. In addition, the client module 1351 or the SDK 1353 may include a framework for processing user input (e.g., voice input, text input, touch input).

[0180] According to an embodiment, the plurality of applications 1355 may be programs for executing a specified function. According to an embodiment, the plurality of applications 1355 may include a first application 1355_1 and a second application 1355_3. According to an embodiment, each of the plurality of applications 1355 may include a plurality of operations for executing a specified function. For example, the plurality of applications 1355 may include at least one of an alarm clock application, a message application, and a scheduling application. According to an embodiment, the plurality of applications 1355 may be executed by the processor 1360 to sequentially execute at least a portion of the plurality of operations.

[0181] According to an embodiment, the processor 1360 may control the overall operation of the user terminal 1300. For example, the processor 1360 may be electrically connected to the communication interface 1310, the microphone 1320, the speaker 1330, the display 1340, and the memory 1350 to perform a designated operation.

[0182] According to an embodiment, the processor 1360 may also execute a program stored in the memory 1350 to perform a specified function. For example, the processor 1360 may execute at least one of the client module 1351 or the SDK 1353 to perform the following operations to process user input. For example, the processor 1360 may control the operations of the plurality of applications 1355 through the SDK 1353. The following operations described as operations of the client module 1351 or the SDK 1353 may be operations performed by the processor 1360.

[0183] According to an embodiment, the client module 1351 may receive user input. For example, the client module 1351 may generate a voice signal corresponding to the user's speech detected by the microphone 1320. Alternatively, the client module 1351 may receive a touch input detected by the display 1340. Alternatively, the client module 1351 may receive a text input detected by a keyboard or an image keyboard. In addition, the client module 1351 may receive various types of user input detected by an input module included in the user terminal 1300 or an input module connected to the user terminal 1300. The client module 1351 may send the received user input to the intelligent server 1400. According to an embodiment, the client module 1351 may send the state information of the user terminal 1300 together with the received user input to the intelligent server 1400. For example, the state information may be the execution state information of the application.

[0184] According to an embodiment, the client module 1351 may receive a result corresponding to the received user input. For example, the client module 1351 may receive a result corresponding to the user input from the intelligent server 1400. The client module 1351 may display the received result on the display 1340. In addition, the client module 1351 may output the received result as audio through the speaker 1330.

[0185] According to an embodiment, the client module 1351 may receive a plan corresponding to the received user input. The client module 1351 may display the result of executing multiple operations of the application according to the plan on the display 1340. For example, the client module 1351 may sequentially display the result of executing multiple operations on the display and output audio through the speaker 1330. For another example, the user terminal 1300 may display a part of the result of executing multiple operations (e.g., the result of the last operation) on the display and output audio through the speaker 1330.

[0186] According to an embodiment, the client module 1351 may receive a request for obtaining information required to calculate a result corresponding to a user input from the smart server 1400. For example, the information required to calculate the result may be state information of the user terminal 1300. According to an embodiment, the client module 1351 may send the required information to the smart server 1400 in response to the request.

[0187] According to an embodiment, the client module 1351 may transmit result information of performing a plurality of operations according to the plan to the smart server 1400. The smart server 1400 may recognize that the user input received through the result information is correctly processed.

[0188] According to an embodiment, the client module 1351 may include a voice recognition module. According to an embodiment, the client module 1351 may recognize a voice input for executing a limited function through the voice recognition module. For example, the client module 1351 may execute a smart application to process a voice input for executing an organic operation through a specified input (e.g., wake up!).

[0189] According to an embodiment, the intelligent server 1400 may receive information related to the user voice input from the user terminal 1300 through a communication network. According to an embodiment, the intelligent server 1400 may convert the data related to the received voice input into text data. According to an embodiment, the intelligent server 1400 may generate a plan for executing a task corresponding to the user voice input based on the text data.

[0190] According to an embodiment, the plan may be generated by an artificial intelligence (AI) system. The AI ​​system may be a rule-based system or a neural network-based system (e.g., a feedforward neural network (FNN), or a recurrent neural network (RNN)). Alternatively, it may be a combination of the above or a different artificial intelligence system. According to an embodiment, a plan may be selected from a set of predetermined plans, or a plan may be generated in real time in response to a user request. For example, the AI ​​system may select at least one plan from a plurality of predetermined plans.

[0191] According to an embodiment, the intelligent server 1400 may send the result calculated according to the generated plan to the user terminal 1300, or send the generated plan to the user terminal 1300. According to an embodiment, the user terminal 1300 may display the result calculated according to the plan on the display. According to an embodiment, the user terminal 1300 may display the result of performing the operation according to the plan on the display.

[0192] The intelligent server 1400 according to an embodiment may include a front end 1410 , a natural language platform 1420 , a capsule DB 1430 , an execution engine 1440 , an end user interface 1450 , a management platform 1460 , a big data platform 1470 , and an analysis platform 1480 .

[0193] According to an embodiment, the front end 1410 may receive a user input received from the user terminal 1300. The front end 1410 may send a response corresponding to the user input.

[0194] According to an embodiment, the natural language platform 1420 may include an automatic speech recognition module (ASR module) 1421 , a natural language understanding module (NLU module) 1423 , a planner module 1425 , a natural language generator module (NLG module) 1427 , and a text-to-speech module (TTS module) 1429 .

[0195] According to an embodiment, the automatic speech recognition module 1421 may convert the voice input received from the user terminal 1300 into text data. According to an embodiment, the natural language understanding module 1423 may use the text data of the voice input to understand the user's intention. For example, the natural language understanding module 1423 may perform syntactic analysis or semantic analysis on the user input in the form of text data to understand the user's intention. According to an embodiment, the natural language understanding module 1423 may understand the meaning of the words extracted from the user input by using the language features (e.g., grammatical elements) of morphemes or phrases, and determine the user's intention by matching the meaning of the recognized words with the intention. The natural language understanding module 1423 may obtain the intention information corresponding to the user's speech. The intention information may be information indicating the user's intention determined by interpreting the text data. The intention information may include information indicating the operation or function that the user wants to perform using the device.

[0196] According to an embodiment, the planner module 1425 can generate a plan by using the intent and parameters determined by the natural language understanding module 1423. According to an embodiment, the planner module 1425 can determine multiple fields required for performing the task based on the determined intent. The planner module 1425 can determine multiple operations included in each of the multiple fields determined based on the present invention. According to an embodiment, the planner module 1425 can determine the parameters required for performing the determined multiple operations or the result values ​​output by performing multiple operations. Parameters and result values ​​can be defined as concepts related to a specified format (or category). Therefore, the plan can include multiple operations and multiple concepts determined by the user's intent. The planner module 1425 can determine the relationship between multiple operations and multiple concepts in stages (or layers). For example, based on multiple concepts, the planner module 1425 can determine the order of performing multiple operations determined based on the user's intent. In other words, based on the parameters required for performing multiple operations and the results output by performing multiple operations, the planner module 1425 can determine the order of performing multiple operations. Therefore, the planner module 1425 can generate a plan including association information (eg, ontology) between a plurality of operations and a plurality of concepts. The planner module 1425 can generate a plan by using information stored in the capsule database 1430, in which a set of relationships between concepts and operations is stored.

[0197] According to an embodiment, the natural language generation module 1427 may convert the specified information into a text form. The information converted into a text form may be in the form of a natural language speech. According to an embodiment, the text-to-speech conversion module 1429 may convert text type information into speech type information.

[0198] According to an embodiment, the capsule database 1430 may store information related to the relationship between multiple concepts corresponding to multiple fields and operations. For example, the capsule database 1430 may store multiple capsules including multiple action objects (or action information) and concept objects (or concept information) planned. According to an embodiment, the capsule database 1430 may store multiple capsules in the form of a concept action network (CAN). According to an embodiment, multiple capsules may be stored in a function registration table included in the capsule database 1430.

[0199] According to an embodiment, the capsule database 1430 may include a policy registry in which policy information required when determining a plan corresponding to a voice input is stored. The policy information may include reference information for determining a plan when there are multiple plans corresponding to the user input. According to an embodiment, the capsule database 1430 may include a subsequent registry in which information related to subsequent operations for suggesting subsequent operations to the user in a specified situation is stored. The subsequent operation may include, for example, a subsequent speech. According to an embodiment, the capsule database 1430 may include a layout registry for storing layout information of information output through the user terminal 1300. According to an embodiment, the capsule database 1430 may include a vocabulary registry in which vocabulary information included in the capsule information is stored. According to an embodiment, the capsule database 1430 may include a dialogue registry in which dialogue (or interaction) information with the user is stored.

[0200] According to an embodiment, the capsule database 1430 may update objects stored by the developer tool. For example, the developer tool may include a function editor for updating an action object or a concept object. The developer tool may include a vocabulary editor for updating a vocabulary. The developer tool may include a strategy editor for generating and registering a strategy for determining a plan. The developer tool may include a dialogue editor for generating a dialogue with a user. The developer tool may include a subsequent operation editor for activating a subsequent goal and editing a subsequent speech that provides a prompt. The subsequent goal may be determined based on a currently set goal, a user preference, or an environmental condition.

[0201] According to an embodiment, the capsule database 1430 may also be implemented in the user terminal 1300. In other words, the user terminal 1300 may include the capsule database 1430 storing information for determining an operation corresponding to a voice input.

[0202] According to an embodiment, the execution engine 1440 may calculate the result by using the generated plan. According to an embodiment, the terminal user interface 1450 may send the calculated result to the user terminal 1300. Therefore, the user terminal 1300 may receive the result and provide the received result to the user. According to an embodiment, the management platform 1460 may manage the information used in the intelligent server 1400. According to an embodiment, the big data platform 1470 may collect user data. According to an embodiment, the analysis platform 1480 may manage the quality of service (QoS) of the intelligent server 1400. For example, the analysis platform 1480 may manage the components and processing speed (or efficiency) of the intelligent server 1400.

[0203] According to an embodiment, the service server 1500 may provide a specified service (e.g., a food order or a hotel reservation) to the user terminal 1300. According to an embodiment, the service server 1500 may be a server operated by a third party. For example, the service server 1500 may include a first service server 1501, a second service server 1503, and a third service server 1505 operated by different third parties. According to an embodiment, the service server 1500 may provide information for generating a plan corresponding to the received voice input to the intelligent server 1400. For example, the provided information may be stored in the capsule database 1430. In addition, the service server 1500 may provide result information to the intelligent server 1400 according to the plan.

[0204] In the above artificial intelligence system 13, the user terminal 1300 can provide various intelligent services to the user in response to user input. For example, the user input can include input through a physical button, touch input, or voice input.

[0205] According to an embodiment, the user terminal 1300 can provide a voice recognition service through an intelligent application (or voice recognition application) stored therein. In this case, for example, the user terminal 1300 can recognize the user's utterance or voice input received through a microphone, and provide the user with a service corresponding to the recognized voice input.

[0206] According to an embodiment, based on the received voice input, the user terminal 1300 can perform a specified operation alone, or perform a specified operation together with the intelligent server and / or the service server. For example, the user terminal 1300 can execute an application corresponding to the received voice input, and perform the specified operation through the executed application.

[0207] According to an embodiment, when the user terminal 1300 provides a service together with the intelligent server 1400 and / or the service server, the user terminal can detect the user's voice using the microphone 1320 and generate a signal (or voice data) corresponding to the detected user's voice. The user terminal can send the voice data to the intelligent server 1400 by using the communication interface 1310.

[0208] According to an embodiment, in response to a voice input received from a user terminal 1300, the intelligent server 1400 may generate a plan for performing a task corresponding to the voice input or a result of performing an operation according to the plan. For example, a plan may include multiple operations for performing a task corresponding to the user's voice input and multiple concepts related to the multiple operations. A concept may define parameters to be input to perform multiple operations or result values ​​output by performing multiple operations. A plan may include association information between multiple operations and multiple concepts.

[0209] The user terminal 1300 according to the embodiment may receive a response using the communication interface 1310. The user terminal 1300 may output a voice signal generated within the user terminal 1300 to the outside using the speaker 1330 or may output an image generated within the user terminal 1300 to the outside using the display 1340.

[0210] Fig.14 is a diagram illustrating a scheme for storing relationship information between concepts and actions in a database according to an embodiment.

[0211] Intelligent servers (e.g. Fig.13 The capsule database of the intelligent server 1400 (e.g., Fig.13 The capsule database 1430 of the embodiment may store a plurality of capsules in the form of a concept-action network (CAN) 1600. The capsule database may store operations for processing a task corresponding to a user's voice input and parameters required for the operations in the form of a concept-action network (CAN). The CAN may represent an organic relationship between an action and a concept defining parameters required to perform the action.

[0212] The capsule database may store a plurality of capsules (e.g., capsule A 1601, capsule B 1604) corresponding to each of a plurality of fields (e.g., applications). According to an embodiment, one capsule (e.g., capsule A 1601) may correspond to one field (e.g., application). In addition, one capsule may correspond to at least one service provider (e.g., CP1 1602, CP2 1603, CP3 1606, or CP4 1605) to perform a function of a field related to the capsule. According to an embodiment, one capsule may include at least one operation 1610 and at least one concept 1620 for performing a specified function.

[0213] According to an embodiment, a natural language platform (e.g., Fig.13 The natural language platform 1420 of the present invention can generate a plan for performing a task corresponding to the received speech input using capsules stored in the capsule database. For example, a planner module of the natural language platform (e.g., Fig.13 The planner module 1425 in the embodiment of the present invention can generate a plan by using capsules stored in the capsule database. For example, the plan 1607 can be generated by using the operations 1711 and 1713 and the concepts 1712 and 1714 of capsule A 1601 and the operations 1741 and 1742 of capsule B 1604.

[0214] Fig.15 is a diagram illustrating a user terminal displaying a screen to process a voice input received through a smart application according to an embodiment.

[0215] The user terminal 1300 may execute a smart application for processing a user input through a smart server (eg, the smart server 1400 of FIG. 16 ).

[0216] According to an embodiment, in screen 1513, the user terminal 1300 may execute a smart application for processing a voice input when a specified voice input (e.g., wake up!) is recognized, or an input is received through a hardware key (e.g., a dedicated hardware key). For example, the user terminal 1300 may execute a smart application in a state where a scheduling application is executed. According to an embodiment, the user terminal 1300 may display a screen (e.g., Fig.13 An object (e.g., icon) 1514 corresponding to the smart application is displayed on the display 1340 of the user terminal 1300. According to an embodiment, the user terminal 1300 may receive a voice input by a user uttering a voice. For example, the user terminal 1300 may receive a voice input "Tell me about this week's schedule!". According to an embodiment, the user terminal 1300 may display a user interface (UI) 1513 (e.g., an input window) of the smart application, wherein text data of the received voice input is displayed on the display.

[0217] According to an embodiment, the user terminal 1300 may display a result corresponding to the received voice input on the display on screen 1520. For example, the user terminal 1300 may receive a plan corresponding to the received user input and display "this week's schedule" on the display according to the plan.

[0218] In an embodiment where an electronic device recognizes a preset speech for executing multiple functions (e.g., a quick command), a method for executing multiple functions more quickly may be required. As described above, according to an embodiment, an electronic device (e.g., Figure 3 The electronic device 101 may include a communication circuit (eg, Figure 3 communication circuit 330), a microphone (e.g., Figure 3 microphone 320), a speaker (e.g., Figure 3 A speaker 310) and a processor (e.g., Figure 3 The processor 120 may be configured to recognize a first audio signal received through a microphone. Based on recognizing speech (e.g., Figure 2 Based on the recognized speech, the processor can be configured to perform at least one first function among the plurality of functions by using an external electronic device (e.g., Figure 3The processor may be configured to sequentially output, through the speaker, a second audio signal representing a result of executing at least one first function and at least one second function according to an order associated with the speech. According to an embodiment, based on the speech for executing the multiple functions, the electronic device may execute the multiple functions more quickly by using the electronic device and the external electronic device.

[0219] For example, the processor may be configured to obtain a first speech engine (eg, Figure 3 A first voice engine 370) and a second voice engine executed by an external electronic device (eg, Figure 3 The processor may be configured to divide the plurality of functions into at least one first function executed by the first voice engine and at least one second function executed by the second voice engine based on the state.

[0220] For example, the processor may be configured to divide each of the plurality of functions into at least one first function and at least one second function based on whether each of the first voice engine or the second voice engine is capable of executing the function indicated by the status.

[0221] For example, the processor may be configured to assign at least one first voice command corresponding to at least one first function to a first queue corresponding to a first voice engine (eg, Figure 7 The processor may be configured to assign at least one second voice command corresponding to at least one second function to a second queue corresponding to a second voice engine (e.g., Figure 7 of the second queue 720).

[0222] The processor may be configured to execute the first voice engine based on at least one first voice command among the voice commands accumulated in the first queue.

[0223] The processor may be configured to transmit at least one signal of at least one second voice command among the voice commands accumulated in the second queue to the external electronic device.

[0224] For example, the electronic device may further include a memory (eg, Figure 3 The processor may be configured to store information including the results arranged according to the sequence in the memory.

[0225] For example, the information may include a queue in which the utterances included in the second audio signal are arranged according to a sequence.

[0226] For example, the electronic device may also include a display (e.g., Figure 3The processor may be configured to display different visual objects (eg, visual object 882, visual object 884, visual object 886, visual object 888 of FIG. 8) including the result in parallel in the display.

[0227] As described above, according to an embodiment, the method of the electronic device may include identifying (eg, Fig.12 Operation 1210) includes speech in the first audio signal. The method may include obtaining (for example, based on recognizing multiple voice commands from the speech) Fig.12 The method may include obtaining (for example, based on the state) a first voice engine executed by the processor to process the voice command and a second voice engine executed by an external electronic device connected through the communication circuit of the electronic device. Fig.12 Operation 1230) is used to divide the plurality of voice commands into a first voice command corresponding to the first voice engine and a second voice command corresponding to the second voice engine. The method may include, based on the information, performing (for example, Fig.12 Operation 1240) corresponds to the first function of the first voice command, and requests the external electronic device (eg, Fig.12 The method may include outputting (for example, outputting, through a speaker of the electronic device, a plurality of voice commands indicated by the speech) a second function corresponding to the second voice command in operation 1250. Fig.12 Operation 1260) represents a second audio signal of a result of performing the first function and the second function.

[0228] For example, recognizing the speech may include: requesting text data corresponding to the first audio signal from the external electronic device based on receiving the first audio signal. Recognizing the speech may include: recognizing the speech based on the text data received from the external electronic device.

[0229] For example, obtaining the state may include: identifying, in a memory of the electronic device, a preset speech for triggering execution of multiple voice commands based on a sequence. Obtaining the state may include: based on whether the speech included in the first audio signal corresponds to the preset speech, identifying multiple voice commands that match the speech.

[0230] For example, the request may include: after requesting the external electronic device to execute the second function, receiving information including a result of executing the second function from the external electronic device. The request may include storing the information received from the external electronic device in the memory based on the order.

[0231] For example, obtaining the information may include obtaining the information based on whether each of the first voice engine or the second voice engine is capable of executing the plurality of voice commands based on the status.

[0232] For example, the outputting may include displaying visual objects corresponding to results of performing the first function and the second function, respectively, in a display of the electronic device.

[0233] As described above, according to an embodiment, a method of an electronic device may include identifying (e.g., Fig.10 The method may include: based on recognizing speech for sequentially executing multiple functions from the first audio signal, executing (for example, Fig.10 The method may include: based on the recognition speech, executing (for example, Fig.10 The method may include: sequentially outputting (for example, Fig.10 Operation 1050) generates a second audio signal representing a result of performing the at least one first function and the at least one second function.

[0234] For example, the method may include: obtaining the status of a first voice engine executed by a processor and a second voice engine executed by an external electronic device. The method may include: dividing a plurality of functions into at least one first function executed by the first voice engine and at least one second function executed by the second voice engine based on the status.

[0235] For example, the dividing may include dividing each of the plurality of functions into at least one first function and at least one second function based on whether each of the first voice engine or the second voice engine is capable of executing the function indicated by the status.

[0236] For example, the division may include: assigning at least one first voice command corresponding to at least one first function to a first queue corresponding to a first voice engine. The division may include: assigning at least one second voice command corresponding to at least one second function to a second queue corresponding to a second voice engine.

[0237] For example, the outputting may include: outputting a second audio signal based on the representation result and according to the speech queue arranged in the order.

[0238] As described above, according to an embodiment, an electronic device (eg, Figure 3 The electronic device 101 may include a communication circuit (eg, Figure 3communication circuit 330), a microphone (e.g., Figure 3 microphone 320), a speaker (e.g., Figure 3 A speaker 310) and a processor (e.g., Figure 3 The processor 120 may be configured to recognize speech (eg, Figure 2 The processor may be configured to recognize a plurality of voice commands from the speech (e.g., Figure 2 230 and 240), to obtain a first speech engine (eg, Figure 3 The state of the first speech engine 370) and the external electronic device connected through the communication circuit (for example, Figure 3 The external electronic device 210) executes a second speech engine (eg, Figure 3 The processor may be configured to obtain information for dividing the plurality of voice commands into a first voice command corresponding to the first voice engine and a second voice command corresponding to the second voice engine based on the status. The processor may be configured to execute a first function corresponding to the first voice command by using the first voice engine, and to request the external electronic device to execute a second function corresponding to the second voice command based on the information. The processor may be configured to output a second audio signal representing the result of executing the first function and the second function through a speaker based on the order of the plurality of voice commands indicated by the speech.

[0239] For example, the electronic device may further include a memory (eg, Figure 3 The processor may be configured to identify, in the memory, a preset speech for triggering execution of a plurality of voice commands based on a sequence. The processor may be configured to identify a plurality of voice commands matching the speech based on whether the speech included in the first audio signal corresponds to the preset speech.

[0240] For example, the processor may be configured to, after requesting the external electronic device to execute the second function, receive information including a result of executing the second function from the external electronic device. The processor may be configured to store the information received from the external electronic device in the memory based on the order.

[0241] For example, the preset speech may include another word different from the one or more words associated with the plurality of voice commands.

[0242] For example, the processor may be configured to obtain information based on whether each of the first voice engine or the second voice engine is capable of executing a plurality of voice commands based on a state.

[0243] For example, the processor may be configured to request text data corresponding to the first audio signal from the external electronic device in response to receiving the first audio signal.The processor may be configured to recognize speech based on the text data received from the external electronic device.

[0244] For example, the electronic device may further include a display. The processor may be configured to display visual objects corresponding to results of executing the first function and the second function, respectively, in the display using the second audio signal output through the speaker.

[0245] For example, the processor may be configured to obtain information by matching each of a plurality of voice commands with one of the first voice engine or the second voice engine based on a delay time correspondence state between the first voice engine and the second voice engine.

[0246] For example, the processor may be configured to obtain information indicating a schedule for executing a plurality of voice commands by using at least one of the first voice engine or the second voice engine based on the state.

[0247] As described above, according to an embodiment, an electronic device may include a communication circuit, a microphone, a speaker, a memory storing at least one instruction, and at least one processor. The at least one processor may be configured to execute at least one instruction to identify a first audio signal received through a microphone. The at least one processor may be configured to perform at least one first function among multiple functions based on recognizing that the first audio signal may include speech for sequentially performing multiple functions. The at least one processor may be configured to perform at least one second function among multiple functions on an external electronic device through a communication circuit based on recognizing that the first audio signal may include speech. The at least one second function may be different from the at least one first function. The at least one processor may be configured to sequentially output at least one second audio signal representing at least one result of performing at least one first function and at least one second function through a speaker based on an order associated with the speech.

[0248] For example, at least one processor may be configured to execute at least one instruction to obtain a state of a first voice engine executed on the electronic device and a state of a second voice engine executed on the external electronic device. At least one processor may be configured to divide a plurality of functions into at least one first function and at least one second function based on the state of the first voice engine and the state of the second voice engine, the at least one first function being executed by the first voice engine, and the at least one second function being executed by the second voice engine.

[0249] For example, the state of the first voice engine may indicate whether the first voice engine is capable of performing one of a plurality of functions, and the state of the second voice engine may indicate whether the second voice engine is capable of performing one of a plurality of functions.

[0250] For example, at least one processor may be configured to execute at least one instruction to assign at least one first voice command corresponding to at least one first function to a first queue corresponding to a first voice engine. At least one processor may be configured to assign at least one second voice command corresponding to at least one second function to a second queue corresponding to a second voice engine.

[0251] For example, at least one processor may be configured to execute at least one instruction based on at least one first voice command accumulated in the first queue to execute the first voice engine.

[0252] For example, the at least one processor may be configured to execute at least one instruction to transmit at least one signal indicating that at least one second voice command is accumulated in the second queue to the external electronic device.

[0253] For example, at least one processor may be configured to execute at least one instruction to store information including at least one result in a memory based on an order.

[0254] For example, the information may include a queue that arranges at least one speech in the at least one second audio signal based on an order.

[0255] For example, the electronic device may include a display. The at least one processor may be configured to execute at least one instruction to control the display to display at least one different visual object showing at least one result in parallel.

[0256] As described above, according to an embodiment, a method of an electronic device may include recognizing speech in a first audio signal based on receiving a first audio signal through a microphone of the electronic device. The method may include obtaining a state of a first voice engine executed on the electronic device and a state of a second voice engine executed on an external electronic device based on recognizing multiple voice commands from the speech, and the external electronic device is connected through a communication circuit of the electronic device. The method may include obtaining information for dividing multiple voice commands into at least one first voice command executed by the first voice engine and at least one second voice command executed by the second voice engine based on the state of the first voice engine and the state of the second voice engine. The method may include executing a first function corresponding to at least one first voice command using the first voice engine, and requesting the external electronic device to execute a second function corresponding to at least one second voice command. The method may include outputting at least one second audio signal through a speaker of the electronic device based on the order of multiple voice commands indicated by the speech, and the at least one second audio signal represents at least one result of executing the first function and the second function.

[0257] For example, recognizing speech in the first audio signal may include receiving text data corresponding to the first audio signal from an external electronic device. Recognizing speech in the first audio signal may include recognizing speech in the first audio signal based on the text data.

[0258] For example, obtaining the state of the first voice engine and the state of the second voice engine may include identifying preset speech in the memory of the electronic device for triggering execution of multiple voice commands based on a sequence. Obtaining the state of the first voice engine and the state of the second voice engine may include identifying multiple voice commands based on whether the speech in the first audio signal corresponds to the preset speech.

[0259] For example, requesting the external electronic device to perform the second function may include receiving information including a result of performing the second function from the external electronic device. Requesting the external electronic device to perform the second function may include storing the information in a memory of the electronic device based on the order.

[0260] For example, the state of the first voice engine may indicate whether the first voice engine is capable of performing one of a plurality of functions, and the state of the second voice engine may indicate whether the second voice engine is capable of performing one of a plurality of functions.

[0261] For example, outputting the at least one second audio signal may include controlling a display of the electronic device to display at least one visual object showing a result of performing the first function and a result of performing the second function.

[0262] As described above, according to an embodiment, a method of an electronic device may include identifying a first audio signal received through a microphone of the electronic device. The method may include, based on identifying that the first audio signal may include speech for sequentially executing the multiple functions, executing at least one first function among the multiple functions on an external electronic device connected through a communication circuit of the electronic device based on identifying that the first audio signal may include speech. The method may include sequentially outputting at least one second audio signal representing at least one result of executing at least one first function and at least one second function through a speaker of the electronic device based on a sequence associated with the speech.

[0263] For example, the method may include obtaining a state of a first voice engine executed on the electronic device and a state of a second voice engine executed on the external electronic device. The method may include, based on the state of the first voice engine and the state of the second voice engine, dividing a plurality of functions into at least one first function executed by the first voice engine and at least one second function executed by the second voice engine.

[0264] For example, the state of the first voice engine may indicate whether the first voice engine is capable of performing one of a plurality of functions, and the state of the second voice engine may indicate whether the second voice engine is capable of performing one of a plurality of functions.

[0265] For example, the partitioning may include assigning at least one first voice command corresponding to at least one first function to a first queue corresponding to a first voice engine. The partitioning may include assigning at least one second voice command corresponding to at least one second function to a second queue corresponding to a second voice engine.

[0266] For example, the outputting may include outputting at least one speech representing at least one result according to a sequence based on the queue of at least one voice in the at least one second audio signal.

[0267] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those described above.

[0268] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalent forms or alternative forms for the corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include any one or all possible combinations of the items listed together with the corresponding one of the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish the corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “coupled with another element (e.g., the second element)”, “coupled to another element (e.g., the second element)”, “connected with another element (e.g., the second element)”, or “connected to another element (e.g., the second element)” with or without the terms “operably” or “communicatively” being used, it means that the element may be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0269] As used in connection with various embodiments of the present disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0270] The various embodiments described herein may be implemented as software (e.g., program #40) including one or more instructions stored in a storage medium (e.g., internal memory #36 or external memory #38) that can be read by a machine (e.g., electronic device #01). For example, under the control of a processor, a processor (e.g., processor #20) of the machine (e.g., electronic device #01) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" only means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but the term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0271] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be released in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be downloaded via an application store (e.g., Play Store TM ) the computer program product may be published (e.g., downloaded or uploaded) online, or the computer program product may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, a server of an application store, or a forwarding server).

[0272] According to various embodiments, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities, and some of the multiple entities may be separately arranged in different components. According to various embodiments, one or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., a module or a program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as a corresponding one of the multiple components performing one or more functions before integration. According to various embodiments, the operations performed by a module, a program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more of the operations may be run or omitted in a different order, or one or more other operations may be added.

[0273] The above-mentioned device can be implemented as a combination of hardware components, software components, and / or hardware components and software components. For example, the devices and components described in the embodiments can be implemented using one or more general-purpose computers or special-purpose computers, such as processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable gate arrays (FPGAs), PLUs (programmable logic units), microprocessors, and any other devices capable of executing and responding to instructions. The processing device can execute an operating system OS and one or more software applications executed on the operating system. In addition, the processing device can access, store, operate, process, and generate data in response to software execution. For ease of understanding, although a processing device can be described as being in use, it can be understood by those skilled in the art that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors, or a processor and a controller. In addition, other processing configurations such as parallel processors are also possible.

[0274] Software may include computer programs, codes, instructions, or a combination of one or more of them, and configures a processing device to operate as desired, or commands a processing device independently or in combination. Software and / or data may be built into any type of machine, component, physical device, computer storage medium, or a device that is interpreted by a processing device or provides instructions or data to a processing device. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

[0275] The method according to the present embodiment can be implemented in the form of program instructions, which can be executed by various computer devices and recorded in a computer-readable medium. In this case, the medium can store computer executable programs continuously or temporarily store programs for execution or downloading. In addition, the medium can be a variety of recording devices or storage devices in which a single or several hardware are combined, and is not limited to a medium directly connected to any computer system, and can be distributed on a network. Examples of media can include magnetic media such as hard disks, floppy disks, and tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floppy disks, ROMs, RAMs, flash memories, etc., to store program instructions. Examples of other media include application stores that distribute applications, websites that provide or distribute various software, and recording media or storage media managed by servers.

[0276] Although the embodiments are described based on the above limited embodiments and drawings, it is apparent to those skilled in the art that various modifications and changes may be made based on the above description. For example, even if the described techniques are performed in a different order than the described methods, and / or components such as the described systems, structures, devices, circuits, etc. are combined or combined in a different form than the described methods, or are replaced or replaced by other components or equivalents, appropriate results may be obtained.

[0277] Accordingly, other implementations, other embodiments, and equivalents of the claims are within the scope of the claims.

Claims

1. An electronic device, comprising: Communication circuits; microphone; speaker; A memory storing at least one instruction; as well as at least one processor configured to execute the at least one instruction to: identifying a first audio signal received via the microphone, performing at least one first function of the plurality of functions based on recognizing that the first audio signal includes speech for sequentially performing a plurality of functions; executing at least one second function of the plurality of functions on the external electronic device through the communication circuit based on recognizing that the first audio signal includes the speech, the at least one second function being different from the at least one first function; as well as At least one second audio signal is sequentially outputted through the speaker based on an order associated with the speech, the at least one second audio signal representing at least one result of performing the at least one first function and the at least one second function.

2. The electronic device according to claim 1, wherein: The at least one processor is further configured to execute the at least one instruction to: Obtaining a state of a first speech engine executed on the electronic device and a state of a second speech engine executed on the external electronic device; as well as Based on a state of the first voice engine and a state of the second voice engine, the plurality of functions are divided into the at least one first function executed by the first voice engine and the at least one second function executed by the second voice engine.

3. The electronic device according to claim 2, wherein: The state of the first voice engine indicates whether the first voice engine is capable of performing one of the plurality of functions, and the state of the second voice engine indicates whether the second voice engine is capable of performing one of the plurality of functions.

4. The electronic device according to claim 2, wherein: The at least one processor is further configured to execute the at least one instruction to: assigning at least one first voice command corresponding to the at least one first function to a first queue corresponding to the first voice engine; as well as At least one second voice command corresponding to the at least one second function is assigned to a second queue corresponding to the second voice engine.

5. The electronic device according to claim 4, wherein: The at least one processor is further configured to execute the at least one instruction to: The first voice engine is executed based on the at least one first voice command accumulated in the first queue.

6. The electronic device according to claim 4, wherein: The at least one processor is further configured to execute the at least one instruction to: At least one signal indicating that the at least one second voice command is accumulated in the second queue is sent to the external electronic device.

7. The electronic device according to claim 1, wherein: The at least one processor is further configured to execute the at least one instruction to: Based on the order, information including the at least one result is stored in the memory.

8. The electronic device according to claim 7, wherein: The information includes arranging a queue of at least one speech in the at least one second audio signal based on the order.

9. The electronic device according to claim 1, further comprising: monitor, Wherein, the at least one processor is further configured to execute the at least one instruction to: The display is controlled to display in parallel at least one different visual object showing the at least one result.

10. A method of an electronic device, the method comprising: Based on receiving a first audio signal through a microphone of the electronic device, recognizing speech in the first audio signal; obtaining a state of a first speech engine executed on the electronic device and a state of a second speech engine executed on an external electronic device connected via a communication circuit of the electronic device based on recognizing a plurality of speech commands from the speech; Based on a state of the first voice engine and a state of the second voice engine, obtaining information for dividing the plurality of voice commands into at least one first voice command to be executed by the first voice engine and at least one second voice command to be executed by the second voice engine; using the first voice engine to execute a first function corresponding to the at least one first voice command, and requesting the external electronic device to execute a second function corresponding to the at least one second voice command; as well as At least one second audio signal is output through a speaker of the electronic device based on the order of the plurality of voice commands indicated by the speech, the at least one second audio signal indicating at least one result of executing the first function and the second function.

11. The method according to claim 10, wherein: Recognizing the speech in the first audio signal includes: receiving text data corresponding to the first audio signal from the external electronic device; and Based on the text data, the speech in the first audio signal is recognized.

12. The method according to claim 10, wherein: Obtaining the state of the first voice engine and the state of the second voice engine includes: identifying, in a memory of the electronic device, preset words for triggering execution of the plurality of voice commands based on the sequence; and The plurality of voice commands are identified based on whether the speech in the first audio signal corresponds to the preset speech.

13. The method according to claim 12, wherein: Requesting the external electronic device to execute the second function includes: receiving information including a result of performing the second function from the external electronic device; and Based on the sequence, the information is stored in a memory of the electronic device.

14. The method according to claim 10, wherein: The state of the first voice engine indicates whether the first voice engine is capable of performing one of a plurality of functions, and the state of the second voice engine indicates whether the second voice engine is capable of performing one of the plurality of functions.

15. The method according to claim 10, wherein: Outputting the at least one second audio signal comprises: A display of the electronic device is controlled to display at least one visual object showing a result of executing the first function and a result of executing the second function.

Images