Electronic device for executing function corresponding to command by using caching and method thereof

Abstract

An electronic device according to an embodiment may detect a user input including a first command expressed in a natural language. The electronic device may determine whether the first command corresponds to a second command that was executed prior to the first command. On the basis of determining that the first command corresponds to the second command, the electronic device may output, as a response to the user input, information generated in a memory of the electronic device by executing the second command. The electronic device may execute one or more functions related to the first command on the basis of determining that the second command different from the second command is detected.

Description

Electronic device and method for executing a function corresponding to a command using caching

[0001] The present disclosure relates to an electronic device and method for executing a function corresponding to a command using caching.

[0002] Natural language (or ordinary language) refers to the language used in human daily life. Electronic devices that analyze the intent contained in natural language are being developed. For example, electronic devices can perform functions indicated by natural language detected from audio signals containing utterances and / or text entered via a keyboard (or software keyboard).

[0003] The above information may be provided as background information to aid in understanding the present disclosure. None of the above is claimed to be prior art related to the present disclosure or can be used in making decisions related to prior art.

[0004] According to an embodiment, an electronic device may include at least one processor including a memory and a processing circuit, the memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to detect a user input including a first command expressed in natural language. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine whether the first command corresponds to a second command that was executed prior to the first command. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to output information generated in a memory of the electronic device by executing the second command based on determining that the first command corresponds to the second command, in response to the user input. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute one or more functions associated with the first command based on a determination that a first command different from the second command has been detected.

[0005] According to one embodiment, a method of an electronic device may include detecting a user input including a first command expressed in natural language. The method may include determining whether the first command corresponds to a second command that was executed prior to the first command. The method may include outputting information generated in a memory of the electronic device by executing the second command in response to the user input, based on determining that the first command corresponds to the second command. The method may include executing one or more functions related to the first command based on determining that a first command different from the second command has been detected.

[0006] In one embodiment, a non-transitory computer-readable storage medium storing instructions may be provided. The instructions, when executed by a processor of an electronic device including communication circuitry, may cause the electronic device to detect a user input comprising a first command expressed in natural language. The instructions, when executed by the processor, may cause the electronic device to identify, using an external electronic device connected via the communication circuitry, one or more functions associated with the first command. The instructions, when executed by the processor, may cause the electronic device to determine, using metadata of the one or more functions, whether the one or more functions are executable by the electronic device or the external electronic device. The instructions, when executed by the processor, may cause the electronic device to store, in a memory of the electronic device, information indicating a result of executing at least one function, based on detecting at least one function executable by the electronic device from the one or more functions using the metadata.

[0007] In one embodiment, an electronic device may include a communication circuit, a memory for storing instructions, and a processor. The instructions, when executed by the processor of the electronic device, may cause the electronic device to detect a user input including a first command expressed in natural language. The instructions, when executed by the processor of the electronic device, may cause the electronic device to identify one or more functions associated with the first command using an external electronic device connected via the communication circuit. The instructions, when executed by the processor of the electronic device, may cause the electronic device to determine, using metadata of the one or more functions, whether the one or more functions are executable by the electronic device or the external electronic device. The instructions, when executed by a processor of the electronic device, may cause the electronic device to store, in a memory of the electronic device, information indicating a result of executing at least one function, based on detecting at least one function executable by the electronic device from the one or more functions using the metadata.

[0008] Figure 1 illustrates the operation of an electronic device that executes a command expressed in natural language.

[0009] Figures 2a, 2b, and 2c illustrate exemplary structures of a system for processing natural language.

[0010] FIG. 3 illustrates an exemplary operation of an electronic device according to one embodiment.

[0011] FIG. 4 illustrates an exemplary operation of a server that generates metadata for at least one function executable by an electronic device.

[0012] FIGS. 5A, 5B, and 5C illustrate exemplary operations of an electronic device that executes one or more functions corresponding to commands recognized from natural language.

[0013] FIG. 6 illustrates an exemplary operation of an electronic device that stores the result of executing one or more functions corresponding to a command recognized from natural language.

[0014] FIG. 7 illustrates an exemplary operation of an electronic device that executes one or more functions corresponding to commands recognized from natural language.

[0015] Figure 8 illustrates an exemplary operation of an electronic device for obtaining another command similar to a command recognized from natural language.

[0016] FIG. 9 illustrates an exemplary operation of an electronic device displaying a screen related to quick commands for executing multiple functions.

[0017] FIG. 10 is a block diagram of an electronic device within a network environment according to various embodiments.

[0018] FIG. 11 is a block diagram illustrating an integrated intelligence (AI) system according to one embodiment.

[0019] FIG. 12 is a diagram showing a form in which relationship information between concepts and actions is stored in a database according to one embodiment.

[0020] FIG. 13 is a diagram illustrating a user terminal displaying a screen for processing voice input received through an intelligent app, according to one embodiment.

[0021] Hereinafter, various embodiments of this document are described with reference to the attached drawings.

[0022] FIG. 1 illustrates the operation of an electronic device (101) that executes a command expressed in natural language. The electronic device (101) of FIG. 1 may be a terminal owned by a user. The terminal may include, for example, a personal computer (PC) such as a laptop or desktop, a smartphone, a smartpad, or a tablet PC. The terminal may include a smart accessory such as a smartwatch, a smart ring, and / or a head-mounted device (HMD). The terminal may be referred to as a mobile device, a user equipment (UE), a multi-function device, a portable communication device, and / or a portable device. The terminal may be included as an electronic control unit (ECU) in a vehicle (e.g., an electric vehicle (EV)). The terminal may have a form factor that is wearable by a user, or may have a form factor that is implantable on a body part of the user.

[0023] According to one embodiment, the electronic device (101) can interact with a user using the display (105), a speaker, and / or a microphone. For example, the electronic device (101) that receives a touch input of the display (105) can execute one or more functions related to the touch input. For example, through the microphone, the electronic device (101) can obtain an audio signal, which is an electrical signal dependent on sounds in an external space. The electronic device (101) can detect or identify an exemplary utterance (120) of the user (e.g., “Tell me the weather today.”) from the audio signal. The electronic device (101) that detects the utterance (120) can execute one or more functions related to the utterance (120).

[0024] The electronic device (101) can display the results of executing one or more functions on a display (105) or output them through a speaker. An exemplary hardware configuration of the electronic device (101) including a display (105), a speaker, and / or a microphone is described with reference to FIG. 2A.

[0025] In one embodiment, the electronic device (101) can detect or identify a command from an utterance (120) that triggers one or more functions associated with the electronic device (101). The command may include natural language spoken by a user or input by touch input to the display (105). The electronic device (101) may communicate with a server (110) to detect the command in natural language and / or to execute one or more functions corresponding to the command. In the present disclosure, a command may be a unit of a function and / or a task executed by the electronic device (101) and / or the server (110) connected to the electronic device (101). The function and / or the result of performing an operation performed by the electronic device (101) and / or the server (110) that detected the command may be included in information referred to as a capsule. For example, one or more functions executable by the electronic device (101) and / or the server (110) may be mapped to, or may be associated with, a single command.

[0026] Referring to FIG. 1, an electronic device (101) that detects an exemplary utterance (120) can detect a command included in the utterance (120). The electronic device (101) that detects a command corresponding to the utterance (120) can execute one or more functions related to the command. The electronic device (101) that detects a command for displaying weather information from the utterance (120) can obtain the weather information using a network. For example, the electronic device (101) can transmit a signal (130) indicating detection of the command to a server (110). The signal (130) can include information for requesting weather information. The embodiment is not limited thereto, and the signal (130) may include information related to the utterance (120) (e.g., an audio signal corresponding to the utterance (120), and / or text obtained from the utterance (120) using speech-to-text (STT).

[0027] Referring to FIG. 1, a server (110) that receives a signal (130) may transmit a signal (140) to an electronic device (101) as a response to the signal (130). The signal (140) may include information indicating a command indicated by the signal (130) and / or a result of executing one or more functions corresponding to the command. The electronic device (101) that receives the signal (140) may output an audio signal expressing a statement (150) (e.g., “It is sunny today. The lowest temperature is 3°C and the highest temperature is 12°C”) based on the information included in the signal (140). The embodiment is not limited thereto, and the electronic device (101) may display a visual object including at least a portion of the statement (150) on the display (105).

[0028] While the electronic device (101) is activated, natural language-based interactions between the electronic device (101) and the user may occur repeatedly. For example, after outputting a utterance (150) corresponding to a utterance (120) including a natural language sentence requesting weather information, the electronic device (101) may detect or identify a utterance (160) (e.g., "How's the weather?") that includes a natural language sentence similar to the natural language sentence included in the utterance (120). The electronic device (101) that detects the utterance (160) from the audio signal may execute at least one function related to the utterance (160).

[0029] Referring to exemplary utterances (120, 160) of FIG. 1, utterances (120, 160) may include different natural language sentences, but may have in common the intention of checking weather information. According to one embodiment, the electronic device (101) may detect similar commands for checking weather information from the utterances (120, 160). When commands that match or are similar to each other are detected, the electronic device (101) may provide a result corresponding to the command by utilizing the history of executing the command in the past. For example, the electronic device (101) may detect a user input that includes a first command expressed in natural language (e.g., a command for obtaining weather information expressed in the utterance (160). The electronic device (101) can determine whether the first command corresponds to a second command (e.g., a command included in the utterance (120)) that was executed prior to the first command.

[0030] Based on determining that the first command corresponds to the second command, the electronic device (101) may execute the second command to output information generated in the memory of the electronic device (101) as a response to a utterance (160) including the first command. For example, a result of executing a function corresponding to the command expressed by the utterance (120) (e.g., information included in the signal (140)) may be output as a response to the utterance (160). For example, the electronic device (101) may execute the second command to output information generated in the memory of the electronic device (101) as a response to a user input including the first command.

[0031] Referring to FIG. 1, an electronic device (101) that receives a utterance (160) similar to a utterance (120) may, instead of transmitting a signal related to a first command included in the utterance (160), provide a utterance (170) including a natural language sentence that was output in response to a second command included in the utterance (120) (e.g., “Today is sunny. The lowest temperature is 3°C and the highest temperature is 12°C.”). For example, the electronic device (101) may output an audio signal representing the utterance (170) through a speaker. For example, the electronic device (101) may display a visual object representing the utterance (170) on the display (105).

[0032] Based on determining that the first command, which is different from the second command, has been detected, the electronic device (101) may execute one or more functions related to the first command. The result of executing the one or more functions related to the first command may be stored (e.g., cached) at least temporarily in the electronic device (101). The result stored at least temporarily in the electronic device (101) may be used to respond to another command similar to the first command after executing the one or more functions.

[0033] As described above, according to one embodiment, the electronic device (101) may correspond to at least a portion of a system for executing one or more functions related to a command expressed in natural language. The electronic device (101) may communicate with the server (110) to obtain a result of executing one or more functions. The result may be stored in the electronic device (101) to respond to a command similar to the command. For example, the electronic device (101) that detects another command similar to the command may output the result stored in the electronic device (101) or provide it to the user, instead of communicating again with the server (110). In the above example, the electronic device (101) may reduce the amount of computation consumed by the server (110) for processing the other command, and may save network resources occupied for transmitting a signal related to the other command to the server (110). Within the above example, the electronic device (101) can reduce the delay that occurs while communicating with the server (110) and respond more quickly to a user who has spoken natural language expressing the other command.

[0034] Hereinafter, an exemplary hardware configuration of the electronic device (101) and / or server (110) of FIG. 1 is described with reference to FIG. 2a, FIG. 2b, and / or FIG. 2c.

[0035] FIGS. 2A, 2B, and 2C illustrate exemplary architectures of a system for processing natural language. Referring to FIGS. 2A to 2C, an exemplary system including the electronic device (101) and / or the server (110) of FIG. 1 is illustrated. While the electronic device (101) and / or the server (110) are illustrated as being directly connected, the embodiment is not limited thereto. For example, the electronic device (101) and the server (110) may be connected to each other via one or more routers and / or an access point (AP).

[0036] Referring to FIG. 2A, according to one embodiment, the server (110) may include at least one of a processor (210-1), a memory (215-1), and a communication circuit (220-1). The processor (210-1), the memory (215-1), and the communication circuit (220-1) may be electrically and / or operably coupled with each other by electronic components, such as a communication bus (202-1). Hereinafter, operably coupled electronic components may mean that a direct connection or an indirect connection is established between the electronic components, either wired or wireless, such that a second electronic component is controlled by a first electronic component among the electronic components. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the electronic components of FIG. 2A (e.g., at least a portion of the processor (210-1), the memory (215-1), and the communication circuit (220-1)) may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of electronic components included in the server (110) is not limited to that illustrated in FIG. 2A. For example, the server (110) may include only some of the electronic components illustrated in FIG. 2A. One or more clusters in which a plurality of electronic devices are grouped may be referred to as a server (110).

[0037] Referring to FIG. 2A, according to one embodiment, the electronic device (101) may include at least one of a processor (210-2), a memory (215-2), a communication circuit (220-2), a display (225), a microphone (226), and a speaker (227). Similar to the server (110), the processor (210-2), the memory (215-2), the communication circuit (220-2), the display (225), the microphone (226), and / or the speaker (227) may be electrically and / or operatively connected to each other by electronic components such as a communication bus (202-2). In the present disclosure, among the descriptions of the processor (210-2), the memory (215-2), and the communication circuit (220-2) of the electronic device (101), any descriptions that overlap with those of the processor (210-1), the memory (215-1), and the communication circuit (220-1) of the server (110) may be omitted.

[0038] In one embodiment, a processor (e.g., processors 210-1 and 210-2) may include circuitry (e.g., processing circuitry) for processing data based on one or more instructions. The circuitry for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), and / or an application processor (AP). For example, the number of processors may be one or more. The processing circuitry of the processor that loads (or fetches) instructions and performs calculations corresponding to the loaded instructions may be referred to as or referred to as a core circuit (or core). For example, the processor may have a multi-core processor architecture including a plurality of core circuits, such as a dual core, a quad core, a hexa core, or an octa core. The functions and / or operations described with reference to the present disclosure may be collectively performed by one or more processing circuits included in a processor.

[0039] In one embodiment, memory (e.g., memories (215-1, 215-2)) may include circuitry for storing data and / or instructions input to and / or output from a processor (e.g., processors (210-1, 210-2)). The memory may include, for example, volatile memory, such as random-access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, a hard disk, a compact disc, a solid state drive (SSD), and an embedded multi media card (eMMC). The processor (210-2) of the electronic device (101) can execute instructions of the memory (215-2) within the electronic device (101) to perform functions and / or operations indicated by the instructions.

[0040] In one embodiment, the communication circuit (e.g., the communication circuits 220-1, 220-2) may include hardware for supporting transmission and / or reception of electrical signals between the electronic device and an external electronic device. The communication circuit may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), near field communication (NFC), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), fifth generation (5G) new radio (NR), sixth generation (6G), and / or above-6G.

[0041] In one embodiment, the display (225) of the electronic device (101) can output visualized information to the user. The display (105) of FIG. 1 may be an example of the display (225) of FIG. 2A. For example, the display (225) may be configured to visualize information provided from a graphic processing unit (GPU) and / or a processor (210-2). The display (225) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). The display (225) may include a flat panel display (FPD) and / or electronic paper. The embodiment is not limited thereto, and the display (225) may have an at least partially curved shape or a deformable shape. A display (225) having a deformable shape may be referred to as a flexible display.

[0042] In one embodiment, the electronic device (101) may include a sensor (e.g., a touch sensor panel (TSP)) for detecting an external object (e.g., a user's finger) on the display (225). For example, using the TSP, the electronic device (101) may detect an external object that is in contact with the display (225) or floating on the display (225). In response to detecting the external object, the electronic device (101) may execute a function associated with a specific visual object among visual objects displayed within the display (225) that corresponds to a location of the external object on the display (225).

[0043] In one embodiment, the processor (210-2) of the electronic device (101) may obtain an electrical signal representing vibration of the atmosphere from the microphone (226). The processor (210-2) may transmit an audio signal to the speaker (227), thereby causing the speaker (227) to cause vibration of the atmosphere. The electronic device (101) may include one or more microphones and / or one or more speakers.

[0044] Referring to FIG. 2A, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (e.g., processors (210-1, 210-2)) may be stored in a memory (e.g., memories (215-1, 215-2)). A set of one or more instructions may be referred to as firmware, an operating system, a process, a routine, a sub-routine, a program, and / or a software application (hereinafter, an application). Hereinafter, installation of an application may mean that one or more instructions provided in the form of an application are stored in a memory, and that the one or more applications are stored in a format executable by the processor (e.g., a file having an extension designated by the operating system of the electronic device (101)).

[0045] Referring to FIG. 2A, one or more programs (e.g., a cloud executor (240), a capsule context service (242), a metadata provider (243), and / or an on-device list service (244)) installed in a server (110) and one or more programs (e.g., a request transmitter (230), an on-device executor (231), and / or an on-device capsule manager (232)) installed in an electronic device (101) are illustrated. According to one embodiment, the electronic device (101) can efficiently execute a natural language processing program (or natural language processing engine) (e.g., a cloud executor (240) and / or an on-device executor (231)) of the electronic device (101) and the server (110) that are connected to each other to process natural language by reusing the result of executing a recognized command by processing natural language. Because the electronic device (101) recycles the above results, the electronic device (101) can operate without requiring installation (or downloading) of additional software applications and / or information for natural language processing.

[0046] For example, the electronic device (101) can identify or detect characteristics (or dependencies) of functions (or events) corresponding to a command (e.g., prompt, time, location, device context, and / or on-device support) of the functions using a list of the functions. The electronic device (101) can use the characteristics to classify or obtain, among the functions, functions executable by the electronic device (101) without the server (110) and / or commands corresponding to the functions. The electronic device (101) can generate or obtain a command similar to the obtained command. If the command obtained by the electronic device (101) is identified from a natural language sentence uttered by the user, the electronic device (101) can execute one or more functions corresponding to the command without causing the cloud executor (240) of the server (110) to execute.

[0047] Referring to FIG. 2A, a cloud executor (240) may be installed in the server (110) to execute a function corresponding to a command (or an utterance including a natural language sentence) transmitted from an electronic device (101). The cloud executor (240) may be referred to as cloud AI (artificial intelligence). In order to support caching of the electronic device (101) (e.g., caching of the results of executing one or more functions related to the command), a capsule context service (242), a metadata provider (243), and / or an on-device list service (244) may be installed in the server (110). The processor (210-1) of the server (110) may execute at least one of the software applications described above, and may determine whether the result of executing functions corresponding to one or more commands detected by the electronic device (101) can be utilized by the on-device executor (231) of the electronic device (101).

[0048] In one embodiment, the result (e.g., a capsule) of executing functions corresponding to one or more commands may include at least one of information obtained or generated by executing the function, a deep link related to the function, and / or information output from a neural network (e.g., a large language model (LLM)) used for the function. The processor (210-1) of the server (110) may store capsules corresponding to functions executable by the cloud executor (240) in a capsule database (DB) (241).

[0049] In one embodiment, the processor (210-1) of the server (110) may execute a capsule context service (242) to obtain or generate information related to the capsule. The processor (210-1) executing the capsule context service (242) may modify or manage components of the capsule. For example, the processor (210-1) may execute the capsule context service (242) to obtain or generate another capsule corresponding to a specific function, which has been modified to be suitable for execution by the electronic device (101) among the server (110) or the electronic device (101), from a specific capsule.

[0050] As the electronic device (101) receives a command from a user, one or more capsules may be accumulated in the memory (215-2). The processor (210-2) may execute the on-device capsule manager (232) to classify or collect one or more commands executable by the electronic device (101). The one or more commands collected based on the execution of the on-device capsule manager (232) may be used to configure an AI engine personalized for the user of the electronic device (101).

[0051] The programs installed on the server (110) and / or the electronic device (101), as illustrated in FIG. 2A, may be configured to process natural language identified through the electronic device (101). The natural language may be included in an audio signal received through the microphone (226), or may be identified based on a combination of one or more touch inputs performed on the display (225). When the electronic device (101) detects natural language, the electronic device (101) may generate or transmit a request to process the natural language using the request transmitter (230).

[0052] Referring to FIG. 2B, programs executed in each of an electronic device (101) and a server (110) for processing natural language are illustrated. An electronic device (101) executing a request transmitter (230) may transmit a request for processing natural language to a server (110). To generate the request, the electronic device (101) may recognize natural language included in an audio signal by operating a model (e.g., an artificial intelligence model) referred to as an automatic speech recognition (ASR). When a text input such as a software keyboard is received, the electronic device (101) may detect or identify natural language included in the text input independently of the operation of the ASR. An operation of recognizing natural language included in an audio signal may include an operation of the electronic device (101) obtaining or generating text information (e.g., a string including one or more characters represented by a binary code such as Unicode) that is related to the audio signal and represents the natural language included in the audio signal.

[0053] The electronic device (101) that executes the request transmitter (230) may further execute the on-device executor (231) to perform a function and / or operation for processing natural language. For example, the electronic device (101) may invoke or call an intent object that causes the execution of the on-device executor (231) and / or an application programming interface (API) provided by the on-device executor (231) to perform a function and / or operation for processing natural language. While executing the request transmitter (230), the electronic device (101) may sequentially perform a first operation for executing the function of the on-device executor (231) and a second operation for transmitting the request to the server (110), or may perform them substantially simultaneously.

[0054] For example, the electronic device (101) may first perform the first operation. If all functions related to natural language are performed by performing the first operation, the electronic device (101) may not transmit a request for processing natural language to the server (110), or may refrain from doing so. After performing the first operation, if all functions related to natural language are not performed, or if a function supported by the server (110) among the electronic device (101) or the server (110) must be executed, the electronic device (101) may perform the second operation and transmit a request for processing natural language to the server (110).

[0055] In one embodiment, the electronic device (101) may support the execution of a quick command to collectively execute one or more commands. A quick command may be referred to as an abbreviation. Referring to FIG. 2B, instructions and / or sub-routines of a quick command processor (250) for identifying whether natural language included in a user input includes a quick command are illustrated. For example, the instructions and / or sub-routines of the quick command processor (250) may be divided into a comparator (251), an executor (252), and a quick command generator (253). Based on the execution of the quick command processor (250), the electronic device (101) may manage a quick command database (254) containing information about quick commands within the memory (215-2). In a state where a quick command is registered by a user, the electronic device (101) may store a natural language corresponding to the quick command, a language type (e.g., English), an identifier uniquely assigned to the quick command, a time at which the quick command is registered, and / or one or more functions to be executed by the quick command (e.g., when executing multiple functions, an order of the multiple functions, and / or a connection between the multiple functions) in a quick command database (254). The electronic device (101) may execute a quick command generator (253) to store the above-described information related to the quick command in the quick command database (254) in response to a user input for registering a quick command. The information stored in the quick command database (254) may be stored in an external electronic device connected to the electronic device (101) (e.g., a memory (215-1) of the server (110) of FIG. 2A).

[0056] Referring to FIG. 2B, the electronic device (101) may execute a comparator (251) to compare an utterance included in a user input with an utterance stored in a quick command database (254) and corresponding to a quick command. When an utterance corresponding to a quick command is detected, the electronic device (101) may execute one or more functions related to the quick command using an executor (252). When multiple functions are linked to a quick command, the electronic device (101) may execute the multiple functions in a chain. While executing one or more functions related to a quick command, the electronic device (101) may not perform additional operations for processing natural language (e.g., operations related to NLU, which will be described later).

[0057] If a utterance corresponding to a quick command is not included in the natural language included in the user input, the electronic device (101) may execute the on-device executor (231) to detect at least one command expressed by the natural language. The electronic device (101) may drive a model referred to as a natural language unit (NLU) to identify at least one command from the natural language. The electronic device (101) that has identified at least one command may execute one or more functions corresponding to the at least one command. The electronic device (101) may display the result of executing the one or more functions through the display (225) or output it through the speaker (227). In order to output the result through the speaker (227), the electronic device (101) may drive a model referred to as a text-to-speech (TTS) to generate an audio signal including one or more natural language sentences expressing the result. The electronic device (101) can transmit the generated audio signal to the speaker (227).

[0058] In one embodiment, functions executable using the on-device executor (231) may be included in a set of functions executable using the cloud executor (240). For example, an electronic device (101) that identifies a command that cannot be executed using the on-device executor (231) may transmit a signal related to the command (or a natural language including the command) to the server (110).

[0059] In order to recognize natural language included in user input, resources (e.g., one or more files) for driving the above-described models (e.g., ASR, NLU, and / or TTS) may be installed in the electronic device (101). Since the electronic device (101) includes a non-volatile memory of a finite size, the electronic device (101) may download the resource conditioned on a user input indicating installation of the resource. According to one embodiment, the electronic device (101) may provide the user with a result of executing one or more functions related to the natural language while reducing the size of the resource downloaded to the electronic device (101).

[0060] Referring to FIG. 2C, according to one embodiment, the electronic device (101) can at least temporarily store natural language recognized from a user input received in the past and / or the result of executing at least one command expressed by the natural language. The electronic device (101) can execute the on-device capsule manager (232) to determine the result stored in the electronic device (101) as the result of executing at least one command identified from the currently received user input. For example, the electronic device (101) that detects a user input including a utterance (120) by executing the request transmitter (230) can execute the on-device capsule manager (232) to confirm at least one command included in the utterance (120).

[0061] Referring to FIG. 2c, functions and / or sub-routines supported by the on-device capsule manager (232) are illustrated as different blocks (e.g., a command analyzer (260) and / or a command storage (270)). For example, an electronic device (101) that executes a command analyzer (260) can check a command included in a statement (120). For example, an electronic device (101) that executes a command storage (270) can store the result of executing at least one command expressed by the statement (120) and one or more functions corresponding to the at least one command in a memory (215-2) of the electronic device (101).

[0062] In one embodiment of FIG. 2C, the electronic device (101) that detects a utterance (120) from a user input can execute a command comparator (264) to check a command included in the utterance (120). For example, the electronic device (101) can search for a command included in the utterance (120) or similar to the command expressed by the utterance (120) among other commands prestored in the cache manager (271). If a command corresponding to the command expressed by the utterance (120) is not found among other commands cached by the cache manager (271), the electronic device (101) can search for a command that is the same as or similar to the command expressed by the utterance (120) within the command DB (272). The operation of comparing the command expressed by the utterance (120) with other commands using the command storage (270) including the cache manager (271) and the command DB (272) of the electronic device (101) may be performed using the Knuth Morris Part algorithm, the Rabin-Karp algorithm, and / or the Boyer-Moore algorithm.

[0063] When a command that is the same as or similar to the command expressed by the utterance (120) is retrieved from the command storage (270), the electronic device (101) may use the information (e.g., capsule) stored in the command storage (270) to execute one or more functions related to the command expressed by the utterance (120). For example, the electronic device (101) may use the information stored in the command storage (270) to perform an action related to the utterance (120) or provide information, instead of executing the on-device launcher (231) and / or the cloud launcher (240). In the example, the electronic device (101) may disconnect the communication link that was established between the server (110) and the electronic device (101). The communication link may be pre-established to respond to natural language of user input while receiving the utterance (120) or before receiving the utterance (120).

[0064] If a command that is the same as or similar to the command expressed by the utterance (120) is not retrieved from the command storage (270), the electronic device (101) may execute the on-device executor (231) and / or the cloud executor (240). The electronic device (101) may execute the on-device executor (231) together with the utterance (120), as described above with reference to FIG. 2B, and / or transmit information representing the utterance (120) to the cloud executor (240) of the server (110).

[0065] In one embodiment where information related to a utterance (120) is transmitted to a server (110), the electronic device (101) may receive information obtained by executing a cloud executor (240) of the server (110) from the server (110) (e.g., information indicating a result of executing at least one function related to a command expressed by the utterance (120). The electronic device (101) that has received the information may execute a metadata analyzer (262) to determine whether at least one function related to the utterance (120) is executable by the electronic device (101) or the server (110).

[0066] For example, the server (110) can execute a metadata provider (243) to generate metadata (e.g., policy verifier manifest data) for at least one function related to a command expressed by an utterance (120) from information provided by an on-device list service (244). The server (110) can transmit the generated metadata to the electronic device (101). By executing the on-device list service (244), the server (110) can generate or manage information indicating whether each of the functions, referred to as microservices, is executable using the electronic device (101) (or the on-device executor (231)). The electronic device (101) can execute a metadata analyzer (262) together with the metadata to determine whether at least one function related to the utterance (120) is executable solely by the electronic device (101).

[0067] The metadata received by the electronic device (101) from the server (110) may include conditions under which the result of executing the function corresponding to the metadata is valid (e.g., version, expiration time, and / or location where the result can be provided). The metadata may indicate conditions under which the result of executing the function by the cloud executor (240) can be reused without additional execution of the cloud executor (240). The operation of the server (110) executing the metadata provider (243) to create or update metadata is described with reference to FIG. 4.

[0068] The electronic device (101) may obtain similar commands related to the function by receiving metadata about the function executable by the electronic device (101) from the server (110). The electronic device (101) may obtain similar commands by executing the similar command detector (261). For example, the electronic device (101) that executes the similar command detector (261) may obtain information generated by the capsule context service (242) of the server (110). The information may include a command similar to a specific function and / or a command expressed by the utterance (120).

[0069] As described above, while the electronic device (101) responds to various utterances including the utterance (120), the results of executing the utterances may be accumulated in the command storage (270). The results stored in the command storage (270) may include the results of executing a capsule corresponding to one or more functions (e.g., a capsule event list). The electronic device (101) including a non-volatile memory and a volatile memory may execute a cache manager (271) to store (e.g., cache) a portion of the information (e.g., a capsule event list) of the command DB (272) stored in the non-volatile memory in the volatile memory. The electronic device (101) may determine whether to cache the information by using at least one of the expiration time, whether it is a favorite, and / or the last executed time of the information stored in the command DB (272).

[0070] In one embodiment, one or more commands executable by the electronic device (101), and the results of executing functions corresponding to the one or more commands, may be stored in the command DB (272). The command DB (272) may include a first table indicating similar commands and a time period during which information corresponding to the commands is valid. The first table may include a list of similar commands. Each record of the first table may store attributes (or columns) indicating a unique ID, a list of functions for executing a command corresponding to the record, a time period during which the command corresponding to the record is stored, and a period during which the command corresponding to the record is valid. A designated value (e.g., null) indicating an unlimited period during which the command corresponding to the record is valid may be recorded. The command DB (272) may be connected to the first table and may include a second table for each of the functions corresponding to the commands of the first table. An operation of an electronic device (101) using the above tables to execute one or more functions related to a command expressed by a statement (120) is described with reference to FIGS. 5a to 5c.

[0071] Hereinafter, with reference to FIG. 3, exemplary operations of the electronic device (101) (e.g., operations of processing one or more commands corresponding to a statement (120)) described with reference to FIGS. 2A to 2C are described.

[0072] FIG. 3 illustrates an exemplary operation of an electronic device according to one embodiment. The electronic device (101) of FIGS. 1, 2A to 2C and / or the processor (210-2) may perform the operation of the electronic device described with reference to FIG. 3.

[0073] Referring to FIG. 3, in operation (310), according to one embodiment, a processor of an electronic device may detect a user input including a first command. The processor may execute the request transmitter (230) of FIGS. 2A to 2C to detect natural language included in the user input. The processor may detect the first command expressed by the detected natural language.

[0074] Referring to FIG. 3, in operation (320), a processor of an electronic device according to an embodiment may determine whether a first command corresponds to a second command that was executed prior to the first command. The processor may execute a command comparator (264) of FIGS. 2A to 2C to search for a command that is the same as or similar to the first command within a list of commands stored in the electronic device. The list may be stored in a command storage (270) of FIGS. 2A to 2C. If a command that is the same as or similar to the first command is found within the list (320—Yes), the processor may perform operation (330). If no command that is the same as or similar to the first command is found from the list (320—No), the processor may perform operation (340).

[0075] Referring to FIG. 3, in operation (330), according to one embodiment, the processor of the electronic device may output information generated by executing a second command as a response to a user input. The processor, which searches for a second command that is the same as or similar to the first command, may output the information generated by executing the second command through a display (e.g., display (225) of FIG. 2A) and / or a speaker (e.g., speaker (227) of FIG. 2A) of the electronic device. The processor may transmit the first command to a server (e.g., server (110) of FIGS. 2A to 2C) or output the information of operation (330) without executing an AI engine (e.g., on-device executor (231) of FIGS. 2A to 2C). By performing operation (330), the processor may transmit the first command to the server or may save resources consumed for executing the AI ​​engine.

[0076] Referring to FIG. 3, in operation (340), according to one embodiment, a processor of an electronic device may execute one or more functions related to a first command using the electronic device and / or a server (e.g., server 110 of FIGS. 2A to 2C). For example, the processor may execute an AI engine of the electronic device (e.g., on-device executor 231 of FIGS. 2A to 2C) to execute one or more functions related to the first command. For example, the processor may transmit a signal including the first command to the server via a communication circuit (e.g., communication circuit 220-2 of FIG. 2A). For example, the processor may transmit a signal related to a user input to the server. After transmitting the signal, the processor may receive, from the server via the communication circuit, a result of executing the first command and / or a list of one or more functions related to the first command. The processor that receives the above results can output the results through a display and / or speaker. The processor that receives the list can execute one or more functions included in the list. The processor can output the results of executing the one or more functions through a display and / or speaker.

[0077] The processor may store, at least temporarily, the result of executing one or more functions of operation (340) in the memory of the electronic device (e.g., the memory (215-2) of FIGS. 2A to 2C and / or the command storage (270)). When the result of executing one or more functions is received from the server, the processor may store information related to the result in the memory. The result stored in the memory may be received after receiving the first command and may be used to execute another command that matches or is similar to the first command. Before storing the result of executing one or more functions of operation (340), the processor may determine whether to store the result using metadata corresponding to the one or more functions. Hereinafter, an exemplary operation in which the metadata is generated is described with reference to FIG. 4.

[0078] FIG. 4 illustrates exemplary operations of a server that generates metadata for at least one function executable by an electronic device (e.g., the electronic device (101) of FIGS. 1 and 2A to 2C). The server (110) of FIGS. 1 and 2A to 2C and / or the processor (210-1) may perform the exemplary operations described with reference to FIG. 4.

[0079] Referring to FIG. 4, in operation (420), according to one embodiment, a processor of a server may receive a request to generate metadata. The processor may receive or obtain the request by executing a metadata provider (243) or through a user interface (UI) provided by the metadata provider (243). The metadata provider (243) may be referred to as a Personal OnDevice AI Policy Service (POPS). The metadata provider (243) may be a process (e.g., a microservice) running on the server.

[0080] While the metadata provider (243) is running, the processor may perform operation (430). Within operation (430), the processor of the server may obtain or request information (e.g., capsule information) about functions executable by a user terminal (e.g., electronic device (101) of FIGS. 1, 2A to 2C) different from the server from the on-device list service (244). Within operation (440), the processor may obtain capsule information managed by the on-device list service (244).

[0081] In one embodiment, within operation (450), the processor of the server may obtain or request information about capsules convertible into executable on-device capsules on the user terminal by executing the capsule context service (242). The processor executing the capsule context service (242) may obtain or generate information about capsules executable on the user terminal (e.g., convertible capsule information) by using machine learning or by using a flag value indicating a capsule executable on the user terminal. Within operation (460), the processor may obtain information about capsules executable on the user terminal as information managed by the capsule context service (242). Operations (430, 450) of FIG. 4 may be performed in the reverse order illustrated in FIG. 4 or may be performed substantially simultaneously.

[0082] Within the operation (470) of FIG. 4, the processor may receive an input for changing a parameter of metadata obtained based on operations (440, 460). The processor may generate metadata that at least partially includes information of operations (440, 460). For example, the metadata may correspond to a capsule executable on a user terminal obtained by performing operation (460) and may include capsule information of operation (440). The processor may receive an input for changing a parameter of the metadata (e.g., at least a portion of information included in the metadata) through a UI that displays the generated metadata.

[0083] Within the operation (480) of FIG. 4, the processor may store metadata including parameters of the operation (470) in the metadata DB (410). The metadata DB (410) may be included in the capsule DB (241) of FIGS. 2A to 2C. Based on the operation (480), the processor of the server may generate metadata for at least one function related to the command. The generated metadata may be provided to a user terminal (e.g., the electronic device (101) of FIGS. 1 and 2A to 2C) connected to the server.

[0084] In one embodiment, metadata stored on the server based on the operation (480) may have a format of JSON (JavaScript Object Notation). Table 1 may include an exemplary portion of metadata stored on the server.

[0085]

[0086] Referring to Table 1, the JSON format shows the names of data separated by colons (:) and the contents of data having the names. The names of the data may be written to the left of the colon, and the contents of the data may be written to the right of the colon. To the right of the colon, using curly brackets, sub-data included in the data having the name to the left of the colon may be written. To the right of the colon, using square brackets, the array of data included in the data having the name to the left of the colon may be written.

[0087] Referring to Table 1, within the metadata, data with the name "mdata" may store the version of the metadata (e.g., data with the name "version" where "0.0.1-5" is stored) and the expiration date (data with the name "expiredTimestamp" where "2023-07-31T20:14:40.5732" is stored).

[0088] Referring to Table 1, in the data having the name "files", a list of files in which a list of functions executable on a specific type of user terminal is stored may be stored as another file linked to the metadata of Table 1. For example, as sub data included in the data having the name "files", sub data targeting a mobile device (e.g., a smartphone, a tablet PC, and / or a smartwatch) (e.g., data having the name "target" in which "mobile" is stored) may include the location of a file in which a list of functions (e.g., capsules) that can be independently executed on the mobile device is stored (e.g., data having the name "ondeviceSupportList" in which a file name of "mobile-ondevice-support-list.json" is stored), and the location of a file in which conditions for providing the functions stored (or cached) on the mobile device are stored (e.g., data having the name "capsuleSpecificVariation" in which a file name of "mobile-capsule-specific-variation.json" is stored). Similarly, as sub-data included in data having the name "files", sub-data targeting a television (e.g., a smart television) (e.g., data having the name "target" where "tv" is stored) may include the location of a file storing a list of functions (e.g., capsules) that can be independently executed on the television (e.g., data having the name "ondeviceSupportList" where the file name "tv-ondevice-support-list.json" is stored), and the location of a file storing conditions under which results of functions cached by the television are to be provided (e.g., data having the name "capsuleSpecificVariation" where the file name "tv-capsule-specific-variation.json" is stored).

[0089] In one embodiment, metadata stored on the server based on the operation (480) may include a file having an exemplary format of Table 1, along with other files linked to the metadata of Table 1 (e.g., other files referenced by data having a name of "files" in Table 1). For example, Table 2 may be stored on the server to represent a list of functions that can be independently executed on a user terminal, as an exemplary portion of the metadata stored on the server.

[0090]

[0091] Referring to Table 2, in data having an exemplary name such as "capsuleInfos", sub-data corresponding to each function executable on a specific type of user terminal may be stored in the format of an array. For example, in sub-data in which a variable having a name of "capsuleId" in which "viv.clockApp" is stored, a list of one or more functions corresponding to a command for registering an alarm may be stored together (e.g., data stored in a variable having a name of "capsuleIdGoalIds"). Referring to Table 2, in the list of one or more functions corresponding to a command for registering an alarm, a function for setting an alarm (e.g., data having a name of "setAlarm"), a function for setting an alarm sound (e.g., data having a name of "AlarmActionResult"), and a function for setting a condition for disabling an alarm (e.g., data having a name of "DisableAlarms") may be stored. Data having the name of "capsuleId" in Table 2 may be stored as a name assigned to a capsule provided from a server to a user terminal (e.g., information storing the result of executing functions related to a command).

[0092] In one embodiment, the metadata stored on the server based on the operation (480) may further include a condition for maintaining the results stored (or cached) on the user terminal, as in Table 3, along with the metadata of Tables 1 and 2.

[0093] {"capsuleInfos": [{"capsuleId": "contentsgate.dailyhoroscope","capsuleGoalIds": ["contentsgate.dailyhoroscope.dailyinfo"]"capsuleVersion": "2.49.3","dependencyMap": ["time": {"maintainable": "86400"}]},{"capsuleId": "agent.weatherInfoResolver_KR","capsuleGoalIds": ["agent.weatherInfoResolver_KR.resolveWeatherInfo"]"capsuleVersion": "1.1.1","dependencyMap": ["time": {"maintainable": "600"},"location": {"marginOfError": {"longtitude": "1.5","latitude": "1.5"}}]},...(omitted)...]}

[0094] Referring to Table 3, the condition (e.g., data with the name "dependencyMap") for storing the result of executing the function assigned to the command for providing today's horoscope (e.g., data including sub data with the name "capsuleId" in which data of "contentesgate.dailyhoroscope" is stored), the time for maintaining the result (e.g., data with the name "time" in which "86400", a numerical value in seconds representing 24 hours, is stored and sub data with the name "maintainable" is stored), and / or the version (e.g., data with the name "capsuleVersion" in which "2.49.3" is stored) may be stored.

[0095] Similarly, the condition (e.g., data with the name of "dependencyMap") for maintaining the result of executing the function assigned to the command for providing weather (e.g., data including sub data with the name of "capsuleId" where the data of "agent.weatherInfoResolver_KR" is stored) may store the time for maintaining the result (e.g., data with the name of "time" where "600" is stored and sub data with the name of "maintainable" is stored), and the location (e.g., data with the name of "location"). Referring to Table 3, in the data with the name of "location", as the condition for maintaining the result of executing the function in the user terminal, data indicating the margin of the location error (e.g., the difference between the location of the user terminal when the function is executed and the current location of the user terminal) according to latitude (e.g., data with the name of "latitude") and / or longitude (e.g., data with the name of "longtitude") may be stored.

[0096] Referring to Table 3, data named "dependencyMap" may indicate conditions under which the results of executing a function corresponding to the data are maintained or reusable by the user terminal. Table 4 may indicate exemplary conditions that may be stored in data named "dependencyMap."

[0097] "dependencyMap": {"time": {"required": / "false" or "true","maintainable": / integer representing seconds and / or hours},"prompt": {"destination": / "none" or "cloud" or "onDevice",},"location": {"required": / "false" or "true","marginOfError": {"allow": / "false" or "true","longtitude": "1.5","latitude": "1.5"}},"deviceContext": {"locationEneabled": / "false" or "true","requiredOsVersion": {"minimum": / minimum supported OS version,"maximum": / maximum supported OS version},"clientVersion": / supported OS version,"deviceModel": / supported user device model Name,...(omitted)...}}

[0098] Referring to Table 4, " / " can be understood as a comment, and as a description of the data of the line where the comment is written. The data of Table 4 can be understood as an example. The data named "prompt" in Table 4 can indicate whether additional data is required to execute a function corresponding to the data. The data named "time" in Table 4 can indicate whether the result corresponding to the data varies over time. The data named "time" in Table 4 can indicate whether the result corresponding to the data varies depending on the location of the user terminal. The data named "deviceContext" in Table 4 can indicate whether the result corresponding to the data varies depending on the state of the user terminal.

[0099] The metadata described with reference to Tables 1 to 4 may be provided to one or more user terminals (e.g., electronic devices (101) of FIGS. 1, 2A to 2C) connected to a server. The user terminal may use the metadata to determine whether to cache the result of executing a command detected from a user input and / or the conditions under which the result will be reused. Hereinafter, exemplary operations of an electronic device executing a function corresponding to the metadata are described with reference to FIGS. 5A to 5C.

[0100] FIGS. 5A, 5B, and 5C illustrate exemplary operations of an electronic device that executes one or more functions corresponding to commands recognized from natural language. The electronic device (101) and / or the processor (210-2) of FIGS. 1, 2A, and 2C may perform the exemplary operations described with reference to FIGS. 5A, 5B, and 5C.

[0101] Referring to FIG. 5A, in operation (501), according to one embodiment, an electronic device may receive a user input. For example, the electronic device may detect the user input of operation (501) using an audio signal received through a microphone (e.g., microphone 226 of FIG. 2A). The electronic device may receive the user input through a software keyboard (or a physical keyboard connected to the electronic device). The user input may include one or more natural language sentences. The electronic device that detects the user input of operation (501) may display a first visual object (e.g., a visual object referred to as a bubble) that includes a first command included in the user input on a display (e.g., display 225 of FIG. 2A).

[0102] In response to a user input, in operation (502), according to one embodiment, the electronic device may execute the on-device capsule manager (232) to request a capsule event corresponding to the user input. In operation (503), the electronic device executing the on-device capsule manager (232) may execute a command comparator (264) to verify a command included in the user input. By executing the command comparator (264), the electronic device may determine whether a command included in a natural language sentence of the user input is a command executable by the electronic device.

[0103] For example, the electronic device can determine similarities between a first command included in a user input and one or more second commands stored in a command comparator (264). By comparing the determined similarities with a specified threshold, the electronic device can determine whether each of the one or more second commands corresponds to the first command. When the electronic device determines a second command having a similarity exceeding the specified threshold, the electronic device can execute the on-device capsule manager (232) together with the determined second command by performing operation (504).

[0104] Referring to FIG. 5A, in operation (505), according to one embodiment, the electronic device may execute the command storage (270) or access the memory managed by the command storage (270) to obtain a result of executing the second command identified based on operation (504) (e.g., a second command similar to or corresponding to the first command included in the user input). The result may include a list (e.g., an event list) of results of executing one or more functions that were executed at the time the second command was detected. In operation (506), the electronic device may execute the command storage (270) to search for or obtain a result of executing one or more functions corresponding to the second command. The electronic device may execute the on-device capsule manager (232) using the obtained result.

[0105] Based on the execution of the on-device capsule manager (232), the electronic device may output or provide to the user the result of executing one or more functions within operation (507). Within the state of acquiring the result of operation (506), the electronic device may determine whether to output information including the result of operation (506) in response to the first command, using metadata corresponding to one or more functions related to the first command. The electronic device performing operation (507) may display a second visual object for expressing the information including the result on the display together with the first visual object related to the user input.

[0106] For example, the metadata may include data named "dependencyMap" as described with reference to Table 1 and / or Table 4. An electronic device that detects a condition indicated by the metadata may compare the detected condition with the result to determine whether to output the result. For example, the electronic device may determine whether to output the information by using the time at which the information including the result was stored and the period for which the information indicated by the metadata is maintained (e.g., data named "maintainable").

[0107] In the operation (504) of FIG. 5A, if no command similar to the first command included in the user input is identified using the command comparator (265), the electronic device may execute a natural language processing engine (e.g., the on-device executor (231) and / or the cloud executor (240) of FIG. 2A) running on the electronic device and / or a server to execute one or more functions related to the first command. Hereinafter, with reference to FIG. 5B, an exemplary operation of the electronic device executing the command comparator (264) to perform operations (503, 504) of FIG. 5A will be described.

[0108] Referring to FIG. 5B, within operation (511), the electronic device executing the command comparator (264) may obtain at least one command included in the user input. Within operation (512), the electronic device may execute the cache manager (271) to obtain a list of one or more commands cached in the memory of the electronic device (e.g., the memory (215-2) of FIG. 2A). Within operation (513), based on the execution of the cache manager (271), the electronic device may use the list of operation (512) to execute the command comparator (264).

[0109] Within the operation (514), the electronic device can compare the list obtained using the cache manager (271) and the command of the operation (511). The electronic device can compare the commands by performing the Knuth Morris Part algorithm, the Rabin-Karp algorithm, and / or the Boyer-Moore algorithm. If the electronic device obtains the command of the operation (511) within the list, or detects another command similar to the command of the operation (511), the electronic device can perform the operation (504) of FIG. 5A to provide the command obtained from the list and / or the result of executing one or more functions corresponding to the command.

[0110] In operation (514), if no command related to the command of operation (511) is found from the list obtained using the cache manager (271), the electronic device may perform operation (515). In operation (515), the electronic device may further request a list of commands stored in the memory of the electronic device using the cache manager (271). In operation (516), the electronic device that has executed the cache manager (271) may access the command DB (272) to obtain a list of commands stored in the non-volatile memory and / or storage of the electronic device. The electronic device that has obtained the list of operation (516) may perform operation (517) to cache commands corresponding to the list of operation (516) and the results of executing functions corresponding to the commands. Within the operation (518), the electronic device can execute the command comparator (264) using the list of operations (516) additionally obtained using the cache manager (271).

[0111] In one embodiment, information obtained from the command DB (272) based on the operation (516) may include exemplary properties of Table 5. The properties of Table 5 may include information related to functions to be executed on a server connected to the electronic device among functions corresponding to the command.

[0112] Attribute name Value Description ID Integer value Unique value of command DB (272) utternceList "Turn on flashlight", "Turn off flashlight", "Turn on flashlight", "Turn on flashlight" List of similar commands responseList [Message, RenderEvent, CapsuleExecutionStarting, CapsuleExecutionFinishing, Applaunch, ...] List of one or more functions corresponding to the command savedTime Timestamp Time when the command was saved in the command DB (272) expiredTime Timestamp Expired time of the command saved in the command DB (272)

[0113] For example, information corresponding to a command that triggers a function of an electronic device related to a flashlight, as illustrated in Table 5, may be stored in the command DB (272). If the value of the attribute having "expiredTime" in Table 5 is "null," information related to the function having the attribute may be permanently cached or stored in the command DB (272). Detailed information stored in the command DB (272) to execute the exemplary function of Table 5 may have the exemplary attributes of Table 6.

[0114]

[0115] Referring to Table 6, when executing one or more functions related to the flashlight, the electronic device may display exemplary text such as “Flashlight turned on” on the display.

[0116] In one embodiment, information corresponding to one or more functions associated with a command for providing a daily horoscope may have exemplary properties of Table 7 within the command DB (272).

[0117] Attribute name Value Description ID Integer value Unique value of command DB (272) utternceList "Today's fortune", "Tell me the fortune", "Fortune", "How is the fortune", "How is the fortune today" List of similar commands responseList [Message, RenderEvent, CapsuleExecutionStarting, CapsuleExecutionFinishing, Applaunch, ...] List of one or more functions corresponding to the command savedTime Timestamp Time when the command was saved in the command DB (272) expiredTime Timestamp Valid time of the command saved in the command DB (272)

[0118] Detailed information stored in the command DB (272) to execute the exemplary function of Table 7 may have the exemplary properties of Table 8.

[0119]

[0120] Using the operations described with reference to FIG. 5b, the electronic device can obtain information related to a command included in a user input and cached in the electronic device. Below, exemplary operations of the electronic device performed to manage a command DB (272) are described with reference to FIG. 5c. At least one of the operations of FIG. 5c may be performed by a process running in the background (e.g., the command storage (270) and / or the cache manager (271) of FIG. 2c).

[0121] Referring to FIG. 5C, in operation (521), according to an embodiment, a processor of an electronic device may obtain metadata related to information stored in a cache. The metadata of operation (521) may include metadata corresponding to functions stored in the electronic device, as described with reference to Tables 5 to 8. The embodiment is not limited thereto, and the metadata of operation (521) may include metadata corresponding to functions executable by the electronic device, as described with reference to Tables 1 to 4.

[0122] Referring to FIG. 5C, in operation (522), the processor of the electronic device according to one embodiment may determine whether an expiration date is detected from the metadata. For example, the electronic device may detect attributes having the names "expiredTime" and / or "savedTime" of Tables 5 and / or 7. If the processor does not detect an expiration date from the metadata, or detects a designated value indicating that the expiration date is not limited (e.g., a "null" value stored in the attribute having the name "expiredTime") (522-No), the processor may perform operation (525). If the processor detects an expiration date from the metadata (522-Yes), the processor may perform operation (523).

[0123] Referring to FIG. 5C , in operation (523), the processor of the electronic device according to one embodiment may determine whether the detected expiration date has expired. Using the properties named "expiredTime" and / or "savedTime" of Table 5 and / or Table 7, the processor may determine whether information stored in the electronic device has expired. For example, if the current time of the electronic device is a point in time after the time indicated by the property named "savedTime" and the time period indicated by the property named "expiredTime," the processor may determine that the information stored in the electronic device has expired. In response to the expiration of operation (523) (523-Yes), the processor may perform operation (524). Prior to the expiration of operation (523), the processor may perform operation (525).

[0124] Referring to FIG. 5c, within operation (524), according to one embodiment, the processor of the electronic device may remove information stored in the cache. By performing operation (524), the processor may remove the information of operation (524) from the memory of the electronic device (e.g., memory (215-2) of FIG. 2a).

[0125] Referring to FIG. 5c, within operation (525), according to one embodiment, the processor of the electronic device may maintain information within a cache. The information maintained by operation (525) may be maintained in the memory of the electronic device until an expiration point indicated by attributes included in the information.

[0126] As described above, according to one embodiment, if a first command detected from a user input is similar to or matches a second command that has been executed in the past, the electronic device may reuse the result of executing one or more functions corresponding to the second command. The result may be stored in the memory of the electronic device. The electronic device may use metadata corresponding to the result to determine whether to reuse the result and / or whether to remove the result from the memory of the electronic device. If the first command does not match the second command, the electronic device may directly (or indirectly using a server) execute one or more functions related to the first command. The result of executing the one or more functions may be stored in the memory of the electronic device to be output as a response to another command that is similar to or matches the first command. Hereinafter, with reference to FIG. 6, an exemplary operation of the electronic device that detects a command that is different from a command previously stored in the electronic device is described.

[0127] FIG. 6 illustrates exemplary operations of an electronic device that stores the results of executing one or more functions corresponding to commands recognized from natural language. The electronic device (101) of FIG. 1, FIG. 2A to FIG. 2C and / or the processor (210-2) can perform the exemplary operations described with reference to FIG. 6.

[0128] Referring to FIG. 6, in operation (610), a processor of an electronic device according to one embodiment may detect a user input that includes a first command that is different from at least one second command cached in the electronic device. Operation (610) may be performed in response to detecting a user input that includes a first command expressed in natural language. The processor may compare a second command associated with information cached in a memory (e.g., memory 215-2 of FIG. 2A) with the first command. By comparing the second command with the first command, the processor may determine whether to transmit a signal to an external electronic device (e.g., server 110 of FIGS. 1, 2A-2C) connected to the electronic device to identify one or more functions associated with the first command. For example, if the processor detects a first command that is different from at least one second command cached in the electronic device, the processor may transmit the signal or perform operation (620).

[0129] Referring to FIG. 6, in operation (620), according to one embodiment, a processor of an electronic device may identify one or more functions associated with a first command using an external electronic device. The processor may identify one or more functions associated with the first command using an external electronic device connected via a communication circuit (e.g., communication circuit 220-2 of FIG. 2A). For example, the processor may request information about one or more functions associated with the first command from the external electronic device. After transmitting the request, the electronic device may obtain information (e.g., a capsule) about the one or more functions from the external electronic device. The electronic device may visualize the information using a display (e.g., display 225 of FIG. 2A) or output the information through a speaker (e.g., speaker 227 of FIG. 2A).

[0130] Referring to FIG. 6, in operation (630), according to one embodiment, a processor of an electronic device may identify at least one function executable by the electronic device using metadata of one or more functions. For example, the processor may identify at least one function executable by the electronic device using metadata having the properties described with reference to Tables 1 to 4. For example, the processor may determine whether the one or more functions are executable by the electronic device among an external electronic device or the electronic device using metadata of one or more functions related to a first command.

[0131] Referring to FIG. 6, in operation 640, a processor of an electronic device according to an embodiment may store information transmitted from an external electronic device and indicating a result of executing at least one function in a memory of the electronic device. For example, the processor may store information indicating a result of executing at least one function in a memory of the electronic device based on detecting at least one function executable by the electronic device, identified using metadata, from one or more functions associated with a first command. The information stored in the memory may be stored to be output in response to another user input detected after the user input of operation 610. In operation 640, the processor may store the information including a first command corresponding to the at least one function and one or more other commands similar to the first command in the memory of the electronic device. Hereinafter, exemplary operations of an electronic device performing the operations of FIG. 6 will be described with reference to FIG. 7.

[0132] FIG. 7 illustrates exemplary operations of an electronic device that executes one or more functions corresponding to commands recognized from natural language. The electronic device (101) of FIG. 1, FIG. 2A to FIG. 2C and / or the processor (210-2) may perform the exemplary operations described with reference to FIG. 7. The operations of the electronic device described with reference to FIG. 7 may be related to at least one of the operations of the electronic device described with reference to FIG. 5A to FIG. 6 .

[0133] Referring to FIG. 7, in operation (501), the electronic device may receive a user input. Upon detecting a first command from the user input, the electronic device may perform operation (502) to detect information about one or more functions related to the first command from the memory of the electronic device. By performing operations (503, 504), the processor may obtain information related to the first command.

[0134] Referring to FIG. 7, if no command related to the user input of operation (501) is detected from the memory of the electronic device, the processor may perform operation (710). In operation (710), the processor may request information (e.g., an event list) regarding one or more functions related to the first command from a server (e.g., the server (110) of FIGS. 1 and 2A to 2C) connected via a communication circuit (e.g., the communication circuit (220-2) of FIG. 2A). The server, upon receiving the request, may execute the cloud executor (240) to obtain information (e.g., an event list) indicating the result of executing the one or more functions. In operation (720), the server may transmit the information to the electronic device. Upon receiving the information of operation (720), the processor of the electronic device may output at least a portion of the information via a display and / or a speaker. For example, the processor may use the server to execute one or more functions associated with a command included in a user input of operation (501).

[0135] Within operation (730), the processor may execute the on-device capsule manager (232) to perform an operation for performing at least one command included in the user input of operation (501) solely using the electronic device or a server. For example, the processor may perform an operation for converting a capsule provided from a server (e.g., an event list of operation (720)) into a format for execution on the electronic device.

[0136] In operation (740), according to one embodiment, the processor of the electronic device may request metadata corresponding to at least one command included in the user input of operation (501) from a server. The server, upon receiving the request, may execute the metadata analyzer (262) to obtain the metadata. In operation (750), the server may transmit the obtained metadata to the electronic device. The electronic device, upon receiving the metadata of operation (750), may determine, in operation (760), whether to store information (e.g., an event list) of operation (720) using an attribute included in the metadata (e.g., at least one of the attributes described with reference to Tables 1 to 4). For example, the processor may determine, using the metadata, whether a result of executing one or more functions related to a command included in the user input of operation (501) is permitted to be reused by the electronic device.

[0137] Within operation (760), a processor that has decided to store information of operation (720) using metadata may perform operation (760). Within operation (760), the processor may request one or more commands similar to the commands included in the user input of operation (501) from the server. In response to the request, the server may execute the similar command detector (261) to obtain a list of one or more commands. Within operation (770), the server may transmit the list of one or more commands to the electronic device. Within operation (770), the information transmitted from the server to the electronic device may include the attributes of Table 9.

[0138] Attribute name value description requestType regex or algorithm or model A value indicating the scheme for determining whether commands are similar command flashlight [ㄱ-ㅎ|ㅏ-ㅣ|갑]*, turn on the flashlight Data used to perform the scheme determined by requestType

[0139] Referring to Table 9, at least one of a regular expression (regex), an algorithm (e.g., an algorithm for comparing sentence similarity such as Levenshtein distance), and / or a computational model for string comparison may be selected as a way to compare commands. The value described in command in Table 9 (e.g., flashlight[ㄱ-ㅎ|ㅏ-ㅣ|갑]*) is an example of a regular expression that may be used to identify other commands similar to the flashlight-related command. For example, based on the value of command (e.g., flashlight[ㄱ-ㅎ|ㅏ-ㅣ|갑]*) in Table 9, natural words containing “flashlight” and other characters combined after “flashlight” (e.g., “turn on the flashlight”, “turn on the flashlight”, “turn on the flashlight”, “please turn on the flashlight”, and / or “turn on the flashlight”) can be identified or determined as commands similar to the specific command.

[0140] A processor that receives information having the properties of Table 9 can store, within operation (780), within a command storage (270), information received within operation (720) (e.g., an event list) and information received within operation (770) (e.g., a list of similar commands). The information stored in the command storage (270) can be compared with commands included in a user input, as described above with reference to FIGS. 5A to 5C .

[0141] The operations described with reference to FIG. 7 may be repeatedly performed whenever a user input is detected. Each time the operations of FIG. 7 are repeatedly performed, a result in response to the user input (e.g., information of operation (720)) and one or more commands causing the result may be accumulated in the command storage (270).

[0142] FIG. 8 illustrates exemplary operations of an electronic device for obtaining another command similar to a command recognized from natural language. The electronic device (101) of FIG. 1, FIG. 2A to FIG. 2C and / or the processor (210-2) may perform the exemplary operations described with reference to FIG. 8. The operations of the electronic device described with reference to FIG. 8 may be related to at least one of the operations of the electronic device described with reference to FIG. 5A to FIG. 7.

[0143] Referring to FIG. 8, within operation (810), a processor of an electronic device executing a metadata analyzer (262) may detect a command executable by the electronic device. For example, the processor may detect a command executable by the electronic device or an external electronic device (e.g., the server (110) of FIGS. 1, 2A to 2C) using metadata corresponding to the command.

[0144] Within operation (820), the processor may perform an operation to detect or confirm a command similar to the command of operation (810). For example, if the command of operation (810) includes natural language such as “turn on the flashlight,” the command of operation (820) may include natural language including words with synonyms (or antonyms) such as “turn on the light,” “turn on the flash,” “activate the flashlight,” and / or “turn on the flashlight.” For example, the processor may execute a similar command detector (261) to detect a command similar to the command of operation (810). Within operation (830), the processor may request a list of commands similar to the command of operation (810) from a server via a communication circuit. In response to the above request, the server may execute the capsule context service (242) to retrieve one or more commands similar to the command of operation (810). In operation (840), the server may transmit a list of one or more commands to the electronic device. The list transmitted from the server to the electronic device by operation (840) may include the attributes of Table 9.

[0145] Referring to FIG. 8, within operation (850), according to one embodiment, the processor of the electronic device may store commands included in the list received by operation (840) within the command storage (270) together with the command of operation (810). Information stored in the command storage (270) based on operation (850) may be compared with other commands included in user input additionally received from the user.

[0146] FIG. 9 illustrates an exemplary operation of an electronic device (101) that displays a screen related to quick commands for executing multiple functions. The electronic device (101) of FIG. 1, FIG. 2A to FIG. 2C, and / or the processor (210-2) may perform the operation of the electronic device (101) described with reference to FIG. 9.

[0147] Referring to FIG. 9, an exemplary state of an electronic device (101) displaying a screen for registering a quick command is illustrated. The electronic device (101) may display a screen for registering a quick command on the display (225). Within the exemplary state of FIG. 9, the electronic device (101) may receive information required for registering a quick command from a user through the display (225). The electronic device (101) may display a text box (910) within the display (225) for obtaining a natural language to be used for identifying the quick command. Together with the text box (910), the electronic device (101) may display a button (915) for obtaining the natural language based on a utterance. In response to an input indicating selection of the button (915), the electronic device (101) may obtain the natural language from an audio signal output from a microphone (e.g., a microphone (226) of FIG. 2A).

[0148] Referring to FIG. 9, the electronic device (101) may display a button (960) for adding one or more commands to be executed by a quick command within the display (225). In response to an input indicating selection of the button (960), the electronic device (101) may obtain the command via text and / or audio signals. Referring to FIG. 9, in an exemplary state in which four commands are obtained based on the quick command, the electronic device (101) may display visual objects (920, 930, 940, 950) corresponding to each of the four commands within the display (225). The order in which the visual objects (920, 930, 940, 950) are arranged within the display (225) may indicate an order in which the four commands are executed based on the quick command.

[0149] Within the exemplary state of FIG. 9, the electronic device (101) can visualize commands to be executed by a quick command using visual objects (920, 930, 940, 950). The visual object (920) can correspond to a command for executing a function related to an alarm. Within the visual object (920), the electronic device (101) can display a natural language sentence expressing the voice command (e.g., “Set an alarm”). Within the visual object (920), the electronic device (101) can display a button (922) for controlling the order in which commands corresponding to the visual object (920) are to be executed. Within the visual object (920), the electronic device (101) can display a button (924) for excluding a voice command corresponding to the visual object (920) from the voice commands to be executed by the quick command.

[0150] Similarly, within the visual object (930), the electronic device (101) may display a natural language sentence expressing a command corresponding to the visual object (930) (e.g., "Read the last message"). Within the visual object (940), the electronic device (101) may display a natural language sentence expressing a command corresponding to the visual object (940) (e.g., "Register a reminder in 10 minutes"). Within the visual object (950), the electronic device (101) may display a natural language sentence expressing a command corresponding to the visual object (950) (e.g., "Tell me the weather"). Referring to the visual objects (920, 930, 940, 950) and the text box (910), the utterance (e.g., "command") designated for the quick command may be different from the natural language sentences required for execution of the visual objects (920, 930, 940, 950). For example, the length of the designated utterance may be shorter than the natural language sentences.

[0151] Referring to FIG. 9, the electronic device (101) can display a button (970) for registering a quick command based on information displayed on the display (225). Together with the button (970), the electronic device (101) can display a button (975) for bypassing the registration of the quick command. In response to an input indicating selection of the button (970), the electronic device (101) can register a quick command executed by text entered in the text box (910). To register the quick command, the electronic device (101) can access the quick command DB (254) described with reference to FIG. 2B.

[0152] Referring to FIG. 9, in a state of registering a quick command for collectively executing one or more commands, the electronic device may display a visual object (e.g., visual objects (926, 956)) indicating whether the result of executing functions corresponding to the one or more commands is stored in the electronic device. For example, by using a visual object (926) displayed together with a visual object (920), the electronic device may indicate that the result of executing one or more functions related to a command corresponding to the visual object (920) (e.g., a command related to setting an alarm) is stored in the electronic device. For example, if the result of executing functions related to a command corresponding to the visual object (950) (e.g., a command related to providing weather information) is stored in the electronic device, the electronic device may display a visual object (956) together with the visual object (950). Visual objects (930, 940) that are not displayed as visual objects (926, 956) may indicate that the results of executing functions corresponding to the visual objects (930, 940) are not stored in the electronic device.

[0153] As described above, according to one embodiment, the electronic device (101) can execute a command expressed in natural language obtained from a user. The electronic device (101) can store the result of executing the command in order to quickly respond to another command detected from the user after the command. For example, while maintaining the result of executing the command, the electronic device (101) that detects another command that matches or is similar to the command can bypass the execution of a model for processing natural language driven by the electronic device (101) and / or a server, and output the stored result.

[0154] FIG. 10 is a block diagram of an electronic device (1001) within a network environment (1000) according to various embodiments. Referring to FIG. 10, in the network environment (1000), the electronic device (1001) may communicate with the electronic device (1002) via a first network (1098) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (1004) or the server (1008) via a second network (1099) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1001) may communicate with the electronic device (1004) via the server (1008). According to one embodiment, the electronic device (1001) may include a processor (1020), a memory (1030), an input module (1050), an audio output module (1055), a display module (1060), an audio module (1070), a sensor module (1076), an interface (1077), a connection terminal (1078), a haptic module (1079), a camera module (1080), a power management module (1088), a battery (1089), a communication module (1090), a subscriber identification module (1096), or an antenna module (1097). In some embodiments, the electronic device (1001) may omit at least one of these components (e.g., the connection terminal (1078)), or may have one or more other components added. In some embodiments, some of these components (e.g., sensor module (1076), camera module (1080), or antenna module (1097)) may be integrated into a single component (e.g., display module (1060)).

[0155] The processor (1020) may, for example, execute software (e.g., a program (1040)) to control at least one other component (e.g., a hardware or software component) of the electronic device (1001) connected to the processor (1020) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1020) may store commands or data received from other components (e.g., a sensor module (1076) or a communication module (1090)) in a volatile memory (1032), process the commands or data stored in the volatile memory (1032), and store result data in a non-volatile memory (1034). According to one embodiment, the processor (1020) may include a main processor (1021) (e.g., a central processing unit or an application processor) or an auxiliary processor (1023) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1021). For example, when the electronic device (1001) includes the main processor (1021) and the auxiliary processor (1023), the auxiliary processor (1023) may be configured to use less power than the main processor (1021) or to be specialized for a given function. The auxiliary processor (1023) may be implemented separately from the main processor (1021) or as a part thereof.

[0156] The auxiliary processor (1023) may control at least a portion of functions or states associated with at least one component (e.g., the display module (1060), the sensor module (1076), or the communication module (1090)) of the electronic device (1001), for example, on behalf of the main processor (1021) while the main processor (1021) is in an inactive (e.g., sleep) state, or together with the main processor (1021) while the main processor (1021) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1023) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1080) or a communication module (1090)). In one embodiment, the auxiliary processor (1023) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1001) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1008)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0157] The memory (1030) can store various data used by at least one component (e.g., the processor (1020) or the sensor module (1076)) of the electronic device (1001). The data can include, for example, software (e.g., the program (1040)) and input data or output data for commands related thereto. The memory (1030) can include volatile memory (1032) or non-volatile memory (1034).

[0158] The program (1040) may be stored as software in memory (1030) and may include, for example, an operating system (1042), middleware (1044), or an application (1046).

[0159] The input module (1050) can receive commands or data to be used in a component of the electronic device (1001) (e.g., a processor (1020)) from an external source (e.g., a user) of the electronic device (1001). The input module (1050) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0160] The audio output module (1055) can output audio signals to the outside of the electronic device (1001). The audio output module (1055) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0161] The display module (1060) can visually provide information to an external party (e.g., a user) of the electronic device (1001). The display module (1060) may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling the device. In one embodiment, the display module (1060) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0162] The audio module (1070) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (1070) can acquire sound through the input module (1050), output sound through the sound output module (1055), or an external electronic device (e.g., electronic device (1002)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1001).

[0163] The sensor module (1076) can detect the operating status (e.g., power or temperature) of the electronic device (1001) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1076) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0164] The interface (1077) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1001) to an external electronic device (e.g., the electronic device (1002)). In one embodiment, the interface (1077) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0165] The connection terminal (1078) may include a connector through which the electronic device (1001) may be physically connected to an external electronic device (e.g., the electronic device (1002)). In one embodiment, the connection terminal (1078) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0166] The haptic module (1079) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (1079) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0167] The camera module (1080) can capture still images and videos. In one embodiment, the camera module (1080) may include one or more lenses, image sensors, image signal processors, or flashes.

[0168] The power management module (1088) can manage power supplied to the electronic device (1001). According to one embodiment, the power management module (1088) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0169] A battery (1089) may power at least one component of the electronic device (1001). In one embodiment, the battery (1089) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0170] The communication module (1090) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1001) and an external electronic device (e.g., electronic device (1002), electronic device (1004), or server (1008)), and the performance of communication through the established communication channel. The communication module (1090) may operate independently from the processor (1020) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1090) may include a wireless communication module (1092) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1094) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (1004) via a first network (1098) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1099) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1092) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1096) to verify or authenticate the electronic device (1001) within a communication network such as the first network (1098) or the second network (1099).

[0171] The wireless communication module (1092) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1092) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1092) may support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1092) may support various requirements specified in the electronic device (1001), an external electronic device (e.g., the electronic device (1004)), or a network system (e.g., the second network (1099)). According to one embodiment, the wireless communication module (1092) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0172] The antenna module (1097) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1097) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1097) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1098) or the second network (1099), may be selected from the plurality of antennas, for example, by the communication module (1090). A signal or power may be transmitted or received between the communication module (1090) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1097).

[0173] According to various embodiments, the antenna module (1097) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0174] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0175] According to one embodiment, commands or data may be transmitted or received between the electronic device (1001) and an external electronic device (1004) via a server (1008) connected to a second network (1099). Each of the external electronic devices (1002, or 704) may be the same or a different type of device as the electronic device (1001). According to one embodiment, all or part of the operations executed in the electronic device (1001) may be executed in one or more of the external electronic devices (1002, 704, or 708). For example, when the electronic device (1001) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1001) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1001). The electronic device (1001) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1001) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (1004) may include an Internet of Things (IoT) device. The server (1008) may be an intelligent server utilizing machine learning and / or a neural network.According to one embodiment, an external electronic device (1004) or server (1008) may be included within the second network (1099). The electronic device (1001) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0176] Figure 11 is a block diagram illustrating an integrated intelligence system according to one embodiment.

[0177] Referring to FIG. 11, an integrated intelligence system (10) of one embodiment may include a user terminal (1100), an intelligent server (1200), and a service server (1300).

[0178] A user terminal (1100) of one embodiment (e.g., electronic device (101) of FIG. 1) may be a terminal device (or electronic device) that can connect to the Internet, and may be, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a TV, white goods, a wearable device, an HMD, or a smart speaker.

[0179] According to one embodiment, the user terminal (1100) may include a communication interface (1110), a microphone (1120), a speaker (1130), a display (1140), a memory (1150), and a processor (1160). The components listed above may be operatively or electrically connected to each other.

[0180] According to one embodiment, the communication interface (1110) may be configured to be connected to an external device to transmit and receive data. According to one embodiment, the microphone (1120) may receive sound (e.g., user speech) and convert it into an electrical signal. According to one embodiment, the speaker (1130) may output the electrical signal as sound (e.g., voice). According to one embodiment, the display (1140) may be configured to display an image or video. According to one embodiment, the display (1140) may display a graphical user interface (GUI) of an app (or application program) being executed.

[0181] The display (1140) of one embodiment may be configured to display an image or video. The display (1140) of one embodiment may also display a graphical user interface (GUI) of a running app (or application program). The display (1140) of one embodiment may receive touch input via a touch sensor. For example, the display (1140) may receive text input via a touch sensor in an on-screen keyboard area displayed within the display (1140).

[0182] According to one embodiment, the memory (1150) may store a client module (1151), a software development kit (SDK) (1153), and a plurality of apps (1155). The client module (1151) and the SDK (1153) may constitute a framework (or solution program) for performing general functions. In addition, the client module (1151) or the SDK (1153) may constitute a framework for processing user input (e.g., voice input, text input, touch input).

[0183] According to one embodiment, the memory (1150) may be a program for performing a specified function of the plurality of apps (1155). According to one embodiment, the plurality of apps (1155) may include a first app (1155_1) and a second app (1155_3). According to one embodiment, each of the plurality of apps (1155) may include a plurality of operations for performing a specified function. For example, the plurality of apps (1155) may include at least one of an alarm app, a message app, and a schedule app. According to one embodiment, the plurality of apps (1155) may be executed by the processor (1160) to sequentially perform at least some of the plurality of operations.

[0184] According to one embodiment, the processor (1160) can control the overall operation of the user terminal (1100). For example, the processor (1160) can be electrically connected to a communication interface (1110), a microphone (1120), a speaker (1130), a display (1140), and a memory (1150) to perform a designated operation.

[0185] According to one embodiment, the processor (1160) may also execute a program stored in the memory (1150) to perform a designated function. For example, the processor (1160) may execute at least one of the client module (1151) or the SDK (1153) to perform the following operations for processing user input. The processor (1160) may control the operations of multiple apps (1155), for example, through the SDK (1153). The following operations described as operations of the client module (1151) or the SDK (1153) may be operations executed by the processor (1160).

[0186] According to one embodiment, the client module (1151) can receive user input. For example, the client module (1151) can generate a voice signal corresponding to a user utterance detected through the microphone (1120). Alternatively, the client module (1151) can receive a touch input detected through the display (1140). Alternatively, the client module (1151) can receive a text input detected through a keyboard or a visual keyboard. In addition, the client module (1151) can receive various forms of user input detected through an input module included in the user terminal (1100) or an input module connected to the user terminal (1100). The client module (1151) can transmit the received user input to the intelligent server (1200). According to one embodiment, the client module (1151) can transmit status information of the user terminal (1100) to the intelligent server (1200) together with the received user input. The above status information may be, for example, the execution status information of the app.

[0187] According to one embodiment, the client module (1151) may receive a result corresponding to the received user input. For example, the client module (1151) may receive a result corresponding to the user input from the intelligent server (1200). The client module (1151) may display the received result on the display (1140). Additionally, the client module (1151) may output the received result as audio through the speaker (1130).

[0188] According to one embodiment, the client module (1151) may receive a plan corresponding to the received user input. The client module (1151) may display the results of executing multiple operations of the app according to the plan on the display (1140). For example, the client module (1151) may sequentially display the results of executing multiple operations on the display and output audio through the speaker (1130). As another example, the user terminal (1100) may display only some results of executing multiple operations (e.g., the result of the last operation) on the display and output audio through the speaker (1130).

[0189] According to one embodiment, the client module (1151) may receive a request from the intelligent server (1200) to obtain information necessary to produce a result corresponding to a user input. The information necessary to produce the result may be, for example, status information of the user terminal (1100). According to one embodiment, the client module (1151) may transmit the necessary information to the intelligent server (1200) in response to the request.

[0190] According to one embodiment, the client module (1151) can transmit result information of executing multiple operations according to a plan to the intelligent server (1200). The intelligent server (1200) can confirm that the received user input has been processed correctly through the result information.

[0191] In one embodiment, the client module (1151) may include a voice recognition module. In one embodiment, the client module (1151) may recognize voice inputs that perform limited functions through the voice recognition module. For example, the client module (1151) may execute an intelligent app to process voice inputs to perform organic actions through designated inputs (e.g., "Wake up!").

[0192] According to one embodiment, the intelligent server (1200) can receive information related to user voice input from the user terminal (1100) via a communication network. According to one embodiment, the intelligent server (1200) can convert data related to the received voice input into text data. According to one embodiment, the intelligent server (1200) can generate a plan for performing a task corresponding to the user voice input based on the text data.

[0193] In one embodiment, the plan may be generated by an artificial intelligence (AI) system. The AI ​​system may be a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN) or a recurrent neural network (RNN)), or a combination of the above or a different AI system. In one embodiment, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the AI ​​system may select at least one plan from a plurality of predefined plans.

[0194] According to one embodiment, the intelligent server (1200) may transmit the results calculated according to the generated plan to the user terminal (1100), or may transmit the generated plan to the user terminal (1100). According to one embodiment, the user terminal (1100) may display the results calculated according to the plan on a display. According to one embodiment, the user terminal (1100) may display the results of executing an operation according to the plan on a display.

[0195] An intelligent server (1200) of one embodiment may include a front end (1210), a natural language platform (1220), a capsule DB (1230), an execution engine (1240), an end user interface (1250), a management platform (1260), a big data platform (1270), and an analytic platform (1280).

[0196] According to one embodiment, the front end (1210) can receive user input from the user terminal (1100). The front end (1210) can transmit a response corresponding to the user input.

[0197] According to one embodiment, the natural language platform (1220) may include an automatic speech recognition module (ASR module) (1221), a natural language understanding module (NLU module) (1223), a planner module (1225), a natural language generator module (NLG module) (1227), and a text to speech module (TTS module) (1229).

[0198] According to one embodiment, the automatic speech recognition module (1221) can convert voice input received from the user terminal (1100) into text data. According to one embodiment, the natural language understanding module (1223) can identify the user's intention using the text data of the voice input. For example, the natural language understanding module (1223) can identify the user's intention by performing syntactic analysis or semantic analysis on user input in the form of text data. According to one embodiment, the natural language understanding module (1223) can identify the meaning of words extracted from the user input using linguistic features (e.g., grammatical elements) of morphemes or phrases, and can determine the user's intention by matching the meaning of the identified words to the intent. The natural language understanding module (1223) can obtain intent information corresponding to the user's utterance. The intent information can be information indicating the user's intent determined by interpreting text data. Intent information may include information indicating an action or function that a user wishes to perform using the device.

[0199] According to one embodiment, the planner module (1225) can generate a plan using the intent and parameters determined by the natural language understanding module (1223). According to one embodiment, the planner module (1225) can determine a plurality of domains necessary to perform a task based on the determined intent. The planner module (1225) can determine a plurality of operations included in each of the plurality of domains determined based on the intent. According to one embodiment, the planner module (1225) can determine parameters necessary to execute the determined plurality of operations or result values ​​output by the execution of the plurality of operations. The parameters and the result values ​​can be defined as concepts related to a specified format (or class). Accordingly, the plan can include a plurality of operations and a plurality of concepts determined by the user's intent. The planner module (1225) can determine the relationships between the plurality of operations and the plurality of concepts in a stepwise (or hierarchical) manner. For example, the planner module (1225) can determine the execution order of a plurality of actions based on the user's intention based on a plurality of concepts. In other words, the planner module (1225) can determine the execution order of a plurality of actions based on parameters required for the execution of the plurality of actions and results output by the execution of the plurality of actions. Accordingly, the planner module (1225) can generate a plan including association information (e.g., ontology) between the plurality of actions and the plurality of concepts. The planner module (1225) can generate the plan using information stored in a capsule database (1230) in which a set of relationships between concepts and actions is stored.

[0200] According to one embodiment, the natural language generation module (1227) can convert specified information into text format. The information converted into text format may be in the form of natural language speech. The text-to-speech conversion module (1229) of one embodiment can convert information in text format into information in speech format.

[0201] According to one embodiment, the capsule database (1230) can store information about the relationship between a plurality of concepts and actions corresponding to a plurality of domains. For example, the capsule database (1230) can store a plurality of capsules including a plurality of action objects (or action information) and concept objects (or concept information) of a plan. According to one embodiment, the capsule database (1230) can store the plurality of capsules in the form of a concept action network (CAN). According to one embodiment, the plurality of capsules can be stored in a function registry included in the capsule database (1230).

[0202] According to one embodiment, the capsule database (1230) may include a strategy registry that stores strategy information required when determining a plan corresponding to a voice input. The strategy information may include reference information for determining a single plan when there are multiple plans corresponding to a user input. According to one embodiment, the capsule database (1230) may include a follow-up registry that stores information on follow-up actions for suggesting follow-up actions to a user in a given situation. The follow-up actions may include, for example, follow-up utterances. According to one embodiment, the capsule database (1230) may include a layout registry that stores layout information of information output through the user terminal (1100). According to one embodiment, the capsule database (1230) may include a vocabulary registry that stores vocabulary information included in capsule information. According to one embodiment, the capsule database (1230) may include a dialog registry in which information about a dialog (or interaction) with a user is stored.

[0203] According to one embodiment, the capsule database (1230) can update stored objects through a developer tool. The developer tool may include, for example, a function editor for updating action objects or concept objects. The developer tool may include a vocabulary editor for updating vocabulary. The developer tool may include a strategy editor for creating and registering strategies that determine plans. The developer tool may include a dialog editor for creating a dialogue with a user. The developer tool may include a follow-up editor for activating follow-up goals and editing follow-up utterances that provide hints. The follow-up goals may be determined based on currently set goals, user preferences, or environmental conditions.

[0204] According to one embodiment, the capsule database (1230) may also be implemented within the user terminal (1100). In other words, the user terminal (1100) may include a capsule database (1230) that stores information for determining an action corresponding to a voice input.

[0205] According to one embodiment, the execution engine (1240) can produce a result using the generated plan. According to one embodiment, the end user interface (1250) can transmit the produced result to the user terminal (1100). Accordingly, the user terminal (1100) can receive the result and provide the received result to the user. According to one embodiment, the management platform (1260) can manage information used in the intelligent server (1200). According to one embodiment, the big data platform (1270) can collect user data. According to one embodiment, the analysis platform (1280) can manage the quality of service (QoS) of the intelligent server (1200). For example, the analysis platform (1280) can manage the components and processing speed (or efficiency) of the intelligent server (1200).

[0206] According to one embodiment, the service server (1300) may provide a designated service (e.g., food ordering or hotel reservation) to the user terminal (1100). According to one embodiment, the service server (1300) may be a server operated by a third party. For example, the service server (1300) may include a first service server (1301), a second service server (1303), and a third service server (1305) operated by different third parties. According to one embodiment, the service server (1300) may provide information for generating a plan corresponding to the received voice input to the intelligent server (1200). The provided information may be stored, for example, in a capsule database (1230). In addition, the service server (1300) may provide result information according to the plan to the intelligent server (1200).

[0207] In the integrated intelligence system (11) described above, the user terminal (1100) can provide various intelligent services to the user in response to user input. The user input may include, for example, input via a physical button, touch input, or voice input.

[0208] According to one embodiment, the user terminal (1100) may provide a voice recognition service through an intelligent app (or voice recognition app) stored internally. In this case, for example, the user terminal (1100) may recognize a user utterance or voice input received through the microphone and provide the user with a service corresponding to the recognized voice input.

[0209] According to one embodiment, the user terminal (1100) may perform a designated action based on the received voice input, either alone or in conjunction with the intelligent server and / or service server. For example, the user terminal (1100) may execute an app corresponding to the received voice input and perform a designated action through the executed app.

[0210] According to one embodiment, when a user terminal (1100) provides a service together with an intelligent server (1200) and / or a service server, the user terminal may detect user speech using the microphone (1120) and generate a signal (or voice data) corresponding to the detected user speech. The user terminal may transmit the voice data to the intelligent server (1200) using a communication interface (1110).

[0211] According to one embodiment, the intelligent server (1200) may generate a plan for performing a task corresponding to the voice input received from the user terminal (1100), or a result of performing an operation according to the plan. The plan may include, for example, a plurality of operations for performing a task corresponding to the user's voice input, and a plurality of concepts related to the plurality of operations. The concept may define parameters input to the execution of the plurality of operations, or result values ​​output by the execution of the plurality of operations. The plan may include association information between the plurality of operations and the plurality of concepts.

[0212] In one embodiment, the user terminal (1100) can receive the response using the communication interface (1110). The user terminal (1100) can output a voice signal generated within the user terminal (1100) to the outside using the speaker (1130), or can output an image generated within the user terminal (1100) to the outside using the display (1140).

[0213] FIG. 12 is a diagram showing a form in which relationship information between concepts and actions is stored in a database according to various embodiments.

[0214] The capsule database (e.g., the capsule database (1230) of FIG. 11) of the intelligent server (e.g., the intelligent server (1200) of FIG. 11) may store multiple capsules in the form of a CAN (concept action network) (1600). The capsule database may store an action for processing a task corresponding to a user's voice input and parameters required for the action in the form of a CAN (concept action network). The CAN may represent an organic relationship between an action and a concept that defines parameters required to perform the action.

[0215] The capsule database may store a plurality of capsules (e.g., Capsule A (1601), Capsule B (1604)) corresponding to each of a plurality of domains (e.g., applications). According to one embodiment, one capsule (e.g., Capsule A (1601)) may correspond to one domain (e.g., application). In addition, one capsule may correspond to at least one service provider (e.g., CP 1 (1602), CP 2 (1603), CP 3 (1606), or CP 4 (1605)) for performing a function of a domain related to the capsule. According to one embodiment, one capsule may include at least one operation (1610) and at least one concept (1620) for performing a specified function.

[0216] According to one embodiment, a natural language platform (e.g., the natural language platform (1220) of FIG. 11) can generate a plan for performing a task corresponding to a received speech input using capsules stored in a capsule database. For example, a planner module of the natural language platform (e.g., the planner module (1225) of FIG. 11) can generate a plan using capsules stored in a capsule database. For example, a plan (1607) can be generated using actions (1711, 1713) and concepts (1712, 1714) of Capsule A (1601) and actions (1741) and concepts (1742) of Capsule B (1604).

[0217] FIG. 13 is a diagram showing a screen for processing voice input received through an intelligent app by a user terminal according to various embodiments.

[0218] The user terminal (1100) can execute an intelligent app to process user input through an intelligent server (e.g., the intelligent server (1200) of FIG. 11).

[0219] According to one embodiment, on screen 1310, when the user terminal (1100) recognizes a designated voice input (e.g., wake up!) or receives an input via a hardware key (e.g., a dedicated hardware key), the user terminal (1100) may execute an intelligent app for processing the voice input. For example, the user terminal (1100) may execute an intelligent app while executing a schedule app. According to one embodiment, the user terminal (1100) may display an object (e.g., an icon) (1311) corresponding to the intelligent app on a display (e.g., a display (1140) of FIG. 11). According to one embodiment, the user terminal (1100) may receive a voice input by a user's speech. For example, the user terminal (1100) may receive a voice input such as "Tell me my schedule for this week!" According to one embodiment, the user terminal (1100) can display a user interface (UI) (1313) (e.g., an input window) of an intelligent app on which text data of a received voice input is displayed.

[0220] According to one embodiment, on screen 1320, the user terminal (1100) may display a result corresponding to the received voice input on the display. For example, the user terminal (1100) may receive a plan corresponding to the received user input and display "This Week's Schedule" on the display according to the plan.

[0221] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0222] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0223] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0224] Various embodiments of the present document may be implemented as software (e.g., a program (1040)) including one or more instructions stored in a storage medium (e.g., an internal memory (1036) or an external memory (1038)) readable by a machine (e.g., an electronic device (1001)). For example, a processor (e.g., a processor (1020)) of the machine (e.g., an electronic device (1001)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0225] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0226] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separately arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to integration. According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added. The electronic device (1001) of FIG. 10 may be an example of the electronic device (101) of FIG. 1.

[0227] In one embodiment, when repeatedly performing a specific function in response to natural language, a method for utilizing the result of executing the specific function may be required. In one embodiment, when repeatedly performing a specific function in response to natural language, a method for reducing resources occupied for repeatedly performing the specific function by utilizing the result of executing the specific function may be required. According to an embodiment as described above, an electronic device (e.g., the electronic device (101) of FIGS. 1 and 2A to 2C) may include a memory (e.g., the memory (215-2) of FIG. 2A) and a processor (e.g., the processor (210-2) of FIG. 2A) for storing instructions. The instructions, when executed by the processor of the electronic device, may cause the electronic device to detect a user input including a first command expressed in natural language. The instructions, when executed by a processor of the electronic device, may cause the electronic device to determine whether a first command corresponds to a second command that was executed prior to the first command. The instructions, when executed by the processor of the electronic device, may cause the electronic device to output, in response to the user input, information generated in a memory of the electronic device by executing the second command based on the electronic device determining that the first command corresponds to the second command. The instructions, when executed by the processor of the electronic device, may cause the electronic device to execute one or more functions associated with the first command based on the electronic device determining that a first command different from the second command has been detected. In one embodiment, the electronic device may utilize a result of executing the specific function when repeatedly performing the specific function in response to natural language.According to one embodiment, when an electronic device repeatedly performs a specific function in response to natural language, the electronic device can reduce resources occupied for repeatedly performing the specific function by utilizing the result of executing the specific function.

[0228] For example, the instructions, when executed by a processor of the electronic device, may cause the electronic device to determine whether to output the information in response using metadata corresponding to the one or more functions.

[0229] For example, the instructions, when executed by a processor of the electronic device, may cause the electronic device to determine whether to output the information, using the time at which the information was stored and the period for which the information is retained as indicated by the metadata.

[0230] For example, the electronic device may include a display. The instructions, when executed by a processor of the electronic device, may cause the electronic device to control the display to display a first visual object that includes the first command included in the user input, in response to detecting the user input. The instructions, when executed by the processor of the electronic device, may cause the electronic device to control the display to display a second visual object for representing the information together with the first visual object displayed within the display.

[0231] For example, the electronic device may include a communication circuit. The instructions, when executed by a processor of the electronic device, may cause the electronic device to transmit a signal including the first command to an external electronic device through the communication circuit to perform one or more functions.

[0232] For example, the instructions, when executed by a processor of the electronic device, may cause the electronic device to perform one or more functions using another signal transmitted from the external electronic device that received the signal.

[0233] For example, the instructions, when executed by a processor of the electronic device, may cause the electronic device to store, in a memory of the electronic device, other information related to a result of executing the one or more functions using the other signal.

[0234] For example, the other information may be stored in the memory of the electronic device to be output in response to another user input including the first command detected after the user input.

[0235] For example, the electronic device may include a microphone. The instructions, when executed by a processor of the electronic device, may cause the electronic device to detect the user input using an audio signal received from the microphone.

[0236] For example, the instructions, when executed by a processor of the electronic device, may cause the electronic device to determine a similarity between the first command and the second command. The instructions, when executed by the processor of the electronic device, may cause the electronic device to compare a specified threshold value and the similarity value to determine whether the first command corresponds to the second command.

[0237] As described above, in one embodiment, a method of an electronic device may be provided. The method may include detecting a user input including a first command expressed in natural language. The method may include determining whether the first command corresponds to a second command that was executed prior to the first command. The method may include outputting information generated in a memory of the electronic device by executing the second command in response to the user input, based on determining that the first command corresponds to the second command. The method may include executing one or more functions associated with the first command based on determining that a first command different from the second command has been detected.

[0238] For example, the determining action may include an action of determining whether to output the information as the response using metadata corresponding to the one or more functions.

[0239] For example, the determining action may include an action of determining whether to output the information, using the time at which the information was stored and the period for which the information is maintained as indicated by the metadata.

[0240] For example, the detecting action may include, in response to detecting the user input, controlling a display of the electronic device to display a first visual object including the first command included in the user input. The outputting action may include controlling the display to display a second visual object for representing the information together with the first visual object displayed within the display.

[0241] For example, the executing operation may include an operation of transmitting a signal including the first command to an external electronic device through a communication circuit of the electronic device to execute the one or more functions.

[0242] For example, the executing operation may include executing one or more functions using another signal transmitted from the external electronic device that received the signal.

[0243] For example, the executing operation may include an operation of storing other information related to the result of executing the one or more functions using the other signal in the memory of the electronic device.

[0244] For example, the other information may be stored in the memory of the electronic device to be output in response to another user input including the first command detected after the user input.

[0245] For example, the detecting action may include detecting the user input using an audio signal received from a microphone of the electronic device.

[0246] For example, the determining operation may include an operation of determining a similarity between the first command and the second command. The determining operation may include an operation of comparing a specified threshold value and the similarity to determine whether the first command corresponds to the second command.

[0247] In one embodiment, as described above, a non-transitory computer-readable storage medium storing instructions may be provided. The instructions, when executed by a processor of an electronic device including communication circuitry, may cause the electronic device to detect a user input including a first command expressed in natural language. The instructions, when executed by the processor of the electronic device, may cause the electronic device to identify, using an external electronic device connected via the communication circuitry, one or more functions associated with the first command. The instructions, when executed by the processor of the electronic device, may cause the electronic device to determine, using metadata of the one or more functions, whether the one or more functions are executable by the electronic device among the external electronic device or the electronic device. The instructions, when executed by a processor of the electronic device, may cause the electronic device to store, in a memory of the electronic device, information indicating a result of executing at least one function, based on detecting at least one function executable by the electronic device from the one or more functions using the metadata. The information may be stored to be output in response to another user input detected after the user input.

[0248] For example, the instructions, when executed by a processor of the electronic device, may cause the electronic device to store, in a memory of the electronic device, the information including the first command corresponding to the at least one function and one or more second commands similar to the first command.

[0249] For example, the instructions, when executed by a processor of the electronic device, may cause the electronic device to compare a second command related to other information cached in a memory of the electronic device with the first command and determine whether to transmit a signal to the external electronic device to identify one or more functions related to the first command.

[0250] According to one embodiment, as described above, an electronic device may include a communication circuit, a memory for storing instructions, and a processor. The instructions, when executed by the processor of the electronic device, may cause the electronic device to detect a user input including a first command expressed in natural language. The instructions, when executed by the processor of the electronic device, may cause the electronic device to identify one or more functions associated with the first command using an external electronic device connected via the communication circuit. The instructions, when executed by the processor of the electronic device, may cause the electronic device to determine, using metadata of the one or more functions, whether the one or more functions are executable by the electronic device or the external electronic device. The instructions, when executed by a processor of the electronic device, may cause the electronic device to store, in a memory of the electronic device, information indicating a result of executing at least one function, based on detecting at least one function executable by the electronic device from the one or more functions using the metadata. The information may be stored to be output in response to another user input detected after the user input.

[0251] For example, the instructions, when executed by a processor of the electronic device, may cause the electronic device to store, in a memory of the electronic device, the information including the first command corresponding to the at least one function and one or more second commands similar to the first command.

[0252] For example, the instructions, when executed by a processor of the electronic device, may cause the electronic device to compare a second command related to other information cached in a memory of the electronic device with the first command and determine whether to transmit a signal to the external electronic device to identify one or more functions related to the first command.

[0253] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0254] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0255] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0256] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0257] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In electronic devices, A memory including one or more storage media storing instructions; and comprising at least one processor including a processing circuit; The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Detecting user input including a first command expressed in natural language; Determine whether the first command corresponds to a second command that was executed prior to the first command; Based on determining that the first command corresponds to the second command, executing the second command to output information generated in the memory of the electronic device as a response to the user input; and Causing one or more functions related to the first command to be executed based on determining that the first command is different from the second command. to cause, to cause, Electronic devices.

2. In claim 1, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Causing a determination of whether to output said information as a response, using metadata corresponding to said one or more functions. Electronic devices.

3. In any one of claims 1 to 2, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Causing to determine whether to output the information, by using the time at which the information was stored and the period for which the information is maintained as indicated by the metadata. Electronic devices.

4. In any one of claims 1 to 3, further comprising a display, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: In response to detecting said user input, controlling said display to display a first visual object including said first command included in said user input; causing the display to be controlled to display a second visual object for expressing the information together with the first visual object displayed within the display; Electronic devices.

5. In any one of claims 1 to 4, further comprising a communication circuit, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: To cause a signal including the first command to be transmitted to an external electronic device through the communication circuit, in order to execute one or more of the above functions. Electronic devices.

6. In any one of claims 1 to 5, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Causing said one or more functions to be executed using another signal transmitted from said external electronic device that received said signal. Electronic devices.

7. In any one of claims 1 to 6, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Causing other information related to the result of executing said one or more functions using said other signal to be stored in the memory of said electronic device. Electronic devices.

8. In any one of claims 1 to 7, Other information above, stored in the memory of the electronic device to be output as a response to another user input including the first command detected after the user input; Electronic devices.

9. In any one of claims 1 to 8, further comprising a microphone, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: causing the user input to be detected using an audio signal received from the microphone; Electronic devices.

10. In any one of claims 1 to 9, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Determine the similarity between the first command and the second command; By comparing the specified threshold and the similarity, it is determined whether the first command corresponds to the second command, Electronic devices.

11. In the method of an electronic device, An action of detecting user input including a first command expressed in natural language; An operation for determining whether the first command corresponds to a second command that was executed prior to the first command; An operation of executing the second command and outputting information generated in the memory of the electronic device as a response to the user input based on determining that the first command corresponds to the second command; and An operation including executing one or more functions related to the first command based on determining that the first command is detected to be different from the second command. method.

12. In claim 11, the determining action is: An operation for determining whether to output the information as a response, using metadata corresponding to one or more of the above functions. method.

13. In any one of claims 11 to 12, the determining action comprises: An operation for determining whether to output the information, using the time at which the information is stored and the period for which the information is maintained as indicated by the metadata. method.

14. In any one of claims 11 to 13, the detecting operation comprises: In response to detecting said user input, comprising an action of controlling a display of said electronic device to display a first visual object including said first command included in said user input; The above output action is, Including an operation of controlling the display to display a second visual object for expressing the information together with the first visual object displayed within the display. method.

15. In any one of claims 11 to 14, the executing operation comprises: In order to execute one or more of the above functions, the method comprises transmitting a signal including the first command to an external electronic device through a communication circuit of the electronic device. method.

Images