Electronic device, method, and non-transitory computer-readable storage medium for displaying user interface for request included in utterance

Abstract

This electronic device may comprise a memory for storing instructions, and at least one processor. The instructions may cause the electronic device to: receive, from a first external electronic device, information on a first sentence corresponding to a first user intent and a second sentence subsequent to the first sentence and corresponding to a second user intent, wherein the information is obtained according to recognition of a voice command; on the basis of the received information, identify whether at least some of words of the first sentence are included in the second sentence; on the basis of identifying that at least some of the words of the first sentence are included in the second sentence, determine that the first user intent is related to the second user intent; on the basis of the determination, generate data for a command to execute a first function indicated by the first user intent and to execute a second function indicated by the second user intent according to a result of the execution of the first function; and transmit the data to a second external electronic device.

Description

Electronic device, method, and non-transitory computer-readable storage medium for displaying a user interface for a request included in an utterance

[0001] The present disclosure relates to an electronic device, a method, and a non-transitory computer-readable storage medium for displaying a user interface for a request included in an utterance.

[0002] An electronic device may include a microphone. The electronic device may receive speech through the microphone. The electronic device may perform speech recognition on the received speech. For example, the electronic device may provide feedback in response to the received speech. For example, the electronic device may identify a user's intent or request from the speech. For example, the electronic device may execute a function in response to the user's request.

[0003] The above information may be provided as background art to aid in understanding the present disclosure.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0005] An electronic device is described. The electronic device may include a memory storing instructions and including one or more storage media. The electronic device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive, from a first external electronic device, information about a first sentence corresponding to a first user intent and a second sentence corresponding to a second user intent and subsequent to the first sentence, and to acquire the information based on recognition of a voice command. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify, based on the received information, whether at least some of the words of the first sentence are included in the second sentence. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine that the first user intent is related to the second user intent based on identifying that at least some of the words of the first sentence are included in the second sentence. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate data for instructions for executing a first function indicated by the first user intent and for executing a second function indicated by the second user intent based on the determination. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit the data to a second external electronic device.

[0006] A method is provided. The method may be executed within an electronic device. The method may include an operation of receiving, from a first external electronic device, information regarding a first sentence corresponding to a first user intention and a second sentence following the first sentence corresponding to a second user intention, and acquiring said information upon recognition of a voice command. The method may include an operation of identifying, based on said received information, whether at least some of the words of said first sentence are included in said second sentence. The method may include an operation of determining that said first user intention is related to said second user intention based on identifying that said at least some of the words of said first sentence are included in said second sentence. The method may include an operation of generating, based on said determination, data for a command to execute a first function indicated by said first user intention and to execute a second function indicated by said second user intention according to the result of the execution of said first function. The method may include an operation of transmitting said data to a second external electronic device.

[0007] A non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device, cause the electronic device to receive, from a first external electronic device, information about a first sentence corresponding to a first user intent and a second sentence corresponding to a second user intent and subsequent to the first sentence, and to obtain the information based on recognition of a voice command. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to identify, based on the received information, whether at least a portion of words in the first sentence are included in the second sentence. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to determine, based on identifying that at least a portion of words in the first sentence are included in the second sentence, that the first user intent is related to the second user intent. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to generate data for a command to execute a first function indicated by the first user intent based on the determination, and to execute a second function indicated by the second user intent based on a result of the execution of the first function. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to transmit the data to a second external electronic device.

[0008] An electronic device is described. The electronic device may include a microphone. The electronic device may include a display. The electronic device may include a memory that stores instructions and includes one or more storage media. The electronic device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a voice command via the microphone. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit information about the voice command to a first external electronic device based on the identification. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive data related to the voice command from a second external electronic device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify, from the data, a command for executing a first function of the electronic device, and, based on a result of executing the first function, for executing a second function of the electronic device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the first function based on the identification. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to simultaneously display, through the display, a UI object indicating the result of executing the first function and whether to execute the second function.The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the second function based on the displayed result of the first function, based on a first user input to the UI object indicating to execute the second function. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to refrain from executing the second function based on a second user input to the UI object indicating to cancel executing the second function.

[0009] A method is provided. The method can be executed in an electronic device having a microphone and a display. The method can include an operation of identifying a voice command through the microphone. The method can include an operation of transmitting information about the voice command to a first external electronic device based on the identification. The method can include an operation of receiving data related to the voice command from a second external electronic device. The method can include an operation of identifying, from the data, a command for executing a first function of the electronic device, and a command for executing a second function of the electronic device based on a result of executing the first function. The method can include an operation of executing the first function based on the identification. The method can include an operation of simultaneously displaying, through the display, a UI object indicating the result of executing the first function and whether to execute the second function. The method can include an operation of executing the second function based on the displayed result of the first function based on a first user input to the UI object indicating to execute the second function. The method may include an action of refraining from executing the second function based on a second user input to the UI object indicating to cancel executing the second function.

[0010] A non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device having a microphone and a display, cause the electronic device to identify a voice command via the microphone. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to transmit information about the voice command to a first external electronic device based on the identification. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to receive data related to the voice command from a second external electronic device. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to identify, from the data, a command for executing a first function of the electronic device and, based on a result of executing the first function, for executing a second function of the electronic device. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to execute the first function based on the identification. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to simultaneously display, through the display, a UI object indicating the result of the execution of the first function and whether the second function is to be executed.The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to execute the second function based on the displayed result of the first function, based on a first user input to the UI object indicating to execute the second function. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to refrain from executing the second function, based on a second user input to the UI object indicating to cancel executing the second function.

[0011] An electronic device is described. The electronic device may include a microphone. The electronic device may include a memory storing instructions and including one or more storage media. The electronic device may include at least one processor including a processing circuit. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a voice command via the microphone. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain, based on the identification, information about a first sentence corresponding to a first user intent and a second sentence corresponding to a second user intent and subsequent to the first sentence using text data obtained from the voice command. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify, based on the obtained information, whether at least a portion of words in the first sentence are included in the second sentence. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine that the first user intent is related to the second user intent based on identifying that at least some of the words of the first sentence are included in the second sentence. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to perform a first function indicated by the first user intent based on the determination.The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute a second function indicated by the second user intent based on the execution of the first function as a result of the execution of the first function.

[0012] A method is provided. The method can be executed in an electronic device having a microphone. The method can include an operation of identifying a voice command through the microphone. Based on the identification, the method can include an operation of obtaining information about a first sentence corresponding to a first user intent and a second sentence corresponding to a second user intent and subsequent to the first sentence using text data obtained from the voice command. Based on the information obtained, the method can include an operation of identifying whether at least a portion of words in the first sentence are included in the second sentence. Based on the identification that at least a portion of words in the first sentence are included in the second sentence, the method can include an operation of determining that the first user intent is related to the second user intent. Based on the determination, the method can include an operation of executing a first function indicated by the first user intent. The method may include an operation of executing a second function indicated by the second user intention based on the execution of the first function, according to a result of the execution of the first function.

[0013] A non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device having a microphone, cause the electronic device to identify a voice command via the microphone. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain information about a first sentence corresponding to a first user intent and a second sentence corresponding to a second user intent and subsequent to the first sentence, based on the identification, using text data obtained from the voice command. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to identify, based on the obtained information, whether at least a portion of words in the first sentence are included in the second sentence. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to determine that the first user intent is related to the second user intent based on identifying that at least some of the words of the first sentence are included in the second sentence. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to execute a first function indicated by the first user intent based on the determination. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to execute a second function indicated by the second user intent based on the execution of the first function as a result of the execution of the first function.

[0014] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0015] Figure 1 illustrates an example of an environment including an electronic device that performs speech recognition.

[0016] Figure 2 is a simplified block diagram of an exemplary electronic device.

[0017] Figure 3 is a flowchart illustrating the operation of a second external electronic device that generates data that causes execution of a second function using the execution result of a first function.

[0018] Figures 4a and 4b illustrate exemplary operations of an electronic device for identifying a first request and a second request from an input.

[0019] FIG. 5 illustrates an exemplary operation of an electronic device displaying a screen related to the execution of a function.

[0020] Figures 6a and 6b illustrate exemplary operations of an electronic device displaying visual objects guiding execution of a function.

[0021] FIG. 7 illustrates an exemplary operation of an electronic device displaying a visual indication to indicate that a function is being executed based on natural language recognition.

[0022] Figure 8 illustrates examples of operations executed within a client, a capsule execution service, a first capsule, and a second capsule.

[0023] FIG. 9 illustrates an exemplary operation of an electronic device that obtains the execution result of a first function using a CFC (client function core) response.

[0024] Figure 10 illustrates examples of other operations executed within a client, a capsule execution service, a first capsule, and a second capsule.

[0025] FIG. 11 illustrates an exemplary operation of a second external electronic device transmitting command data causing execution of a second function.

[0026] FIG. 12 is a block diagram of an electronic device within a network environment according to various embodiments.

[0027] FIG. 13 is a block diagram illustrating an integrated intelligence (AI) system according to one embodiment.

[0028] FIG. 14 is a diagram showing a form in which relationship information between concepts and actions is stored in a database according to one embodiment.

[0029] FIG. 15 is a diagram illustrating a user terminal displaying a screen for processing voice input received through an intelligent app, according to one embodiment.

[0030] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0031] Figure 1 illustrates an example of an environment including an electronic device that performs speech recognition.

[0032] Referring to FIG. 1, the environment (150) may include an electronic device (100) and a user (120) of the electronic device (100). For example, the electronic device (100) may be used for natural language recognition. For example, the natural language recognition may be referred to as speech recognition. For example, the electronic device (100) may include a microphone (e.g., the microphone (209) of FIG. 2). For example, the electronic device (100) may receive an utterance (or audio, voice) of the user (120) through the microphone. For example, the electronic device (100) may identify a request included in the utterance based on receiving the utterance. For example, the request may be referred to as an intent or user intent. For example, the electronic device (100) may execute a function corresponding to the request based on identifying the request. For example, the electronic device (100) may include a display (e.g., display (208) of FIG. 2). For example, the electronic device (100) may display the execution result through the display (208) as the function is executed. For example, the electronic device (100) may include an intelligent assistant. For example, execution of the function based on recognition of the natural language may be performed through the intelligent assistant. The intelligent assistant may be referred to as a digital assistant, a voice assistant, a voice agent, or a voice agent.

[0033] For example, a user (120) may request the electronic device (100) to execute two or more functions. For example, an utterance of the user (120) may include two or more request(s). For example, the utterance of the user (120) may include multiple requests. For example, the electronic device (100) may execute functions corresponding to each of the requests based on identifying an utterance of the user (120) that includes two or more requests. For example, the electronic device (100) may identify a first request and a second request from the utterance. For example, the first request and the second request may be independent of each other. For example, the electronic device (100) may simultaneously execute the first request and a second request that is independent of the first request.

[0034] For example, the electronic device (100) can identify a first request and a second request from the user's (120) utterance. For example, the second request may be dependent on the first request. For example, the second request may be in a causal relationship with the first request. For example, the second request may be in a precedence relationship with the first request. For example, the electronic device (100) can use the execution result of the first request to execute a function for the second request. For example, execution of the function for the second request may require using the execution result for the first request. For example, based on identifying that the first and second requests included in one utterance are related to each other, the electronic device (100) can obtain the execution result of the function for the first request and then execute the function for the second request using the execution result. For example, the electronic device (100) may be required to obtain the execution result of the function for the first request in order to execute the function for the second request. For example, if the electronic device (100) executes the function for the second request first among the first and second requests in the order relationship, the execution of the function for the second request may fail. For example, if the electronic device (100) executes the function for the second request first, or executes the first function for the first request and the second function for the second request simultaneously, the execution of the second function for the second request may fail. For example, it may be impossible for the electronic device (100) to execute the second function for the second request without obtaining the execution result of the first function for the first request. For example, the electronic device (100) may be required to refrain from executing the function for the second request until it obtains the execution result of the function for the first request in order to execute the function for the second request.

[0035] For example, the electronic device (100) may be required to display a user interface (UI) object (e.g., the UI object (514) of FIG. 5) indicating the progress of the first request and the second request through a display (e.g., the display (208) of FIG. 2). For example, the electronic device (100) may be required to display, through the display, the UI object for confirming to the user (120) whether requests based on the user's (120) utterance are performed. For example, the user (120) may recognize through the UI object whether requests for the user's (120) utterance are received. For example, when the electronic device (100) executes functions for multiple requests, it may provide to the user (120) whether the first function for the first request has been executed through the UI object. For example, the user (120) may recognize through the UI object that the first function for the first request has been executed. For example, the user (120) may recognize that the execution of the first function for the first request has failed through the UI object. For example, if the electronic device (100) does not display the UI object, the user (120) cannot recognize the progress of the utterance. For example, the electronic device (100) may be required to display the UI object to confirm the execution of the first function and the second function.

[0036] For example, the electronic device (100) may include hardware components used to perform or execute the above operations. The hardware components are described and exemplified with reference to FIG. 2.

[0037] Figure 2 is a simplified block diagram of an exemplary electronic device.

[0038] Referring to FIG. 2, the electronic device (100) may include at least one processor (207), a communication circuit (205), a memory (206), and a microphone (209).

[0039] At least one processor (207) is stored in the memory (206).

[0040] The hardware component for processing data using instructions may include a hardware component for processing data. The hardware component for processing data may include a central processing unit (CPU) (e.g., including processing circuitry). The hardware component for processing data may include a graphics processing unit (GPU) (e.g., including processing circuitry). The hardware component for processing data may include a display processing unit (DPU) (e.g., including processing circuitry). The hardware component for processing data may include a neural processing unit (NPU) (e.g., including processing circuitry).

[0041] At least one processor (207) may include one or more cores. For example, at least one processor (207) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core.

[0042] At least one processor (207) can control the operations of the electronic device (100) by executing instructions stored in the memory (206). For example, the processor (207) may correspond to a plurality of processors that collectively perform a plurality of operations by dividing them among the processors.

[0043] The memory (206) may include hardware components for storing data and / or instructions input to and / or output from at least one processor (207). The memory (206) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multimedia card (EMMC).

[0044] The communication circuit (205) may include hardware components for supporting transmission and / or reception of signals between the electronic device (100) and an external electronic device. The communication circuit (205) may include, for example, at least one of a modem (modulator and demdulator), an antenna, and an optical / electronic (O / E) converter. The communication circuit (205) may support transmission and / or reception of signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), zigbee, long term evolution (LTE), and 5G new radio (NR).

[0045] The microphone (209) may include a hardware component for supporting the reception of audio (e.g., the voice of a user (120)) or an audio signal. The microphone (209) may be used to obtain audio data or an audio signal by receiving audio, voice, speech, or utterance.

[0046] Based on receiving a first input for natural language recognition by at least one processor (207), a UI object related to said natural language recognition (e.g., a UI object (514) of FIG. 5) may be displayed on a part of a display (208). For example, the display (208) may be used to display said UI object. For example, a microphone (209) may receive said first input for natural language recognition. For example, said first input may include the utterance of a user (120). While said UI object is being displayed, at least one processor (207) may receive a second input based on natural language. For example, a microphone (209) may receive a second input based on natural language. For example, said second input may include the utterance of a user (120). For example, at least one processor (207) can identify, from the second input, a first request for executing a first function and a second request for executing a second function based on the result of executing the first function. For example, at least one processor (207) can display a first screen (e.g., the first screen (512) of FIG. 5) related to the execution of the first function, together with the UI object displayed on a part of the display (208). For example, the display (208) can be used to display the first screen. For example, when the result of executing the first function is displayed on the first screen, at least one processor (207) can display a visual object (e.g., the visual object (612) of FIG. 6a) within the UI object that guides the execution of the second function using the result of executing the first function. For example, the display (208) can be used to display the visual object. For example, at least one processor (207) can execute the second function using the result of executing the first function.

[0047] FIG. 3 is a flowchart illustrating the operation of a second external electronic device that generates data that causes the execution of a second function using the execution result of a first function. This method may be executed by the electronic device (100) illustrated in FIG. 2 or at least one processor (207) of the electronic device (100). In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0048] Referring to FIG. 3, in operation 310, a second external electronic device (e.g., the second external electronic device (1120) of FIG. 11) may receive, from the first external electronic device (e.g., the first external electronic device (1110) of FIG. 11), information about a first sentence corresponding to a first user intent and a second sentence corresponding to a second user intent and following the first sentence. For example, the information may be acquired by recognizing a voice command. For example, the second external electronic device may receive the information from the first external electronic device executing a trained model configured to perform natural language processing (e.g., the pre-trained model (420) of FIG. 4A). For example, the first external electronic device may include a model configured to perform natural language processing.

[0049] In one embodiment, the first external electronic device and the second external electronic device may be the same. For example, natural language recognition for audio data for a voice command (e.g., audio data (1102) of FIG. 11) may be performed by the first external electronic device and / or the second external electronic device. For example, the second external electronic device, which is identical to the first external electronic device, may perform text-to-speech (TTS) for the audio data for the voice command. For example, the second external electronic device, which is identical to the first external electronic device, may generate information for a first sentence and a second sentence from the audio data.

[0050] According to one embodiment, the electronic device (100) may display a UI object (e.g., UI object (514) of FIG. 5) related to natural language recognition on a portion of the display (208) based on receiving a first input for natural language recognition. For example, the electronic device (100) may receive the first input including a speech of a user (120). For example, the electronic device (100) may receive the speech of the user (120) through a microphone (209). For example, the electronic device (100) may receive the first input for natural language recognition through a microphone (209).

[0051] According to one embodiment, the electronic device (100) may include an input mechanism (not shown). For example, the input device may include a power button. For example, the input device may include a volume button. For example, the electronic device (100) may receive a first input for natural language recognition through the input device. For example, the first input may include a press input.

[0052] In operation 320, a second external electronic device (e.g., the second external electronic device (1120) of FIG. 11) may identify whether at least some of the words of the first sentence are included in the second sentence based on the information received in operation 310.

[0053] In one embodiment, the second external electronic device may determine that the first user intent is not related to the second user intent based on identifying that words in the first sentence are not included in the second sentence. For example, the second external electronic device may generate other data for a command to execute a first function corresponding to the first user intent and to execute a second function corresponding to the second user intent independently of execution of the first function, based on the determination that the first user intent is not related to the second user intent. For example, the second external electronic device may transmit the other data to the electronic device (100).

[0054] According to one embodiment, the electronic device (100) may receive a second input based on natural language while a UI object (e.g., the UI object (514) of FIG. 5) is displayed. For example, the electronic device (100) may receive a second input based on natural language while a UI object (e.g., the UI object (514) of FIG. 5) indicating that a function for natural language recognition is executed is displayed. For example, the second input may include an utterance (or voice) of the user (120). For example, the electronic device (100) may obtain a text corresponding to the second input based on receiving the second input. For example, the electronic device (100) may obtain a text representing the second input based on receiving the second input. For example, an operation of obtaining the text from the second input may be described as a speech-to-text (STT) operation. For example, the electronic device (100) can perform an STT operation using ASR (auto speech recognition). For example, the user (120) can recognize that natural language recognition is being performed through the UI object.

[0055] In operation 330, a second external electronic device (e.g., the second external electronic device (1120) of FIG. 11) may determine that the first user intent is related to the second user intent based on identifying that at least some of the words of the first sentence are included in the second sentence.

[0056] In one embodiment, the second external electronic device may determine that the first user intent is related to the second user intent using a trained model (e.g., the pre-trained model (420) of FIG. 4A ) configured to receive the first sentence and the second sentence. For example, the second external electronic device may use the trained model to identify that at least some of the words in the first sentence are included in the second sentence.

[0057] In operation 340, a second external electronic device (e.g., the second external electronic device (1120) of FIG. 11) may generate data for a command to execute a first function indicated by the first user intent and to execute a second function indicated by the second user intent based on a determination that the first user intent is related to the second user intent, and as a result of executing the first function.

[0058] According to one embodiment, the second external electronic device may generate the data that causes the electronic device (100) to display a user interface (UI) object based on the execution of a first function corresponding to the first user intention. For example, the electronic device (100) may display the UI object based on receiving the data. For example, the electronic device (100) may display the UI object through the display (208) along with the execution of the first function.

[0059] According to one embodiment, the second external electronic device may generate data that causes the electronic device (100) to display a UI object indicating that the second function will be executed using the result of the first function. For example, the second external electronic device may transmit the data to the electronic device (100). For example, the electronic device (100) may, based on receiving the data, display a UI object indicating that the second function will be executed using the result of the execution of the first function.

[0060] According to one embodiment, the second external electronic device may generate data that causes the electronic device (100) to display a UI object capable of receiving a user input indicating a cancellation of execution of the second function. For example, the second external electronic device may transmit the data to the electronic device (100). For example, the electronic device (100) may receive the data. For example, based on receiving the data, the electronic device (100) may display the UI object through the display (208). For example, while displaying the UI object, the electronic device (100) may receive a user input indicating a cancellation of execution of the second function. For example, the electronic device (100) may indicate a cancellation of execution of the second function and receive a user input for the UI object. For example, based on receiving the user input, the electronic device (100) may cancel or stop execution of the second function.

[0061] In operation 350, a second external electronic device (e.g., the second external electronic device (1120) of FIG. 11) may transmit data for the command to the electronic device (100). For example, the transmission of the data is described and illustrated in more detail with reference to FIG. 11.

[0062] FIG. 11 illustrates an exemplary operation of a second external electronic device transmitting command data causing execution of a second function.

[0063] Referring to FIG. 11, the electronic device (100) can receive a user's (120) speech through the microphone (209). For example, the electronic device (100) can identify a voice command through the microphone (209). For example, the electronic device (100) can identify a second input indicating the voice command. For example, the electronic device (100) can transmit audio data (1102) for the voice command to a first external electronic device (1110) through the communication circuit (205). For example, the audio data (1102) can be audio data corresponding to the user's (120) speech. For example, the audio data (1102) can be audio data corresponding to the user's (120) voice command. For example, the voice command can include a first user intention and a second user intention.

[0064] The first external electronic device (1110) can receive audio data (1102) from the electronic device (100). For example, the first external electronic device (1110) can be used to obtain a first sentence and a second sentence following the first sentence using the audio data (1102). For example, the first external electronic device (1110) can obtain information about the first sentence and the second sentence using the audio data (1102). For example, the first external electronic device (1110) can obtain sentence information (1104) from the audio data (1102). For example, the first external electronic device (1110) can obtain a first sentence corresponding to a first user intention and a second sentence corresponding to a second user intention by providing the audio data (1102) to a trained model (e.g., a pre-trained model (420) of FIG. 4A). However, the present invention is not limited thereto. For example, the first external electronic device (1110) can obtain text data corresponding to the audio data (1102) by performing TTS using the audio data (1102). For example, the first external electronic device (1110) can obtain a first sentence corresponding to a first user intention and a second sentence corresponding to a second user intention by providing the text data to the pre-trained model. For example, the first external electronic device (1110) can obtain sentence information (1104) for the first sentence and the second sentence by providing the text data to the pre-trained model. For example, the first external electronic device (1110) can transmit the sentence information (1104) to the second external electronic device (1120).

[0065] The second external electronic device (1120) can receive sentence information (1104) from the first external electronic device (1110). For example, the second external electronic device (1120) can be used to generate data that causes the first function and the second function to be sequentially executed within the electronic device (100). For example, the second external electronic device (1120) can identify whether a first function corresponding to a first sentence and a second function corresponding to a second sentence are related. For example, the second external electronic device (1120) can identify whether a first sentence corresponding to a first user intention and a second sentence corresponding to a second user intention are related. For example, the second external electronic device (1120) can identify whether at least some of the words of the first sentence are included in the second sentence. For example, the second external electronic device (1120) may determine that the first sentence and the second sentence are related based on a determination that at least some of the words of the first sentence are included in the second sentence. For example, the second external electronic device (1120) may generate command data (1106) that causes the electronic device (100) to execute a first function corresponding to the first sentence and then execute a second function corresponding to the second sentence based on a determination that the first sentence and the second sentence are related. For example, the second external electronic device (1120) may generate data for a command that causes the execution of the first function indicated by the first sentence and the execution of the second function indicated by the second sentence based on a result of the execution of the first function. However, the present invention is not limited thereto. For example, the second external electronic device (1120) may generate data that causes the electronic device (100) to execute the second function based on the elapsed time after the execution of the first function is completed. For example, the second external electronic device (1120) can transmit the data to the electronic device (100).For example, a second external electronic device (1120) can transmit command data (1106) to the electronic device (100).

[0066] For example, the electronic device (100) may receive command data (1106) through the communication circuit (205). For example, based on receiving the command data (1106), the electronic device (100) may execute a first function and a second function corresponding to a first user intention and a second user intention included in the user's (120) utterance, respectively. For example, based on receiving the command data (1106), the electronic device (100) may execute a first function. For example, based on the completion of execution of the first function, the electronic device (100) may execute a second function according to the result of execution of the first function. For example, the electronic device (100) may display a UI object (e.g., UI object (514) of FIG. 5) through the display (208) along with execution of the first function. For example, the electronic device (100) may receive a user input to stop or cancel execution of the second function while displaying the UI object.

[0067] Referring to FIG. 11, the first external electronic device (1110) and the second external electronic device (1120) are distinguished, but the embodiment is not limited thereto. For example, the first external electronic device (1110) and the second external electronic device (1120) may be the same. For example, the electronic device (100) and the first external electronic device (1110) may be the same. For example, the electronic device (100) and the second external electronic device (1120) may be the same. For example, the electronic device (100), the first external electronic device (1110), and the second external electronic device (1120) may be the same.

[0068] According to one embodiment, the electronic device (100) may display a screen related to the execution of the first function together with the UI object displayed on a portion of the display (208) based on identifying a first request for executing a first function and a second request for executing a second function based on a result of executing the first function from a received second input while a UI object (e.g., UI object (514) of FIG. 5) is displayed. For example, the electronic device (100) may identify a first request corresponding to the first function from the second input and a second request corresponding to the second function to be executed based on a result of executing the first function. The identification of the first request and the second request is described and illustrated in more detail with reference to FIGS. 4A and 4B.

[0069] Figures 4a and 4b illustrate exemplary operations of an electronic device for identifying a first request and a second request from an input.

[0070] Referring to FIG. 4A, the electronic device (100) can receive a user's (120) utterance (410) through the microphone (209). For example, the electronic device (100) can provide the utterance (410) to a pre-trained model (420). For example, the electronic device (100) can obtain a first request (430) and a second request (440) from the utterance (410) by providing the utterance (410) to the pre-trained model (420). For example, the request can be described as an intent that the user (120) desires to be executed by the electronic device (100). For example, the request can be referred to as an intent or user intention. For example, the pre-trained model (420) can include a large language model (LLM). For example, the pre-trained model (420) may include a model trained to utilize an artificial neural network. For example, the pre-trained model (420) may include a model trained to recognize natural language.

[0071] LLM refers to an artificial neural network-based language model that has learned from a large amount of text data through pre-training. LLMs can contain significantly more parameters (e.g., over 10 billion) than typical language models. LLMs are a type of machine learning model used in natural language processing. They can be trained on large amounts of text data and used to make predictions about new text data. LLMs can be used for tasks such as natural language understanding, sentence generation, translation, grammatical error correction, and summarization.

[0072] The electronic device (100) can obtain a first request (430) and a second request (440) from an utterance (410) using a pre-trained model (420). For example, the utterance (410) may include the content "Erase the shadow from this photo and save it to a note." For example, the electronic device (100) can divide the utterance (410) into a first request (430) and a second request (440). For example, the operation of dividing the utterance (410) into the first request (430) and the second request (440) may be referred to as a rewrite operation. For example, the pre-trained model (420) can perform the rewrite operation by performing grammar and syntax analysis of the utterance (410). Referring to FIG. 4A, the first request (430) is illustrated as "Erase the shadow from this photo," but is not limited thereto. For example, the first request (430) may include a request contained in the utterance (410). For example, the first request (430) may vary depending on the content of the utterance (410).

[0073] For example, based on receiving the second input, the electronic device (100) can obtain a first sentence corresponding to the first request (430) and a second sentence corresponding to the second request (440) by providing the second input to a pre-trained model (420). For example, based on determining that the first part (435) of the first sentence and the second part (445) of the second sentence are the same, the electronic device (100) can identify a first request (430) for executing a first function and a second request (440) for executing a second function based on the execution result of the first function. For example, based on receiving the second input, the electronic device (100) can obtain a first natural language sentence representing the first request (430) and a second natural language sentence representing the second request (440) by providing the second input to a pre-trained model (420). For example, the second sentence may be a sentence next to the first sentence. For example, the electronic device (100) may determine that the second function is executed based on the result of executing the first function, based on identifying at least one identical word from the first natural language sentence and the second natural language sentence. For example, the natural language sentence may be referred to as a sentence. For example, the natural language sentence may be referred to as a text portion.

[0074] The electronic device (100) can identify whether the first request (430) and the second request (440) are in a causal relationship or a precedence relationship. For example, the electronic device (100) can identify whether the execution result of the first request (430) is required to execute a second function for the second request (440). For example, the electronic device (100) can identify whether the first request (430) and the second request (440) are in a causal relationship (or a precedence relationship) by identifying whether a part of the first request (430) and a part of the second request (440) are identical to each other. For example, the electronic device (100) may determine that the first request (430) and the second request (440) are in a causal relationship based on a first portion (435) of the first request (430) being (substantially) identical to a second portion (445) of the second request (440). For example, the electronic device (100) may determine that the result of executing the first function for the first request (430) is used to execute the second function for the second request (440) based on a determination that the first portion (435) of the first request (430) is (substantially) identical to a second portion (445) of the second request (440). For example, the electronic device (100) may determine the object of the first request (430) to be the first portion (435). For example, the electronic device (100) can determine the object of the second request (440) as the second part (445).

[0075] Below, exemplary operations of an electronic device (100) for identifying a first request and a second request from an input representing a user's (120) speech are illustrated.

[0076] Request 1 Request 2 Remaster this photo and share it with Kim Samsung Remaster this photo and share it with Kim Samsung Color this photo and share it with Mom via text message Color this photo and share it with Mom via text message Remove the shadows from this photo and send it to Ryu Gi-hyeok via message Remove the shadows from this photo and send it to Ryu Gi-hyeok via message Remove the light reflections from this photo and share it via Samsung Note Remove the light reflections from this photo and share it via Samsung Note Apply a 24-hour time-lapse to this photo and send it via Note Apply a 24-hour time-lapse to this photo and send it via Note Apply a long exposure effect to this photo and send it to Grandpa Apply a long exposure effect and send it to Grandpa

[0077] Referring to Table 1, examples of identifying requests from utterances about a gallery or image can be described. For example, the requests may be related to each other. For example, the execution of a second function for a second request may be based on the result of the execution of a first function for the first request.

[0078] Speech Request 1 Request 2 Summarize the current page and share it with Kim Samsung Summarize the current page Share the result with Kim Samsung Translate the current page and save it to Samsung Notes Translate the current page Save the result to Samsung Notes Correct typos on the current page and send it to Hong Gil-dong via email Correct typos on the current page Send the result to Hong Gil-dong via email Check the spelling and share it to my smartphone via the Quick Share app Check the spelling Share the result to my smartphone using the Quick Share app

[0079] Referring to Table 2, an example of identifying requests from utterances in a note software application can be described. For example, the requests may be related to each other. For example, the execution of a second function for a second request may be based on the results of the execution of a first function for the first request.

[0080] Request for ignition 1 Request for ignition 2 Take a video and share it with Kim Sam-seong Take a video Share the video you just shot with Kim Sam-seong Take a video at 0.5x and send it to Kim Sam-seong's notebook Take a video at 0.5x and send the video you just shot to your notebook Take a video in wide-angle and share it with your father Take a video in wide-angle Share the video you just shot with your father Will you turn on the flash and send it to Kim Du-seok via text Take a video with the flash Will you send the video you just shot to Kim Du-seok via text

[0081] Referring to Table 3, an example of identifying requests from utterances to a camera software application may be described. For example, the requests may be related to each other. For example, execution of a second function for a second request may be based on the result of execution of a first function for a first request. Referring to FIG. 4B, the electronic device (100) may include a pre-trained model (420), a multi-intent task manager (450), a runtime manager (452), a multi-intent relationship manager (454), and an intent condition memory (456). However, the present invention is not limited thereto. For example, the first external electronic device (1110) may include at least one of a pre-trained model (420), a multi-intent task manager (450), a runtime manager (452), a multi-intent relationship manager (454), and an intent condition memory (456). For example, the second external electronic device (1120) may include at least one of a pre-trained model (420), a multi-intent task manager (450), a runtime manager (452), a multi-intent relationship manager (454), and an intent condition memory (456).

[0082] A pre-trained model (420) can be used to identify requests from a user's (120) utterance. For example, the pre-trained model (420) can identify multiple requests from the utterance by performing a rewrite operation. The requests can be referred to as intents.

[0083] The multi-intention task manager (450) can obtain requests from the pre-trained model (420). For example, the multi-intention task manager (450) can set priorities for requests based on the order provided by the pre-trained model (420). For example, the multi-intention task manager (450) can sequentially determine priorities for the obtained requests. For example, the multi-intention task manager (450) can determine priorities for each request and then transfer them to the intent condition memory (456). For example, the intent condition memory (456) can be used to store the transferred requests. For example, the multi-intention task manager (450) can obtain information on the execution time intended by the user (120) from the execution time manager (452). For example, the multi-intention task manager (450) can store the information on the execution time in the intent condition memory (456). For example, the multi-intent task manager (450) may transmit the relationship data for each of the multiple requests to the multi-intent relationship manager (454) to evaluate the relationship. For example, the multi-intent task manager (450) may store the relationship data for each of the multiple requests obtained from the multi-intent relationship manager (454) in the intent condition memory (456). For example, the multi-intent task manager (450) may use the relationship data and information about the execution time to determine whether each of the functions corresponding to each of the requests is executable.

[0084] The multi-intent task manager (450) can determine whether a function corresponding to a request is executable based on the completion of relationship analysis for each request and the storage of relationship data in the intent condition memory (456). For example, the multi-intent task manager (450) can immediately execute the first function for the first request (430) based on the determination that the first function can be executed. For example, the multi-intent task manager (450) can execute the second function of the second request (440) based on the information indicating that the execution of the first function is completed being updated in the intent condition memory (456) after the execution of the first function is completed. For example, the multi-intent task manager (450) can determine the start time of the execution of the second function using information (e.g., time interval) about the execution time obtained from the execution time manager (452). For example, if the execution timing mode is automatic mode, the multi-intent task manager (450) can execute the first function and the second function simultaneously based on a determination that the execution of the second function is not required based on the execution result of the first function. For example, if the execution timing mode is automatic mode, the multi-intent task manager (450) can execute the second function upon completion of the execution of the first function based on a determination that the execution of the second function is required based on the execution result of the first function. For example, if the execution timing mode is manual mode, the multi-intent task manager (450) can execute the second function upon the elapse of a specified time (e.g., a time interval) after the execution of the first function is completed.

[0085] The multi-intent relationship manager (454) can obtain or generate relationship data between multiple requests by analyzing the requests.

[0086] The execution time manager (452) can receive information about the execution timing between functions for each of multiple requests from the user (120). The execution time manager (452) can transmit the information to the multi-intent task manager (450).

[0087] According to one embodiment, the electronic device (100) may display a first screen (512) related to the execution of the first function, together with a UI object (e.g., the UI object (514) of FIG. 5) displayed on a portion of the display (208), based on identifying a second request (430) for executing a second function from a second input based on natural language and a second request (440) for executing a second function based on the execution result of the first function. For example, the electronic device (100) may display a UI object (e.g., the UI object (514) of FIG. 5) indicating that a function for natural language recognition is being executed, together with the first screen (e.g., the first screen (512) of FIG. 5) related to the execution of the first function. The display of the first screen and the UI object is described and illustrated in more detail with reference to FIG. 5.

[0088] FIG. 5 illustrates an exemplary operation of an electronic device displaying a screen related to the execution of a function.

[0089] Referring to FIG. 5, a state (510) can be described as a state in which a first screen (512) and a UI object (514) are displayed. For example, the electronic device (100) can display a first screen (512) related to the execution of a first function on the display (208). For example, the electronic device (100) can display the first screen (512) to execute a first function corresponding to a first request (430). For example, the first screen (512) can include a screen on which the first function is executed. For example, the user (120) can recognize that the first function for the first request (430) is executed through the first screen (512) displayed on the display (208).

[0090] For example, the electronic device (100) may display a UI object (514) while displaying a first screen (512). For example, the electronic device (100) may display the first screen (512) while displaying the UI object (514). For example, the electronic device (100) may display a UI object (514) for inputting speech based on receiving the first input. For example, the electronic device (100) may receive a second input while the UI object (514) is displayed on the display (208). For example, the electronic device (100) may display text (516) regarding the second input within the UI object (514) based on receiving the second input. For example, the text (516) may be described as text obtained by performing STT on the speech of the user (120). For example, the text (516) may correspond to the user's (120) utterance. For example, based on receiving the second input for the user's (120) utterance, the electronic device (100) may display text within the UI object (514) indicating the second input for confirming whether the user's (120) utterance (410) has been received.

[0091] The state (520) can be described as a screen on which a first function for a first request (430) is executed. For example, the electronic device (100) can identify the first request (430) and the second request (440) from the second input, and then execute the first function corresponding to the first request (430). For example, the electronic device (100) can execute the first function while the first screen (512) is displayed. For example, the user (120) can recognize that the first function is executed through the first screen (512). For example, the electronic device (100) can remove a shadow from a photo in the gallery based on the first request (430) of the user (120). For example, the electronic device (100) can execute the first function of removing a shadow from a photo in the gallery.

[0092] The state (530) can be described as a state in which the execution of the first function is completed. For example, the electronic device (100) can display a visual object (532) on the display (208) based on the completion of the execution of the first function. For example, the electronic device (100) can display a visual object (532) to indicate that the execution of the first function is completed. For example, the electronic device (100) can display a visual object (532) to indicate that the shadow of a photo specified by the user (120) has been removed.

[0093] According to one embodiment, the electronic device (100) can identify a voice command through the microphone (209). For example, the electronic device (100) can identify a voice command representing a user's (120) utterance through the microphone (209). For example, the electronic device (100) can transmit information about the voice command to a first external electronic device (1110) based on the identification. For example, the electronic device (100) can transmit information about the voice command (e.g., audio data (1102) of FIG. 11) to the first external electronic device (1110) through the communication circuit (205) based on the identification. For example, the information can include information about an audio signal representing the voice command. For example, the electronic device (100) may receive data related to the voice command (e.g., command data (1106) of FIG. 11) from the second external electronic device (1120) through the communication circuit (205). For example, the data may cause the electronic device (100) to execute a function corresponding to a user intention included in the voice command. For example, the electronic device (100) may identify, from the data, a command for executing a first function corresponding to a first user intention included in the voice command and a second function corresponding to a second user intention included in the voice command. For example, the electronic device (100) may identify, from the data, a command for executing the first function and a command for executing the second function based on a result of executing the first function. For example, the electronic device (100) may execute the first function based on identifying the command. For example, the electronic device (100) may display a UI object (e.g., UI object (514) of FIG. 5) indicating the result of execution of the first function and whether or not the second function is executed, through the display (208).For example, the electronic device (100) may display a UI object through the display (208) that indicates whether to execute the second function along with the result of executing the first function.

[0094] For example, the electronic device (100) may execute a second function based on a displayed result of the first function, based on a first user input to the UI object indicating that a second function is to be executed. For example, the electronic device (100) may receive a first user input for executing the second function. For example, the electronic device (100) may receive a first user input to the UI object. For example, the electronic device (100) may execute the second function based on receiving the first user input.

[0095] For example, the electronic device (100) may refrain from executing the second function based on a second user input to the UI object indicating that execution of the second function should be canceled. For example, the electronic device (100) may receive a second user input to cancel execution of the second function. For example, the electronic device (100) may cancel, refrain from, or stop execution of the second function based on receiving the second user input.

[0096] According to one embodiment, the electronic device (100) may display an execution screen including a result of execution of the first function, and a pop-up window overlapping the execution screen and including the UI object. For example, the electronic device (100) may display the pop-up window while displaying an execution screen regarding the result of execution of the first function. For example, the pop-up window may include a result of execution of the first function. For example, the pop-up window may include text indicating that execution of the first function is complete.

[0097] According to one embodiment, the electronic device (100) may, based on receiving a second user input for executing a second function, change the execution screen for the first function to another execution screen related to the second function while the pop-up window is maintained. For example, based on receiving the second user input, the electronic device (100) may, while displaying the pop-up window, switch the screen displayed through the display (208) from the execution screen for the first function to another execution screen related to the execution of the second function. For example, the electronic device (100) may, based on the execution of the first function, display the execution screen of the first software application. For example, the execution screen may be a screen for the first software application. For example, the first software application may be a software application for executing the first function. For example, the electronic device (100) may, based on the first user input, display another execution screen of the second software application for executing the second function. For example, the second software application may be a software application for executing a second function. For example, the other execution screen may be a screen of the second software application related to executing the second function.

[0098] In one embodiment, the electronic device (100) may stop displaying the pop-up window based on the execution of the second function. For example, by executing the second function, the electronic device (100) may switch from a screen including the pop-up window to a screen related to the second function.

[0099] According to one embodiment, the electronic device (100) can identify a command from data indicating a command to execute a second function based on a result of execution of a first function. For example, the electronic device (100) can receive the data from a second external electronic device (e.g., the second external electronic device (1120) of FIG. 11) that receives information including a first sentence indicating to execute a first function and a second sentence indicating to execute a second function from a first external electronic device (e.g., the first external electronic device (1110) of FIG. 11). For example, the second external electronic device can receive information (e.g., sentence information (1104) of FIG. 11) including a first sentence for executing the first function and a second sentence indicating to execute the second function, from the first external electronic device. For example, the first external electronic device can be configured to identify the information including a first sentence and a second sentence following the first sentence based on recognition of a voice command. For example, the first external electronic device can identify the information using a pre-trained model (420). For example, the first external electronic device can obtain information including a first sentence and a second sentence corresponding to each of the first user intention and the second user intention included in the voice command using the pre-trained model (420).

[0100] According to one embodiment, the electronic device (100) can identify a voice command through the microphone (209). For example, based on the identification of the voice command, the electronic device (100) can obtain information about a first sentence corresponding to a first user intent and a second sentence corresponding to a second user intent and following the first sentence using text data obtained from the voice command. For example, the text data can be obtained by performing TTS on an utterance for the voice command. For example, the text data can represent an utterance for the voice command. For example, the information can be obtained based on the text data provided to a pre-trained model (420). For example, the pre-trained model (420) can generate the first sentence and the second sentence from the text data. For example, based on obtaining the information, the electronic device (100) can identify whether at least some of the words of the first sentence are included in the second sentence.

[0101] According to one embodiment, the electronic device (100) can identify a relationship between the first sentence and the second sentence using a language model. For example, the electronic device (100) can identify whether the first sentence is related to the second sentence by providing the first sentence and the second sentence to the language model.

[0102] The electronic device (100) may determine that the first user intent is related to the second user intent based on identifying that at least some of the words of the first sentence are included in the second sentence. For example, the electronic device (100) may execute the first function indicated by the first user intent based on the determination. For example, the electronic device (100) may execute the second function indicated by the second user intent based on the execution of the first function, depending on the result of the execution of the first function.

[0103] According to one embodiment, the electronic device (100) may, based on identifying a second request (440) for executing a second function based on the first request (430) and the execution result of the first function from the second input, when the execution result of the first function is displayed on the first screen (512), display a visual object (e.g., a visual object (612) of FIG. 6A) within a UI object (e.g., a UI object (514) of FIG. 5) that guides execution of the second function using the execution result of the first function. For example, the electronic device (100) may include a microphone (209). For example, the electronic device (100) may, based on a second input detected from a voice signal received through the microphone (209), display a text converted from the voice signal received through the microphone (209) together with the UI object. The display of the visual object is described and exemplified in more detail with reference to FIGS. 6A and 6B.

[0104] Figures 6a and 6b illustrate exemplary operations of an electronic device displaying visual objects guiding execution of a function.

[0105] Referring to FIG. 6A, a state (610) can be described as a state in which a visual object (612) is displayed within a UI object (514). For example, based on receiving a second input, the electronic device (100) can display text (516) indicating the second input within the UI object (514). For example, when the execution result of the first function is displayed on the first screen (512), the electronic device (100) can display a visual object (612) within the UI object (514) that guides execution of the second function using the execution result of the first function. For example, after execution of the first function is completed, the electronic device (100) can display a visual object (612) within the UI object (514) that announces execution of the second function using the execution result of the first function. For example, a user (120) may recognize that a function is being executed based on the user's (120) utterance (410) from a UI object (514) and a visual object (612).

[0106] For example, when the electronic device (100) displays the visual object (612) within the UI object (514), it may further display an executable object (614) to stop execution of the second function. For example, the electronic device (100) may display the executable object (614) while displaying the visual object (612) within the UI object (514). For example, the electronic device (100) may receive an input for the executable object (614). For example, the electronic device (100) may stop execution of the second function based on receiving the input for the executable object (614). For example, the user (120) may cause the electronic device (100) to stop execution of the second function by providing an input for the executable object (614).

[0107] The electronic device (100) may display a visual object (612) to confirm the execution of the second function within the UI object (514). For example, the electronic device (100) may display a visual representation (635) indicating that the execution of the second function is completed within the UI object (514). The display of the visual representation (635) is described and exemplified in more detail with reference to FIG. 6B.

[0108] Referring to FIG. 6B, state (620) can be described as another exemplary state in which text (516) and visual objects (612) are displayed within the UI object (514). For example, the electronic device (100) can receive a second input to add color to a photo being displayed through the display (208) and share it with Kim Samsung using a software application for a message. For example, the electronic device (100) can identify a first request (430) and a second request (440) from the second input. For example, the electronic device (100) can display a screen related to a first function for the first request (430) on the display (208). For example, the electronic device (100) can display a UI object (514) on a portion of the display (208) while the screen is being displayed. For example, the electronic device (100) may display text (516) for the second input within the UI object (514). For example, the electronic device (100) may receive a voice signal through the microphone (209). For example, the electronic device (100) may display text (516) converted from the voice signal received through the microphone (209) along with the UI object (514) based on the second input detected from the voice signal received through the microphone (209).

[0109] For example, while the electronic device (100) displays the UI object (514), the electronic device (100) may display a visual object (612) for confirming the execution of the second function. For example, while the electronic device (100) displays the visual object (612) for confirming the execution of the second function, the electronic device (100) may receive a third input for authorizing the execution of the second function. For example, based on receiving the third input, the electronic device (100) may execute the second function using the execution result of the first function. For example, in order to prevent the second function from being executed for a second request (440) that is different from the intention of the user (120), the electronic device (100) may display a visual object (612) for confirming the execution of the second function. For example, based on receiving the third input for the visual object (612), the electronic device (100) may execute the second function.

[0110] For example, while displaying a visual object (612) for confirming execution of a second function, the electronic device (100) may receive a fourth input that disallows execution of the second function. For example, based on receiving the fourth input, the electronic device (100) may refrain from, bypass, or block execution of the second function using the execution result of the first function.

[0111] State (630) can be described as a state in which execution of the second function is completed based on receiving the third input. For example, the electronic device (100) can display a visual object (612) to confirm execution of the second function. For example, while displaying the visual object (612), the electronic device (100) can execute the second function using the execution result of the first function based on receiving a third input that authorizes execution of the second function. For example, the electronic device (100) can display a visual representation (635) within the UI object (514) based on completion of execution of the second function. For example, after execution of the second function is completed, the electronic device (100) can display a visual representation (635) indicating completion of execution of the second function within the UI object (514). For example, the user (120) may recognize that a second function identical to the user's (120) intention is being executed through a visual representation (635).

[0112] According to one embodiment, the electronic device (100) may execute the second function of the second request (440) using the execution result of the first function of the first request (430). For example, the electronic device (100) may execute the second function using the execution result of the first function based on identifying the first request (430) for executing the first function from the second input and the second request (440) for executing the second function based on the execution result of the first function. For example, the electronic device (100) may display a UI object (514) that can check the execution status of the first function and the second function through the display (208). For example, the electronic device (100) may provide convenience to the user (120) by displaying the UI object (514). For example, the electronic device (100) may improve convenience to the user (120) by displaying the UI object (514). For example, the electronic device (100) may cause the user (120) to select whether to execute the second function by displaying a UI object (514).

[0113] According to one embodiment, the electronic device (100) may execute a second function based on the elapsed time after the execution of the first function based on the first request (430) is completed. For example, the electronic device (100) may display an interface (not shown) for setting the specified time through the display (208). For example, while displaying the interface, the electronic device (100) may receive a user input for determining the specified time. For example, based on receiving the user input, the electronic device (100) may determine the time between the completion of execution of the first function and the execution of the second function. For example, before receiving the first input, the electronic device (100) may receive a user input regarding a time interval (or a specified time) between a first time point at which execution of the first function is completed and a second time point at which the second function is to be executed. For example, the electronic device (100) may receive a user input for determining a time interval between the completion of execution of the first function and the start of execution of the second function before receiving the first input. For example, the electronic device (100) may execute the second function based on the elapsed time of the time interval (e.g., 5 seconds) after the completion of execution of the first function. For example, the electronic device (100) may receive a user input for determining a time interval between the completion time of execution of the first function and the start time of execution of the second function before receiving the first input. For example, the electronic device (100) may execute the second function when the time interval elapses from the completion time of execution of the first function.

[0114] According to one embodiment, the electronic device (100) may display a UI object (514) together with other visual objects representing a time interval on a portion of the display (208) determined based on received user input. For example, the UI object (514) may be displayed together with other visual objects representing a time interval on a portion of the display (208) determined based on received user input. For example, the location of the UI object (514) may be changeable. For example, the location at which the UI object (514) is displayed on a portion of the display (208) may be determined based on user input. For example, the UI object (514) may be movable.

[0115] According to one embodiment, the electronic device (100) may receive another input based on natural language while the UI object (514) is displayed. For example, the other input may include an utterance of the user (120). For example, the other input may include an utterance of the user (120) containing a single request. For example, the electronic device (100) may identify a single request for executing a single function from the other input. For example, based on identifying the single request from the third input, the electronic device (100) may display another screen related to the single function without displaying the UI object (514). For example, the other screen may be described as a screen related to executing the single function.

[0116] For example, while displaying the first screen (512), the electronic device (100) may display a visual indication (e.g., visual indication (715) of FIG. 7) indicating that the first function is executed by natural language recognition. For example, the display of the visual indication is described and exemplified in more detail with reference to FIG. 7.

[0117] FIG. 7 illustrates an exemplary operation of an electronic device displaying a visual indication to indicate that a function is being executed based on natural language recognition.

[0118] Referring to FIG. 7, a state (710) may be described as a state that displays a visual indication (715) indicating that the state is being executed by natural language recognition. The electronic device (100) may display the visual indication (715) through the display (208) while displaying a first screen related to the execution of a first function based on a first request (430). For example, the electronic device (100) may display a visual indication (715) to express that the first function is executed in response to a first input for natural language recognition while displaying the first screen related to the execution of the first function. For example, the electronic device (100) may perform a natural language recognition function based on receiving a first input for natural language recognition. For example, the electronic device (100) may identify the first request and the second request based on receiving a second input requesting that a voice be recorded and translated into English. For example, the first request may include a request to record a voice. For example, the second request may include a request to translate into English. For example, the electronic device (100) may display a first screen related to the execution of a first function corresponding to the first request on the display (208). For example, the electronic device (100) may display a visual indication (715) indicating that the first screen is displayed based on receiving a first input while the first screen is displayed. For example, the electronic device (100) may display a visual indication (715) to indicate that the first function is executed based on a first input for natural language recognition while the first screen is displayed. For example, the electronic device (100) may display a visual indication (715) through the display (208) to indicate that the first function is executed based on receiving a second input based on natural language while the UI object (514) is displayed.For example, the electronic device (100) may display a visual indication on the first screen to indicate that the first function is being executed based on receiving the second input. For example, the user (120) may identify that the natural language recognition function is being executed through the visual indication (715). For example, the user (120) may identify that the first function is being executed by the intelligent assistant through the visual indication (715).

[0119] State (720) can be described as a state in which the execution of the first function is completed. For example, the electronic device (100) can execute the second function after the execution of the first function is completed. For example, the electronic device (100) can execute the second function using the execution result of the first function.

[0120] State (730) can be described as a state in which a visual indication (715) is displayed while a second screen related to a second function is displayed. For example, after the execution of the first function is completed, the electronic device (100) can execute the second function using the execution result of the first function. For example, when executing the second function, the electronic device (100) can display a second screen related to the second function. For example, when executing the second function, the electronic device (100) can switch the screen displayed through the display (208) from the first screen to the second screen. For example, while displaying the second screen, the electronic device (100) can display a visual indication (715) indicating that the second function is executed by the natural language recognition function. For example, while displaying the second screen, the electronic device (100) can display a visual indication (715) to express that the second function is executed according to a first input for natural language recognition. For example, the electronic device (100) may display a visual indication (715) to indicate that the second function is executed based on a second input based on natural language while displaying the second screen. For example, the electronic device (100) may display a visual indication (715) on the second screen to indicate that the second function is executed based on receiving the second input. For example, the user (120) may recognize that the second function is executed based on the second input through the visual indication (715). For example, the electronic device (100) may display a UI object (514) together with another visual object (e.g., the visual indication (715)) representing a time interval on a portion of the display (208) determined based on the received user input.

[0121] The electronic device (100) may utilize capsules within a server (e.g., server (900) of FIG. 9) to execute a second function using the execution result of the first function. For example, the transmission of information, signals, and data between the capsules and the electronic device (100) is described and exemplified in more detail with reference to FIG. 8.

[0122] Figure 8 illustrates examples of operations executed within a client, a capsule execution service, a first capsule, and a second capsule.

[0123] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0124] Referring to FIG. 8, in operation 810, a client (800) may transmit first data to a capsule execution service (802). For example, a server (e.g., server (900) of FIG. 9) may include a first capsule (804) and a first capsule (806). For example, the server may be referred to as an external electronic device. For example, the client (800) may transmit first data about natural language to the capsule execution service (802). For example, the first data may include data about a second input based on natural language.

[0125] In operation 815, the capsule execution service (802) may forward a first request (e.g., the first request (430) of FIG. 4A) to a first capsule (804). For example, a server (e.g., the server (900) of FIG. 9) may include a first capsule (804) and a second capsule (806). For example, a capsule may include information about functions executable by natural language recognition among functions executable in a software application. For example, the capsule may include a deep link for each of the executable functions. For example, the deep link may be described as a link for executing a specific function of the software application.

[0126] In operation 820, the first capsule (804) may execute the capsule. For example, the first capsule (804) may obtain a deep link to the first function corresponding to the first request by executing the capsule.

[0127] In operation 825, the first capsule (804) may transmit first information to the capsule execution service (802). For example, the first information may include a deep link for the first function. For example, the first information may include a signal requesting a client function core (CFC) response for the deep link of the first function. For example, the CFC response may be described as a response requesting the return of the execution result of the function executed using the deep link. For example, a request for a CFC response may be referred to as a CFC request.

[0128] In operation 830, the capsule execution service (802) may transmit a first signal to the client (800). For example, the client (800) may receive the first signal from the capsule execution service (802). For example, the first signal may include a signal requesting a CFC response for a first function. For example, the first signal may include a deep link for the first function. For example, when the capsule requests a CFC response from a software application, the form of the request for the CFC response may be a uniform resource locator (URL) or a uniform resource identifier (URI). For example, a request in the form of a URL may include information about the software application. For example, a request in the form of a URL may include an operation requesting the software application.

[0129] For example, a CFC response may be a signal, information, or data transmitted from a software application (910) to a capsule. For example, the information transmitted from the software application (910) to the capsule may include status information. For example, the status information may include the result of executing a function (e.g., success or failure). For example, the status information may include information about an image or the location where the image is stored.

[0130] In operation 835, the client (800) may store the execution result of the first function. For example, the client (800) may obtain the execution result of the first function using a deep link to the first function. For example, the client (800) may (temporarily) store the execution result in the memory (206).

[0131] At operation 840, the capsule execution service (802) may forward a second request (e.g., the second request (440) of FIG. 4A) to the second capsule (806). For example, the second capsule (806) may receive the second request.

[0132] In operation 845, the second capsule (806) may transmit a second signal to the client (800). For example, the second capsule (806) may transmit the second signal to the client (800) via the capsule execution service (802). For example, the client (800) may receive the second signal from the second capsule (806). For example, the second signal may include a CFC response for reading the execution result of the first function.

[0133] In operation 850, the client (800) may transmit second information to the second capsule (806). For example, the client (800) may transmit the second information to the second capsule (806) via the capsule execution service (802). For example, the second information may include the execution result of the first function. For example, the client (800) may transmit the second information to the second capsule (806) in response to the CFC response.

[0134] In operation 855, the second capsule (806) may execute a capsule. For example, the second capsule (806) may obtain a deep link to the second function using the execution result of the first function. For example, the second capsule (806) may obtain a deep link to the second function corresponding to the second request.

[0135] In operation 860, the second capsule (806) may transmit third information to the capsule execution service (802). For example, the third information may include a deep link to the second function.

[0136] In operation 865, the capsule execution service (802) may transmit fourth information to the client (800). For example, the client (800) may receive the fourth information from the capsule execution service (802). For example, the fourth information may include a deep link to the second function. For example, based on receiving the fourth information, the client (800) may use the deep link to the second function to obtain an execution result for the second function. The transmission of the signal and information is described and illustrated in more detail with reference to FIG. 9.

[0137] FIG. 9 illustrates an exemplary operation of an electronic device that obtains the execution result of a first function using a CFC response.

[0138] Referring to FIG. 9, for example, the server (900) may include a server related to natural language recognition. The server (900) may include a first capsule (804) and a second capsule (806). For example, the electronic device (100) may include a client (800) and a software application (910). For example, the server (900) may include a server related to natural language recognition. For example, the client (800) may include an interface for natural language recognition. For example, the client (800) may be used for natural language recognition. For example, the client (800) may be used to perform operations related to natural language recognition. For example, the first capsule (804) may request a CFC response for a first function from the software application (910) through the capsule execution service (802) and the client (800). For example, the first capsule (804) may transmit a signal requesting a CFC response for the first function to the software application (910) through the capsule execution service (802) and the client (800). For example, the client (800) may temporarily store the execution result of the first function. For example, the first capsule (804) may generate natural language that is provided to the server (900). For example, the software application (910) may, based on receiving the request for the CFC response, obtain or generate the execution result of the first function using a deep link for the first function of the first request. For example, the software application (910) may cause the electronic device (100) to transmit the execution result of the first function to the first capsule (804) within the server (900) based on the request for the CFC response. However, the present invention is not limited thereto. For example, the capsule execution service (802) may store the execution result of the first function.

[0139] For example, the electronic device (100) may set a time limit for a CFC response. For example, a request for a CFC response may include a parameter for the time limit.

[0140] For example, methods for obtaining execution results for functions using capsules may include a first method and a second method. For example, the first method may be referred to as a punchout method. For example, the second method may be referred to as a background method. For example, the first and second methods may be described as methods using deep links. For example, the first method may be described as a method in which the execution result of the function is not output after the function is executed using a deep link. For example, the second method may be described as a method in which the execution result of the function is output after the function is executed using a deep link.

[0141] In one embodiment, an intelligent assistant for natural language recognition may be maintained while executing the first and second functions. For example, the intelligent assistant may be maintained in the background of the operating system of the electronic device (100) while executing the first and second functions. For example, the client (800) may transmit signals to each of the software application (910) and the server (900). For example, the client (800) may transmit a signal including a first request to the server (900) and then be maintained in the background to obtain a response. For example, a capsule within the server (900) may receive an execution result from the software application (910) and then generate an execution result in the form of a natural language. The operation of maintaining the intelligent assistant in the background is described and exemplified in more detail with reference to FIG. 10 .

[0142] Figure 10 illustrates examples of other operations executed within a client, a capsule execution service, a first capsule, and a second capsule.

[0143] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0144] Referring to FIG. 10 , in operation 1010, a client (800) may transmit first data to a capsule execution service (802). For example, the first data may include a second input in natural language. For example, the first data may include data in natural language received via a microphone (209) of the client (800). For example, operation 1010 may correspond to operation 810 of FIG. 8 .

[0145] In operation 1015, the capsule execution service (802) may forward a first request (e.g., the first request (430) of FIG. 4A) to the first capsule (804). For example, the first request may indicate the intent of the user (120). For example, operation 1015 may correspond to operation 815 of FIG. 8.

[0146] In operation 1020, the first capsule (804) may execute a capsule. For example, the first capsule (804) may obtain a deep link for the first function. For example, operation 1020 may correspond to operation 820 of FIG. 8.

[0147] In operation 1025, the first capsule (804) may transmit first information to the capsule execution service (802). For example, the first information may include a deep link to the first function. For example, operation 1025 may correspond to operation 825 of FIG. 8 .

[0148] In operation 1030, the capsule execution service (802) may store the execution result of the first function. For example, the capsule execution service (802) may store the deep link of the first function for the first request. For example, the capsule execution service (802) may analyze the execution result of the first function. For example, the capsule execution service (802) may analyze the deep link of the first function.

[0149] In operation 1035, the capsule execution service (802) may forward a second request (e.g., the second request (440) of FIG. 4A) to the second capsule (806). For example, the second request may cause the second capsule (806) to execute a second function using the execution result of the first function. For example, operation 1035 may correspond to operation 840 of FIG. 8.

[0150] In operation 1040, the second capsule (806) may execute a capsule. For example, the second capsule (806) may obtain a deep link for the second function of the second request based on the execution result of the first function.

[0151] In operation 1045, the second capsule (806) may transmit second information to the capsule execution service (802). For example, the capsule execution service (802) may receive the second information. For example, the second information may include a deep link to a second function to be executed based on the execution result of the first function. For example, the second information may include the execution result of the second function.

[0152] In operation 1050, the capsule execution service (802) may transmit third information to the client (800). For example, the electronic device (100) may receive the third information. For example, the third information may include a deep link to the second function. For example, the third information may include the execution result of the second function.

[0153] FIG. 12 is a block diagram of an electronic device within a network environment according to various embodiments.

[0154] FIG. 12 is a block diagram of an electronic device (1201) within a network environment (1200) according to various embodiments. Referring to FIG. 12 , in the network environment (1200), the electronic device (1201) may communicate with the electronic device (1202) via a first network (1298) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (1204) or the server (1208) via a second network (1299) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (1201) may communicate with the electronic device (1204) via the server (1208). According to one embodiment, the electronic device (1201) may include a processor (1220), a memory (1230), an input module (1250), an audio output module (1255), a display module (1260), an audio module (1270), a sensor module (1276), an interface (1277), a connection terminal (1278), a haptic module (1279), a camera module (1280), a power management module (1288), a battery (1289), a communication module (1290), a subscriber identification module (1296), or an antenna module (1297). In some embodiments, the electronic device (1201) may omit at least one of these components (e.g., the connection terminal (1278)), or may have one or more other components added. In some embodiments, some of these components (e.g., sensor module (1276), camera module (1280), or antenna module (1297)) may be integrated into a single component (e.g., display module (1260)).

[0155] The processor (1220) may control at least one other component (e.g., hardware or software component) of the electronic device (1201) connected to the processor (1220) by executing, for example, software (e.g., program (1240)), and may perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1220) may store commands or data received from other components (e.g., sensor module (1276) or communication module (1290)) in volatile memory (1232), process the commands or data stored in volatile memory (1232), and store result data in non-volatile memory (1234). According to one embodiment, the processor (1220) may include a main processor (1221) (e.g., a central processing unit or an application processor) or an auxiliary processor (1223) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1221). For example, when the electronic device (1201) includes the main processor (1221) and the auxiliary processor (1223), the auxiliary processor (1223) may be configured to use less power than the main processor (1221) or to be specialized for a given function. The auxiliary processor (1223) may be implemented separately from the main processor (1221) or as a part thereof.

[0156] The auxiliary processor (1223) may control at least a portion of functions or states associated with at least one component (e.g., a display module (1260), a sensor module (1276), or a communication module (1290)) of the electronic device (1201), for example, on behalf of the main processor (1221) while the main processor (1221) is in an inactive (e.g., sleep) state, or together with the main processor (1221) while the main processor (1221) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1223) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1280) or a communication module (1290)). In one embodiment, the auxiliary processor (1223) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1201) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1208)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0157] The memory (1230) can store various data used by at least one component (e.g., the processor (1220) or the sensor module (1276)) of the electronic device (1201). The data can include, for example, software (e.g., the program (1240)) and input data or output data for commands related thereto. The memory (1230) can include a volatile memory (1232) or a non-volatile memory (1234).

[0158] The program (1240) may be stored as software in memory (1230) and may include, for example, an operating system (1242), middleware (1244), or an application (1246).

[0159] The input module (1250) can receive commands or data to be used in a component of the electronic device (1201) (e.g., a processor (1220)) from an external source (e.g., a user) of the electronic device (1201). The input module (1250) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0160] The audio output module (1255) can output audio signals to the outside of the electronic device (1201). The audio output module (1255) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0161] The display module (1260) can visually provide information to an external party (e.g., a user) of the electronic device (1201). The display module (1260) may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling the device. In one embodiment, the display module (1260) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0162] The audio module (1270) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (1270) can acquire sound through the input module (1250), output sound through the sound output module (1255), or an external electronic device (e.g., electronic device (1202)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1201).

[0163] The sensor module (1276) can detect the operating status (e.g., power or temperature) of the electronic device (1201) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1276) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0164] The interface (1277) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1201) with an external electronic device (e.g., the electronic device (1202)). In one embodiment, the interface (1277) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0165] The connection terminal (1278) may include a connector through which the electronic device (1201) may be physically connected to an external electronic device (e.g., the electronic device (1202)). In one embodiment, the connection terminal (1278) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0166] The haptic module (1279) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (1279) may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0167] The camera module (1280) can capture still images and videos. According to one embodiment, the camera module (1280) may include one or more lenses, image sensors, image signal processors, or flashes.

[0168] The power management module (1288) can manage the power supplied to the electronic device (1201). According to one embodiment, the power management module (1288) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0169] A battery (1289) may power at least one component of the electronic device (1201). In one embodiment, the battery (1289) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0170] The communication module (1290) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1201) and an external electronic device (e.g., electronic device (1202), electronic device (1204), or server (1208)), and the performance of communication through the established communication channel. The communication module (1290) may operate independently from the processor (1220) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1290) may include a wireless communication module (1292) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1294) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (1204) via a first network (1298) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1299) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1292) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1296) to verify or authenticate the electronic device (1201) within a communication network such as the first network (1298) or the second network (1299).

[0171] The wireless communication module (1292) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1292) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1292) can support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1292) can support various requirements specified in the electronic device (1201), an external electronic device (e.g., the electronic device (1204)), or a network system (e.g., the second network (1299)). According to one embodiment, the wireless communication module (1292) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0172] The antenna module (1297) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1297) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1297) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1298) or the second network (1299), may be selected from the plurality of antennas by, for example, the communication module (1290). A signal or power may be transmitted or received between the communication module (1290) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1297).

[0173] According to various embodiments, the antenna module (1297) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0174] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0175] According to one embodiment, commands or data may be transmitted or received between the electronic device (1201) and an external electronic device (1204) via a server (1208) connected to a second network (1299). Each of the external electronic devices (1202 or 1204) may be the same or a different type of device as the electronic device (1201). According to one embodiment, all or part of the operations executed in the electronic device (1201) may be executed in one or more of the external electronic devices (1202, 1204, or 1208). For example, when the electronic device (1201) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1201) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1201). The electronic device (1201) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1201) may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1204) may include an Internet of Things (IoT) device. The server (1208) may be an intelligent server utilizing machine learning and / or a neural network.According to one embodiment, an external electronic device (1204) or server (1208) may be included within the second network (1299). The electronic device (1201) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0176] FIG. 13 is a block diagram illustrating an integrated intelligence (AI) system according to one embodiment.

[0177] Referring to FIG. 13, an integrated intelligence system (1302) of one embodiment may include a user terminal (1300), an intelligent server (1400), and a service server (1500).

[0178] The user terminal (1300) of one embodiment may be a terminal device (or electronic device) that can connect to the Internet, and may be, for example, a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a TV, white goods, a wearable device, an HMD, or a smart speaker.

[0179] According to one embodiment, the user terminal (1300) may include a communication interface (1310), a microphone (1320), a speaker (1330), a display (1340), a memory (1350), and a processor (1360). The components listed above may be operatively or electrically connected to each other.

[0180] According to one embodiment, the communication interface (1310) may be configured to be connected to an external device to transmit and receive data. According to one embodiment, the microphone (1320) may receive sound (e.g., user speech) and convert it into an electrical signal. According to one embodiment, the speaker (1330) may output the electrical signal as sound (e.g., voice). According to one embodiment, the display (1340) may be configured to display an image or video. According to one embodiment, the display (1340) may display a graphical user interface (GUI) of an app (or application program) being executed.

[0181] According to one embodiment, the memory (1350) may store a client module (1351), a software development kit (SDK) (1353), and a plurality of apps (1355). The client module (1351) and the SDK (1353) may constitute a framework (or solution program) for performing general functions. In addition, the client module (1351) or the SDK (1353) may constitute a framework for processing voice input.

[0182] According to one embodiment, the plurality of apps (1355) stored in the memory (1350) may be programs for performing a specified function. According to one embodiment, the plurality of apps (1355) may include a first app (1355_1) and a second app (1355_3). According to one embodiment, each of the plurality of apps (1355) may include a plurality of operations for performing a specified function. For example, the plurality of apps (1355) may include at least one of an alarm app, a message app, and a schedule app. According to one embodiment, the plurality of apps (1355) may be executed by the processor (1360) to sequentially execute at least some of the plurality of operations.

[0183] According to one embodiment, the processor (1360) can control the overall operation of the user terminal (1300). For example, the processor (1360) can be electrically connected to a communication interface (1310), a microphone (1320), a speaker (1330), a display (1340), and a memory (1350) to perform a designated operation.

[0184] According to one embodiment, the processor (1360) may also execute a program stored in the memory (1350) to perform a designated function. For example, the processor (1360) may execute at least one of the client module (1351) or the SDK (1353) to perform the following operations for processing voice input. The processor (1360) may control the operations of multiple apps (1355), for example, through the SDK (1353). The following operations described as operations of the client module (1351) or the SDK (1353) may be operations executed by the processor (1360).

[0185] According to one embodiment, the client module (1351) can receive a voice input. For example, the client module (1351) can generate a voice signal corresponding to a user utterance detected through the microphone (1320). The client module (1351) can transmit the received voice input to the intelligent server (1400). According to one embodiment, the client module (1351) can transmit status information of the user terminal (1300) to the intelligent server (1400) together with the received voice input. The status information can be, for example, execution status information of an app.

[0186] According to one embodiment, the client module (1351) may receive a result corresponding to the received voice input. For example, the client module (1351) may receive a result corresponding to the voice input from the intelligent server (1400). The client module (1351) may display the received result on the display (1340).

[0187] According to one embodiment, the client module (1351) may receive a plan corresponding to the received voice input. The client module (1351) may display the results of executing multiple operations of the app according to the plan on the display (1340). For example, the client module (1351) may sequentially display the results of executing multiple operations on the display. In another example, the user terminal (1300) may display only some of the results of executing multiple operations (e.g., the result of the last operation) on the display.

[0188] According to one embodiment, the client module (1351) may receive a request from the intelligent server (1400) to obtain information necessary to produce a result corresponding to a voice input. The information necessary to produce the result may be, for example, status information of the electronic device (1300). According to one embodiment, the client module (1351) may transmit the necessary information to the intelligent server (1400) in response to the request.

[0189] According to one embodiment, the client module (1351) can transmit result information of executing multiple operations according to a plan to the intelligent server (1400). The intelligent server (1400) can confirm that the received voice input has been processed correctly through the result information.

[0190] In one embodiment, the client module (1351) may include a voice recognition module. In one embodiment, the client module (1351) may recognize voice inputs that perform limited functions through the voice recognition module. For example, the client module (1351) may execute an intelligent app to process voice inputs to perform organic actions through designated inputs (e.g., "Wake up!").

[0191] According to one embodiment, the intelligent server (1400) can receive information related to user voice input from the user terminal (1300) via a communication network. According to one embodiment, the intelligent server (1400) can convert data related to the received voice input into text data. According to one embodiment, the intelligent server (1400) can generate a plan for performing a task corresponding to the user voice input based on the text data.

[0192] In one embodiment, the plan may be generated by an artificial intelligence (AI) system. The AI ​​system may be a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN) or a recurrent neural network (RNN)), or a combination of the above or a different AI system. In one embodiment, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the AI ​​system may select at least one plan from a plurality of predefined plans.

[0193] According to one embodiment, the intelligent server (1400) may transmit the results calculated according to the generated plan to the user terminal (1300), or may transmit the generated plan to the user terminal (1300). According to one embodiment, the user terminal (1300) may display the results calculated according to the plan on a display. According to one embodiment, the user terminal (1300) may display the results of executing an operation according to the plan on a display.

[0194] An intelligent server (1400) of one embodiment may include a front end (1410), a natural language platform (1420), a capsule database (1430), an execution engine (1440), an end user interface (1450), a management platform (1460), a big data platform (1470), and an analytic platform (1480).

[0195] According to one embodiment, the front end (1410) can receive a voice input from a user terminal (1300). The front end (1410) can transmit a response corresponding to the voice input.

[0196] According to one embodiment, the natural language platform (1420) may include an automatic speech recognition module (ASR module) (1421), a natural language understanding module (NLU module) (1423), a planner module (1425), a natural language generator module (NLG module) (1427), and a text to speech module (TTS module) (1429).

[0197] According to one embodiment, the automatic speech recognition module (1421) can convert voice input received from the user terminal (1300) into text data. According to one embodiment, the natural language understanding module (1423) can use the text data of the voice input to determine the user's intention. For example, the natural language understanding module (1423) can perform syntactic analysis or semantic analysis to determine the user's intention. According to one embodiment, the natural language understanding module (1423) can use linguistic features (e.g., grammatical elements) of morphemes or phrases to determine the meaning of words extracted from the voice input, and can match the meaning of the determined words to the intent to determine the user's intent. The natural language understanding module (1123) can obtain intent information corresponding to the user's utterance. The intent information can be information indicating the user's intent determined by interpreting text data. The intent information can include information indicating an action or function that the user intends to execute using the device.

[0198] According to one embodiment, the planner module (1425) can generate a plan using the intent and parameters determined by the natural language understanding module (1423). According to one embodiment, the planner module (1425) can determine a plurality of domains necessary to perform a task based on the determined intent. The planner module (1425) can determine a plurality of operations included in each of the plurality of domains determined based on the intent. According to one embodiment, the planner module (1425) can determine parameters necessary to execute the determined plurality of operations or result values ​​output by the execution of the plurality of operations. The parameters and the result values ​​can be defined as concepts related to a specified format (or class). Accordingly, the plan can include a plurality of operations and a plurality of concepts determined by the user's intent. The planner module (1425) can determine the relationships between the plurality of operations and the plurality of concepts in a stepwise (or hierarchical) manner. For example, the planner module (1425) can determine the execution order of a plurality of actions based on the user's intention based on a plurality of concepts. In other words, the planner module (1425) can determine the execution order of a plurality of actions based on parameters required for the execution of the plurality of actions and results output by the execution of the plurality of actions. Accordingly, the planner module (1425) can generate a plan including association information (e.g., ontology) between the plurality of actions and the plurality of concepts. The planner module (1425) can generate the plan using information stored in a capsule database (1430) in which a set of relationships between concepts and actions is stored.

[0199] According to one embodiment, the natural language generation module (1427) can convert specified information into text format. The information converted into text format may be in the form of natural language speech. The text-to-speech conversion module (1429) of one embodiment can convert information in text format into information in speech format.

[0200] According to one embodiment, the capsule database (1430) can store information about the relationship between a plurality of concepts and actions corresponding to a plurality of domains. For example, the capsule database (1430) can store a plurality of capsules including a plurality of action objects (or action information) and concept objects (or concept information) of a plan. According to one embodiment, the capsule database (1430) can store the plurality of capsules in the form of a concept action network (CAN). According to one embodiment, the plurality of capsules can be stored in a function registry included in the capsule database (1430).

[0201] According to one embodiment, the capsule database (1430) may include a strategy registry that stores strategy information required when determining a plan corresponding to a voice input. The strategy information may include reference information for determining a single plan when there are multiple plans corresponding to a voice input. According to one embodiment, the capsule database (1430) may include a follow-up registry that stores information on follow-up actions for suggesting follow-up actions to a user in a given situation. The follow-up actions may include, for example, follow-up utterances. According to one embodiment, the capsule database (1430) may include a layout registry that stores layout information of information output through the user terminal (1300). According to one embodiment, the capsule database (1430) may include a vocabulary registry that stores vocabulary information included in capsule information. According to one embodiment, the capsule database (1430) may include a dialog registry in which information about a dialog (or interaction) with a user is stored.

[0202] According to one embodiment, the capsule database (1430) can update stored objects through a developer tool. The developer tool may include, for example, a function editor for updating action objects or concept objects. The developer tool may include a vocabulary editor for updating vocabulary. The developer tool may include a strategy editor for creating and registering strategies that determine plans. The developer tool may include a dialog editor for creating a dialogue with a user. The developer tool may include a follow-up editor for activating follow-up goals and editing follow-up utterances that provide hints. The follow-up goals may be determined based on currently set goals, user preferences, or environmental conditions.

[0203] According to one embodiment, the capsule database (1430) may also be implemented within the user terminal (1300). In other words, the user terminal (1300) may include a capsule database (1430) that stores information for determining an action corresponding to a voice input.

[0204] According to one embodiment, the execution engine (1440) can produce a result using the generated plan. According to one embodiment, the end user interface (1450) can transmit the produced result to the user terminal (1300). Accordingly, the user terminal (1300) can receive the result and provide the received result to the user. According to one embodiment, the management platform (1460) can manage information used in the intelligent server (1400). According to one embodiment, the big data platform (1470) can collect user data. According to one embodiment, the analysis platform (1480) can manage the quality of service (QoS) of the intelligent server (1400). For example, the analysis platform (1480) can manage the components and processing speed (or efficiency) of the intelligent server (1400).

[0205] According to one embodiment, the service server (1500) may provide a designated service (e.g., food ordering or hotel reservation) to the user terminal (1300). According to one embodiment, the service server (1500) may be a server operated by a third party. For example, the service server (1500) may include a first service server (1501), a second service server (1503), and a third service server (1505) operated by different third parties. According to one embodiment, the service server (1500) may provide information for generating a plan corresponding to the received voice input to the intelligent server (1400). The provided information may be stored, for example, in a capsule database (1430). In addition, the service server (1500) may provide result information according to the plan to the intelligent server (1400).

[0206] In the integrated intelligence system (1302) described above, the user terminal (1300) can provide various intelligent services to the user in response to user input. The user input may include, for example, input via a physical button, touch input, or voice input.

[0207] According to one embodiment, the user terminal (1300) may provide a voice recognition service through an intelligent app (or voice recognition app) stored internally. In this case, for example, the user terminal (1300) may recognize a user utterance or voice input received through the microphone and provide the user with a service corresponding to the recognized voice input.

[0208] According to one embodiment, the user terminal (1300) may perform a designated action based on the received voice input, either alone or in conjunction with the intelligent server and / or service server. For example, the user terminal (1300) may execute an app corresponding to the received voice input and perform a designated action through the executed app.

[0209] According to one embodiment, when a user terminal (1300) provides a service together with an intelligent server (1400) and / or a service server, the user terminal may detect user speech using the microphone (1320) and generate a signal (or voice data) corresponding to the detected user speech. The user terminal may transmit the voice data to the intelligent server (1400) using a communication interface (1310).

[0210] According to one embodiment, the intelligent server (1400) may generate a plan for performing a task corresponding to the voice input received from the user terminal (1300), or a result of performing an operation according to the plan. The plan may include, for example, a plurality of operations for performing a task corresponding to the user's voice input, and a plurality of concepts related to the plurality of operations. The concept may define parameters input to the execution of the plurality of operations, or result values ​​output by the execution of the plurality of operations. The plan may include association information between the plurality of operations and the plurality of concepts.

[0211] In one embodiment, the user terminal (1300) can receive the response using the communication interface (1310). The user terminal (1300) can output a voice signal generated within the user terminal (1300) to the outside using the speaker (1330), or can output an image generated within the user terminal (1300) to the outside using the display (1340).

[0212] FIG. 14 is a diagram showing a form in which relationship information between concepts and actions is stored in a database according to various embodiments.

[0213] The capsule database (e.g., capsule database (1430)) of the intelligent server (1400) may store multiple capsules in the form of a CAN (concept action network) (1600). The capsule database may store operations for processing tasks corresponding to a user's voice input and parameters required for the operations in the form of a CAN (concept action network). The CAN may represent an organic relationship between an action and a concept that defines parameters required to perform the action.

[0214] The capsule database may store a plurality of capsules (e.g., Capsule A (1601), Capsule B (1602)) corresponding to each of a plurality of domains (e.g., applications). According to one embodiment, one capsule (e.g., Capsule A (1601)) may correspond to one domain (e.g., application). In addition, one capsule may correspond to at least one service provider (e.g., CP 1 (1603), CP 2 (1604), CP 3 (1605), or CP 4 (1606)) for performing a function of a domain related to the capsule. According to one embodiment, one capsule may include at least one operation (1610) and at least one concept (1620) for performing a specified function.

[0215] According to one embodiment, the natural language platform (1420) can generate a plan for performing a task corresponding to a received speech input using capsules stored in a capsule database. For example, the planner module (1425) of the natural language platform can generate a plan using capsules stored in the capsule database. For example, a plan (1607) can be generated using actions (16011, 16013) and concepts (16012, 16014) of Capsule A (1601) and actions (16041) and concepts (16042) of Capsule B (1602).

[0216] FIG. 15 is a diagram showing a screen for processing voice input received through an intelligent app by a user terminal according to various embodiments.

[0217] The user terminal (1300) can execute an intelligent app to process user input through an intelligent server (1400).

[0218]

[0219] According to one embodiment, on screen 1510, when the user terminal (1300) recognizes a designated voice input (e.g., wake up!) or receives an input via a hardware key (e.g., a dedicated hardware key), the user terminal (1300) may execute an intelligent app for processing the voice input. For example, the user terminal (1300) may execute an intelligent app while executing a schedule app. According to one embodiment, the user terminal (1300) may display an object (e.g., an icon) (1511) corresponding to the intelligent app on the display (1340). According to one embodiment, the user terminal (1300) may receive a voice input by a user's speech. For example, the user terminal (1300) may receive a voice input such as "Tell me my schedule for this week!" According to one embodiment, the user terminal (1300) may display a user interface (UI) (1513) (e.g., an input window) of the intelligent app, in which text data of the received voice input is displayed, on the display.

[0220] According to one embodiment, on screen 1520, the user terminal (1300) may display a result corresponding to the received voice input on the display. For example, the user terminal (1300) may receive a plan corresponding to the received user input and display "This Week's Schedule" on the display according to the plan.

[0221] The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by a person having ordinary knowledge in the technical field to which the present disclosure pertains.

[0222] An electronic device (e.g., electronic device (100)) as described above may include a memory (e.g., memory (206)) that stores instructions. The electronic device may include at least one processor (e.g., at least one processor (207)). The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive, from a first external electronic device, information about a first sentence corresponding to a first user intent and a second sentence corresponding to a second user intent and subsequent to the first sentence, and to acquire the information based on recognition of a voice command. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify, based on the received information, whether at least some of the words of the first sentence are included in the second sentence. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine that the first user intent is related to the second user intent based on identifying that at least some of the words of the first sentence are included in the second sentence. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate data for instructions for executing a first function indicated by the first user intent and for executing a second function indicated by the second user intent based on the determination. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit the data to a second external electronic device.

[0223] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine that the first user intent is not related to the second user intent based on identifying that words in the first sentence are not included in the second sentence. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate another data for executing the first function and for executing the second function independently of the execution of the first function, based on the determination that the first user intent is not related to the second user intent. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit the another data to the second external electronic device.

[0224] In one embodiment, the first external electronic device and the second external electronic device may be identical.

[0225] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine, using a trained model configured to receive the first sentence and the second sentence, that the first user intent is related to the second user intent.

[0226] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive the information from the first external electronic device executing a trained model configured to perform natural language processing.

[0227] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate the data that causes the second external electronic device to display a user interface (UI) object based on the execution of the first function.

[0228] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate the data, which causes the second external electronic device to display the UI object indicating that the second function is to be executed using the result.

[0229] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate the data that causes the second external electronic device to display the UI object capable of receiving a user input indicating to cancel executing the second function.

[0230] A method performed by an electronic device as described above may include receiving, from a first external electronic device, information about a first sentence corresponding to a first user intent and a second sentence corresponding to a second user intent and subsequent to the first sentence, and obtaining the information based on recognition of a voice command. The method may include identifying, based on the received information, whether at least some of the words of the first sentence are included in the second sentence. The method may include determining, based on identifying that the at least some of the words of the first sentence are included in the second sentence, that the first user intent is related to the second user intent. The method may include generating, based on the determination, data for a command to execute a first function indicated by the first user intent and to execute a second function indicated by the second user intent based on a result of executing the first function. The method may include transmitting the data to the second external electronic device.

[0231] In one embodiment, the method may include an operation of determining that the first user intent is not related to the second user intent based on identifying that words in the first sentence are not included in the second sentence. The method may include an operation of generating another data for a command to execute the first function and to execute the second function independently of the execution of the first function, based on the determination that the first user intent is not related to the second user intent. The method may include an operation of transmitting the another data to the second external electronic device.

[0232] In one embodiment, the first external electronic device and the second external electronic device may be identical.

[0233] In one embodiment, the method may include an action of determining, using a trained model configured to receive the first sentence and the second sentence, that the first user intent is related to the second user intent.

[0234] According to one embodiment, the method may include receiving the information from the first external electronic device executing a trained model configured to perform natural language processing.

[0235] According to one embodiment, the method may include an action of generating the data that causes the second external electronic device to display a user interface (UI) object based on the execution of the first function.

[0236] In one embodiment, the method may include generating the data causing the second external electronic device to display the UI object indicating that the second function is to be executed using the result.

[0237] In one embodiment, the method may include generating the data causing the second external electronic device to display the UI object capable of receiving a user input indicating to cancel executing the second function.

[0238] In a computer-readable storage medium having one or more programs stored thereon, as described above, the one or more programs may include instructions that, when executed by an electronic device, cause the electronic device to receive, from a first external electronic device, information about a first sentence corresponding to a first user intent and a second sentence corresponding to a second user intent and subsequent to the first sentence, and to obtain the information based on recognition of a voice command. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to identify, based on the received information, whether at least a portion of words in the first sentence are included in the second sentence. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to determine, based on identifying that at least a portion of words in the first sentence are included in the second sentence, that the first user intent is related to the second user intent. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to generate data for a command to execute a first function indicated by the first user intent based on the determination, and to execute a second function indicated by the second user intent based on a result of the execution of the first function. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to transmit the data to a second external electronic device.

[0239] In one embodiment, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to determine that the first user intent is not related to the second user intent based on identifying that words in the first sentence are not included in the second sentence. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to generate another data for executing the first function and an instruction for executing the second function independently of the execution of the first function based on a determination that the first user intent is not related to the second user intent. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to transmit the another data to the second external electronic device.

[0240] In one embodiment, the first external electronic device and the second external electronic device may be identical.

[0241] According to one embodiment, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to determine, using a trained model configured to receive the first sentence and the second sentence, that the first user intent is related to the second user intent.

[0242] According to one embodiment, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to receive the information from the first external electronic device executing a trained model configured to perform natural language processing.

[0243] According to one embodiment, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to generate the data that causes the second external electronic device to display a user interface (UI) object based on the execution of the first function.

[0244] According to one embodiment, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to generate the data, which causes the second external electronic device to display the UI object indicating that the second function is to be executed using the result.

[0245] According to one embodiment, the one or more programs may include instructions that cause the electronic device to generate the data, which causes the second external electronic device to display the UI object capable of receiving a user input indicating to cancel executing the second function when executed by the electronic device.

[0246] An electronic device (e.g., electronic device (100)) as described above may include a memory (e.g., memory (206)) that stores instructions. The electronic device may include a microphone. The electronic device may include a display. The electronic device may include at least one processor (e.g., at least one processor (207)). The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a voice command via the microphone. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit information about the voice command to a first external electronic device based on the identification. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive data related to the voice command from a second external electronic device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify, from the data, a command for executing a first function of the electronic device, and, based on a result of executing the first function, for executing a second function of the electronic device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the first function based on the identification. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to simultaneously display, through the display, a UI object indicating the result of executing the first function and whether to execute the second function.The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the second function based on the displayed result of the first function, based on a first user input to the UI object indicating to execute the second function. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to refrain from executing the second function based on a second user input to the UI object indicating to cancel executing the second function.

[0247] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display an execution screen including the result of the execution of the first function, and a pop-up window overlaid on the execution screen and including the UI object.

[0248] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to change the execution screen to another execution screen related to the second function while the pop-up window is maintained, based on the first user input.

[0249] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to stop displaying the pop-up window based on execution of the second function.

[0250] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the execution screen of the first software application based on the execution of the first function. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display another execution screen of a second software application for executing the second function based on the first user input.

[0251] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive the data from the second external electronic device, the second external electronic device receiving information including a first sentence indicating to execute the first function from the first external electronic device and a second sentence indicating to execute the second function.

[0252] According to one embodiment, the first external electronic device may be configured to identify the information including the first sentence and the second sentence following the first sentence based on recognition of the voice command.

[0253] A method performed by an electronic device having a microphone and a display as described above may include an operation of identifying a voice command through the microphone. The method may include an operation of transmitting information about the voice command to a first external electronic device based on the identification. The method may include an operation of receiving data related to the voice command from a second external electronic device. The method may include an operation of identifying, from the data, a command for executing a first function of the electronic device, and a command for executing a second function of the electronic device based on a result of executing the first function. The method may include an operation of executing the first function based on the identification. The method may include an operation of simultaneously displaying, through the display, a UI object indicating the result of executing the first function and whether to execute the second function. The method may include an operation of executing the second function based on the displayed result of the first function based on a first user input to the UI object indicating to execute the second function. The method may include an action of refraining from executing the second function based on a second user input to the UI object indicating to cancel executing the second function.

[0254] According to one embodiment, the method may include an operation of displaying an execution screen including the result of the execution of the first function, and a pop-up window overlapping the execution screen and including the UI object.

[0255] According to one embodiment, the method may include an operation of changing the execution screen to another execution screen related to the second function while the pop-up window is maintained based on the first user input.

[0256] According to one embodiment, the method may include an action of stopping displaying the pop-up window based on execution of the second function.

[0257] In one embodiment, the method may include an operation of displaying the execution screen of the first software application based on the execution of the first function. The method may include an operation of displaying another execution screen of the second software application for executing the second function based on the first user input.

[0258] According to one embodiment, the method may include receiving, from the second external electronic device, information including a first sentence indicating to execute the first function from the first external electronic device and a second sentence indicating to execute the second function.

[0259] According to one embodiment, the first external electronic device may be configured to identify the information including the first sentence and the second sentence following the first sentence based on recognition of the voice command.

[0260] In a computer-readable storage medium having one or more programs stored thereon, as described above, the one or more programs may include instructions that, when executed by an electronic device having a microphone and a display, cause the electronic device to identify a voice command via the microphone. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to transmit information about the voice command to a first external electronic device based on the identification. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to receive data related to the voice command from a second external electronic device. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to identify, from the data, a command for executing a first function of the electronic device and, based on a result of executing the first function, for executing a second function of the electronic device. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to execute the first function based on the identification. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to simultaneously display, through the display, a UI object indicating the result of the execution of the first function and whether to execute the second function. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to execute the second function according to the displayed result of the first function based on a first user input to the UI object indicating to execute the second function.The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to refrain from executing the second function based on a second user input to the UI object indicating to cancel executing the second function.

[0261] According to one embodiment, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display an execution screen including the result of the execution of the first function, and a pop-up window overlaid on the execution screen and including the UI object.

[0262] According to one embodiment, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to change the execution screen to another execution screen related to the second function while the pop-up window is maintained based on the first user input.

[0263] According to one embodiment, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to stop displaying the pop-up window based on execution of the second function.

[0264] According to one embodiment, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display the execution screen of the first software application based on execution of the first function. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display another execution screen of a second software application for executing the second function based on the first user input.

[0265] According to one embodiment, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to receive the data from the second external electronic device, the second external electronic device receiving information including a first sentence indicating to execute the first function from the first external electronic device and a second sentence indicating to execute the second function.

[0266] According to one embodiment, the first external electronic device may be configured to identify the information including the first sentence and the second sentence following the first sentence based on recognition of the voice command.

[0267] An electronic device (e.g., electronic device (100)) as described above may include a memory (e.g., memory (206)) that stores instructions. The electronic device may include a microphone. The electronic device may include at least one processor (e.g., at least one processor (207)). The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a voice command via the microphone. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain information about a first sentence corresponding to a first user intent and a second sentence corresponding to a second user intent and subsequent to the first sentence, based on the identification, using text data obtained from the voice command. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify whether at least some of the words of the first sentence are included in the second sentence based on obtaining the information. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine that the first user intent is related to the second user intent based on identifying that the at least some of the words of the first sentence are included in the second sentence. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute a first function indicated by the first user intent based on the determination.The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute a second function indicated by the second user intent based on the execution of the first function as a result of the execution of the first function.

[0268] A method performed by an electronic device having a microphone as described above may include an operation of identifying a voice command through the microphone. Based on the identification, the method may include an operation of obtaining information about a first sentence corresponding to a first user intent and a second sentence corresponding to a second user intent and subsequent to the first sentence using text data obtained from the voice command. Based on the obtaining of the information, the method may include an operation of identifying whether at least some of the words of the first sentence are included in the second sentence. Based on the identification that the at least some of the words of the first sentence are included in the second sentence, the method may include an operation of determining that the first user intent is related to the second user intent. Based on the determination, the method may include an operation of executing a first function indicated by the first user intent. Based on the execution of the first function, the method may include an operation of executing a second function indicated by the second user intent according to a result of the execution of the first function.

[0269] In a computer-readable storage medium having one or more programs stored thereon, as described above, the one or more programs may include instructions that, when executed by an electronic device having a microphone, cause the electronic device to identify a voice command via the microphone. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain, based on the identification, information about a first sentence corresponding to a first user intent and a second sentence corresponding to a second user intent and subsequent to the first sentence using text data obtained from the voice command. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to identify, based on the obtained information, whether at least a portion of words of the first sentence are included in the second sentence. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to determine that the first user intent is related to the second user intent based on identifying that at least some of the words of the first sentence are included in the second sentence. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to execute a first function indicated by the first user intent based on the determination. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to execute a second function indicated by the second user intent based on the execution of the first function as a result of the execution of the first function.

[0270] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

[0271] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0272] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0273] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0274] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0275] Therefore, other implementations, other embodiments, and equivalents of the claims are also within the scope of the claims described below. According to one embodiment, the method according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0276] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a non-transitory computer-readable storage medium storing one or more programs, the one or more programs are: When executed by an electronic device, Receive information about a first sentence corresponding to a first user intent and a second sentence corresponding to a second user intent and following the first sentence, from a first external electronic device, wherein the information is obtained by recognizing a voice command; Based on the received information, identifying whether at least some of the words of the first sentence are included in the second sentence; Based on identifying that at least some of the words of the first sentence are included in the second sentence, determining that the first user intent is related to the second user intent; Based on the above decision, generate data for a command to execute a first function indicated by the first user intent and to execute a second function indicated by the second user intent according to a result of execution of the first function; and To transmit the above data to a second external electronic device; comprising instructions causing the electronic device to operate; Non-transitory computer-readable storage medium.

2. In claim 1, The above one or more programs, when executed by the electronic device, Based on identifying that the words of the first sentence are not included in the second sentence, it is determined that the first user intent is not related to the second user intent, Based on a determination that the first user intent is not related to the second user intent, generating another data for executing the first function and for executing the second function independently of the execution of the first function, and To transmit the above other data to the second external electronic device, comprising instructions causing the electronic device to operate; Non-transitory computer-readable storage medium.

3. In claim 1, the first external electronic device and the second external electronic device are the same. Non-transitory computer-readable storage medium.

4. In claim 1, The above one or more programs, when executed by the electronic device, Using a trained model configured to receive the first sentence and the second sentence, determine that the first user intent is related to the second user intent. comprising instructions causing the electronic device to operate; Non-transitory computer-readable storage medium.

5. In claim 1, The above one or more programs, when executed by the electronic device, To receive the above information from the first external electronic device executing a trained model configured to perform natural language processing, comprising instructions causing the electronic device to operate; Non-transitory computer-readable storage medium.

6. In claim 1, The above one or more programs, when executed by the electronic device, Based on the execution of the first function, generate the data causing the second external electronic device to display a UI (user interface) object. comprising instructions causing the electronic device to operate; Non-transitory computer-readable storage medium.

7. In claim 6, The above one or more programs, when executed by the electronic device, Generate said data to cause said second external electronic device to display said UI object indicating that said second function will be executed using said result; comprising instructions causing the electronic device to operate; Non-transitory computer-readable storage medium.

8. In claim 6, The above one or more programs, when executed by the electronic device, Generate said data to cause said second external electronic device to display said UI object capable of receiving user input indicating to cancel executing said second function; comprising instructions causing the electronic device to operate; Non-transitory computer-readable storage medium.

9. In electronic devices, microphone; display; A memory comprising one or more storage media and storing instructions; and At least one processor comprising processing circuitry, The above instructions, when individually or collectively executed by the at least one processor, Identify voice commands through the microphone, Based on the above identification, information about the voice command is transmitted to the first external electronic device, Receive data related to the above voice command from a second external electronic device, Identifying, from the data, a command for executing a first function of the electronic device, and for executing a second function of the electronic device based on a result of executing the first function; Based on the identification, execute the first function, Simultaneously displaying, through the display, a UI object indicating the result of the execution of the first function and whether the second function is executed; Based on a first user input to the UI object indicating to execute the second function, executing the second function according to the displayed result of the first function, and To refrain from executing said second function based on a second user input to said UI object indicating to cancel executing said second function, causing the above electronic device, Electronic devices.

10. In claim 9, The above instructions, when individually or collectively executed by the at least one processor, To display an execution screen including the result of the execution of the first function, and a pop-up window overlapping the execution screen and including the UI object; causing the above electronic device, Electronic devices.

11. In claim 10, The above instructions, when individually or collectively executed by the at least one processor, Based on the first user input, change the execution screen to another execution screen related to the second function while the pop-up window is maintained. causing the above electronic device, Electronic devices.

12. In claim 10, The above instructions, when individually or collectively executed by the at least one processor, Based on the execution of the above second function, to stop displaying the pop-up window, causing the above electronic device, Electronic devices.

13. In claim 10, The above instructions, when individually or collectively executed by the at least one processor, Based on the execution of the above first function, displaying the execution screen of the first software application, and Based on the first user input, display another execution screen of the second software application for executing the second function. causing the above electronic device, Electronic devices.

14. In claim 9, The above instructions, when individually or collectively executed by the at least one processor, To receive the data from the second external electronic device, which receives information including a first sentence indicating to execute the first function from the first external electronic device and a second sentence indicating to execute the second function, causing the above electronic device, Electronic devices.

15. In electronic devices, microphone; A memory comprising one or more storage media and storing instructions; and At least one processor comprising processing circuitry, The above instructions, when individually or collectively executed by the at least one processor, Identify voice commands through the microphone, Based on the above identification, using the text data obtained from the voice command, information about a first sentence corresponding to the first user intent and a second sentence corresponding to the second user intent and following the first sentence is obtained, Based on obtaining the above information, identifying whether at least some of the words of the first sentence are included in the second sentence, Based on identifying that at least some of the words of the first sentence are included in the second sentence, determining that the first user intent is related to the second user intent; Based on the above decision, the first function indicated by the first user intention is executed, and Based on the execution of the first function, the second function indicated by the second user intention is executed according to the result of the execution of the first function. causing the above electronic device, Electronic devices.

Images