Electronic device for controlling other device on basis of voice trigger, and operating method thereof

Abstract

The present disclosure provides an electronic device. The electronic device can: in response to receiving, through a microphone, a first user utterance including a first voice trigger and a complex command for controlling an external electronic device, transmit the first user utterance to a voice assistant server corresponding to the first voice trigger through a communication circuit; in response to receiving, through the communication circuit, from the voice assistant server, information about a device to be controlled, which corresponds to the external electronic device and has received the first user utterance, perform, through the communication circuit, a short-range wireless communication connection with the device to be controlled; and display, on a display, a control screen including at least one user interface for controlling the device to be controlled.

Description

Electronic device for controlling another device based on voice trigger and method for operating the same

[0001] The present disclosure relates to an electronic device for controlling another device based on a voice trigger and a method of operating the same.

[0002] With the advancement of multimedia and network technologies, users can now access a variety of services using their electronic devices. For example, with the advancement of voice recognition technology, users can input their voice (e.g., speaking) into their electronic devices and receive a response message based on their voice input through a voice assistant (e.g., a voice assistant service).

[0003] Additionally, the electronic device may perform a function of the electronic device or support performing a function on another electronic device based on recognition of voice input via a voice assistant.

[0004] An electronic device according to one embodiment of the present disclosure includes a microphone; a speaker; a display; a communication circuit; a memory storing at least one program; and at least one processor electrically connected to the memory and executing at least one command of the program stored in the memory, wherein the at least one processor is configured to: in response to receiving a first user utterance including a first voice trigger and a composite command for controlling an external electronic device through the microphone, transmit the first user utterance to a voice assistant server corresponding to the first voice trigger through the communication circuit; in response to receiving information about a control target device corresponding to the external electronic device and having received the first user utterance from the voice assistant server through the communication circuit, perform a short-range wireless communication connection with the control target device through the communication circuit; and display a control screen including at least one user interface for controlling the control target device through the display.

[0005] According to one embodiment, the composite command for controlling the external electronic device may be set to correspond to the execution of an application for transmitting a control signal for controlling the external electronic device from the electronic device.

[0006] According to one embodiment, the at least one processor may perform a short-range wireless communication connection with the controlled device based on pairing information.

[0007] According to one embodiment, the control target device may be located physically adjacent to the electronic device.

[0008] According to one embodiment, the electronic device and the controlled device may each correspond to one of a plurality of electronic devices registered to an online user account of a user of the electronic device.

[0009] According to one embodiment, the at least one user interface may include a second cursor synchronized with a first cursor displayed on a screen of the control target device, and may transmit to the control target device at least one of a motion signal for moving the second cursor or a motion signal for clicking the second cursor by a user input.

[0010] According to one embodiment, the at least one user interface includes at least one of a plurality of icons each corresponding to a plurality of control commands, an icon for receiving a voice command, and an input window for receiving a text input, and in response to receiving a user input for any one of the at least one user interface, a control signal corresponding to the user input can be transmitted to the controlled device.

[0011] In another embodiment of the present disclosure, another electronic device includes: a microphone; a speaker; a display; a communication circuit; a memory storing at least one program; and at least one processor electrically connected to the memory and executing at least one command of the program stored in the memory, wherein the at least one processor is configured to, in response to receiving a first user utterance through the microphone, the first user utterance including a first voice trigger and a composite command for allowing control by an external electronic device, transmit the first user utterance to a voice assistant server corresponding to the first voice trigger through the communication circuit, and in response to receiving information about a control device corresponding to the external electronic device and having received the first user utterance from the voice assistant server through the communication circuit, perform a short-range wireless communication connection with the control device through the communication circuit, and output a standby screen for waiting for reception of a control signal from the control device through the display.

[0012] According to one embodiment, the standby screen includes at least one user interface, the at least one user interface including a second cursor synchronized with a first cursor displayed on a screen of the control device, and in response to receiving at least one motion signal from the control device, the second cursor can be controlled according to the at least one motion signal.

[0013] According to one embodiment, the at least one user interface includes at least one of a plurality of icons each corresponding to a plurality of control commands, an icon for receiving a voice command, and an input window for receiving a text input, and in response to receiving at least one control signal corresponding to a user input from the control device, the control device identifies a first user interface corresponding to the at least one control signal, and controls the first user interface according to the at least one control signal.

[0014] According to one embodiment, the composite command for allowing control by the external electronic device may be set corresponding to the execution of an application that converts a control signal received through the communication circuit into an internal control signal of the electronic device.

[0015] In one embodiment, the at least one processor may establish a short-range wireless communication connection with the control device based on pairing information. The control device may be physically located adjacent to the electronic device.

[0016] According to one embodiment, the electronic device and the control device may each correspond to one of a plurality of electronic devices registered to an online user account of a user of the electronic device.

[0017] According to another embodiment of the present disclosure, a voice assistant server comprises: a communication circuit; a database including user information according to an online user account; a memory storing at least one program; and at least one processor electrically connected to the memory and executing at least one command of the program stored in the memory, wherein the at least one processor comprises: Through the communication circuit, a voice signal is received from each of a plurality of electronic devices included in the same wireless network within a set time interval, and based on the database, the voice signals received from at least some of the electronic devices identified by the same online user account among the plurality of electronic devices are converted into text, and the voice signals converted into text are compared to identify at least some of the electronic devices that have transmitted the same voice signal, and the same voice signal is analyzed using a natural language processing model to determine a first electronic device among the at least some of the electronic devices that have transmitted the same voice signal as a control device, and a second electronic device among the at least some of the electronic devices that have transmitted the same voice signal excluding the first electronic device is determined as a control target device, and through the communication circuit, information indicating that the first electronic device is a control device and identification information about the second electronic device are transmitted, and information indicating that the second electronic device is a control target device and identification information about the first electronic device are transmitted to the second electronic device.

[0018] According to one embodiment, the operation of analyzing the same voice signal using the natural language processing model and determining a first electronic device among the at least some electronic devices that transmitted the same voice signal as the control device may include extracting a first composite command from the same voice signal using the natural language processing model and determining the first electronic device as the control device in response to identifying that the first composite command is set to the first electronic device among the at least some electronic devices as a composite command for controlling an external electronic device.

[0019] According to one embodiment, the operation of analyzing the same voice signal using the natural language processing model and determining a first electronic device among at least some electronic devices that transmitted the same voice signal as the control device may include an operation of selecting the control device based on the voice signals having the largest signal size among the received voice signals among at least some electronic devices that transmitted the same voice signal.

[0020] According to one embodiment, the operation of determining a second electronic device among at least some electronic devices that transmitted the same voice signal excluding the first electronic device as the control target device may include extracting a first composite command from the same voice signal using the natural language processing model, and determining the second electronic device among the at least some electronic devices as the control target device in response to identifying that the first composite command is set to a composite command for allowing control by an external electronic device.

[0021] According to one embodiment, the operation of determining a second electronic device among at least some electronic devices that transmitted the same voice signal, excluding the first electronic device, as a control target device may include an operation of calculating a physical distance between the first electronic device and at least some electronic devices that transmitted the same voice signal using at least one of an RSSI signal intensity or a signal size based on received voice signals, and selecting the second electronic device based on the one that has the closest physical distance from the first electronic device.

[0022] According to one embodiment, the user information included in the database may include at least one of an online account ID, a password, a user name, a contact number, an address, and one or more user terminal information.

[0023] According to one embodiment, the user terminal information may include at least one of a user registration name, a product name, a product type, product identification information, or a product location.

[0024] According to one embodiment, the information indicating that it is a control device may include a flag indicating the control device, and the information indicating that it is a controlled target device may include a flag indicating the controlled target device.

[0025] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0026] FIG. 1 illustrates an electronic device for controlling another device based on a voice trigger, according to one embodiment of the present disclosure.

[0027] FIG. 2A is an example of a plurality of electronic devices receiving user speech in a home, according to one embodiment of the present disclosure.

[0028] FIG. 2b is an example of a control device and a controlled device corresponding to a user's utterance according to one embodiment of the present disclosure.

[0029] FIG. 3 is a flowchart illustrating a method for an electronic device to control another device based on a voice trigger, according to one embodiment of the present disclosure.

[0030] FIG. 4 is a flowchart illustrating a method for receiving a control signal from another device based on a user's speech, according to one embodiment of the present disclosure.

[0031] FIG. 5 is a flowchart illustrating a method for a voice assistant server to process a voice signal according to an embodiment of the present disclosure.

[0032] FIG. 6 is an example of user information included in a database according to one embodiment of the present disclosure.

[0033] FIG. 7 is an example of a screen of a controlled device that is synchronized with a screen of a control device according to one embodiment of the present disclosure.

[0034] FIG. 8 is an example of a screen of a controlled device that is executed in accordance with a control command of a control device according to an embodiment of the present disclosure.

[0035] FIG. 9 is an example of switching between multiple control screens of a control device according to one embodiment of the present disclosure.

[0036] FIG. 10 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0037] FIG. 11 is a block diagram of a control target device according to an embodiment of the present disclosure.

[0038] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0039] An embodiment of the present disclosure is described below with reference to the attached drawings.

[0040] FIG. 1 illustrates an electronic device for controlling another device based on voice trigger recognition and proximity, according to one embodiment of the present disclosure.

[0041] Referring to FIG. 1, a first electronic device (101) and a second electronic device (105) may be positioned within physically adjacent spaces. In one embodiment, the physically adjacent space may refer to a space in which a user's speech located within the space can be received through a microphone of the electronic device. For example, a living room space of a home may be considered a physically adjacent space. Multiple electronic devices included within a physically adjacent space may detect the same user speech through their respective microphones. According to one embodiment, the first electronic device (101) may receive a user speech "Hi Bixby (trigger) remote control mode (command)" through the microphone of the electronic device (101). According to one embodiment, the second electronic device (105) may also receive the same user speech "Hi Bixby (trigger) remote control mode (command)" through the microphone.

[0042] In one embodiment, a user utterance may include a voice trigger and a composite command. The voice trigger may include a specific word that can invoke an AI-based voice assistant. For example, "Hi Bixby" or "Hi Bixby" may represent a specific voice trigger that invokes a specific voice assistant (e.g., the Bixby server).

[0043] In one embodiment, a composite command may represent a voice command that controls the operation of multiple electronic devices. For example, a composite command called "remote control mode" may simultaneously control the operation of a user's mobile device (e.g., a smartphone) and the operation of a user's display device (e.g., a smart TV). Specifically, the mobile device may set "remote control mode" as a composite command for controlling an external electronic device (e.g., a display device), establish a short-range wireless communication connection with the display device in response to receiving the "remote control mode," and perform an operation of displaying a screen capable of controlling the display device. The display device may set "remote control mode" as a composite command for allowing control by an external electronic device (e.g., a mobile device), establish a short-range wireless communication connection with the mobile device in response to receiving the "remote control mode," and perform at least one operation for waiting for reception of a control signal from the mobile device. For example, the display device may establish a communication connection with the mobile device based on short-range wireless communication (e.g., WiFi communication) to receive a control signal directly from the mobile device, and output a screen waiting for reception of a control signal. Two electronic devices can be configured to interact with each other using the same compound command. One of the two electronic devices can be configured to use the same compound command to control an external electronic device, and the other can be configured to use the same compound command to allow control by the external electronic device. For example, if a mobile device sets "remote control mode" as a compound command to control another electronic device, and a display device sets "remote control mode" as a compound command to allow control by the other electronic device, the mobile device and the display device that receive a user utterance including "remote control mode" can become objects that can interact with each other (send and receive control signals).

[0044] According to one embodiment, the first electronic device (101) may set an application execution for transmitting a control signal for controlling an external electronic device in response to a compound command for controlling the external electronic device. For example, the first electronic device (101) may set an application execution for transmitting a control signal for controlling the external electronic device in response to a “remote control mode,” and in response to receiving the “remote control mode,” may execute the remote control application to transmit a control signal to the external electronic device.

[0045] According to one embodiment, the second electronic device (105) may set an application execution to convert a control signal received through a communication circuit into an internal control signal of the second electronic device (105) in response to a compound command for allowing control by an external electronic device. For example, the second electronic device (105) may set an application execution to convert a control signal received through the communication circuit into an internal control signal of the second electronic device (105) in response to a “remote control mode,” and in response to receiving the “remote control mode,” may execute the remote control mode application to receive a control signal from the external electronic device.

[0046] According to one embodiment, an AI-based voice assistant may be provided in the form of a cloud server. The voice assistant server (e.g., a Bixby server) may be a cloud server capable of speech recognition and natural language processing (NLP). The voice assistant server may receive a voice command corresponding to a user's utterance from an electronic device, analyze the voice command through natural language processing, and provide a response or execute a requested task corresponding to the analyzed voice command. The voice assistant server may utilize a database containing user information based on an online user account to perform voice processing on the user's electronic device (user terminal). In one embodiment, the user information may include an online account ID, password, user name, contact information, address, and one or more pieces of user terminal information. The user terminal information may include information about the electronic device owned by the user (e.g., user registration name, product name, product type, product identification information, and product location).

[0047] Referring to FIG. 1, a first electronic device (101) can receive a user utterance “Hi Bixby remote control mode” through a microphone and identify “Hi Bixby” as a voice trigger. The first electronic device (101) can transmit a “Hi Bixby remote control mode” voice signal to a voice assistant server (108) corresponding to the identified voice trigger. Similarly, a second electronic device (105) located in an adjacent space to the first electronic device (101) can receive a user utterance “Hi Bixby remote control mode” through a microphone and identify “Hi Bixby” as a voice trigger. The second electronic device (105) can transmit a “Hi Bixby remote control mode” voice signal to a voice assistant server (108) corresponding to the identified voice trigger.

[0048] According to one embodiment, the voice assistant server (108) may receive voice signals from each of the first electronic device (101) and the second electronic device (105), and determine that the same user utterance has been received through voice recognition. The voice assistant server (108) may perform a locality check to indicate whether the first electronic device (101) and the second electronic device (105) that received the same user utterance are physically adjacent to each other. The voice assistant server (108) may determine whether the first electronic device (101) and the second electronic device belong to a physically adjacent space, for example, based on whether they belong to the same wireless network. If there are multiple electronic devices that have received the same user utterance, the voice assistant server (108) may identify the electronic device that is closest to the first electronic device (101). For example, the voice assistant server (108) can determine the degree of distance from the first electronic device by comparing the RSSI (received signal strength indicator) signal strength between the first electronic device and other electronic devices, or by comparing the volume of voice utterances received by each of the plurality of electronic devices.

[0049] According to one embodiment, the voice assistant server (108) analyzes a user utterance to identify a compound command, and selects a control device (target device) to perform the command and a control target device (target device) corresponding to the command execution from among a plurality of electronic devices that transmitted the user utterance. At this time, the control device and the controlled target device may be set to be able to interact with each other using the same compound command. The control device may set the same compound command to control an external electronic device (target device), and the controlled target device may set the same compound command to allow control by the external electronic device (control device). For example, the voice assistant server (108) may identify the compound command "remote control mode" in the user utterance "Hi Bixby remote control mode." The voice assistant server (108) can select the first electronic device (101) as a control device capable of executing the “remote control mode” and can select the second electronic device (105) as a control target device that is the target of the “remote control mode”.

[0050] According to one embodiment, the voice assistant server (108) can analyze and process user utterances and transmit composite commands and identified control device / controlled device information to the first electronic device (101) and the second electronic device (105).

[0051] Referring to FIG. 1, the voice assistant server (108) may transmit information about a control target device (e.g., a second electronic device (105)) to the first electronic device (101) and transmit a composite command to cause the second electronic device (105) to perform a remote control function. The voice assistant server (108) may transmit information about a control device (e.g., the first electronic device (101)) to the second electronic device (105) and transmit a composite command to cause the second electronic device (105) to receive a control signal from the first electronic device (101). For example, the composite command may include one or more commands generated based on a command analyzed from a user utterance (e.g., "remote control mode"). Alternatively, the composite command may be the command analyzed from the user utterance (e.g., "remote control mode") itself.

[0052] FIG. 2A is an example of a plurality of electronic devices receiving user speech in a home, according to one embodiment of the present disclosure.

[0053] According to one embodiment, a plurality of electronic devices (101, 201, 202, 203, 204) may be positioned within a physically adjacent space (200). Some of the electronic devices (e.g., mobile device (101)) may be movable devices, and some of the electronic devices (e.g., air conditioner (201), TV (202), washing machine (203), refrigerator (204)) may be positioned in a fixed location. The plurality of electronic devices included within the same space (200) may each receive a user's speech within the same space. Each of the plurality of electronic devices may include a microphone and may be capable of voice recognition. Alternatively, some of the electronic devices within the same space may not receive the user's speech depending on the magnitude of the user's speech. Referring to FIG. 2A, the mobile device (101), the air conditioner (201), the TV (202), and the washing machine (203) may each receive a user's speech ("Hi Bixby remote control mode"). However, a refrigerator (204) that is far from the user's speech may not be able to receive the user's speech. Alternatively, the refrigerator (204) may not include a microphone or may not include a voice recognition function.

[0054] FIG. 2b is an example of a control device and a controlled device identified in response to a user utterance according to one embodiment of the present disclosure.

[0055] According to one embodiment, among multiple electronic devices included in the same space (200), the electronic devices may be identified as a control device and a control target device based on a user's utterance. For example, the control device may be selected from among electronic devices capable of executing a control command included in the user's utterance. The control target device may be selected from among electronic devices that may be the target of a control command included in the user's utterance. After the control device is determined first, the control target device may be determined in response to the control device. If there are multiple control target devices, the electronic device closest to the control device may be identified as the control target device.

[0056] Referring to FIG. 2B, a control device capable of executing a control command may be determined based on a composite command included in the user utterance "Hi Bixby remote control mode." For example, a mobile device (101) capable of controlling other electronic devices with the function of the composite command "remote control mode" may be determined as the control device. The mobile device (101) may perform the remote control mode function for an air conditioner (201), a TV (202), a washing machine (203), or a refrigerator (204). An air conditioner (201), a TV (202), a washing machine (203), or a refrigerator (204) capable of allowing control by other electronic devices with the function of the composite command "remote control mode" may be determined as the controlled device. Since the refrigerator (204) did not recognize the user utterance, it cannot be the controlled device. The air conditioner (201), TV (202), and washing machine (203) that can be controlled devices and recognize user speech, when listed in order of physical proximity to the mobile device (101), can be a TV (202) at a distance of d2, an air conditioner (201) at a distance of d1, and a washing machine (203) at a distance of d3. The TV (202) that is closest to the mobile device (101) can be determined as the controlled device.

[0057] FIG. 3 is a flowchart illustrating a method for an electronic device to control another electronic device based on a voice trigger, according to one embodiment of the present disclosure.

[0058] According to one embodiment, an electronic device (101) (e.g., the first electronic device (101) of FIG. 1) may control another electronic device (e.g., the second electronic device (105) of FIG. 1) using a user utterance that includes a voice trigger and a composite command for controlling an external electronic device. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0059] In operation 310, an electronic device (101) according to an embodiment may detect a voice signal including a voice trigger and a compound command through a microphone. The electronic device (101) may include information about a user's voice and may be configured to respond to the user's voice. The electronic device (101) may automatically detect a voice signal including a voice trigger during a user's speech. In an embodiment, the voice signal may include a voice trigger and a compound command. There may be more than one voice trigger. A specific voice trigger may be designated to correspond to a specific voice assistant. The electronic device (101) may include information about a specific voice trigger and a specific voice assistant corresponding to the voice trigger. The compound command may include commands that cause a plurality of electronic devices (e.g., the electronic device (101) and another electronic device (105)) to perform at least one operation. For example, the compound command may be configured to cause the electronic device (101) to control another electronic device (105).

[0060] In operation 320, an electronic device (101) according to an embodiment may transmit a voice signal to a server. The electronic device (101) may transmit a voice signal including a specific voice trigger to a specific voice assistant (e.g., the voice assistant server (108) of FIG. 1) corresponding to a specific voice trigger. In response to receiving the voice signal, the server (108) may identify an online user account of the electronic device (101) based on a database containing information about online user accounts. The server (108) may process the results of analyzing the voice signal based on the online user account of the electronic device (101).

[0061] By means of operations 330 and 340, the electronic device (101) according to one embodiment can receive information about whether it corresponds to a control device and information about a controlled target device from the server (108).

[0062] In operation 330, the electronic device (101) according to one embodiment may determine whether the electronic device (101) corresponds to the control device of the compound command by receiving information from the server (108) indicating that the electronic device (101) corresponds to the control device. In response to the electronic device (101) identifying that it corresponds to the control device, operation 340 may be performed to identify the control target device. If the electronic device (101) does not correspond to the control device, the operation based on the user utterance may be terminated without further performing. In one embodiment, the electronic device (101) may output information on the display indicating that the user utterance is unrelated to the electronic device (101). For example, the electronic device (101) may display a message such as "I cannot execute what you said" on the display or output it as a voice signal through a speaker.

[0063] In operation 340, the electronic device (101) according to one embodiment may identify the controlled device by receiving information about the controlled device from the server (108). In response to not receiving information about the controlled device or receiving information that there is no controlled device, the electronic device (101) may perform operation 351.

[0064] In operation 351, the electronic device (101) according to one embodiment may indicate the absence of a control target device through a display. After moving close to the control target device, the electronic device (101) may output a message through the display that induces utterance including a voice trigger and a compound command. For example, the electronic device (101) may display the message "Move close enough to the device you want to control, then speak again" through the display or output it as a voice signal through a speaker.

[0065] In operation 350, an electronic device (101) according to an embodiment may perform a short-range wireless communication connection with an identified control target device. A server (108) may analyze a voice signal to identify multiple electronic devices that will perform a complex command based on user device information included in an online user account. The server (108) may identify a control device that will perform a control command and a control target device that is a target of the control command based on the complex command included in the voice signal. The server (108) may transmit identification information (e.g., a control device flag) indicating that the electronic device (101) is a control device, and may transmit information about the control target device (e.g., an identification value of the control target device). The electronic device (101) may explicitly receive identification information indicating that it is a control device, but may also confirm that it is a control device by implicitly receiving information about the control target device. The control target device information may include identification information (e.g., a user registration name, a product name, a product type, product identification information, and a product location) of an electronic device registered in an online user account. For example, the electronic device (101) can receive “living room TV” corresponding to the user registration name as information about the controlled device.

[0066] In one embodiment, the electronic device (101) may be paired with a controlled device, and when the electronic device (101) receives information about the controlled device, it may immediately perform a short-range wireless communication connection with the controlled device based on the pairing information.

[0067] In a 360 motion, an electronic device (101) according to an embodiment can output a control screen for a control target device through a display. The control screen can include a user interface capable of receiving control commands for the control target device. The electronic device (101) can control the control target device by transmitting a control signal input through the user interface included in the control screen to the control target device.

[0068] FIG. 4 is a flowchart illustrating a method for an electronic device to operate as a controlled device based on a voice trigger according to an embodiment of the present disclosure.

[0069] According to one embodiment, an electronic device (105) (e.g., the second electronic device (105) of FIG. 1) may operate as a control target device of another electronic device (hereinafter, referred to as a control device) (e.g., the first electronic device (101) of FIG. 1) using a user utterance including a voice trigger. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0070] In operation 410, an electronic device (105) according to an embodiment may detect a voice signal including a voice trigger and a compound command through a microphone. The electronic device (105) may include information about a user's voice and may be configured to respond to the user's voice. The electronic device (105) may automatically detect a voice signal including a voice trigger during a user's speech. In an embodiment, the voice signal may include a voice trigger and a compound command. There may be more than one voice trigger. A specific voice trigger may be designated to correspond to a specific voice assistant. The electronic device (105) may include information about a specific voice trigger and a specific voice assistant corresponding to the voice trigger. The compound command may include commands that cause a plurality of electronic devices (e.g., the electronic device (105) and another electronic device (101)) to perform at least one operation. For example, the compound command may include commands that allow another electronic device (101) to control the electronic device (105).

[0071] In operation 420, the electronic device (105) according to one embodiment may transmit a voice signal to a server. The electronic device (105) may transmit the voice signal including the specific voice trigger to a specific voice assistant (e.g., the voice assistant server (108) of FIG. 1) corresponding to the specific voice trigger. In response to receiving the voice signal, the server (108) may identify the online user account of the electronic device (105) based on a database containing information about the online user account. The server (108) may process the results of analyzing the voice signal based on the online user account of the electronic device (105).

[0072] By means of operations 430 and 440, the electronic device (105) according to one embodiment can receive information about whether the electronic device corresponds to a control target device and information about the control device from the server (108).

[0073] In operation 430, the electronic device (105) according to one embodiment may determine whether the electronic device (105) corresponds to a control target device of a compound command by receiving information from the server (108) indicating that the electronic device (105) corresponds to a control target device. In response to the electronic device (105) identifying that the electronic device (105) corresponds to the control target device, the electronic device may perform operation 440 to identify the control device. If the electronic device (105) does not correspond to the control target device, the electronic device may terminate the operation based on the user utterance without performing any further operations. In one embodiment, the electronic device (105) may output information on the display indicating that the user utterance is unrelated to the electronic device (105). For example, the electronic device (105) may display a message such as "I cannot execute what you said" on the display or output it as a voice signal through a speaker.

[0074] In operation 440, the electronic device (105) according to one embodiment may identify the control device by receiving information about the control device from the server (108). In response to not receiving information about the control device or receiving information that the control device is not present, the electronic device (105) may perform operation 451.

[0075] In operation 451, the electronic device (105) according to one embodiment may indicate the absence of a control device through a display. After moving in proximity to the control device, the electronic device (105) may output a message through the display that induces utterance, including a voice trigger and a compound command. For example, the electronic device (105) may display the message "Move close enough to the device to send a control signal to me, and then say it again" through the display or output it as a voice signal through a speaker.

[0076] In operation 450, an electronic device (105) according to an embodiment may perform a short-range wireless communication connection with a control device identified by a server (108). The server (108) according to an embodiment may transmit information about the control device (e.g., an identification value of the control device) to the electronic device (105) identified as the control target device. The electronic device (105) may explicitly receive identification information indicating that it is a control target device, but may also implicitly receive information about the control device to confirm that it is a control target device. The information about the control device may include identification information of the electronic device registered in an online user account (e.g., a user registration name, a product name, a product type, product identification information, a product location). For example, the electronic device (105) may receive “My Galaxy Z Flip 6” (e.g., a mobile device (101)) corresponding to the user registration name as information about the control device. Based on the information about the control device, the electronic device (105) may identify the user’s mobile device (101) as the control device.

[0077] In one embodiment, the electronic device (105) may be paired with a control device, and when the electronic device (105) receives information about the control device, it may immediately perform a short-range wireless communication connection with the control device based on the pairing information.

[0078] In operation 460, the electronic device (105) according to one embodiment may wait for receiving a control signal from the control device. The electronic device (105) may perform a connection based on short-range wireless communication (e.g., WiFi communication) with the control device to directly receive the control signal from the control device. For example, the electronic device (105) may perform a short-range wireless communication connection with a mobile device among the paired electronic devices based on information about the control device. By connecting to short-range wireless communication, the electronic device (105) may directly receive a control signal from the control device.

[0079] An electronic device (105) according to an embodiment may output a screen for waiting to receive a control signal from a control device through a display. The electronic device (105) may display a standby screen including a user interface including information about the control device. For example, the electronic device (105) may display a start screen including a first cursor that is synchronized with a user interface (e.g., a cursor) displayed on the control device. The first cursor may display information (e.g., “Flip 6”) next to the first cursor indicating that it is being synchronized with the cursor of the control device.

[0080] FIG. 5 is a flowchart illustrating a method for a voice assistant server to process a voice signal according to an embodiment of the present disclosure.

[0081] According to one embodiment, a voice assistant server (108) (e.g., server (108) of FIG. 1) may receive voice signals from a plurality of electronic devices, identify a control device and a controlled device, and transmit the identification results to the plurality of electronic devices.

[0082] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0083] In operation 510, the voice assistant server (108) according to one embodiment may receive a voice signal from at least one electronic device included in the same network within a predetermined time interval. The predetermined time interval is a time for identifying electronic devices that have received the same voice signal and may be set to a relatively short time. However, the predetermined time interval may be determined by considering the time required for multiple electronic devices within physically adjacent spaces to each receive the same voice signal and the time required for multiple electronic devices to transmit the same voice signal to the voice assistant server (108) according to different communication functions. For example, the predetermined time interval may be 0.3 seconds.

[0084] In operation 520, the voice assistant server (108) according to one embodiment can determine whether a device belongs to the same user based on user information contained in a database, based on whether the devices are identified by the same online user account. The database can include user information according to online user accounts. Each user can have one online user account, and the user can be identified by the account information. The user information can include, for example, at least one of an online account ID, a password, a user name, a contact information, an address, and one or more pieces of user terminal information. The user terminal information can indicate information about an electronic device owned by the user. The user terminal information can include, for example, at least one of a user registration name, a product name, a product type, product identification information, or a product location. The voice assistant server (108) can identify the user by searching the database for identification information of the electronic device that transmitted the voice signal. Some of the multiple electronic devices contained within a physically adjacent space can be owned by more than one user, and can be registered to online user accounts by different users. For example, for a TV and a display device contained within the same space, the TV may be registered to the online user accounts of a first user (e.g., mother) and a second user (e.g., father), respectively. The display device contained within the same space may only be registered to the online user account of a third user (e.g., son).

[0085] In operation 530, the voice assistant server (108) according to one embodiment can convert voice signals received from electronic devices identified as devices of the same user into text.

[0086] In operation 540, the voice assistant server (108) according to one embodiment can compare voice signals converted to text to determine whether they are the same voice signal. The voice assistant server (108) may detect voice signals from different electronic devices, and the volume and quality of the voice signals may differ. For example, the volume of the voice signals received by each electronic device may differ depending on the distance from the user's speech, and each electronic device may contain different noise depending on the surrounding environment. The voice assistant server (108) can convert the voice signals into text and determine whether they correspond to the same text, thereby making a more accurate judgment. The voice assistant server (108) can identify only the electronic devices that transmitted the same voice among the electronic devices of the same user and consider them as processing targets for the same voice signal. If the voice signals are not the same, the voice assistant server (108) can exclude the corresponding electronic device from the processing targets for the voice signal.

[0087] According to one embodiment, the voice assistant server (108) can analyze voice signals transmitted by electronic devices to determine a control device and a controlled device. Since the voice assistant server (108) can process requests from an unspecified number of electronic devices, the voice assistant server (108) can determine whether the received voice signals are from electronic devices included in the same wireless communication network, electronic devices received within a certain interval, electronic devices included in the same online user account, or the same voice signal, thereby identifying the electronic devices that received the same voice signal. For convenience of explanation, the same voice signal may be referred to as a first voice signal hereinafter.

[0088] In operation 550, a voice assistant server (108) according to an embodiment may analyze a first voice signal using a natural language processing (NLP) model to determine a control device among electronic devices that transmitted the first voice signal. The first voice signal may include a composite command. The composite command may represent a command for controlling operations of multiple electronic devices. In this case, the control device and the controlled device may be set to be able to interact with each other using the same composite command. The control device may set the same composite command to control an external electronic device (the controlled device), and the controlled device may set the same composite command to allow control by the external electronic device (the control device). An electronic device that includes a function for executing a control command included in the composite command may be the control device. For example, if the composite command is "remote control mode," an electronic device (e.g., a mobile device) capable of performing a remote control mode function may be the control device. An electronic device (e.g., an air conditioner) that cannot perform a remote control mode function cannot be the control device. The voice assistant server (108) can analyze the first voice signal, extract a control command, and distinguish between electronic devices that can execute the extracted control command and electronic devices that cannot execute the command.

[0089] According to one embodiment, the voice assistant server (108) may select, as the control device, the electronic device corresponding to the loudest signal among the received voice signals among the electronic devices that can be the control device. Considering the user's intention, the electronic device that the user wants to use as the control device may be located closest to the point where the user's utterance occurs. For example, the user may hold a mobile device (e.g., a smartphone) and approach the control target device (e.g., a TV) and utter "Hi Bixby remote control mode." In this case, the electronic device that receives the user's utterance closest to the control target device may be the mobile device, and the electronic device that receives the user's utterance next closest to the control target device may be the TV.

[0090] In operation 560, the voice assistant server (108) according to one embodiment may analyze the same voice signal (the first voice signal) to determine a control target device. The voice assistant server (108) may select a control target device from among electronic devices that have transmitted the same voice signal and that can be the target of execution of the control command included in the voice signal. For example, if the compound command is "remote control mode," a mobile device may be excluded from the control target devices because it is difficult to execute with a remote control.

[0091] According to one embodiment, the voice assistant server (108) may determine the electronic device that is physically closest to the control device among the electronic devices that transmitted the same voice signal as the control device. The voice assistant server (108) may calculate the distance between the control device and other devices by comparing the received signal strength indicator (RSSI) signal strength or signal size of the received voice signals. The voice assistant server (108) may determine the electronic device located at the closest distance to the control device determined in operation 550 as the control target device.

[0092] In operation 570, the voice assistant server (108) according to one embodiment may transmit identification information to the control device and the controlled device, respectively. The voice assistant server (108) may transmit information (e.g., a flag) indicating that the control device is the control device, and may transmit identification information (e.g., a registered name included in an online user account) for the controlled device. In addition, the voice assistant server (108) may transmit information (e.g., a flag) indicating that the control device is the control device, and may transmit identification information (e.g., a registered name included in an online user account) for the controlled device.

[0093] According to one embodiment, the voice assistant server (108) may transmit unidentified information to the control device or the controlled device when the control device or the controlled device is not identified.

[0094] According to another embodiment of the present disclosure, a voice assistant server comprises: a communication circuit; a database including user information according to an online user account; a memory storing at least one program; and at least one processor electrically connected to the memory and executing at least one command of the program stored in the memory, wherein the at least one processor comprises: Through the communication circuit, a voice signal is received from each of a plurality of electronic devices included in the same wireless network within a set time interval, and based on the database, the voice signals received from at least some of the electronic devices identified by the same online user account among the plurality of electronic devices are converted into text, and the voice signals converted into text are compared to identify at least some of the electronic devices that have transmitted the same voice signal, and the same voice signal is analyzed using a natural language processing model to determine a first electronic device among the at least some of the electronic devices that have transmitted the same voice signal as a control device, and a second electronic device among the at least some of the electronic devices that have transmitted the same voice signal excluding the first electronic device is determined as a control target device, and through the communication circuit, information indicating that the first electronic device is a control device and identification information about the second electronic device are transmitted, and information indicating that the second electronic device is a control target device and identification information about the first electronic device are transmitted to the second electronic device.

[0095] According to one embodiment, the operation of analyzing the same voice signal using the natural language processing model and determining a first electronic device among at least some electronic devices that transmitted the same voice signal as the control device may include extracting a first composite command from the same voice signal using the natural language processing model and determining the first electronic device as the control device in response to identifying that the first composite command is set to the first electronic device among the at least some electronic devices as a composite command for controlling an external electronic device.

[0096] According to one embodiment, the operation of analyzing the same voice signal using the natural language processing model and determining a first electronic device among at least some electronic devices that transmitted the same voice signal as the control device may include an operation of selecting the control device based on the voice signals having the largest signal size among the received voice signals among at least some electronic devices that transmitted the same voice signal.

[0097] According to one embodiment, the operation of determining a second electronic device among at least some electronic devices that transmitted the same voice signal excluding the first electronic device as the control target device may include extracting a first composite command from the same voice signal using the natural language processing model, and determining the second electronic device as the control target device in response to identifying that the first composite command is set to a composite command for allowing control by an external electronic device in the second electronic device among the at least some electronic devices.

[0098] According to one embodiment, the operation of determining a second electronic device among at least some electronic devices that transmitted the same voice signal, excluding the first electronic device, as a control target device may include an operation of calculating a physical distance between the first electronic device and at least some electronic devices that transmitted the same voice signal using at least one of an RSSI signal intensity or a signal size based on received voice signals, and selecting the second electronic device based on the one that has the closest physical distance from the first electronic device.

[0099] According to one embodiment, the user information included in the database may include at least one of an online account ID, a password, a user name, a contact number, an address, and one or more user terminal information.

[0100] According to one embodiment, the user terminal information may include at least one of a user registration name, a product name, a product type, product identification information, or a product location.

[0101] According to one embodiment, the information indicating that it is a control device may include a flag indicating the control device, and the information indicating that it is a controlled target device may include a flag indicating the controlled target device.

[0102] FIG. 6 is an example of user information included in a database according to one embodiment of the present disclosure.

[0103] According to one embodiment, a voice assistant server (e.g., server (108) of FIG. 1) can identify users of electronic devices using a database containing user information.

[0104] In one embodiment, the database may include user information matching an online user account. Each user may have one online user account, and the user may be identified by the account information. The user information may include, for example, at least one of an online account ID, a password, a user name, a contact information, an address, and one or more user device information. The user device information may indicate information about an electronic device owned by the user. The user device information may include, for example, at least one of a user registration name, a product name, a product type, product identification information, or a product location.

[0105] Referring to FIG. 6, a database according to one embodiment may include first user (User1) information (610), second user (User2) information (620), and third user (User3) information (630). The database may register and manage a voice recognition device among the user information.

[0106] The first user (User1) information (610) includes information about three voice recognition devices of the first user. The second user (User2) information (620) includes information about four voice recognition devices of the second user. The third user (User3) information (630) includes information about two voice recognition devices of the third user.

[0107] According to one embodiment, a voice assistant server (108) may search a database for an online user account corresponding to a user of a first electronic device that has transmitted a voice signal using the identification information of the first electronic device. The first electronic device may be one of the voice recognition devices registered to the online user account.

[0108] The same electronic device may be registered to an online user account by multiple users. For example, Device 1 registered by User 1 may be the same electronic device as Device 4 registered by User 2.

[0109] FIG. 7 is an example of a screen of a controlled device that is synchronized with a screen of a control device according to one embodiment of the present disclosure.

[0110] An electronic device (101) according to one embodiment (e.g., the first electronic device (101) of FIG. 1) may be a control device and display a screen for controlling a control target device (105) (e.g., the second electronic device (105) of FIG. 1).

[0111] An electronic device (101) according to one embodiment may display a first screen (710) including at least one user interface (e.g., a first cursor (711)) for controlling a control target device (105). The first cursor (711) of the control screen (710) may be synchronized with a user interface (e.g., a second cursor (721)) included in a second screen (720) of the control target device (105).

[0112] An electronic device (101) according to one embodiment may transmit, to a control target device (105), at least one of a motion signal for moving a first cursor or a motion signal for clicking the first cursor by user input for synchronization.

[0113] According to one embodiment, the control target device (105) can move a second cursor in response to a motion signal for moving a first cursor received from the electronic device (101). The control target device (105) can be controlled to click a second cursor in response to a motion signal for clicking the first cursor received from the electronic device (101).

[0114] Referring to FIG. 7, a user can simultaneously check a second screen (720) of a control target device (105) and a first screen (710) of an electronic device (101), and, through a user input to a first cursor (711) included in the first screen (710), can substantially control a second cursor included in a second screen (720) of the control target device (105).

[0115] FIG. 8 is an example of a screen of a controlled device that is executed in accordance with a control command of a control device according to an embodiment of the present disclosure.

[0116] According to one embodiment, a control device (101) (e.g., the first electronic device (101) of FIG. 1) can control a control target device (105) (e.g., the second electronic device (105) of FIG. 1) according to a user input.

[0117] A control device (101) according to one embodiment may display a control screen including at least one user interface for controlling a control target device (105). In one embodiment, the at least one user interface may include at least one of a plurality of icons each corresponding to a plurality of control commands, an icon for receiving a voice command, or an input window for receiving a text input.

[0118] A control device (101) according to one embodiment may, in response to receiving a user input for at least one user interface, transmit a control signal corresponding to the user input to a controlled device (105).

[0119] In one embodiment, the control device (101) and the controlled device (105) can receive a user utterance such as “Hi Bixby remote control mode, play the movie Rocketman.” The control device (101) and the controlled device (105) can each transmit a voice signal including a user utterance to a voice assistant server (e.g., the server (108) of FIG. 1). The control device (101) can receive from the voice assistant server (108) information identifying the control device and information indicating that the controlled device is a TV. The control device (101) can analyze a complex command included in the user utterance to extract a control command. For example, the control device (101) can generate a “control signal for searching for the movie Rocket Man” and a “control signal for clicking a play button among the search results” in response to “Play the movie Rocket Man”, and transmit the generated control signals to the TV identified as the controlled device (105). The control device (101) can generate a control signal according to the complex command, transmit the control signal to the controlled device (105), and output information related to the generated control signal on the screen (810). Referring to FIG. 8, the control The device (101) can display a screen (810) that displays “TV remote control mode” and the text “Play the movie Rocket Man on TV” in response to “Play the movie Rocket Man on TV.”

[0120] According to one embodiment, the control target device (105) may receive from the voice assistant server (108) information identifying the control target device and information indicating that the control device is a mobile device. The control target device (105) may perform a communication connection based on short-range wireless communication with the control device (101) in order to wait for receiving a control signal from the control device (101). The control target device (105) may process the control signal in response to receiving a “control signal for searching for the movie Rocketman” and a “control signal for clicking a play button among the search results” from the control device (101). The control target device (105) may process the control signal to output a search result for the movie Rocketman and a screen (820) for playing the Rocketman movie.

[0121] FIG. 9 is an example of switching between multiple control screens of a control device according to one embodiment of the present disclosure.

[0122] According to one embodiment, a control device (101) (e.g., the first electronic device (101) of FIG. 2A) may be a control device that controls a plurality of electronic devices based on a voice trigger. Referring to FIG. 2A together, in a first situation, a user may carry the first electronic device (101) (hereinafter, referred to as the control device (101)), move close to an air conditioner (201), and then utter “Hi Bixby remote control mode.” Thereafter, in a second situation, the user may carry the control device (101), move close to a TV (202), and then utter “Hi Bixby remote control mode.”

[0123] In the first situation, the control device (101) transmits a user utterance to a voice assistant server (e.g., server (108) of FIG. 1) corresponding to a voice trigger (“Hi Bixby”), and as a control device for the air conditioner (201), can output a control screen (910) for the air conditioner (201) on the display.

[0124] In a second situation, the control device (101) may transmit a user utterance to a voice assistant server (e.g., server (108) of FIG. 1) corresponding to a voice trigger (“Hi Bixby”), and, as a control device for the TV (202), may output a control screen (920) for the TV (202) to the display.

[0125] A control device (101) according to one embodiment may include a control screen (910) capable of performing a remote control function for an air conditioner (201) and a control screen (920) capable of performing a remote control function for a TV (202), and may switch and display the control screen on a display by user input (e.g., swipe gesture).

[0126] FIG. 10 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0127] Referring to FIG. 10, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0128] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0129] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0130] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0131] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0132] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0133] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0134] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0135] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0136] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0137] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) to an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multi-media interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0138] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0139] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0140] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0141] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0142] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0143] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0144] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for realizing URLLC.

[0145] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0146] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0147] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0148] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0149] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0150] An electronic device according to one embodiment of the present disclosure includes a microphone; a speaker; a display; a communication circuit; a memory storing at least one program; and at least one processor electrically connected to the memory and executing at least one command of the program stored in the memory, wherein the at least one processor is configured to: in response to receiving a first user utterance including a first voice trigger and a composite command for controlling an external electronic device through the microphone, transmit the first user utterance to a voice assistant server corresponding to the first voice trigger through the communication circuit; in response to receiving information about a control target device corresponding to the external electronic device and having received the first user utterance from the voice assistant server through the communication circuit, perform a short-range wireless communication connection with the control target device through the communication circuit; and display a control screen including at least one user interface for controlling the control target device through the display.

[0151] According to one embodiment, the composite command for controlling the external electronic device may be set to correspond to the execution of an application for transmitting a control signal for controlling the external electronic device from the electronic device.

[0152] According to one embodiment, the at least one processor may perform a short-range wireless communication connection with the controlled device based on pairing information.

[0153] According to one embodiment, the control target device may be located physically adjacent to the electronic device.

[0154] According to one embodiment, the electronic device and the controlled device may each correspond to one of a plurality of electronic devices registered to an online user account of a user of the electronic device.

[0155] According to one embodiment, the at least one user interface may include a second cursor synchronized with a first cursor displayed on a screen of the control target device, and may transmit to the control target device at least one of a motion signal for moving the second cursor or a motion signal for clicking the second cursor by a user input.

[0156] According to one embodiment, the at least one user interface includes at least one of a plurality of icons each corresponding to a plurality of control commands, an icon for receiving a voice command, and an input window for receiving a text input, and in response to receiving a user input for any one of the at least one user interface, a control signal corresponding to the user input can be transmitted to the controlled device.

[0157] FIG. 11 is a block diagram of a control target device according to an embodiment of the present disclosure.

[0158] According to one embodiment, a control target device (105) (e.g., the second electronic device (105) of FIG. 1, the TV (202) of FIG. 2a, the TV (202) of FIG. 2b, the control target device (105) of FIG. 4, the control target device (105) of FIG. 7, the control target device (105) of FIG. 8) may be a control target device that receives and processes a control signal from another electronic device (e.g., the first electronic device (101) of FIG. 1, the electronic device (101) of FIG. 2a, the electronic device (101) of FIG. 2b, the control device (101) of FIG. 4, the control device (101) of FIG. 7, the control device (101) of FIG. 8, the electronic device (101) of FIG. 10) based on a user utterance including a voice trigger.

[0159] According to one embodiment, the control target device (105) may include at least some of a processor (1101), a memory (1102), a communication module (1103), a power management module (1104), a display (1105), a speaker (1106), a microphone (1107), and an input / output interface (1108). Among the components of the electronic device (101) described in FIG. 10, the same components as those of the control target device (105) may be included in whole or in part in the components of the control target device (105). For example, the processor (1101) of the control target device (105) may be at least partly or entirely identical to the processor (120) of the electronic device (101).

[0160] In another embodiment of the present disclosure, another electronic device includes: a microphone; a speaker; a display; a communication circuit; a memory storing at least one program; and at least one processor electrically connected to the memory and executing at least one command of the program stored in the memory, wherein the at least one processor is configured to, in response to receiving a first user utterance through the microphone, the first user utterance including a first voice trigger and a composite command for allowing control by an external electronic device, transmit the first user utterance to a voice assistant server corresponding to the first voice trigger through the communication circuit, and in response to receiving information about a control device corresponding to the external electronic device and having received the first user utterance from the voice assistant server through the communication circuit, perform a short-range wireless communication connection with the control device through the communication circuit, and output a standby screen for waiting for reception of a control signal from the control device through the display.

[0161] According to one embodiment, the standby screen includes at least one user interface, the at least one user interface including a second cursor synchronized with a first cursor displayed on a screen of the control device, and in response to receiving at least one motion signal from the control device, the second cursor can be controlled according to the at least one motion signal.

[0162] According to one embodiment, the at least one user interface includes at least one of a plurality of icons each corresponding to a plurality of control commands, an icon for receiving a voice command, and an input window for receiving a text input, and in response to receiving at least one control signal corresponding to a user input from the control device, the control device identifies a first user interface corresponding to the at least one control signal, and controls the first user interface according to the at least one control signal.

[0163] According to one embodiment, the composite command for allowing control by the external electronic device may be set corresponding to the execution of an application that converts a control signal received through the communication circuit into an internal control signal of the electronic device.

[0164] In one embodiment, the at least one processor may establish a short-range wireless communication connection with the control device based on pairing information. The control device may be physically located adjacent to the electronic device.

[0165] According to one embodiment, the electronic device and the control device may each correspond to one of a plurality of electronic devices registered to an online user account of a user of the electronic device.

[0166] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0167] The term "module" used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0168] One embodiment of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0169] According to one embodiment, the method according to one embodiment disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0170] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, mike; speaker; display; communication circuit; memory for storing at least one program; and At least one processor electrically connected to the memory and executing at least one instruction of a program stored in the memory, At least one processor; In response to receiving a first user utterance including a first voice trigger and a composite command for controlling an external electronic device through the microphone, transmitting the first user utterance to a voice assistant server corresponding to the first voice trigger through the communication circuit; In response to receiving information about a control target device corresponding to the external electronic device and receiving the first user utterance from the voice assistant server through the communication circuit, performing a short-range wireless communication connection with the control target device through the communication circuit, and An electronic device configured to display a control screen including at least one user interface for controlling the control target device through the display.

2. In paragraph 1, An electronic device, wherein the composite command for controlling the external electronic device is set to correspond to the execution of an application for transmitting a control signal for controlling the external electronic device from the electronic device.

3. In paragraph 1, An electronic device, wherein at least one processor performs a short-range wireless communication connection with the controlled device based on pairing information.

4. In paragraph 1, An electronic device, wherein the electronic device and the control target device each correspond to one of a plurality of electronic devices registered to an online user account of a user of the electronic device.

5. In paragraph 1, wherein said at least one user interface comprises a second cursor synchronized with a first cursor displayed on a screen of said controlled device; An electronic device that transmits at least one of a motion signal for moving the second cursor or a motion signal for clicking the second cursor to the controlled device by user input.

6. In paragraph 1, The at least one user interface includes at least one of a plurality of icons each corresponding to a plurality of control commands, an icon for receiving a voice command, and an input window for receiving a text input, An electronic device, in response to receiving a user input for any one of the at least one user interface, transmitting a control signal corresponding to the user input to the controlled device.

7. In electronic devices, mike; speaker; display; communication circuit; memory for storing at least one program; and At least one processor electrically connected to the memory and executing at least one instruction of a program stored in the memory, At least one processor, In response to receiving a first user utterance including a first voice trigger and a composite command for allowing control by an external electronic device through the microphone, transmitting the first user utterance to a voice assistant server corresponding to the first voice trigger through the communication circuit; In response to receiving information about a control device corresponding to the external electronic device and receiving the first user utterance from the voice assistant server through the communication circuit, performing a short-range wireless communication connection with the control device through the communication circuit, and An electronic device set to output a standby screen for waiting for receiving a control signal from the control device through the display.

8. In paragraph 7, The above waiting screen includes at least one user interface, wherein said at least one user interface comprises a second cursor synchronized with a first cursor displayed on a screen of said control device; An electronic device, in response to receiving at least one operation signal from the control device, for controlling the second cursor according to the at least one operation signal.

9. In paragraph 7, The at least one user interface includes at least one of a plurality of icons each corresponding to a plurality of control commands, an icon for receiving a voice command, or an input window for receiving a text input, In response to receiving at least one control signal corresponding to a user input from the control device, identifying a first user interface corresponding to the at least one control signal, and An electronic device configured to control the first user interface according to at least one control signal.

10. In paragraph 7, An electronic device, wherein the composite command for allowing control by the external electronic device is set in response to the execution of an application that converts a control signal received through the communication circuit into an internal control signal of the electronic device.

11. In the voice assistant server, communication circuit; A database containing user information according to online user accounts; Memory that stores at least one program; At least one processor electrically connected to the memory and executing at least one instruction of a program stored in the memory, At least one processor; Through the above communication circuit, voice signals are received from a plurality of electronic devices included in the same wireless network communication network within a set time interval, Based on the database, for at least some of the electronic devices identified by the same online user account among the plurality of electronic devices, converting voice signals received from the at least some of the electronic devices into text, By comparing the voice signals converted to the above text, identifying at least some electronic devices that transmitted the same voice signal, By analyzing the same voice signal using a natural language processing model, the first electronic device among at least some electronic devices that transmitted the same voice signal is determined as the control device, Among at least some electronic devices that transmitted the same voice signal excluding the first electronic device, a second electronic device is determined as a control target device, and A voice assistant server that transmits, through the communication circuit, information indicating that the first electronic device is a control device and identification information about the second electronic device, and transmits, through the communication circuit, information indicating that the second electronic device is a control target device and identification information about the first electronic device.

12. In paragraph 11, An operation of analyzing the same voice signal using the natural language processing model and determining a first electronic device among at least some electronic devices that transmitted the same voice signal as a control device is performed. Using the above natural language processing model, extracting a first composite command from the same speech signal, A voice assistant server, in response to identifying that the first composite command is set to a composite command for controlling an external electronic device among at least some of the electronic devices, to determine the first electronic device as the control device.

13. In paragraph 12, An operation of analyzing the same voice signal using the natural language processing model and determining a first electronic device among at least some electronic devices that transmitted the same voice signal as a control device is performed. A voice assistant server, comprising an operation for causing the control device to select, based on the largest signal size among the received voice signals among at least some electronic devices that transmitted the same voice signal.

14. In paragraph 12, An operation of determining a second electronic device among at least some electronic devices that transmitted the same voice signal excluding the first electronic device as a control target device is as follows: Using the above natural language processing model, extracting a first composite command from the same speech signal, A voice assistant server, in response to identifying that the first composite command is set to a composite command for allowing control by an external electronic device among at least some of the electronic devices, to determine the second electronic device as the control target device.

15. In paragraph 11 or 14, An operation of determining a second electronic device among at least some electronic devices that transmitted the same voice signal excluding the first electronic device as a control target device is as follows: Calculating the physical distance between the first electronic device and at least some electronic devices that transmitted the same voice signal using at least one of the received signal strength indicator (RSSI) signal strength or the volume of the voice signal based on the received voice signals, A voice assistant server comprising an action that causes the second electronic device to be selected based on the closest physical distance to the first electronic device.

Images