Electronic device and method for controlling external electronic device based on usage pattern information corresponding to user

Abstract

An electronic device is provided. The electronic device may include: a microphone; a communication module including a communication circuit configured to be connected to one or more external electronic devices; and a processor. The processor may be configured to: identify a user based on at least voice data including phoneme information acquired using a microphone; identify first usage pattern information corresponding to the first user based on a specified artificial neural network (ANN) algorithm when the user is a first user, and control a first specified electronic device belonging to one or more external electronic devices and corresponding to the phoneme information via the communication module based on the identified first usage pattern information to perform a first specified operation; and identify second usage pattern information corresponding to the second user based on a specified ANN algorithm when the user is a second user, and control a second specified electronic device belonging to one or more external electronic devices and corresponding to the phoneme information via the communication module based on the identified second usage pattern information to perform a second specified operation. Various embodiments of the present disclosure may use an ANN algorithm to easily control at least one electronic device desired by a user based on user pattern information corresponding to the user.

Description

Technical Field

[0001] The present disclosure relates to a method of controlling an external electronic device based on usage pattern information corresponding to a user. Background Art

[0002] With the development of Internet of Things (IoT) technology, the number of IoT devices that provide various services is increasing. For example, IoT devices that have been combined with IoT technology may include products related to real life, such as TVs, refrigerators, washing machines, indoor lights, air conditioning and heating systems, and audio. In addition to personal products, products used by the public may also be included in IoT devices. Multiple IoT devices may receive commands for performing specific functions and perform specific functions corresponding to the received commands.

[0003] Multiple IoT devices can be controlled by a specific device (e.g., a hub device) and a specific program. Multiple IoT devices can be controlled in response to previous training information based on artificial neural network (ANN) technology. ANN technology has a good possibility that it will develop in the near future and can be used throughout the industry. ANN training algorithms corresponding to ANN technology may include recursive neural network (RNN) algorithms and convolutional neural network (CNN) algorithms. RNN algorithms may be algorithms for identifying the order of operations over time and performing training based on the continuity of corresponding information. CNN algorithms may be algorithms for analyzing static images. Summary of the invention

[0004] Technical issues

[0005] As the number of IoT devices to which IoT technology has been applied increases, a process of determining one of the plurality of IoT devices and issuing a command in order to execute a specific function in the determined IoT device may be difficult.

[0006] Solution to the problem

[0007] According to various example embodiments, in order to more efficiently control a plurality of IoT devices, a user may issue a command to at least one of the plurality of IoT devices using voice.

[0008] Various embodiments of the present disclosure may provide a method for at least one IoT device to perform a specific function based on an ANN algorithm when using a voice command to control at least one of a plurality of IoT devices.

[0009] Advantageous Effects of the Invention

[0010] Various embodiments of the present disclosure may include using voice commands to determine at least one of a plurality of IoT devices and controlling the determined at least one IoT device to perform a specific function. In various embodiments of the present disclosure, a user may more conveniently control at least one IoT device by performing the above operations based on an ANN algorithm.

[0011] In various embodiments of the present disclosure, the ANN algorithm can be used to easily control the functions of the IoT device desired by the user based on the usage pattern information corresponding to the user. Therefore, the convenience of the user's control over the IoT device can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0013] Figure 1 is a block diagram illustrating an electronic device within a network environment according to various embodiments;

[0014] Figure 2 is a diagram showing a pairing state between an electronic device and a plurality of IoT devices according to various embodiments;

[0015] Figure 3 is a flow chart illustrating an example method of controlling an IoT device according to voice commands according to various embodiments;

[0016] Figure 4 is a flowchart illustrating an example method of controlling an external electronic device based on configuration information corresponding to a user according to various embodiments;

[0017] Figure 5 is a diagram illustrating an example method of controlling an external electronic device based on configuration information corresponding to a user according to various embodiments;

[0018] Figure 6 is a flow chart illustrating an example method of controlling an external electronic device based on a voice command and a gaze direction of a pupil according to various embodiments; and

[0019] Figure 7 is a diagram illustrating an example method of controlling an external electronic device based on a voice command and a gaze direction of a pupil according to various embodiments. DETAILED DESCRIPTION

[0020] Figure 1 1 is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments. Figure 1, the electronic device 101 in the network environment 100 may communicate with the electronic device 102 via the first network 198 (e.g., a short-range wireless communication network), or communicate with the electronic device 104 or the server 108 via the second network 199 (e.g., a long-range wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 via the server 108. According to an embodiment, the electronic device 101 may include a processor 120, a memory 130, an input device 150, a sound output device 155, a display device 160, an audio module 170, a sensor module 176, an interface 177, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a user identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the components (e.g., the display device 160 or the camera module 180) may be omitted from the electronic device 101, or one or more other components may be added to the electronic device 101. In some embodiments, some of the components may be implemented as a single integrated circuit.For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented to be embedded in the display device 160 (eg, a display).

[0021] The processor 120 may run, for example, software (e.g., program 140) to control at least one other component (e.g., hardware component or software component) of the electronic device 101 connected to the processor 120, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, the processor 120 may load a command or data received from another component (e.g., sensor module 176 or communication module 190) into the volatile memory 132, process the command or data stored in the volatile memory 132, and store the resultant data in the non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is independent of or combined with the main processor 121 in operation. Additionally or alternatively, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to be specifically used for a specified function. The auxiliary processor 123 may be implemented separately from the main processor 121 , or as part of the main processor 121 .

[0022] When the main processor 121 is in an inactive (e.g., sleep) state, the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., display device 160, sensor module 176, or communication module 190) among the components of the electronic device 101 (not the main processor 121), or when the main processor 121 is in an active state (e.g., running an application), the auxiliary processor 123 may control at least some of the functions or states related to at least one component (e.g., display device 160, sensor module 176, or communication module 190) among the components of the electronic device 101 together with the main processor 121. According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 180 or communication module 190) that is functionally related to the auxiliary processor 123.

[0023] The memory 130 may store various data used by at least one component of the electronic device 101 (e.g., the processor 120 or the sensor module 176). The various data may include, for example, software (e.g., the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

[0024] The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system (OS) 142 , middleware 144 , or applications 146 .

[0025] The input device 150 may receive commands or data to be used by other components (eg, the processor 120) of the electronic device 101 from outside the electronic device 101 (eg, a user). The input device 150 may include, for example, a microphone, a mouse, or a keyboard.

[0026] The sound output device 155 can output a sound signal to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records, and the receiver can be used for incoming calls. Depending on the embodiment, the receiver can be implemented as a separate part from the speaker, or as a part of the speaker.

[0027] The display device 160 may visually provide information to the outside of the electronic device 101 (e.g., a user). The display device 160 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to an embodiment, the display device 160 may include a touch circuit adapted to detect a touch or a sensor circuit (e.g., a pressure sensor) adapted to measure the strength of a force caused by a touch.

[0028] The audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 may obtain sound via the input device 150, or output sound via the sound output device 155 or an earphone of an external electronic device (e.g., electronic device 102) directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0029] The sensor module 176 may detect an operating state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) outside the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0030] The interface 177 may support one or more specific protocols to be used to connect the electronic device 101 directly (e.g., wired) or wirelessly to an external electronic device (e.g., the electronic device 102). According to an embodiment, the interface 177 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0031] The connection end 178 may include a connector, wherein the electronic device 101 can be physically connected to an external electronic device (e.g., the electronic device 102) via the connector. According to an embodiment, the connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0032] The haptic module 179 may convert the electrical signal into mechanical stimulation (eg, vibration or movement) or electrical stimulation that can be recognized by the user via his sense of touch or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0033] The camera module 180 may capture still images or moving images. According to embodiments, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

[0034] The power management module 188 may manage power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0035] The battery 189 may power at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0036] The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. The communication module 190 may include one or more communication processors capable of operating independently from the processor 120 (e.g., an application processor (AP)) and supporting direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip), or these various types of communication modules can be implemented as multiple components separated from each other (e.g., multiple chips). The wireless communication module 192 can identify and authenticate the electronic device 101 in a communication network (such as the first network 198 or the second network 199) using user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0037] The antenna module 197 may transmit or receive a signal or power to or from the outside of the electronic device 101 (e.g., an external electronic device). According to an embodiment, the antenna module 197 may include one or more antennas, and thus, at least one antenna suitable for a communication scheme used in a communication network (such as the first network 198 or the second network 199) may be selected by, for example, the communication module 190 (e.g., the wireless communication module 192). A signal or power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna.

[0038] At least some of the above components may be connected to each other via an inter-peripheral communication scheme (e.g., a bus, general purpose input output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and communicatively send signals (e.g., commands or data) therebetween.

[0039] According to an embodiment, a command or data may be sent or received between the electronic device 101 and the external electronic device 104 via the server 108 connected to the second network 199. Each of the electronic device 102 and the electronic device 104 may be a device of the same type as the electronic device 101, or a device of a different type from the electronic device 101. According to an embodiment, all or some operations to be executed in the electronic device 101 may be executed in one or more of the external electronic device 102, the external electronic device 104, or the server 108. For example, if the electronic device 101 should automatically execute a function or service or should execute a function or service in response to a request from a user or another device, the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service instead of executing the function or service, or the electronic device 101 may request the one or more external electronic devices to execute at least part of the function or service in addition to executing the function or service. The one or more external electronic devices that receive the request may execute the requested at least part of the function or service, or execute another function or another service related to the request, and send the result of the execution to the electronic device 101. The electronic device 101 may provide the result as at least a partial reply to the request with or without further processing the result. To this end, for example, cloud computing technology, distributed computing technology, or client-server computing technology may be used.

[0040] Figure 2 is a diagram illustrating a pairing status between an electronic device and a plurality of IoT devices according to various embodiments.

[0041] Reference Figure 2 , the electronic device 101 may use a communication module including, for example, various communication circuits (eg, Figure 1 The communication module 190 of the present invention performs communication with multiple IoT devices through a wired communication channel or a wireless communication channel. Multiple IoT devices may include, for example, but are not limited to, a washing machine 201, a refrigerator 202, a television (e.g., TV1 203 and TV2 204), an audio device 205, a curtain 206 (e.g., an automation system capable of opening and closing curtains), a closed-circuit television (CCTV) 207, an intercom 208, a lighting system 209, and an air conditioning and heating system 210 (e.g., an air conditioner). The IoT devices may be two or more identical products. The products included in the same category may be two or more. For example, a TV (e.g., TV1 203) may be located in a living room, while another TV (e.g., TV2 204) may be located in a study. The functions of the lighting system 209 and the air conditioning and heating system 210 may be performed according to at least one room.

[0042] According to various embodiments, the electronic device 101 can determine at least one IoT device in response to a user's command, and can control the determined IoT device to perform at least one function. According to various embodiments, when a voice command is issued to an IoT device and the command is processed, the user's usage pattern (e.g., usage pattern information and / or configuration information) can be identified based on an ANN algorithm (e.g., an RNN algorithm and / or a long short-term memory (LSTM) algorithm). For example, an RNN algorithm, for example, may involve a deep learning model for training data that changes over time, such as time series data. An LSTM algorithm, for example, may refer to a deep learning model with the supplemented shortcomings of an RNN algorithm. An LSTM algorithm, for example, may refer to an algorithm that can maintain training information for a long period of time by supplementing an RNN algorithm whose training information may be lost. The electronic device 101 according to various embodiments can identify a user who has sent a voice command, can identify trained usage pattern information (e.g., configuration information) based on the usage pattern information identified based on the ANN algorithm according to the identified user, and can control at least one IoT device. According to various embodiments, the usage pattern information may, for example, include but is not limited to information about the frequency of using at least one external electronic device based on the user, information about the time when the external electronic device is used, information about external electronic devices of the same type that are used relatively more frequently, information about the user, information about the user's preferred external electronic devices, time information, weather information, etc.

[0043] Figure 3 is a flow chart illustrating an example method of controlling an IoT device according to voice commands according to various embodiments.

[0044] Reference Figure 3 In operation 301, an electronic device (eg Figure 1 A processor (eg, Figure 1 The processor 120 can be connected via a microphone (e.g. Figure 1 The input device 150 receives the user's voice to detect the voice command. For example, but not limited to, the electronic device 101 may be a hub device configured to manage or control IoT devices, and may maintain a pairing state with at least one IoT device. IoT devices may include electronic products used in real life and related to individual users, such as, but not limited to, washing machines, refrigerators, and TVs, and may include electronic products used in real life and publicly used by multiple users.

[0045] In operation 303, the processor 120 of the electronic device 101 may recognize the user based on the detected voice command. For example, the electronic device 101 may have a user program that has been stored in a memory (e.g., Figure 1 The processor 120 may identify the user corresponding to the voice command by comparing the detected voice command with the audio information stored in the memory 130. According to various embodiments, when a voice is received, the processor 120 may analyze and predict the input voice based on the ANN algorithm. The processor 120 may identify whether a specific pattern is repeated based on the input voice by analysis and prediction. When a specific pattern is repeated, the processor 120 may store information about the specific pattern and the frequency of the input voice in the memory 130, and may assign a unique identification (ID). According to various embodiments, when a voice command is detected, the processor 120 may identify the user corresponding to the detected voice command based on information about the specific pattern and the frequency of the voice corresponding to the unique ID.

[0046] In operation 305, the processor 120 of the electronic device 101 can use a camera (eg, Figure 1 The camera module 180) uses, for example, a facial recognition method to identify the user.

[0047] According to various embodiments, the processor 120 may identify the user based on at least one of operation 303 and operation 305. For example, the processor 120 may identify the user based on the voice command detected in operation 303, and operation 305 may be omitted. In operation 305, the processor 120 may omit operation 303, and may identify the user using a facial recognition method through a camera. According to various embodiments, in order to accurately identify the user, operations 303 and 305 may be performed, and the order of performing operations 303 and 305 may be changed. However, it should be understood that any means for identifying the user may be used.

[0048] At operation 307, the processor 120 may control at least one external electronic device (e.g., IoT device) corresponding to the detected voice command based on the usage pattern information (e.g., configuration information) corresponding to the user identified in operations 303 to 305. For example, the usage pattern information may be determined based on an ANN algorithm (e.g., an RNN algorithm and an LSTM algorithm). The RNN algorithm may, for example, refer to an ANN model for training time-varying data such as time series data. The LSTM algorithm may, for example, refer to an ANN model having the supplemented shortcomings of the RNN algorithm. The LSTM algorithm may, for example, refer to an algorithm capable of storing training information over a long period of time by supplementing the RNN algorithm whose training information may be lost. The usage pattern information may include specific usage pattern information corresponding to a specific user. For example, in the case of a first user, the first usage pattern information (e.g., first configuration information) may be determined based on the first user. In the case of a second user, the second usage pattern information (e.g., second configuration information) may be determined based on the second user.

[0049] According to various embodiments, the usage pattern information may include information about the frequency of using IoT devices, information about the time when the IoT devices are used, information about IoT devices that are used more frequently if there are IoT devices of the same type (e.g., TV1 and TV2), information about users (e.g., location, family relationship, and dwelling time), information about functions of IoT devices preferred by users, time information, or weather information.

[0050] According to an example embodiment, when a user (e.g., a first user) issues a voice command "turn off the lights in my room and turn on my favorite music", the processor 120 of the electronic device 101 can determine which room corresponds to the user's room based on the usage pattern information, and can turn off the lights in the user's room. In addition, the processor 120 can identify the user's location and determine an IoT device that can play music at the user's location. The processor 120 can identify the user's preferred music based on the usage pattern information, and can control the IoT device to play the user's preferred music. If a different user (e.g., a second user) issues a voice command, the processor 120 can determine the IoT device corresponding to the voice command based on the usage pattern information corresponding to the different user, and can perform the configuration function of the determined IoT device.

[0051] According to another embodiment, when the user issues a voice command "turn on TV", the processor 120 can identify the user's location based on the usage pattern information, and can control the TV near the user to turn on. The processor 120 can identify the current time and the user's preferred TV program based on the usage pattern information, and can control the TV to broadcast the preferred TV program.

[0052] According to various embodiments, if a new user who does not correspond to the usage pattern information (e.g., a user not recorded by the ANN algorithm or an unregistered user) issues a voice command, the processor 120 does not execute the voice command and may use a speaker (e.g., Figure 1 sound output device 155) or a display (e.g., Figure 1 The processor 120 may control the IoT device based on the voice command of the existing user trained according to the ANN algorithm. According to various embodiments, the processor 120 may register a new user with the permission of the existing user. The usage pattern information corresponding to the new user may be continuously updated.

[0053] According to various embodiments, the processor 120 may continuously update the usage pattern information corresponding to each of the multiple users. For example, when a voice is received, the processor 120 may analyze and predict the input voice based on an ANN algorithm (e.g., an RNN algorithm and an LSTM algorithm). If a specific pattern is repeated in the data acquired by analysis and prediction, the processor 120 may store information related to the specific pattern and the audio frequency of the speech in a server or a specific memory, and may assign a unique identification (ID). The processor 120 may identify similarities with existing stored information based on information related to the specific pattern and the frequency of the voice corresponding to the unique ID. If the information related to the unique ID is similar to the existing stored information, the processor 120 may update the existing stored information based on the information related to the unique ID.

[0054] Figure 4 is a flowchart illustrating an example method of controlling an external electronic device based on configuration information corresponding to a user according to various embodiments.

[0055] Reference Figure 4 In operation 401, an electronic device (eg Figure 1 A processor (eg, Figure 1 The processor 120 may use a microphone (e.g. Figure 1 The input device 150) receives the user's voice command.

[0056] In operation 403, the processor 120 may identify who the user is based on the received voice command. For example, the processor 120 may identify audio information about the received voice command (e.g., a specific pattern of the voice and frequency information of the voice). The processor 120 may store the recognized audio information in a memory (e.g., Figure 1The user corresponding to the voice command is compared with the audio information in the memory 130) to identify the user.

[0057] In operation 405, the processor 120 may identify (determine) whether the user who has sent the voice command is the first user. When the user is identified (determined) as the first user, in operation 407, the processor 120 may identify first configuration information (e.g., first usage mode information) corresponding to the first user, and may determine at least one external electronic device corresponding to the voice command based on the first configuration information.

[0058] In operation 409 , the processor 120 may control the determined external electronic device to perform a function corresponding to the voice command.

[0059] If the user is not the first user at operation 405, the processor 120 may identify (determine) whether the user is the second user at operation 411. If the user is identified (determined) as the second user, the processor 120 may identify second configuration information (e.g., second usage mode information) corresponding to the second user at operation 413, and may determine at least one external electronic device corresponding to the voice command based on the second configuration information.

[0060] At operation 415 , the processor 120 may control the determined external electronic device to execute a function corresponding to the voice command.

[0061] If the user is not the second user at operation 411, the processor 120 may ignore the received voice command. According to various embodiments, when the audio information to be compared with the audio information corresponding to the received voice command is not stored in the memory 130, the processor 120 may store the audio information corresponding to the voice command in the memory 130, and may assign a unique ID. According to various embodiments, the electronic device 101 may store new audio information in the memory 130 under the control of the user corresponding to the ID previously stored in the memory 130, and may assign a unique ID. For example, when the electronic device 101 assigns a unique ID, it may request permission from the user corresponding to the previously stored ID.

[0062] Figure 5 is a diagram illustrating an example method of controlling an external electronic device based on configuration information corresponding to a user according to various embodiments.

[0063] Reference Figure 5 , the electronic device 101 can use a microphone (e.g., Figure 1The electronic device 101 may receive the voice of at least one user and may identify who the user corresponding to the received voice is. For example, the electronic device 101 may identify audio information (e.g., specific pattern and frequency information) about the received voice. The electronic device 101 may store the recognized audio information and the audio information stored in the memory (e.g., Figure 1 The electronic device 101 can identify the user corresponding to the received voice by comparing the audio information in the memory 130 of the electronic device 101. The electronic device 101 can identify the usage pattern information corresponding to the identified user based on the ANN algorithm, and can control at least one external electronic device (e.g., IoT device) to execute the voice command for the received voice based on the usage pattern information.

[0064] According to various embodiments, user A 510 (e.g., a first user) and user B 520 (e.g., a second user) may live in the same space 501 (e.g., a house). User A 510 may live in room A 511 included in the same space 501, and user B 520 may live in room B 521 included in the same space 501. Figure 1 The electronic device 101 may identify user A 510 and user B 520 located in the same space 501 based on the ANN algorithm. For example, the electronic device 101 may have a memory (eg, Figure 1 The electronic device 101 may store the audio information of the user A 510 and the audio information of the user B 520 in the memory 130 of the electronic device 101. In addition, the electronic device 101 may store the usage pattern information corresponding to the user A 510 and the usage pattern information corresponding to the user B 520 in the memory 130. The electronic device 101 may receive the voice of at least one of the user A 510 and the user B 520, and may recognize the user corresponding to the voice based on the audio information stored in the memory.

[0065] For example, when user A 510 issues a voice command 513 "Raise the temperature in my room", the electronic device 101 may receive the voice command 513 and may recognize that the user corresponding to the voice command 513 is user A 510. The electronic device 101 may recognize room A 511, i.e., the room of user A 510, based on the usage pattern information corresponding to user A 510. The electronic device 101 may recognize information related to outdoor weather and a set temperature preferred by user A 510 based on the usage pattern information. The electronic device 101 may control the air conditioning and heating system 503 (e.g., Figure 2According to various embodiments, the electronic device 101 may identify the indoor set temperature preferred by the user A 510 based on the usage pattern information of the user A 510, and may adjust the temperature of the room A 511 based on the identified temperature.

[0066] According to various embodiments, when user B 520 instead of user A 510 issues a voice command, the electronic device 101 may control at least one external electronic device (eg, air conditioning and heating system 503) as in the above example.

[0067] For example, when user B 520 issues a voice command 523 "Raise the temperature in my room", the electronic device 101 may receive the voice command 523 and may recognize that the user corresponding to the voice command 523 is user B 520. The electronic device 101 may recognize room B 521, i.e., the room of user B 520, based on the usage pattern information corresponding to user B 520. The electronic device 101 may control the air conditioning and heating system 503 (e.g., Figure 2 The air conditioning and heating system 210 of the electronic device 101 can adjust the temperature of the room B 521. According to various embodiments, the electronic device 101 can identify the indoor set temperature preferred by the user B 520 based on the usage pattern information of the user B 520, and can adjust the temperature of the room B 521 based on the identified temperature.

[0068] Figure 6 is a flow chart illustrating an example method of controlling an external electronic device based on a voice command and a gaze direction of a pupil according to various embodiments.

[0069] Reference Figure 6 In operation 601, an electronic device (eg, Figure 1 A processor (eg, Figure 1 The processor 120 of the electronic device 101 can detect surrounding movement in the operating sensor mode. The operating sensor mode can be that the processor 120 of the electronic device 101 uses a sensor module (e.g., Figure 1 The processor 120 of the electronic device 101 may use a camera (e.g., Figure 1 According to various embodiments, the processor 120 may acquire an image of an external object while the camera is always activated, may acquire an image of an external object by activating the camera in an operation sensor mode, or may acquire an image of an external object by activating the camera when it detects a moving object for photographing.

[0070] In operation 603 , the processor 120 may recognize (determine) whether an external object (eg, an object for photographing) is a person based on the acquired image.

[0071] If the external object is recognized (determined) as a person in operation 603, the processor 120 may use a microphone (e.g., Figure 1 The input device 150) acquires voice from an external object (e.g., a person).

[0072] At operation 607, the processor 120 may identify the user based on the acquired voice. For example, the processor 120 may determine whether the user is an existing registered user or a new user. The processor 120 may identify whether the user is a registered user or an unregistered user based on the audio information stored in the memory 130. Although not shown, when the user is an unregistered user, the processor 120 may register the unregistered user with the permission of the registered user.

[0073] At operation 609, the processor 120 may use a camera (eg, Figure 1 The camera module 180 of the processor 120 can be used to identify the user's pupil movement. For example, the processor 120 can use a camera to identify the direction in which the user is looking.

[0074] At operation 611, the processor 120 may determine at least one of a plurality of external electronic devices based on the identified pupil movement. For example, if a plurality of external electronic devices are arranged in a direction (e.g., coordinates) that the user is gazing at, the processor 120 may group the plurality of external electronic devices. The processor 120 may determine at least one external electronic device capable of executing a function corresponding to the voice based on the plurality of grouped external electronic devices.

[0075] At operation 613, the processor 120 may determine whether the function corresponding to the voice command can be executed by the determined external electronic device based on the usage pattern information (e.g., configuration information) corresponding to the user. According to various embodiments, the usage pattern information (e.g., configuration information) may be determined according to a specific user, and the usage pattern information may be periodically updated based on the ANN algorithm. The processor 120 may determine at least one external electronic device based on the usage pattern information according to the user, and may control the determined external electronic device to execute the function desired by the user.

[0076] At operation 615 , the processor 120 may control the determined external electronic device to perform a function corresponding to the voice command.

[0077] If the external electronic device determined in operation 613 does not perform a function corresponding to the voice command, the processor 120 may provide a notification to the user using methods such as audio signals, light, vibration, and message notification in a display in operation 617. The processor 120 may send a message to the user that the voice command has not been recognized.

[0078] Figure 7 is a diagram illustrating an example method of controlling an external electronic device based on a voice command and a gaze direction of a pupil according to various embodiments.

[0079] Figure 7 700 (e.g., a house) of a user 701 as seen from the top. For example, the living space 700 of the user 701 may include an audio device 711 (e.g., Figure 2 audio device 205), two TVs (e.g., TV1 712 (e.g., Figure 2 TV1 203) and TV2 713 (e.g., Figure 2 TV2 204)), curtains 714 (e.g., Figure 2 curtains 206), lights (e.g., lights (left) 716 and lights (right) 715), washing machines 717 (e.g., Figure 2 washing machine 201) and refrigerator 718 (e.g., Figure 2 Refrigerator 202). Multiple external electronic devices may be included in the IoT device. Although not shown, the living space 700 may also include multiple IoT devices (e.g., air conditioning and heating systems (e.g., Figure 2 Air conditioning and heating system 210)). External electronic devices are not limited to Figure 7 External electronics shown.

[0080] Reference Figure 7 , electronic device 710 (eg, Figure 1 The electronic device 101) has been shown as being located at the center of the living space 700, but the location of the electronic device 710 is not limited. The electronic device 710 can be used as a hub device for managing or controlling multiple external electronic devices (e.g., audio devices 711, TL1, 712, TL2, 713, lights 715 and 716, a washing machine 717, and a refrigerator 718), and can be synchronized with at least one external electronic device. The electronic device 710 can determine at least one of the synchronized external electronic devices and control the function of the determined external electronic device. For example, the electronic device 710 can control the refrigerator 718 to control the refrigeration temperature of the refrigerator 718, and can remotely control the on / off and volume of the audio device 711. The electronic device 710 may include a camera (e.g., Figure 1A camera module 180 is provided, and the surrounding objects for shooting are photographed using the camera.

[0081] Reference Figure 7 , the electronic device 710 can track the user's pupil movement using a camera, and identify at least one external electronic device located in the direction of the user's gaze based on the pupil movement. The electronic device 710 can predict the user's intention based on the user's pupil movement, and control at least one external electronic device located in the user's gaze direction.

[0082] According to various embodiments, the user may gaze at at least one external electronic device corresponding to the voice command while issuing a voice command to the electronic device 710. The electronic device 710 may recognize the pupil movement of the user while receiving the voice command through the microphone. The electronic device 710 may predict the direction in which the user is gazing and recognize at least one external electronic device located in the direction in which the user is gazing. The electronic device 710 may apply the issued voice command to the recognized external electronic device. The electronic device 710 may remotely control the recognized external electronic device to perform the function corresponding to the voice command.

[0083] According to various embodiments of the present disclosure, an electronic device (eg, Figure 1 The electronic device 101 may include a microphone (eg, Figure 1 An input device 150 for connecting to one or more external electronic devices (eg, Figure 1 communication module 190) and a processor (e.g., Figure 1 The processor 120 may identify the user based on at least the voice data including the phoneme information acquired using the microphone 150. When the user is a first user, the processor may identify the first usage pattern information corresponding to the first user based on the specified ANN algorithm, and may control the first specified electronic device belonging to one or more external electronic devices and corresponding to the phoneme information via the communication module 190 based on the identified first usage pattern information to perform a first specified operation. When the user is a second user, the processor may identify the second usage pattern information corresponding to the second user based on the specified ANN algorithm, and may control the second specified electronic device in the one or more external electronic devices via the communication module 190 using the phoneme information based on the identified second usage pattern information to perform a second specified operation.

[0084] According to various embodiments, the first usage pattern information may include usage pattern information of the first user corresponding to one or more external electronic devices. The second usage pattern information may include usage pattern information of the second user corresponding to one or more external electronic devices.

[0085] According to various embodiments, the first usage pattern information may include usage pattern information of the first user to which a weight corresponding to the first user has been applied based on a specified ANN algorithm. The second usage pattern information may include usage pattern information of the second user to which a weight corresponding to the second user has been applied based on a specified ANN algorithm.

[0086] According to various embodiments, the first usage pattern information and the second usage pattern information may include, for one or more external electronic devices, information about the frequency of use of each external electronic device, information about the time when the external electronic device is used, information about external electronic devices that are used more when the same type of external electronic devices are included, information about the user, at least one of time information and weather information.

[0087] According to various embodiments, the processor 120 may determine at least one external electronic device that belongs to one or more external electronic devices and is capable of performing a first designated operation as a first designated electronic device. The processor may determine at least one different external electronic device that belongs to one or more external electronic devices and is capable of performing a second designated operation as a second designated electronic device.

[0088] According to various embodiments, if multiple external electronic devices among one or more external electronic devices correspond to a first designated operation or a second designated operation, the processor 120 can determine a first designated electronic device among the multiple external electronic devices based at least on the first usage mode information, and can determine a second designated electronic device among the multiple external electronic devices based at least on the second usage mode information.

[0089] According to various embodiments, the first designated operation and the second designated operation may perform the same function.

[0090] According to various embodiments, an electronic device (e.g., Figure 1 The electronic device 101 may also include a camera (eg, Figure 1 The processor 120 may use the camera 180 to identify pupil movement of the user, may determine a direction in which the user is looking based at least on the pupil movement, and may determine a first designated electronic device or a second designated electronic device based at least on the direction.

[0091] According to various embodiments, if the user does not belong to the first user and the second user, the processor 120 may provide a notification for identifying whether the user's control over the one or more external electronic devices is permitted.

[0092] According to various embodiments of the present disclosure, an electronic device (eg, Figure 1 The electronic device 101 may include a microphone (eg, Figure 1An input device 150 including a communication module (eg, Figure 1 communication module 190) and a processor (e.g., Figure 1 The processor 120 may identify the user based on at least the voice data including the phoneme information acquired using the microphone 150. When the user is a first user, the processor may identify the first usage pattern information corresponding to the first user based on the specified ANN algorithm, and may perform a specified operation by controlling a first specified electronic device belonging to one or more external electronic devices and corresponding to the phoneme information via the communication module 190 based on the identified first usage pattern information. When the user is a second user, the processor may identify the second usage pattern information corresponding to the second user based on the specified ANN algorithm, and may perform a specified operation by controlling a second specified electronic device belonging to one or more external electronic devices and corresponding to the phoneme information via the communication module 190 based on the identified second usage pattern information.

[0093] According to various example embodiments, the first usage pattern information may include usage pattern information of the first user corresponding to one or more external electronic devices. The second usage pattern information may include usage pattern information of the second user corresponding to one or more external electronic devices.

[0094] According to various example embodiments, the first usage pattern information may include usage pattern information of a first user, wherein a weight corresponding to the first user has been applied to the usage pattern information of the first user based on a specified ANN algorithm. The second usage pattern information may include usage pattern information of a second user, wherein a weight corresponding to the second user has been applied to the usage pattern information of the second user based on a specified ANN algorithm.

[0095] According to various exemplary embodiments, the first usage pattern information and the second usage pattern information may include at least one of the following information for one or more external electronic devices: information about the frequency of using each external electronic device, information about the time when the external electronic device is used, information about the external electronic devices that are used more when the same type of external electronic devices are used, information about the user, information about the external electronic devices preferred by the user, time information, and weather information.

[0096] According to various example embodiments, the processor 120 may determine at least one external electronic device that belongs to one or more external electronic devices and is capable of performing a first designated operation as a first designated electronic device, and may determine at least one different external electronic device that belongs to one or more external electronic devices and is capable of performing a second designated operation as a second designated electronic device.

[0097] The method according to various embodiments of the present disclosure may include: identifying a user based at least on voice data including phoneme information acquired using a microphone; when the user is a first user, identifying first usage pattern information corresponding to the first user based on a specified ANN algorithm; performing a first specified operation by controlling a first specified electronic device belonging to one or more external electronic devices and corresponding to the phoneme information via a communication module based on the identified first usage pattern information, and when the user is a second user, identifying second usage pattern information corresponding to the second user based on a specified ANN algorithm, and controlling a second specified electronic device belonging to one or more external electronic devices and corresponding to the phoneme information via the communication module based on the identified second usage pattern information to perform a second specified operation.

[0098] The method according to various example embodiments may include: determining at least one external electronic device that belongs to one or more external electronic devices and is capable of performing a first designated operation as a first designated electronic device; and determining at least one different external electronic device that belongs to the one or more external electronic devices and is capable of performing a second designated operation as a second designated electronic device.

[0099] The method according to various exemplary embodiments may include: if multiple external electronic devices among the one or more external electronic devices correspond to a first designated operation or a second designated operation, determining a first designated electronic device among the multiple external electronic devices based at least on first usage mode information; and determining a second designated electronic device among the multiple external electronic devices based at least on second usage mode information.

[0100] According to various example embodiments, the first designated operation and the second designated operation may perform the same function.

[0101] According to various example embodiments, performing a first designated operation or performing a second designated operation may include: using a camera to identify pupil movement of a user; determining a direction of the user's gaze based at least on the pupil movement; and determining a first designated electronic device or a second designated electronic device among one or more external electronic devices based at least on the direction.

[0102] A method according to various example embodiments may include, when a user does not belong to the first user and the second user, providing a notification for identifying whether control of one or more external electronic devices corresponding to the user is permitted.

[0103] The electronic device according to various embodiments may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to an embodiment of the present disclosure, the electronic device is not limited to those electronic devices described above.

[0104] It should be understood that the various embodiments of the present disclosure and the terms used therein are not intended to limit the technical features set forth herein to specific embodiments, but include various changes, equivalents or alternative forms for the corresponding embodiments. For the description of the accompanying drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that the nouns in the singular form corresponding to the term may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include all possible combinations of items listed together with the corresponding one of the multiple phrases. As used herein, terms such as "1st" and "2nd" or "first" and "second" may be used to simply distinguish the corresponding component from another component, and do not limit the component in other aspects (e.g., importance or order). It will be understood that if an element (e.g., a first element) is referred to as being “combined with another element (e.g., the second element)”, “combined to another element (e.g., the second element)”, “connected with another element (e.g., the second element)”, or “connected to another element (e.g., the second element)” when the term “operably” or “communicatively” is used or when the term “operably” or “communicatively” is not used, it means that the element may be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0105] As used herein, the term "module" may include units implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "portion," or "circuit"). A module may be a single integrated component adapted to perform one or more functions or a minimum unit or portion of the single integrated component. For example, according to an embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0106] The various embodiments described herein may be implemented as software (e.g., program 140) including one or more instructions stored in a storage medium (e.g., internal memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) may call at least one of the one or more instructions stored in the storage medium and execute the at least one instruction with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be run by an interpreter. A machine-readable storage medium may be provided in the form of a non-transitory storage medium. Among them, the term "non-transitory" only means that the storage medium is a tangible device and does not include a signal (e.g., an electromagnetic wave), but the term does not distinguish between data being semi-permanently stored in a storage medium and data being temporarily stored in a storage medium.

[0107] According to an embodiment, the method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be released in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)), or may be downloaded via an application store (e.g., Play Store TM ) The computer program product may be published (e.g., downloaded or uploaded) online, or the computer program product may be distributed (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If published online, at least part of the computer program product may be temporarily generated, or at least part of the computer program product may be at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a forwarding server).

[0108] According to various embodiments, each component (e.g., module or program) in the above-mentioned components may include a single entity or multiple entities. According to various embodiments, one or more components in the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each component in the multiple components in the same or similar manner as a corresponding component in the multiple components before integration. According to various embodiments, the operations performed by a module, program or another component may be performed sequentially, in parallel, repeatedly or in a heuristic manner, or one or more operations in the operations may be run or omitted in different orders, or one or more other operations may be added.

[0109] Although the present disclosure has been shown and described with reference to various exemplary embodiments, it should be understood that the various exemplary embodiments are illustrative rather than restrictive. Those skilled in the art will appreciate that various modifications, substitutions and changes may be made without departing from the spirit and scope of the present invention as defined in, for example, but not limited to, the appended claims and their equivalents.

Claims

1. Electronic devices, including: Memory; a camera configured to identify pupil movements of an individual user; microphone; a communication module including communication circuitry configured to connect to one or more external electronic devices; as well as A processor, wherein the processor is configured to: in response to receiving voice data including phoneme information acquired using the microphone, identifying the single user as a first user or a second user from a plurality of registered users allowed to control the one or more external electronic devices, wherein the single user is identified from the plurality of registered users based at least on the voice data including the phoneme information acquired using the microphone and audio information of each of the plurality of registered users stored in the memory, identifying a direction in which the single user is looking based on the pupil movement of the single user, If the single user is identified as the first user, then: identifying first usage pattern information corresponding to the first user based on a specified artificial neural network (ANN) algorithm corresponding to the plurality of registered users, identifying a first designated electronic device that belongs to the one or more external electronic devices and corresponds to the phoneme information based on the identified first usage pattern information and the direction in which the first user is looking, When there are multiple first designated electronic devices, determining the nearest electronic device among the multiple first designated electronic devices as the first designated electronic device based on the location of the first user, and controlling the first designated electronic device to perform a first designated operation corresponding to the phoneme information through the communication module, and If the single user is identified as the second user, then: identifying second usage pattern information corresponding to the second user based on the specified ANN algorithm, identifying a second designated electronic device that belongs to the one or more external electronic devices and corresponds to the phoneme information based on the identified second usage pattern information and the direction in which the second user is looking, When there are a plurality of second designated electronic devices, determining the nearest electronic device among the plurality of second designated electronic devices as the second designated electronic device based on the location of the second user, and The communication module controls the second designated electronic device to perform a second designated operation corresponding to the phoneme information.

2. The electronic device according to claim 1, wherein: The first usage pattern information includes usage pattern information of the first user corresponding to the one or more external electronic devices, and The second usage pattern information includes usage pattern information of the second user corresponding to one or more external electronic devices.

3. The electronic device according to claim 1, wherein: The first usage pattern information includes usage pattern information of the first user, wherein a weight corresponding to the first user has been applied to the usage pattern information of the first user based on the specified ANN algorithm, and The second usage pattern information includes usage pattern information of the second user, wherein a weight corresponding to the second user has been applied to the usage pattern information of the second user based on the specified ANN algorithm.

4. The electronic device according to claim 1, wherein: The first usage pattern information and the second usage pattern information include at least one of the following information about the one or more external electronic devices: Information about the frequency of using each external electronic device, information about the time when the external electronic device is used, information about more used external electronic devices including the same type of external electronic devices, information about users, time information, and weather information.

5. The electronic device according to claim 1, wherein: The processor is configured as follows: determining at least one external electronic device that belongs to the one or more external electronic devices and is capable of performing the first designated operation as the first designated electronic device, and At least one external electronic device other than the first designated electronic device, which belongs to the one or more external electronic devices and is capable of performing the second designated operation, is determined as the second designated electronic device.

6. The electronic device according to claim 1, wherein: The processor is configured to determine the first designated electronic device among the multiple external electronic devices based on at least the first usage mode information based on multiple external electronic devices among the one or more external electronic devices corresponding to the first designated operation or the second designated operation, and to determine the second designated electronic device among the multiple external electronic devices based on at least the second usage mode information.

7. The electronic device according to claim 1, wherein: The first designation operation and the second designation operation perform the same function.

8. The electronic device according to claim 1, wherein: The processor is configured to provide a notification for identifying whether the user's control over the one or more external electronic devices has been permitted based on the user not belonging to one of the first user and the second user.

9. A method for controlling an external electronic device, comprising: identifying a single user as a first user or a second user from a plurality of registered users allowed to control one or more external electronic devices in response to receiving voice data including phoneme information acquired using a microphone, wherein the single user is identified from the plurality of registered users based at least on the voice data including the phoneme information acquired using the microphone and audio information of each of the plurality of registered users stored in a memory; identifying a gaze direction of the single user based on pupil movement of the single user, If the single user is identified as the first user, then: identifying first usage pattern information corresponding to the first user based on a specified artificial neural network (ANN) algorithm corresponding to the plurality of registered users; Based on the identified first usage pattern information and the direction in which the first user is looking, identifying a first designated electronic device that belongs to the one or more external electronic devices and corresponds to the phoneme information; When there are multiple first designated electronic devices, determining the nearest electronic device among the multiple first designated electronic devices as the first designated electronic device based on the location of the first user, and controlling the first designated electronic device to execute a first designated operation corresponding to the phoneme information through the communication module, and If the single user is identified as the second user, then: identifying second usage pattern information corresponding to the second user based on the specified ANN algorithm; identifying, based on the identified second usage pattern information and the direction in which the second user is looking, a second designated electronic device that belongs to the one or more external electronic devices and corresponds to the phoneme information; When there are a plurality of second designated electronic devices, determining the nearest electronic device among the plurality of second designated electronic devices as the second designated electronic device based on the location of the second user; and The communication module controls the second designated electronic device to perform a second designated operation corresponding to the phoneme information.

10. The method according to claim 9, further comprising: determining at least one external electronic device that belongs to the one or more external electronic devices and is capable of performing the first designated operation as the first designated electronic device; as well as At least one external electronic device other than the first designated electronic device, which belongs to the one or more external electronic devices and is capable of performing the second designated operation, is determined as the second designated electronic device.

11. The method according to claim 9, further comprising: Based on multiple external electronic devices among the one or more external electronic devices corresponding to the first designated operation or the second designated operation, the first designated electronic device among the multiple external electronic devices is determined at least based on the first usage mode information, and the second designated electronic device among the multiple external electronic devices is determined at least based on the second usage mode information.

12. The method according to claim 9, wherein: The first designation operation and the second designation operation perform the same function. 13 . The method according to claim 9 , further comprising providing a notification for identifying whether control of the one or more external electronic devices corresponding to the user is permitted based on the fact that the user does not belong to the first user and / or the second user.

Images