Wearable electronic apparatus and method for controlling electronic devices using vision information

Abstract

A wearable device and method for controlling an electronic device using vision information are disclosed. An electronic device according to one embodiment includes a camera that captures the surroundings of the electronic device to acquire a current image, a voice input device that receives user utterance, a processor, and at least one memory that stores instructions to be executed by the processor. When instructions are executed by the processor, the processor performs an operation of determining a user's intention based on the received user utterance, an operation of determining a target device for control corresponding to the determined intention, and an operation of controlling the target device for control determined based on the determined intention. The operation of determining a target device for control may include an operation of determining whether it is possible to determine a target device for control from the user utterance, and, if it is not possible to determine a target device for control from the user utterance, an operation of determining whether it is possible to determine a target device for control based on the current image acquired through the camera.

Description

Technology Field

[0001] Various embodiments of the present invention relate to a technology for controlling electronic devices using vision information. Background Technology

[0002] Wearable devices such as augmented reality glasses are next-generation devices that display virtual images (e.g., information about objects) over the image that the user is actually looking at. To this end, the wearable device may include a camera that recognizes the surroundings, a sensor, and an optical display that analyzes the information acquired through the camera and sensor to display a virtual image over the screen that is actually being viewed.

[0003] A wearable device can provide a voice assistant service to a user that performs tasks or services for the user based on commands or questions. The wearable device can receive a user's voice command through a voice input device, such as a microphone, and execute the user's voice command by understanding the intent of the voice command. For example, a user can speak a voice command to control another electronic device and input it into the wearable device through a voice input device, and the wearable device can control the electronic device by transmitting a control signal to the other electronic device according to the voice command. Prior art literature

[65535] Korean Published Patent Application No. 10-2019-0134975 (December 5, 2019) The problem to be solved

[0004] Voice commands from a user for controlling electronic devices input into a wearable device typically include information about the device to be controlled, but if the user omits information about the device to be controlled when speaking, or if the device to be controlled is not identified from the user's speech, it may be difficult for the wearable device to execute the command.

[0005] According to a wearable device and method for controlling an electronic device using vision information according to one embodiment, even when the device to be controlled is not recognized from a user's speech for controlling the electronic device, the device to be controlled can be identified. means of solving the problem

[0006] An electronic device for controlling an electronic device using vision information according to one embodiment comprises a camera that captures the surroundings of the electronic device to acquire a current image, a voice input device that receives user utterance, a processor, and at least one memory that stores instructions to be executed by the processor. When the instructions are executed by the processor, the processor performs an operation of determining a user's intention based on the received user utterance, an operation of determining a target device for control corresponding to the determined intention, and an operation of controlling the determined target device for control based on the determined intention. The operation of determining the target device for control may include an operation of determining whether it is possible to determine the target device for control from the user utterance, and, if it is not possible to determine the target device for control from the user utterance, an operation of determining whether it is possible to determine the target device for control based on the current image acquired through the camera.

[0007] A method for controlling an electronic device using vision information according to one embodiment includes: receiving a user utterance through a voice input device of a wearable electronic device; determining a user's intention based on the received user utterance; determining a target device for control corresponding to the determined intention; and controlling the determined target device for control based on the determined intention. The operation of determining the target device for control may include determining whether it is possible to determine the target device for control from the user utterance; and, if it is not possible to determine the target device for control from the user utterance, determining whether it is possible to determine the target device for control based on the current image obtained through a camera that captures the surroundings of the wearable electronic device to obtain a current image. Effects of the invention

[0008] According to a wearable device and method for controlling an electronic device using vision information according to one embodiment, when a device to be controlled is not recognized from a user's speech for controlling an electronic device, the device to be controlled can be identified by recognizing a space using vision information acquired through a camera of the wearable device and identifying an electronic device corresponding to that space. Brief explanation of the drawing

[0009] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments. FIG. 2 illustrates a wearable device according to one embodiment. FIG. 3 is a diagram illustrating an example of a voice command spoken by a user wearing an electronic device that controls an electronic device using vision information according to one embodiment. FIG. 4 is a diagram illustrating operations performed to determine a target control target device in an electronic device according to one embodiment. FIG. 5 is a flowchart illustrating a method for controlling an electronic device using vision information according to one embodiment. FIG. 6 is a flowchart illustrating the operation of determining a target control target device based on spatial information of a method according to one embodiment. FIGS. 7(A) and FIGS. 7(B) are drawings for illustrating an embodiment in which the target control target device is determined differently depending on the user's location. FIG. 8 is a block diagram illustrating the configuration of an electronic device according to one embodiment. FIG. 9 is a flowchart of a method for controlling an electronic device using vision information according to one embodiment. Specific details for implementing the invention

[0010] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0011] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0012] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0013] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0014] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0015] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0016] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0017] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0018] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0019] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0020] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0021] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0022] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0023] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0024] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0025] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0026] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0027] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0028] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0029] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0030] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0031] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0032] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0033] FIG. 2 is a drawing illustrating the structure of a wearable device according to one embodiment.

[0034] Referring to FIG. 2, a wearable device (200) (e.g., the electronic device (101) of FIG. 1) is worn on the face of a user and can provide the user with images related to augmented reality services and / or virtual reality services.

[0035] In one embodiment, the wearable device (200) comprises a first display (205), a second display (210), a screen display unit (215a, 215b), an input optical member (220), a first transparent member (225a), a second transparent member (225b), a lighting unit (230a, 230b), a first PCB (235a), a second PCB (235b), a first hinge (240a), a second hinge (240b), a first camera (245a, 245b, 245c, 245d), a plurality of microphones (e.g., a first microphone (250a), a second microphone (250b), a third microphone (250c)), a plurality of speakers (e.g., a first speaker (255a), a second speaker (255b)), a battery (260), a second camera (275a, 275b), and a third It may include a camera (265) and a visor (270a, 270b).

[0036] In one embodiment, the display (e.g., first display (205) and second display (210)) may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). Although not illustrated, if the display is one of a liquid crystal display, a digital mirror device, or a silicon liquid crystal display, the wearable device (200) may include a light source that irradiates light onto a screen output area of ​​the display. In another embodiment, if the display is capable of generating light itself, for example, one of an organic light emitting diode or a micro LED, the wearable device (200) may provide a good quality virtual image to the user without including a separate light source. In one embodiment, if the display is implemented with an organic light-emitting diode or a micro LED, a light source is unnecessary, so the wearable device (200) can be made lighter. Hereinafter, a display capable of generating light on its own is referred to as a self-emissive display, and the description is based on the premise of a self-emissive display.

[0037] A display according to various embodiments of the present invention (e.g., a first display (205) and a second display (210)) may be composed of at least one micro light emitting diode (micro LED). For example, a micro LED can express red (R), green (G), and blue (B) through self-emission, and because it is small in size (e.g., 100 μm or less), one chip can implement one pixel (e.g., one of R, G, and B). Accordingly, when the display is composed of micro LEDs, high resolution can be provided without a backlight unit (BLU).

[0038] This is not limited thereto, and a single pixel may include R, G, and B, and a single chip may be implemented with multiple pixels including R, G, and B.

[0039] In one embodiment, a display (e.g., a first display (205) and a second display (210)) may be composed of a display area composed of pixels for displaying a virtual image and light receiving pixels (e.g., photo sensor pixels) positioned between the pixels that receive light reflected from the eye, convert it into electrical energy, and output it.

[0040] In one embodiment, the wearable device (200) can detect the user's gaze direction (e.g., eye movement) through light-receiving pixels. For example, the wearable device (200) can detect and track the gaze direction of the user's right eye and the gaze direction of the user's left eye through one or more light-receiving pixels constituting the first display (205) and one or more light-receiving pixels constituting the second display (210). The wearable device (200) can determine the position of the center of the virtual image according to the gaze direction of the user's right eye and left eye detected through one or more light-receiving pixels (e.g., the direction in which the pupils of the user's right eye and left eye are gazing).

[0041] In one embodiment, light emitted from a display (e.g., a first display (205) and a second display (210)) can pass through a lens (not shown) and a waveguide to reach a screen display portion (215a, 215b) formed on a first transparent member (225a) positioned facing the user's right eye and a screen display portion (215a, 215b) formed on a second transparent member (225b) positioned facing the user's left eye. For example, light emitted from a display (e.g., a first display (205) and a second display (210)) can pass through a waveguide and be reflected by a grating area formed on an input optical member (220) and a screen display portion (215a, 215b) and transmitted to the user's eye. The first transparent member (225a) and / or the second transparent member (225b) may be formed of a glass plate, a plastic plate, or a polymer, and may be made transparent or translucent.

[0042] In one embodiment, a lens (not shown) may be positioned on the front of a display (e.g., a first display (205) and a second display (210)). The lens (not shown) may include a concave lens and / or a convex lens. For example, the lens (not shown) may include a projection lens or a collimation lens.

[0043] In one embodiment, the screen display unit (215a, 215b) or the transparent member (e.g., the first transparent member (225a), the second transparent member (225b)) may include a lens including a waveguide or a reflective lens.

[0044] In one embodiment, the waveguide may be made of glass, plastic, or polymer and may include a nano-pattern formed on an inner or outer surface, for example, a grating structure of a polygonal or curved shape. According to one embodiment, light incident on one end of the waveguide may propagate within the display waveguide by the nano-pattern and be provided to the user. In one embodiment, the waveguide composed of a free-form prism may provide the incident light to the user through a reflective mirror. The waveguide may include at least one diffractive element, such as a DOE (diffractive optical element), a HOE (holographic optical element), or a reflective element (e.g., a reflective mirror). In one embodiment, the waveguide may guide light emitted from the display (205, 210) to the user's eye using at least one diffractive element or reflective element included in the waveguide.

[0045] According to various embodiments, the diffraction element may include an input optical member (220) and an output optical member (not shown). For example, the input optical member (220) may refer to an input grating area, and the output optical member (not shown) may refer to an output grating area. The input grating area may serve as an input terminal for diffracting (or reflecting) light output from a display (e.g., a first display (205) and a second display (210)) (e.g., a micro LED) to transmit light to a transparent member (e.g., a first transparent member (250a), a second transparent member (250b)) of a screen display unit (215a, 215b). The output grating area may serve as an output terminal for diffracting (or reflecting) light transmitted to a transparent member of a waveguide (e.g., a first transparent member (250a), a second transparent member (250b)) to transmit light to the user's eye.

[0046] According to various embodiments, the reflection element may include a total internal reflection optical element or a total internal reflection waveguide for total internal reflection (TIR). For example, total internal reflection is a method of inducing light, which may mean creating an angle of incidence such that light (e.g., a virtual image) input through an input grating area is 100% reflected from one side (e.g., a specific side) of the waveguide and is transmitted 100% to an output grating area.

[0047] In one embodiment, light emitted from a display (205, 210) can be guided along a light path to a waveguide through an input optical member (220). Light traveling inside the waveguide can be guided toward the user's eye through an output optical member. A screen display (215a, 215b) can be determined based on the light emitted toward the eye.

[0048] In one embodiment, the first camera (245a, 245b, 245c, 245d) may include a camera used for 3 degrees of freedom (3DoF), 6DoF head tracking, hand detection and tracking, gesture and / or spatial recognition. For example, the first camera (245a, 245b, 245c, 245d) may include a GS (global shutter) camera to detect and track head and hand movements.

[0049] For example, the first camera (245a, 245b, 245c, 245d) may be a stereo camera for head tracking and spatial recognition, or a camera of the same specifications and performance may be used. For the first camera (245a, 245b, 245c, 245d), a GS camera with excellent performance (e.g., image drag) may be used to detect fast hand movements and fine movements such as fingers and to track movements.

[0050] According to various embodiments, the first camera (245a, 245b, 245c, 245d) may be a rolling shutter (RS) camera. The first camera (245a, 245b, 245c, 245d) can perform spatial recognition for 6 DOF and SLAM functions through depth capture. The first camera (245a, 245b, 245c, 245d) can perform user gesture recognition functions.

[0051] In one embodiment, the second camera (275a, 275b) may be used to detect and track the pupil. The second camera (275a, 275b) may be referred to as an ET (eye tracking) camera. The second camera (275a, 275b) may track the direction of the user's gaze. The wearable device (200) may take into account the direction of the user's gaze so that the center of the virtual image projected onto the screen display (215a, 215b) is positioned according to the direction in which the user's pupil gazes.

[0052] For tracking the direction of gaze, a GS camera may be used for the second camera (275a, 275b) to detect the pupil and track rapid pupil movements. The second camera (265a) may be installed for the left eye and the right eye, respectively, and cameras with the same performance and specifications may be used for the second camera (265a) for the left eye and the right eye.

[0053] In one embodiment, the third camera (265) may be referred to as HR (high resolution) or PV (photo video) and may include a high-resolution camera. The third camera (265) may include a color camera equipped with functions for obtaining high-quality images, such as AF (auto focus) and shake correction (OIS (optical image stabilizer)). Not limited thereto, the third camera (265) may include a GS (global shutter) camera or an RS (rolling shutter) camera.

[0054] In one embodiment, at least one sensor (e.g., gyroscope sensor, accelerometer, geomagnetic sensor, touch sensor, light sensor and / or gesture sensor) and a first camera (245a, 245b, 245c, 245d) can perform at least one of head tracking for 6DoF, pose estimation and prediction, gesture and / or spatial recognition, and slam function through depth capture.

[0055] In another embodiment, the first camera (245a, 245b, 245c, 245d) may be used separately as a camera for head tracking and a camera for hand tracking.

[0056] In one embodiment, the lighting unit (230a, 230b) may have different uses depending on the location where it is attached. For example, the lighting unit (230a, 230b) may be attached together with the first camera (245a, 245b, 245c, 245d) mounted around a hinge (e.g., first hinge (240a), second hinge (240b)) connecting the frame and the temple, or around a bridge connecting the frame. When shooting with a GS camera, the lighting unit (230a, 230b) may be used as a means to supplement ambient brightness. For example, the lighting unit (230a, 230b) may be used when it is difficult to detect the subject to be photographed due to a dark environment or the mixing of various light sources and reflected light.

[0057] In one embodiment, a lighting unit (230a, 230b) attached around the frame of a wearable device (200) may be used as an auxiliary means to facilitate eye gaze detection when photographing the pupil with a second camera (275a, 275b). When the lighting unit (230a, 230b) is used as an auxiliary means to detect the eye gaze, it may include an IR (infrared) LED of an infrared wavelength.

[0058] In one embodiment, the PCB (e.g., first PCB (235a), second PCB (235b)) may include a processor (not shown), memory (not shown), and a communication module (not shown) for controlling the components of the wearable device (200). The communication module may be configured in the same way as the communication module (290) of FIG. 2, and the same description of the communication module (290) may apply. For example, the communication module may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the wearable device (200) and an external electronic device, and the performance of communication through the established communication channel. The PCB may transmit electrical signals to the components constituting the wearable device (200).

[0059] A communication module (not shown) operates independently of a processor and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (not shown) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (e.g., a LAN (local area network) communication module, or a power line communication module). The corresponding communication module (not shown) among these communication modules may communicate with an external electronic device through a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association), or a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).

[0060] The wireless communication module can support 5G networks following 4G networks and next-generation communication technologies, such as new radio access technology (NR access technology). NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module can support high-frequency bands (e.g., mmWave band) to achieve high data transmission rates, for example. Wireless communication modules can support various technologies for securing performance in high-frequency bands, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beamforming, or large-scale antenna.

[0061] The wearable device (200) may further include an antenna module (not shown). The antenna module may transmit a signal or power to an external source (e.g., an external electronic device) or receive it from an external source. According to one embodiment, the antenna module may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a first PCB (235a), a second PCB (235b)). According to one embodiment, the antenna module may include a plurality of antennas (e.g., an array antenna).

[0062] In one embodiment, a plurality of microphones (e.g., a first microphone (250a), a second microphone (250b), and a third microphone (250c)) can process an external acoustic signal into electrical voice data. The processed voice data can be utilized in various ways depending on the function (or application running) being performed on the wearable device (200).

[0063] In one embodiment, a plurality of speakers (e.g., a first speaker (255a), a second speaker (255b)) can output audio data received from a communication module or stored in memory.

[0064] In one embodiment, the battery (260) may include one or more batteries and may supply power to the components constituting the wearable device (200).

[0065] In one embodiment, the visor (270a, 270b) can adjust the amount of external light transmitted to the user's eyes according to the transmittance. The visor (270a, 270b) may be positioned in front of or behind the screen display (215a, 215b). The front of the screen display (215a, 215b) may mean the direction opposite to the user wearing the wearable device (200), and the back may mean the direction toward the user wearing the wearable device (200). The visor (270a, 270b) can protect the screen display (215a, 215b) and adjust the amount of external light transmitted.

[0066] For example, the visor (270a, 270b) may include an electrochromic element that changes color according to the applied power source to control the transmittance. Electrochromism is a phenomenon in which color changes due to an oxidation-reduction reaction caused by the applied power source. The visor (270a, 270b) can control the transmittance of external light by utilizing the change in color of the electrochromic element.

[0067] For example, the visor (270a, 270b) may include a control module and an electrochromic element. The control module can control the electrochromic element to adjust the transmittance of the electrochromic element.

[0068] FIG. 3 is a diagram illustrating an example of a voice command spoken by a user wearing an electronic device that controls an electronic device using vision information according to one embodiment.

[0069] Referring to FIG. 3, a scene is illustrated in which a user (305) of an electronic device (300) according to one embodiment (e.g., a wearable device (200) of FIG. 2) wears the electronic device (300) and speaks a voice command to control a surrounding electronic device.

[0070] In one embodiment, the electronic device (300) can display augmented reality content (310, 315) (e.g., a cooking recipe (313), a cooking video) that a user (305) can view, such as images and text, through the display of the electronic device (300) (e.g., the first display (205) and the second display (210) of FIG. 2).

[0071] In one embodiment, electronic devices that can be remotely controlled (e.g., the smartphone (335) and oven (325) of FIG. 3) may be present around the user (305). The electronic device (300) and the electronic devices (e.g., the smartphone (335) and oven (325) of FIG. 3) may be wirelessly connected.

[0072] In one embodiment, the electronic device (300) may provide a voice assistant service to the user (305) to perform a task or service for the user (305) based on a command or question. In one embodiment, the user (305) may utter a voice command for controlling another electronic device. The electronic device (300) may receive the user's utterance through a voice input device such as a microphone (e.g., the first microphone (250a), the second microphone (250b), and the third microphone (250c) of FIG. 2), determine the intent of the user's utterance to control a target device, and perform control on the determined target device according to the intent of the user's utterance. The target device may refer to an electronic device corresponding to the target that the user (305) intends to control through the user's utterance.

[0073] For example, a user (305) can input a voice command to control the temperature of an oven (325) into an electronic device (300) through a voice input device, and the electronic device (300) can determine the oven (325) as the target device to be controlled according to the user's voice command and transmit a control signal to the oven (325) to control the temperature of the oven (325).

[0074] In general, the utterance of a user (305) for controlling an electronic device input into an electronic device (300) includes information about the target device to be controlled, but if the user (305) omits information about the target device to be controlled when uttering, or if the device to be controlled is not specified from the user's utterance, it may be difficult for the electronic device (300) to execute the command.

[0075] For example, referring to FIG. 3, a user (305) may speak "Hi Bixby, set the temperature to 38 degrees" to control the temperature of the oven. In one embodiment, "Hi Bixby" may be a command to call a voice assistant service installed in the electronic device (300), and the command to call the voice assistant service may be set in various ways in various embodiments. Although the user (305) speaks "set the temperature to 38 degrees" with the intention of controlling the temperature of the oven, the user speech itself does not include the oven, which is the target device for control, so the electronic device (300) may not be able to identify the target device for control based solely on the user speech.

[0076] In the case where the target device for control is not determined from user speech, the electronic device (300) that controls the electronic device using vision information according to one embodiment can determine the target device for control by using an image obtained through a camera that captures the surroundings of the electronic device (300) to obtain an image.

[0077] In one embodiment, the electronic device (300) can recognize objects in an image acquired through a camera and determine a target device for control among the recognized objects. For example, in FIG. 3, the electronic device (300) can recognize objects included in an image acquired through a camera (e.g., a dough roller (320), an oven (325), an apple (330), a smartphone (335), a cutting board (340)). The electronic device (300) can identify controllable devices (e.g., an oven (325), a smartphone (335)) among the recognized objects (e.g., a dough roller (320), an oven (325), an apple (330), a smartphone (335), a cutting board (340)). The controllable devices may be Internet of Things (IOT) devices. The operation of an electronic device (300) recognizing objects from an image and identifying controllable devices can be performed using a deep learning model trained for object recognition and identification of controllable devices.

[0078] In one embodiment, the electronic device (300) determines the oven (325), which is capable of controlling temperature among the controllable devices (e.g., oven (325), smartphone (335)), as the target device for control and can control the oven (325).

[0079] In one embodiment, a target control target device may not be determined from an image acquired from a camera. If the target control target device is not determined from the acquired image, the electronic device (300) may determine the space where the user (305) is located based on a rule-based database in which information about an object and spatial information are mapped and stored together, and the object recognized in the image. Once the space where the user (305) is located is determined, the electronic device (300) may receive information about a controllable device corresponding to the space from an Internet of Things server (not shown) (e.g., server (108) of FIG. 1) and determine the target control target device based on the received information about the controllable device. An Internet of Things server communicating wirelessly with the electronic device (300) may distinguish users using user accounts and store information about at least one controllable device registered with the Internet of Things server by each user assigning a space tag.

[0080] Hereinafter, a method for controlling an electronic device using vision information will be explained in detail with reference to FIG. 4.

[0081] FIG. 4 is a diagram illustrating operations performed to determine a target control target device in an electronic device according to one embodiment.

[0082] Referring to FIG. 4, in operation (405), the electronic device (300) can receive a user utterance, “Set the temperature to 38 degrees.” In operation (410), the electronic device (300) can determine the intent of the user utterance by performing natural language understanding (NLU) on the received user utterance. By determining the intent of the user utterance, “Set the temperature to 38 degrees,” the electronic device (300) can determine that the target device for temperature control cannot be determined solely by the user utterance and that more information about the target device for temperature control is required.

[0083] If the target control target device is not determined in operation (410), the electronic device (300) may determine the target control target device using vision information in operation (415). The vision information may include information about an image from a camera, information about an object recognized in the image, and / or information about the space of the image.

[0084] The electronic device (300) can determine whether it is possible to determine a target control target device based on current vision information, which is vision information after user speech input, in operation (420). In one embodiment, the electronic device (300) can determine whether it is possible to determine a target control target device based on a current image acquired through the camera of the electronic device (300) during a first time (e.g., a specific time such as 2 seconds) after the time when user speech is input into the voice input device of the electronic device (300).

[0085] In one embodiment, the electronic device (300) can acquire a current image of the area around the electronic device (300) during a first time period using a camera, and can recognize an object from the current image.

[0086] In one embodiment, the electronic device (300) can identify at least one controllable device among the objects recognized in the current image. The electronic device (300) can determine that a target control target device can be determined if, among the identified at least one controllable device, there is a controllable device corresponding to the intention determined in the operation (410). If a target control target device can be determined from the current image, the electronic device (300) can determine the target control target device determined from the current image as the target control target device corresponding to the intention determined in the operation (410).

[0087] In one embodiment, the electronic device (300) may determine that it is not possible to determine a target control target device when there is no controllable device identified in the current image, or when there is a controllable device identified in the current image but there is no controllable device corresponding to the intention determined in the operation (410).

[0088] In another embodiment, the electronic device (300) can determine a target control target device by tracking the user's gaze through an eye-tracking camera (e.g., the second camera (275a, 275b) of FIG. 2). The electronic device (300) can determine the control target device as a target control target device if the user's focus remains on a controllable device included in the current image for a period of time greater than or equal to a first time.

[0089] If it is determined in operation (420) that the target control target device cannot be determined, the electronic device (300) may determine in operation (425) whether the target control target device can be determined based on vision information prior to user speech input. The electronic device (300) may determine whether it is possible to determine the target control target device based on a previous image, which is an image acquired through the camera of the electronic device (300) during a second time (e.g., a specific time such as 1 minute) prior to the point in time when user speech is input into the voice input device of the electronic device (300). In one embodiment, the first time and the second time may be the same or different.

[0090] In one embodiment, the electronic device (300) can identify at least one controllable device among the objects recognized in the previous image. The electronic device (300) can determine that a target control target device can be determined if, among the at least one identified controllable device, there is a controllable device corresponding to the intention determined in the operation (410). If a target control target device can be determined from the previous image, the electronic device (300) can determine the target control target device determined from the previous image as the target control target device corresponding to the intention determined in the operation (410).

[0091] In one embodiment, the electronic device (300) may determine that it is not possible to determine the target control target device in the case where there is no controllable device identified in the previous image, or in the case where there is a controllable device identified in the previous image but there is no controllable device corresponding to the intention determined in the operation (410).

[0092] If it is determined in operation (425) that it is not possible to determine the target control target device, the electronic device (300) can determine the space where the user is located based on the object recognized in the image and the rule-based database (433) in operation (430) where information about the object and spatial information are mapped and stored together, and determine whether it is possible to determine the target control target device from the information about the controllable device corresponding to the space.

[0093] In one embodiment, the electronic device (300) can determine the space where the user is located based on an object recognized in the current image. The operation of determining the space where the user is located can be performed using a rule-based database (433). Object information regarding objects that can be recognized in the current image and the previous image by the electronic device (300) may be tagged with a space tag corresponding to each object information and stored in the rule-based database (433). A space tag may refer to space information tagged with object information.

[0094] In one embodiment, the electronic device (300) can create a database (433) by learning from an image acquired through a camera of the electronic device (300) using a deep learning model. For example, the electronic device (300) can acquire images of various parts of the house while a user moves around the house and can recognize objects from the images. The electronic device (300) can create and update the database (433) by learning information about the space captured in the images and the objects recognized in the images.

[0095] In one embodiment, there may be multiple spatial tags. The electronic device (300) can determine a spatial tag corresponding to the space where the user is located based on spatial tags corresponding to objects included in the acquired image. For example, spatial tags for 'kitchen', 'master bedroom', and 'living room' may be stored in the database (433), and a frying pan may be tagged as the spatial tag for 'kitchen' among the spatial tags and stored in the database (433). When the frying pan is recognized in the current image, the electronic device (300) can determine the spatial tag corresponding to the space where the user is located as 'kitchen' based on the spatial tag 'kitchen' corresponding to the frying pan in the database (433).

[0096] In one embodiment, the electronic device (300) may request information about a controllable device registered with a space tag corresponding to the space where the user is located from an Internet of Things server (not shown). An Internet of Things server communicating wirelessly with the electronic device (300) may distinguish users using user accounts and store information about controllable devices registered with the Internet of Things server by each user assigning a space tag.

[0097] For example, a user may register an oven with an IoT server by assigning a space tag of 'kitchen', and register an air conditioner and a TV with an IoT server by assigning a space tag of 'living room'. When the space tag corresponding to the space where the user is located is determined to be 'kitchen', the electronic device (300) may request information from the IoT server regarding a controllable device corresponding to 'kitchen' among the controllable devices registered by the user. The IoT server may identify the user through a user account and provide the electronic device (300) with information regarding an 'oven' corresponding to 'kitchen' among the controllable devices registered by the user.

[0098] In one embodiment, when the electronic device (300) receives information about a controllable device corresponding to a spatial tag of the current image from an Internet of Things server, it can find a controllable device corresponding to an intention determined in operation (410) and determine the controllable device as a target control device. For example, if an 'oven' and a 'light' are registered to the Internet of Things server for a spatial tag of 'kitchen', the controllable device corresponding to the intention of the voice command "set the temperature to 38 degrees" may be the 'oven', and the electronic device (300) can determine the 'oven' as a target control device.

[0099] In one embodiment, there are multiple controllable devices corresponding to the spatial tag of the current image, and among the multiple controllable devices, there may also be multiple controllable devices corresponding to the intention determined in the operation (410). For example, if an 'air conditioner' and an 'oven' are registered to the Internet of Things server for the spatial tag of 'kitchen', both the 'air conditioner' and the 'oven' may be targets for temperature control. When there are multiple controllable devices corresponding to the spatial tag of the current image, and among the multiple controllable devices, there are multiple controllable devices corresponding to the intention determined in the operation (410), the electronic device (300) may query the user to control which of the multiple controllable devices using at least one of the display of the electronic device (300) and the voice output device of the electronic device (300). When the user selects one of the multiple controllable devices, the electronic device (300) may determine that controllable device as the target control device.

[0100] When a target device for control is determined, the electronic device (300) can complete a command in operation (435). For example, a voice command "Set the temperature to 38 degrees" entered with missing information about the target device for control can be completed as "Set the oven temperature to 38 degrees."

[0101] In operation (440), the electronic device (300) can control the target device to be controlled according to the intention determined in operation (410). For example, in the embodiment of FIG. 4, the temperature of the 'oven', which is the target device to be controlled, can be controlled to 38 degrees.

[0102] Hereinafter, a flowchart of a method for controlling an electronic device using vision information will be described with reference to FIG. 5.

[0103] FIG. 5 is a flowchart illustrating a method for controlling an electronic device using vision information according to one embodiment.

[0104] Referring to FIG. 5, in operation (505), the electronic device (300) can receive a user utterance through a voice input device of the electronic device (300). For example, the voice input device may include a microphone. In operation (510), the electronic device (300) can determine the intent of the user utterance by performing natural language understanding (NLU) on the received user utterance. For example, the electronic device (300) may include a natural language processing module (not shown) for natural language processing of the user utterance. The natural language processing module may include an automatic speech recognition module (not shown) and a natural language understanding module (not shown). The natural language processing module can generate text data from the utterance data by performing speech recognition on the user utterance data using the automatic speech recognition module. The natural language processing module can identify the user's intent from the text data using the natural language understanding module. For example, the natural language processing module can identify the intent corresponding to the user utterance by comparing a plurality of predefined intents with the text data.

[0105] The natural language processing module can extract additional information (e.g., information about the target control target device) from the utterance data. As another example, the electronic device (300) can transmit utterance data corresponding to a user utterance received through a voice input device to a server (e.g., the server (108) of FIG. 1). The electronic device (300) can determine the intent of the user utterance based on the result of natural language processing corresponding to the utterance data received from the server. For example, the server may be the same server as the Internet of Things server or a separate, different server.

[0106] In operation (515), the electronic device (300) can determine whether it is possible to determine a target device for control from a user's utterance. The electronic device (300) can determine a target device for control if the user's utterance contains information about the target device for control, but if the user utters without information about the target device for control or if the device for control is not specified from the user's utterance, it may be difficult for the electronic device (300) to execute the command.

[0107] If a target control target device can be determined in operation (515) (operation 515 - 'yes'), the electronic device (300) can determine the target control target device determined from user utterance as a target control target device corresponding to the intention determined in operation (510). When the target control target device is determined, the electronic device (300) can control the determined target control target device in operation (535) according to the intention determined in operation (510).

[0108] If it is not possible to determine the target control target device in operation (515) (operation 515 - 'No'), the electronic device (300) may determine whether it is possible to determine the target control target device in operation (520) based on the current image obtained through a camera that captures the surroundings of the electronic device (300). The current image may refer to an image obtained during a first time period after the point in time when a user utterance is input into the voice input device.

[0109] In one embodiment, the electronic device (300) can identify at least one controllable device in the current image. The identification of the controllable device can be performed using a deep learning model. The electronic device (300) can determine that among the at least one controllable device identified in the current image, if there is a controllable device corresponding to the intention determined in the operation (510), it is possible to determine the target control target device.

[0110] In one embodiment, the electronic device (300) may determine that it is not possible to determine a target control target device if there is no controllable device identified in the current image, or if there is no controllable device among at least one controllable device identified in the current image that corresponds to the intention determined in the operation (510).

[0111] If a target control target device can be determined in operation (520) (operation 520 - 'yes'), the electronic device (300) can determine the target control target device determined from the current image as a target control target device corresponding to the intention determined in operation (510). When the target control target device is determined, the electronic device (300) can control the determined target control target device according to the intention determined in operation (510) in operation (535).

[0112] If it is not possible to determine the target control target device in operation (520) (operation 520 - 'No'), the electronic device (300) may determine in operation (525) whether it is possible to determine the target control target device based on a previous image acquired prior to the current image. The previous image may refer to an image acquired during a second time period prior to the point in time when a user utterance is input into the voice input device.

[0113] In one embodiment, the electronic device (300) can identify at least one controllable device in a previous image. The identification of the controllable device can be performed using a deep learning model. The electronic device (300) can determine that among the at least one controllable device identified in the previous image, if there is a controllable device corresponding to the intention determined in the operation (510), it is possible to determine the target control target device.

[0114] In one embodiment, the electronic device (300) may determine that it is not possible to determine the target control target device if there is no controllable device identified in the previous image, or if there is no controllable device among at least one controllable device identified in the previous image that corresponds to the intention determined in the operation (510).

[0115] If a target control target device can be determined in operation (525) (operation 525 - 'yes'), the electronic device (300) can determine the target control target device determined from the previous image as the target control target device corresponding to the intention determined in operation (510). When the target control target device is determined, the electronic device (300) can control the determined target control target device according to the intention determined in operation (510) in operation (535).

[0116] If it is not possible to determine the target control target device in operation (525) (operation 525 - 'No'), the electronic device (300) may, in operation (530), determine information about the space where the user is located and determine whether it is possible to determine the target control target device based on the information about the space. In one embodiment, the information about the space where the user is located may be a space tag corresponding to the space where the user is located. In one embodiment, the information about the space where the user is located may be determined based on space information corresponding to objects recognized in the current image and the previous image. In one embodiment, the space information corresponding to objects recognized in the current image and the previous image may be a space tag tagged to objects recognized in the current image and the previous image.

[0117] In one embodiment, the electronic device (300) determines spatial information corresponding to the space where the user is located based on an object recognized in an image (e.g., current image or previous image) and a rule-based database (433) in which information about an object and spatial information are mapped to each other and stored, and determines whether it is possible to determine a target control target device from information about a controllable device corresponding to the spatial information.

[0118] If a target control target device can be determined in operation (530) (operation 530 - 'yes'), the electronic device (300) can determine the target control target device determined from the spatial information where the user is located as the target control target device corresponding to the intention determined in operation (510). When the target control target device is determined, the electronic device (300) can control the determined target control target device in operation (535) according to the intention determined in operation (510).

[0119] If it is not possible to determine the target control target device in operation (530) (operation 530 - 'No'), the electronic device (300) may determine in operation (540) that there is no target control target device. In one embodiment, the electronic device (300) may receive the user utterance again if there is no target control target device.

[0120] Hereinafter, the operation (530) will be described in more detail with reference to FIG. 6.

[0121] FIG. 6 is a flowchart illustrating the operation of determining a target control target device based on spatial information of a method according to one embodiment.

[0122] Referring to FIG. 6, if it is not possible to determine the target control target device in operation (525), the electronic device (300) can determine in operation (605) whether it is possible to determine a space tag corresponding to the space where the user is located.

[0123] In one embodiment, the electronic device (300) can recognize an object in the current image. The recognition of the object can be performed using a deep learning model mounted on the electronic device (300). The electronic device (300) can determine whether it is possible to determine a spatial tag corresponding to the space where the user is located based on a spatial tag in a database (433) corresponding to the object recognized in the current image. In one embodiment, the electronic device (300) can determine that it is possible to determine a spatial tag corresponding to the space where the user is located if a spatial tag corresponding to the object recognized in the current image is stored in the database (433).

[0124] In one embodiment, the electronic device (300) may determine whether it is possible to determine a space tag corresponding to the space where the user is located based on a space tag in the database (433) corresponding to an object recognized in a previous image, when there is no object recognized in the current image or when a space tag corresponding to an object recognized in the current image is not stored in the database (433). In one embodiment, the electronic device (300) may determine that it is possible to determine a space tag corresponding to the space where the user is located when a space tag corresponding to an object recognized in a previous image is stored in the database (433).

[0125] In one embodiment, the electronic device (300) may determine that it is not possible to determine a space tag corresponding to the space where the user is located if there is no object recognized in the previous image or if a space tag corresponding to an object recognized in the previous image is not stored in the database (433).

[0126] In another embodiment, if there is no object recognized in the current image or if a spatial tag corresponding to an object recognized in the current image is not stored in the database (433), the electronic device (300) may determine that it is not possible to determine a spatial tag corresponding to the space where the user of the operation (605) is located without referring to a spatial tag corresponding to an object recognized in a previous image.

[0127] If it is determined in operation (605) that it is not possible to determine a space tag corresponding to the space where the user is located (operation 605 - 'No'), the electronic device (300) may determine in operation (540) that there is no target control target device. In one embodiment, the electronic device (300) may receive the user utterance again if there is no target control target device.

[0128] In operation (605), if it is determined that it is possible to determine a space tag corresponding to the space where the user is located (operation 605 - 'yes'), the electronic device (300) can receive information about a controllable device registered as a space tag corresponding to the space where the user is located from an Internet of Things server that stores information about a controllable device registered by the user specifying a space tag in operation (610).

[0129] In operation (615), the electronic device (300) can determine whether it can determine a target device for control from information received from an Internet of Things server. In one embodiment, the electronic device (300) can determine that the target device for control can be determined if the information received from the Internet of Things server includes a controllable device corresponding to the intention determined in operation (510). In one embodiment, the electronic device (300) can determine that the target device for control cannot be determined if the information received from the Internet of Things server does not include a controllable device corresponding to the intention determined in operation (510).

[0130] In one embodiment, if the electronic device (300) includes two or more controllable devices corresponding to the intention determined in the operation (510) in the information received from the Internet of Things server, the electronic device (300) may query the user to control which of the two or more controllable devices using at least one of the display of the electronic device (300) and the voice output device of the electronic device (300). The electronic device (300) receives the user's response to the query, and if the user selects one of the two or more controllable devices, it may determine that the target control device can be determined.

[0131] If it is determined in operation (615) that the target control target device cannot be determined (operation 615 - 'No'), the electronic device (300) may determine in operation (540) that there is no target control target device. In one embodiment, the electronic device (300) may receive user speech again when there is no target control target device.

[0132] If it is determined in operation (615) that a target control target device can be determined (operation 615 - 'yes'), the electronic device (300) can control the target control target device determined in operation (535) according to the intention determined in operation (510).

[0133] FIGS. 7(A) and FIGS. 7(B) are drawings for illustrating an embodiment in which the target control target device is determined differently depending on the user's location.

[0134] Referring to FIGS. 7(A) and FIGS. 7(B), in one embodiment, an exemplary structure of a house in which a user (305) resides and a space in which the user (305) utters a voice command, “Hi Bixby, lower the temperature.” For example, the user’s (305) house includes room A, room B, a kitchen, a living room, and a master bedroom, and in FIG. 7(A), the user (305) utters a voice command in the living room, and in FIG. 7(B), the user (305) utters a voice command in the kitchen.

[0135] In the embodiments of FIGS. 7(A) and FIGS. 7(B), it is assumed that the electronic device (300) has performed the operations (505, 510, 515, 520 and 525) of FIGS. 5, but it is determined that the target control target device cannot be determined.

[0136] In the example of FIG. 7(A), the electronic device (300) can recognize a TV located in the living room (e.g., TV (710)) from the current image. In one embodiment, the rule-based database (433) may store information about space tags for 'Room A', 'Room B', 'Kitchen', 'Living Room', and 'Master Bedroom', and objects corresponding to each space tag. In one embodiment, the rule-based database (433) may store 'Living Room' as a space tag corresponding to the TV (e.g., TV (710)). The electronic device (300) can refer to the database (433) to determine 'Living Room', which is a space tag corresponding to the TV (e.g., TV (710)), as a space tag corresponding to the space where the user (305) is located.

[0137] The user (305) can register controllable devices, such as a TV (e.g., TV (710)) and an air conditioner (e.g., air conditioner (705)), with the space tag of 'living room' on an Internet of Things server. The electronic device (300) can receive information about the TV (e.g., TV (710)) and the air conditioner (e.g., air conditioner (705)) from the Internet of Things server as information about controllable electronic devices corresponding to the space tag 'living room' where the user (305) is located.

[0138] When the electronic device (300) receives information about a TV (e.g., TV (710)) and an air conditioner (e.g., air conditioner (705)) as information about a controllable electronic device corresponding to a ‘living room’ from an Internet of Things server, it can determine the air conditioner (e.g., air conditioner (705)) that corresponds to the intent of the user’s utterance “Hi Bixby, lower the temperature” among the TV (e.g., TV (710)) and the air conditioner (e.g., air conditioner (705)) as the target control device.

[0139] In the example of FIG. 7(B), the electronic device (300) can recognize a frying pan (e.g., frying pan (715)) located in the kitchen from the current image. In one embodiment, the rule-based database (433) may store 'kitchen' as a space tag corresponding to the frying pan (e.g., frying pan (715)). The electronic device (300) can refer to the database (433) to determine 'kitchen', which is a space tag corresponding to the frying pan (e.g., frying pan (715)), as a space tag corresponding to the space where the user (305) is located.

[0140] The user (305) can register an IoT server by assigning a space tag of 'kitchen' to a controllable device, such as an oven (e.g., oven (720)). The electronic device (300) can receive information about the oven (e.g., oven (720)) from the IoT server as information about a controllable electronic device corresponding to the space tag 'kitchen' where the user (305) is located.

[0141] When the electronic device (300) receives information about an oven (e.g., oven (720)) as information about a controllable electronic device corresponding to a ‘kitchen’ from an Internet of Things server, it can determine the oven (e.g., oven (720)) as a target control device corresponding to the intent of the user’s utterance, “Hi Bixby, lower the temperature.”

[0142] FIG. 8 is a block diagram illustrating the configuration of an electronic device according to one embodiment.

[0143] Referring to FIG. 8, an electronic device (800) (e.g., the electronic device (101) of FIG. 1 or the electronic device (300) of FIG. 3) may include a camera (815) that captures the surroundings of the electronic device (800) to acquire a current image, a voice input device (820) that receives user speech, a processor (805), and / or at least one memory (810) that stores instructions to be executed by the processor (805). In one embodiment, the electronic device (800) may be a wearable electronic device. In one embodiment, the memory (810) may include a rule-based database (not shown) (e.g., the database (433) of FIG. 4) that stores object information about an object that can be recognized in an image captured through the camera (815) and at least one spatial tag corresponding to the object information.

[0144] In one embodiment, when instructions stored in memory (810) are executed by the processor (805), the processor (805) can perform natural language understanding (NLU) on the received user utterance to determine the intent of the user utterance.

[0145] In one embodiment, the processor (805) can determine whether it is possible to determine a target device for control from a user's utterance. The processor (805) can determine a target device for control if the user's utterance contains information about the target device for control, but if the user utters without information about the target device for control or if the device for control is not specified from the user's utterance, it may be difficult for the processor (805) to execute a command.

[0146] If a target device for control can be determined from a user utterance, the processor (805) can determine the target device for control determined from the user utterance as a target device for control corresponding to the determined intention of the user utterance. Once the target device for control is determined, the processor (805) can control the determined device for control according to the determined intention of the user utterance. For example, the processor (805) can transmit control data according to the determined intention of the user utterance to the target device for control through a network (e.g., the first network (198) or the second network (199) of FIG. 1), and the target device for control can be set (or controlled) to perform an action according to the control data. As another example, the processor (805) can transmit control data according to the determined intention of the user utterance to an Internet of Things server through a network. The Internet of Things server can transmit the control data received from the electronic device (800) to the target device for control and set the target device for control to perform an action according to the control data.

[0147] If it is not possible to determine the target control target device from the user's utterance, the processor (805) may determine whether it is possible to determine the target control target device based on the current image obtained through the camera (815) that captures the surroundings of the electronic device (800). The current image may refer to an image obtained during a first time period after the point in time when the user's utterance is input into the voice input device (820).

[0148] In one embodiment, the processor (805) can identify at least one controllable device in the current image. The identification of the controllable device can be performed using a deep learning model. The processor (805) can determine that among the at least one controllable device identified in the current image, if there is a controllable device corresponding to the determined intention of the user utterance, it is possible to determine the target controllable device.

[0149] In one embodiment, the processor (805) may determine that it is not possible to determine a target control target device if there is no controllable device identified in the current image, or if there is no controllable device among at least one controllable device identified in the current image that corresponds to the determined intention of the user utterance.

[0150] If a target control target device can be determined from the current video, the processor (805) can determine the target control target device determined from the current video as the target control target device corresponding to the determined intention of the user's utterance. Once the target control target device is determined, the processor (805) can control the determined target control target device according to the determined intention of the user's utterance.

[0151] If it is not possible to determine the target control target device from the current image, the processor (805) may determine whether it is possible to determine the target control target device based on a previous image acquired prior to the current image. The previous image may refer to an image acquired during a second time period prior to the point in time when a user utterance is input into the voice input device (820).

[0152] In one embodiment, the processor (805) can identify at least one controllable device in the previous image. The identification of the controllable device can be performed using a deep learning model. The processor (805) can determine that among the at least one controllable device identified in the previous image, if there is a controllable device corresponding to the determined intention of the user utterance, it is possible to determine the target controllable device.

[0153] In one embodiment, the processor (805) may determine that it is not possible to determine the target control target device if there is no controllable device identified in the previous image, or if there is no controllable device among at least one controllable device identified in the previous image that corresponds to the determined intention of the user utterance.

[0154] If a target control target device can be determined from a previous video, the processor (805) can determine the target control target device determined from the previous video as the target control target device corresponding to the determined intention of the user's utterance. Once the target control target device is determined, the processor (805) can control the determined target control target device according to the determined intention of the user's utterance.

[0155] If it is not possible to determine the target control target device from the previous video, the processor (805) can determine whether it is possible to determine the space tag corresponding to the space where the user is located.

[0156] In one embodiment, the processor (805) can recognize an object in the current image. The recognition of the object may be performed using a deep learning model. The processor (805) can determine whether it is possible to determine a spatial tag corresponding to the space where the user is located based on a spatial tag in a database corresponding to the object recognized in the current image. Since the database has been described in detail with reference to FIGS. 4 to 6, a redundant description is omitted.

[0157] In one embodiment, the processor (805) may determine that it is possible to determine a space tag corresponding to the space where the user is located if a space tag corresponding to an object recognized in the current image is stored in the database.

[0158] In one embodiment, the processor (805) may determine whether it is possible to determine a space tag corresponding to the space where the user is located based on a space tag in the database corresponding to an object recognized in a previous image, when there is no object recognized in the current image or when a space tag corresponding to an object recognized in the current image is not stored in the database. In one embodiment, the processor (805) may determine that it is possible to determine a space tag corresponding to the space where the user is located if a space tag corresponding to an object recognized in a previous image is stored in the database.

[0159] In one embodiment, the processor (805) may determine that it is not possible to determine a space tag corresponding to the space where the user is located if there is no object recognized in the previous image or if a space tag corresponding to an object recognized in the previous image is not stored in the database.

[0160] If it is determined that it is not possible to determine a space tag corresponding to the space where the user is located, the processor (805) may determine that there is no target control target device. In one embodiment, the processor (805) may receive the user utterance again if there is no target control target device.

[0161] When it is determined that it is possible to determine a space tag corresponding to the space where the user is located, the processor (805) can receive information about a controllable device registered as a space tag corresponding to the space where the user is located from an Internet of Things server that stores information about a controllable device registered by the user specifying a space tag.

[0162] In one embodiment, the processor (805) can determine whether a target device for control can be determined from information received from an Internet of Things server. In one embodiment, the processor (805) can determine that a target device for control can be determined if the information received from the Internet of Things server includes a controllable device corresponding to the determined intention of the user's utterance. In one embodiment, the processor (805) can determine that a target device for control cannot be determined if the information received from the Internet of Things server does not include a controllable device corresponding to the determined intention of the user's utterance.

[0163] In one embodiment, if the information received from the Internet of Things server contains two or more controllable devices corresponding to the determined intention of the user's utterance, the processor (805) may query the user to control which of the two or more controllable devices using at least one of the display (not shown) of the electronic device (800) and the voice output device (not shown) of the processor (805). The processor (805) receives the user's response to the query and, if the user selects one of the two or more controllable devices, may determine that the target control device can be determined.

[0164] If it is determined that it is not possible to determine the target control target device from the information received from the Internet of Things server, the processor (805) may determine that there is no target control target device. In one embodiment, if there is no target control target device, the processor (805) may receive the user utterance again.

[0165] If it is determined that a target device for control can be determined from the information received from the Internet of Things server, the processor (805) can control the determined target device for control according to the determined intention of the user's utterance.

[0166] FIG. 9 is a flowchart of a method for controlling an electronic device using vision information according to one embodiment.

[0167] Referring to FIG. 9, a method for controlling an electronic device using vision information according to one embodiment includes an operation (905) of receiving a user utterance through a voice input device (820) of a wearable electronic device (800), an operation (910) of determining a user's intention based on the received user utterance, an operation (915) of determining a target device for control corresponding to the determined intention, and an operation (920) of controlling the target device for control determined based on the determined intention. The operation (915) of determining a target device for control may include an operation of determining whether it is possible to determine a target device for control from the user utterance, and, if it is not possible to determine a target device for control from the user utterance, an operation of determining whether it is possible to determine a target device for control based on a current image obtained through a camera (815) that captures the surroundings of the wearable electronic device (800) to obtain a current image.

[0168] The operation (915) for determining a target control target device may include, if it is possible to determine a target control target device from a current image, the operation of determining the target control target device determined from the current image as a target control target device corresponding to the determined intention.

[0169] The operation of determining whether it is possible to determine a target control target device based on the current image may include the operation of identifying at least one controllable device in the current image, and the operation of determining that it is possible to determine a target control target device if there is a controllable device corresponding to a determined intention among the identified at least one controllable device.

[0170] The operation (915) for determining a target control target device may further include determining whether it is possible to determine a target control target device based on at least one controllable device identified in a previous image acquired prior to the current image, when it is not possible to determine a target control target device from the current image.

[0171] The operation (915) of determining a target control target device may further include: determining whether it is possible to determine a space tag corresponding to the space where the user is located among at least one space tag stored in a database that stores object information for objects recognizable in the current image and the previous image and at least one space tag corresponding to the object information, when it is not possible to determine a target control target device from at least one controllable device identified in the previous image; when it is possible to determine a space tag, receiving information about a controllable device registered as a space tag corresponding to the space where the user is located from an Internet of Things server that stores information about a controllable device registered by the user by specifying a space tag, and when the information received from the Internet of Things server includes a controllable device corresponding to the determined intention, determining that it is possible to determine a target control target device.

[0172] The operation of determining that it is possible to determine a target control device may further include, when there are two or more controllable devices corresponding to the intention determined in the information received from the Internet of Things server, an operation of querying the user to control which of the two or more controllable devices using at least one of the display of the electronic device (800) and the voice output device of the electronic device (800).

[0173] The operation of determining whether it is possible to determine a spatial tag may include the operation of recognizing an object in the current image, and the operation of determining whether it is possible to determine a spatial tag corresponding to the space where the user is located based on a spatial tag in a database corresponding to the object recognized in the current image.

[0174] An electronic device (800) that controls an electronic device using vision information according to one embodiment includes a camera (815) that captures the surroundings of the electronic device (800) to acquire a current image, a voice input device (820) that receives user speech, a processor (805), and at least one memory (810) that stores instructions to be executed by the processor (805). When instructions are executed by the processor (805), the processor (805) performs an operation of determining a user's intention based on the received user speech, an operation of determining a target device for control corresponding to the determined intention, and an operation of controlling the target device for control determined based on the determined intention. The operation of determining a target device for control may include an operation of determining whether it is possible to determine a target device for control from the user speech, and, if it is not possible to determine a target device for control from the user speech, an operation of determining whether it is possible to determine a target device for control based on the current image acquired through the camera (815).

[0175] The operation of determining a target control target device may further include, if it is possible to determine a target control target device from the current image, an operation of determining the target control target device determined from the current image as the target control target device corresponding to the determined intention.

[0176] The operation of determining whether it is possible to determine a target control target device based on the current image may include the operation of identifying at least one controllable device in the current image, and the operation of determining that it is possible to determine a target control target device if there is a controllable device corresponding to a determined intention among the identified at least one controllable device.

[0177] The operation of determining a target control target device may further include, when it is not possible to determine a target control target device from the current image, an operation of determining whether it is possible to determine a target control target device based on at least one controllable device identified in a previous image acquired prior to the current image.

[0178] The memory (810) stores a database that stores object information for objects recognizable in the current image and the previous image, and at least one space tag corresponding to the object information. The operation of determining a target control target device may further include determining whether it is possible to determine a space tag corresponding to the space where the user is located among at least one space tag stored in the database when it is not possible to determine a target control target device from at least one controllable device identified in the previous image.

[0179] The operation of determining a target device for control may further include, when it is possible to determine a space tag, receiving information about a controllable device registered with a space tag corresponding to the space where the user is located from an Internet of Things server that stores information about a controllable device registered by a user specifying a space tag, and determining that it is possible to determine a target device for control if the information received from the Internet of Things server includes a controllable device corresponding to the determined intention.

[0180] The operation of determining that it is possible to determine a target control device may further include, when there are two or more controllable devices corresponding to the intention determined in the information received from the Internet of Things server, an operation of querying the user to control which of the two or more controllable devices using at least one of the display of the electronic device (800) and the voice output device of the electronic device (800).

[0181] The current image is an image acquired during a first time period after the point in time when user speech is input into the voice input device (820).

[0182] The previous video is a video acquired during a second time period prior to the point in time when the user's speech was input into the voice input device (820).

[0183] The operation of determining whether it is possible to determine a spatial tag may include the operation of recognizing an object in the current image, and the operation of determining whether it is possible to determine a spatial tag corresponding to the space where the user is located based on a spatial tag in a database corresponding to the object recognized in the current image.

[0184] The operation of determining whether it is possible to determine a spatial tag may include, when there is no object recognized in the current image, determining whether it is possible to determine a spatial tag corresponding to the space where the user is located based on a spatial tag in a database corresponding to an object recognized in a previous image.

[0185] The operation of identifying at least one controllable device in the current video can be performed using a deep learning model.

[0186] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0187] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., first) component is referred to as “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., by wire), wirelessly, or through a third component.

[0188] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0189] Various embodiments of this document may be implemented as software (e.g., program (#40)) comprising one or more instructions stored in a storage medium (e.g., internal memory (#36) or external memory (#38)) readable by a machine (e.g., electronic device (#01)). For example, a processor (e.g., processor (#20)) of the machine (e.g., electronic device (#01)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0190] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0191] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

Claim 1 An electronic device comprises: a camera that captures the surroundings of the electronic device to acquire a current image; a voice input device that receives user utterances; one or more processors; and at least one memory that stores instructions. When the instructions are executed by the one or more processors, the instructions cause the electronic device to perform the following operations: determining a user's intention based on the received user utterances; determining a target device for control corresponding to the determined intention; and controlling the determined device for control based on the determined intention. The operation of determining the target device for control includes: determining whether it is possible to determine the target device for control from the user utterances; and, if it is not possible to determine the target device for control from the user utterances, determining whether it is possible to determine the target device for control based on the current image acquired through the camera. An electronic device comprising an operation of determining whether it is possible to determine the target control target device based on at least one controllable device identified in a previous image acquired prior to the current image, when it is not possible to determine the target control target device from the current image. Claim 2 An electronic device according to claim 1, wherein the operation of determining the target control target device further includes, when it is possible to determine the target control target device from the current image, the operation of determining the target control target device determined from the current image as the target control target device corresponding to the determined intention. Claim 3 An electronic device according to claim 1, wherein the operation of determining whether it is possible to determine the target control target device based on the current image comprises: an operation of identifying at least one controllable device in the current image; and an operation of determining that the target control target device is possible if, among the at least one identified controllable device, there is a controllable device corresponding to the determined intention. Claim 4 delete Claim 5 An electronic device according to claim 1, wherein the memory stores a database storing object information for an object recognizable in the current image and the previous image and at least one spatial tag corresponding to the object information, and the operation of determining the target control target device further includes, when it is not possible to determine the target control target device from at least one controllable device identified in the previous image, the operation of determining whether it is possible to determine a spatial tag corresponding to the space where the user is located among at least one spatial tag stored in the database. Claim 6 In claim 5, the operation of determining the target control target device further comprises: an operation of receiving information about a controllable device registered as a space tag corresponding to the space where the user is located from an Internet of Things server that stores information about a controllable device registered by the user by designating a space tag, when it is possible to determine the space tag; and an operation of determining that it is possible to determine the target control target device when the information received from the Internet of Things server includes a controllable device corresponding to the determined intention. Claim 7 An electronic device according to claim 6, wherein the operation of determining that it is possible to determine the target control target device further includes, when the information received from the Internet of Things server includes two or more controllable devices corresponding to the determined intention, an operation of querying the user to control which of the two or more controllable devices using at least one of the display of the electronic device and the voice output device of the electronic device. Claim 8 An electronic device according to claim 1, wherein the current image is an image acquired during a first time period after the point in time when the user utterance is input into the voice input device, and the previous image is an image acquired during a second time period prior to the point in time when the user utterance is input into the voice input device. Claim 9 An electronic device according to claim 5, wherein the operation of determining whether it is possible to determine the spatial tag comprises: the operation of recognizing an object in the current image; and the operation of determining whether it is possible to determine a spatial tag corresponding to the space where the user is located based on a spatial tag in the database corresponding to an object recognized in the current image, and wherein the operation of determining whether it is possible to determine the spatial tag comprises, when there is no object recognized in the current image, the operation of determining whether it is possible to determine a spatial tag corresponding to the space where the user is located based on a spatial tag in the database corresponding to an object recognized in the previous image. Claim 10 A method for controlling an electronic device using vision information performed by an electronic device, comprising: receiving a user utterance through a voice input device of a wearable electronic device; determining a user's intention based on the received user utterance; determining a target device for control corresponding to the determined intention; and controlling the determined target device for control based on the determined intention, wherein the operation of determining the target device for control includes: determining whether it is possible to determine the target device for control from the user utterance; if it is not possible to determine the target device for control from the user utterance, determining whether it is possible to determine the target device for control based on a current image obtained through a camera that captures the surroundings of the wearable electronic device and obtains a current image; and if it is not possible to determine the target device for control from the current image, determining whether it is possible to determine the target device for control based on at least one controllable device identified in a previous image obtained prior to the current image. Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images