Device for outputting object-based feedback, operating method thereof, and recording medium

Abstract

An electronic device according to an embodiment may comprise: a memory in which computer-executable instructions are stored; and a processor which accesses the memory to execute the instructions. The instructions may, when executed, cause the processor to: on the basis of obtaining a user's input, generate first feedback for the input; generate second feedback subsequent to the first feedback on the basis of an object detected in at least one of a display area, a surrounding physical space, and a virtual space; and output the first feedback and the second feedback.

Description

Device for outputting object-based feedback, method of operation thereof, and recording medium

[0001] Hereinafter, a device for outputting object-based feedback, its operating method, and a recording medium are disclosed.

[0002] Recently, virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies that utilize computer graphics technology are being developed. VR technology, for example, uses computers to create a virtual space that doesn't exist in the real world and then makes that virtual space feel real. AR or MR, on the other hand, refers to technologies that superimpose computer-generated information on top of the real world. In other words, they combine the real and virtual worlds to enable real-time user interaction.

[0003] Among these, AR and MR technologies are being integrated and utilized in various fields, such as broadcasting, medical technology, and gaming technology. Representative examples of AR being integrated into broadcasting include the natural change of weather maps in front of a weathercaster giving a weather forecast on TV, or the insertion of non-existent advertising images into the stadium during a sports broadcast, making them appear as if they were actually there.

[0004] A representative service that provides users with augmented reality or mixed reality is the metaverse. This metaverse, a portmanteau of "meta," meaning "fictional" or "abstract," and "universe," signifies a three-dimensional virtual world. A more advanced concept than the existing term "virtual reality environment," the metaverse offers an augmented reality environment where the virtual world of the web and the internet is integrated into the real world.

[0005] The above information is provided as background information to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0006] An electronic device according to various embodiments may include a memory having computer-executable instructions stored therein; and a processor accessing the memory to execute the instructions. The instructions, when executed, may cause the processor to generate a first feedback for a user input based on obtaining the input, generate a second feedback subsequent to the first feedback based on an object detected in at least one of a display area, a surrounding physical space, or a virtual space, and output the first feedback and the second feedback.

[0007] A method performed by an electronic device according to various embodiments may include: generating a first feedback for an input based on obtaining a user input; generating a second feedback subsequent to the first feedback based on an object detected in at least one of a display area, a surrounding physical space, or a virtual space; and outputting the first feedback and the second feedback.

[0008] FIG. 1 is a block diagram illustrating an exemplary configuration of an electronic device according to various embodiments.

[0009] FIG. 2 illustrates an example of an optical see-through device according to various embodiments.

[0010] FIG. 3 illustrates examples of optical systems for an eye tracking camera, a transparent member, and a display according to various embodiments.

[0011] FIGS. 4A and 4B are drawings showing examples of the front and back of an electronic device according to various embodiments.

[0012] FIG. 5 illustrates examples of construction of a virtual space, input from a user within the virtual space, and output to the user according to various embodiments.

[0013] FIGS. 6A and 6B are diagrams illustrating examples of operations in which an electronic device according to various embodiments outputs first feedback and second feedback based on an object detected in a display area.

[0014] FIG. 7 is a drawing illustrating an example of a configuration of an electronic device according to various embodiments.

[0015] FIG. 8 is a flowchart illustrating an example of an operation of an electronic device outputting feedback according to various embodiments.

[0016] FIG. 9 is a diagram illustrating an operation of analyzing additional input when an electronic device obtains additional input according to various embodiments.

[0017] FIGS. 10A and 10B are diagrams illustrating examples of operations in which an electronic device outputs an interface to an object according to various embodiments.

[0018] FIG. 11 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to output a preview of an object's operation.

[0019] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0020] FIG. 1 is a block diagram illustrating an exemplary configuration of an electronic device according to various embodiments.

[0021] FIG. 1 is a block diagram of an example of an electronic device (101) within a network environment (100), according to various embodiments. Referring to FIG. 1 , in the network environment (100), the electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to various embodiments, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0022] The processor (120) includes various processing circuits and / or multiple processors. For example, as used herein, including in the claims, the term "processor" may include various processing circuits including at least one processor, one or more of which may be individually and / or collectively configured to perform various functions as described herein, in a distributed manner. As used herein, "processor," "at least one processor," and "one or more processors" may be described as being configured to perform numerous functions, and these terms cover, by way of example and without limitation, situations in which one processor performs some of the listed functions, other processor(s) perform other of the listed functions, and a single processor may perform all of the listed functions. Additionally, the at least one processor may include a combination of processors that perform various of the listed and / or disclosed functions in a distributed manner. The at least one processor may execute program instructions to perform or accomplish various functions. The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134).According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0023] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0024] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0025] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0026] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0027] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0028] A display module (160) (e.g., a display) can visually provide information to an external device (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0029] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0030] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0031] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0032] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0033] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0034] The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0035] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0036] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0037] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0038] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0039] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0040] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0041] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0042] According to one embodiment, commands or data can be transmitted or received between an electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199).

[0043] Each of the external electronic devices (102, 104) and the server (108) may be the same type of device as or different from the electronic device (101). According to one embodiment, all or part of the operations executed by the electronic device (101) may be executed by one or more of the external electronic devices (102, 104) or the server (108). For example, when the electronic device (101) needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of executing the function or service itself or in addition, request one or more external electronic devices to execute at least a part of the function or service. The one or more external electronic devices that receive the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a part of a response to the request. In this specification, an example is mainly described in which an electronic device (101) is an augmented reality device (e.g., an electronic device (201) of FIG. 2, an electronic device (301) of FIG. 3, or an electronic device (401) of FIG. 4), and an external electronic device (102, 104) or a server (108) among the servers transmits the results of executing a virtual space and additional functions or services related to the virtual space to the electronic device (101).

[0044] The server (108) may include a processor (181) (e.g., a processor including a processing circuit), a communication module (182) (e.g., a communication module including a communication circuit), and a memory (183). The processor (181), the communication module (182), and the memory (183) may be configured similarly to the processor (120), the communication module (190), and the memory (130) of the electronic device (101). For example, the processor (181) may provide a virtual space and interaction between users within the virtual space by executing instructions stored in the memory (183). The processor (181) may generate at least one of visual information, auditory information, or tactile information of the virtual space and objects within the virtual space. For example, as visual information, the processor (181) may generate rendering data (e.g., visual rendering data) that renders the appearance (e.g., shape, size, color, and / or texture) of a virtual space and the appearance (e.g., shape, size, color, and / or texture) of an object positioned within the virtual space. In addition, the processor (181) may also generate rendering data that renders a change (e.g., change in the appearance of an object, generation of a sound, and / or generation of a tactile sensation) based on at least one of an interaction between objects (e.g., physical objects, virtual objects, and / or avatar objects) within the virtual space, or a user input to an object (e.g., physical objects, virtual objects, or avatar objects). The communication module (182) may establish communication with a first electronic device (e.g., electronic device (101)) of a user and a second electronic device (e.g., electronic device (102)) of another user. The communication module (182) can transmit at least one of the aforementioned visual information, tactile information, or auditory information to the first electronic device and / or the second electronic device. For example, the communication module (182) can transmit rendering data.

[0045] For example, the server (108) renders content data executed in an application and transmits it to the electronic device (101), and the electronic device (101) that receives the data can output the content data to the display module (160). If the electronic device (101) detects user movement through an IMU (inertial measurement unit) sensor, the processor (120) of the electronic device (101) can correct the rendering data received from the external electronic device (102) based on the movement information and output it to the display module (160). Alternatively, the movement information can be transmitted to the server (108) to request rendering so that the screen data is updated accordingly. However, the present invention is not limited thereto, and the above-described rendering can be performed by various forms of external electronic devices (102, 104), such as a case device or dock that can store and charge a smartphone or the electronic device (101). Rendering data corresponding to the aforementioned virtual space generated by an external electronic device (102, 104) may be provided to the electronic device (101). As another example, the electronic device (101) may receive virtual space information (e.g., vertex coordinates, texture, color defining the virtual space) and object information (e.g., vertex coordinates, texture, color defining the appearance of an object) from a server (108) and perform rendering on its own based on the received data.

[0046] FIG. 2 illustrates an example of an optical see-through device according to various embodiments.

[0047] The electronic device (201) may include at least one of a display (e.g., a display module (160) of FIG. 1), a vision sensor, a light source (230a, 230b), an optical element, or a substrate. An electronic device (201) that has a transparent display and provides an image through the transparent display may be referred to as an optical see-through device (OST device).

[0048] The display may include, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0049] In one embodiment, when the display is formed of one of a liquid crystal display (LCD), a digital mirror display (DMD), or a silicon liquid crystal display (SLCD), the electronic device (201) may include a light source (230a, 230b) that irradiates light to a screen output area of ​​the display (e.g., a screen display unit (215a, 215b)). In another embodiment, when the display can generate light on its own, for example, when it is formed of one of an organic light emitting diode (OLED) or a micro LED, the electronic device (201) may provide a good quality virtual image to the user even without including a separate light source (230a, 230b). In one embodiment, if the display is formed of an organic light emitting diode (OLED) or a micro LED, the light source (230a, 230b) is unnecessary, and thus the electronic device (201) may be lightweight.

[0050] Referring to FIG. 2, the electronic device (201) may include a display, a first transparent member (225a) and / or a second transparent member (225b), and a user may use the electronic device (201) while wearing it on his or her face. The first transparent member (225a) and / or the second transparent member (225b) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent. According to one embodiment, the first transparent member (225a) may be arranged to face the user's right eye, and the second transparent member (225b) may be arranged to face the user's left eye. The display may include a first display (205) that outputs a first image (e.g., a right image) corresponding to the first transparent member (225a), and a second display (210) that outputs a second image (e.g., a left image) corresponding to the second transparent member (225b). In one embodiment, if each display is transparent, each display and transparent member may be positioned at a position facing the user's eyes to form a screen display unit (215a, 215b).

[0051] In one embodiment, light emitted from the display (205, 210) may be guided along an optical path through an input optical member (220a, 220b) into a waveguide. Light traveling within the waveguide may be guided toward a user's eyes through an output optical member (e.g., the output optical member (340) of FIG. 3). The screen display units (215a, 215b) may be determined based on the light emitted toward the user's eyes.

[0052] For example, light emitted from the display (205, 210) may be reflected in the grating region of the waveguide formed in the input optical member (220a, 220b) and the screen display portions (215a, 215b) and transmitted to the user's eyes.

[0053] The optical element may include at least one of a lens or an optical waveguide.

[0054] The lens can adjust the focus so that the screen output to the display can be viewed by the user. The lens may include, for example, at least one of a Fresnel lens, a pancake lens, or a multi-channel lens.

[0055] An optical waveguide can transmit image light generated from a display to a user's eyes. For example, the image light may represent light emitted by a light source (230a, 230b) that passes through a screen output area of ​​the display. The optical waveguide can be made of glass, plastic, or polymer. The optical waveguide can include a nano-pattern formed on a portion of an internal or external surface, for example, a grating structure having a polygonal or curved shape. An exemplary structure of the optical waveguide is described below in FIG. 3.

[0056] The vision sensor may include at least one of a camera sensor or a depth sensor.

[0057] The first camera (265a, 265b) is a recognition camera, and may be used for 3DoF (degrees of freedom), 6DoF head tracking, hand detection, hand tracking, and spatial recognition. The first camera (265a, 265b) may include, for example, a GS (global shutter) camera. Since a stereo camera is required for head tracking and spatial recognition, the first camera (265a, 265b) may include two or more GS cameras. The GS camera may have superior performance compared to the RS (rolling shutter) camera in terms of detecting rapid hand movements and fine finger movements and tracking movements. For example, the GS camera may have low image blur. The first camera (265a, 265b) can capture image data used for spatial recognition for 6DoF and SLAM (simultaneous localization and mapping) functions through depth shooting. In addition, a user gesture recognition function can be performed based on the image data captured by the first camera (265a, 265b).

[0058] The second camera (270a, 270b) is an ET (eye tracking) camera that can be used to capture image data for detecting and tracking the user's pupils. The second camera (270a, 270b) is described later in FIG. 3.

[0059] The third camera (245) may be a camera for photography. The third camera (245) may include a high-resolution camera for capturing high-resolution (HR) or photo video (PV) images. The third camera (245) may include a color camera equipped with functions for obtaining high-quality images, such as an auto focus (AF) function and optical image stabilization (OIS). The third camera (245) may be a GS camera or an RS camera.

[0060] The fourth camera unit (e.g., the face recognition camera (425, 426) of FIG. 4 below) is a face recognition camera, and the FT (face tracking) camera can be used to detect and track the user's facial expression.

[0061] A depth sensor (not shown) may be configured to sense information for determining the distance to an object, such as, for example, time of flight (TOF). TOF is a technology for measuring the distance to an object using, for example, signals (e.g., near-infrared, ultrasound, laser). A depth sensor based on TOF technology can measure the time of flight of a signal by transmitting a signal and measuring the signal at a receiving unit.

[0062] The light source (230a, 230b) (e.g., an illumination module) may include a device (e.g., a light emitting diode (LED)) that emits light of various wavelengths. The illumination module may be attached to various locations depending on the application. In one example, a first illumination module (e.g., an LED device) attached around the frame of an augmented reality glasses device may emit light to assist in gaze detection when tracking eye movements with an ET camera. The first illumination module may include, for example, an IR LED of an infrared wavelength. As another example of use, a second lighting module (e.g., an LED element) may be attached adjacent to a camera mounted around a hinge (240a, 240b) connecting a frame (e.g., a first frame (290a), a second frame (290b)) and a temple (e.g., a first temple (280a), a second temple (280b)), or around a bridge (295) connecting the frames. The second lighting module may emit light to supplement the ambient brightness when the camera is taking a picture. In a dark environment where subject detection is not easy, the second lighting module may emit light.

[0063] A substrate (235a, 235b) (e.g., a printed circuit board (PCB)) can support the aforementioned components.

[0064] A printed circuit board (PCB) may be disposed on the temple portion of the glasses. The flexible printed circuit board (FPCB) may transmit electrical signals to each module (e.g., a camera, a display, an audio module, a sensor module) and other printed circuit boards. In one embodiment, at least one printed circuit board may include a first substrate, a second substrate, and an interposer disposed between the first substrate and the second substrate. Electrical signals may be transmitted to each module and other printed circuit boards.

[0065] Other components may include, for example, at least one of a plurality of microphones (e.g., a first microphone (250a), a second microphone (250b), a third microphone (250c)), a plurality of speakers (e.g., a first speaker (255a), a second speaker (255b)), a battery (260), an antenna, or a sensor (e.g., an acceleration sensor, a gyro sensor, a touch sensor).

[0066] FIG. 3 illustrates examples of optical systems for an eye tracking camera, a transparent member, and a display according to various embodiments.

[0067] FIG. 3 is a diagram for explaining the operation of an eye-tracking camera included in an electronic device according to various embodiments. Referring to FIG. 3, a process is illustrated in which an eye-tracking camera (310) of an electronic device (301) according to one embodiment (e.g., the first eye-tracking camera (270a) and the second eye-tracking camera (270b) of FIG. 2) tracks a user's eye (309), that is, the user's gaze, using light (e.g., infrared light) output from a display (320) (e.g., the first display (205) and the second display (210) of FIG. 2).

[0068] The second camera (e.g., the second cameras (270a, 270b) of FIG. 2) may be an eye tracking camera (310) that collects information to position the center of a virtual image projected onto the electronic device (301) according to the direction in which the pupil of the wearer of the electronic device (301) is looking. The second camera may also include a GS camera to detect the pupil and track rapid eye movement. ET cameras may also be installed for the left and right eyes, respectively, and camera performance and specifications may be substantially the same. The eye tracking camera (310) may include a gaze tracking sensor (315). The gaze tracking sensor (315) may be included inside the eye tracking camera (310). Infrared light output from the display (320) may be transmitted to the user's eyes (309) as infrared reflected light (303) by a half mirror. The gaze tracking sensor (315) can detect infrared reflected light (303) and infrared transmitted light (305) reflected from the user's eye (309). The eye tracking camera (310) can track the user's eye (309), or in other words, the user's gaze, based on the detection result of the gaze tracking sensor (315).

[0069] The display (320) may include a plurality of visible light pixels and a plurality of infrared pixels. The visible light pixels may include R, G, and B pixels. The visible light pixels may output visible light corresponding to a virtual object image. The infrared pixels may output infrared light. The display (320) may include, for example, micro light emitting diodes (LEDs) or organic light emitting diodes (OLEDs).

[0070] The display optical pipe (350) and the eye tracking camera optical pipe (360) may be included inside a transparent member (370) (e.g., the first transparent member (225a) and the second transparent member (225b) of FIG. 2). The transparent member (370) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent. The transparent member (370) may be positioned to face the user's eyes. At this time, the distance between the transparent member (370) and the user's eyes (309) may be referred to as 'eye relief' (380).

[0071] The transparent member (370) may include optical waveguides (350, 360). The transparent member (370) may include an input optical member (330) and an output optical member (340). In addition, the transparent member (370) may include an eye-tracking splitter (375) that separates input light into multiple waveguides.

[0072] According to one embodiment, light incident on one end of the display light pipe (350) can be propagated inside the display light pipe (350) by the nano-pattern and provided to the user. In addition, the display light pipe (350) composed of a free-form prism can provide image light to the user through the reflected mirror by reflecting the incident light. The display light pipe (350) can include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). The display light pipe (350) can guide display light (e.g., image light) emitted from a light source to the user's eyes by using at least one diffractive element or reflective element included in the display light pipe (350). For reference, also, in FIG. 3, the output optical member (340) is expressed as being separate from the eye-tracking optical waveguide (360), but the output optical member (340) may be included inside the eye-tracking optical waveguide (360).

[0073] According to various embodiments, the diffractive element may include an input optical member (330) and an output optical member (340). For example, the input optical member (330) may mean an input grating region. The output optical member (340) may mean an output grating region. The input grating region may serve as an input terminal that diffracts (or reflects) light output from (e.g., a micro LED) to transmit the light to a transparent member (e.g., a first transparent member, a second transparent member) of the screen display unit. The output grating region may serve as an outlet that diffracts (or reflects) light transmitted to a transparent member (e.g., a first transparent member, a second transparent member) of the waveguide to a user's eye.

[0074] According to various embodiments, the reflective element may include a total internal reflection (TIR) ​​optical element or a total internal reflection waveguide. For example, total internal reflection may refer to a method of guiding light such that light (e.g., a virtual image) entering through an input grating region is 100% reflected from one surface (e.g., a specific surface) of the waveguide, thereby transmitting 100% of the light to the output grating region.

[0075] In one embodiment, light emitted from the display (320) may be guided along an optical path through an input optical element (330) into a waveguide. Light traveling within the waveguide may be guided toward the user's eyes through an output optical element (340). The screen display may be determined based on the light emitted toward the user's eyes.

[0076] FIGS. 4A and 4B are diagrams illustrating examples of the front and back of an electronic device according to various embodiments. FIG. 4A may be an external appearance of the electronic device (401) as viewed from a first direction (①), and FIG. 4B may be an external appearance of the electronic device (401) as viewed from a second direction (②). When a user wears the electronic device (401), the external appearance viewed by the user's eyes may be FIG. 4B.

[0077] Referring to FIG. 4A, according to various embodiments, an electronic device (401) (e.g., the electronic device (101) of FIG. 1, the electronic device (201) of FIG. 2, and the electronic device (301) of FIG. 3) may provide a service that provides an extended reality (XR) experience to a user. For example, the XR or XR service may be defined as a service that collectively refers to virtual reality (VR), augmented reality (AR), and / or mixed reality (MR), for example.

[0078] According to one embodiment, the electronic device (401) may mean, for example, a head-mounted device or a head-mounted display worn on the user's head, but may also be configured in the form of at least one of glasses, goggles, a helmet, or a hat. The electronic device (401) may include an OST (optical see-through) type configured to allow external light to reach the user's eyes through the glasses when worn, or a VST (video see-through) type configured to block external light so that, when worn, light emitted from the display reaches the user's eyes but external light does not reach the user's eyes.

[0079] According to one embodiment, the electronic device (401) may be worn on the user's head and may provide the user with an image related to an extended reality (XR) service. For example, the electronic device (401) may provide XR content (hereinafter referred to as an XR content image) that outputs at least one virtual object to be superimposed on a display area or an area determined to be the user's field of view (FoV). According to one embodiment, the XR content may refer to an image or image related to a real space acquired through a camera (e.g., a camera for taking pictures) or an image or image in which at least one virtual object is superimposed on a virtual space. According to one embodiment, the electronic device (401) may provide XR content based on a function being performed by the electronic device (401) and / or a function being performed by one or more external electronic devices (e.g., the electronic devices (102, 104) of FIG. 1, the server (108) of FIG. 1).

[0080] According to one embodiment, the electronic device (401) is at least partially controlled by an external electronic device (e.g., electronic devices (102 or 104) of FIG. 1), and may perform at least one function under the control of the external electronic device, but may also perform at least one function independently.

[0081] Referring to FIG. 4A, a vision sensor may be placed on a first surface of a housing of a main body (410) of an electronic device (401). The vision sensor may include cameras (e.g., cameras for second functions (411, 412), cameras for first functions (415)) and / or a depth sensor (417) for obtaining information related to the surrounding environment of the electronic device (401).

[0082] In one embodiment, the second function cameras (411, 412) can obtain images related to the surrounding environment of the electronic device (401). The first function cameras (415) can obtain images when the wearable electronic device is worn by the user. The first function cameras (415) can be used for hand detection and tracking, and recognition of user gestures (e.g., hand movements). The first function cameras (415) can be used for 3DoF, 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. In one embodiment, the second function cameras (411, 412) can also be used for hand detection and tracking, and user gestures.

[0083] In one embodiment, the depth sensor (417) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Instead of or in addition to the depth sensor (417), cameras (411, 412, 415) may determine the distance to an object.

[0084] Referring to FIG. 4b, a camera (425, 426) for facial recognition and / or a display (421) (and / or lens) may be placed on the second surface (420) of the housing of the main body (410).

[0085] In one embodiment, a face recognition camera (425, 426) adjacent to the display may be used to recognize the user's face, or may recognize and / or track the user's two eyes.

[0086] In one embodiment, the display (421) (and / or lens) may be disposed on the second side (420) of the electronic device (401). In one embodiment, the electronic device (401) may not include some of the plurality of cameras (415). Although not shown in FIGS. 4A and 4B , the electronic device (401) may further include at least one of the configurations illustrated in FIG. 2 .

[0087] According to one embodiment, the electronic device (401) may include a main body (410) that mounts at least some of the components of FIG. 1, a display (421) (e.g., a display module (160) of FIG. 1) disposed in a first direction (①) of the main body (410), a first function camera (e.g., a recognition camera) (415) disposed in a second direction (②) of the main body (410), a second function camera (e.g., a shooting camera) (411, 412) disposed in a second direction (②), a third function camera (e.g., a gaze tracking camera) (428) disposed in the first direction (①), a fourth function camera (e.g., a face recognition camera) (425, 426) disposed in the first direction (①), a depth sensor (417) disposed in the second direction (②), and a touch sensor (413) disposed in the second direction (②). Although not shown in the drawing, the main body (410) includes a memory (e.g., memory (130) of FIG. 1) and a processor (e.g., processor (120) of FIG. 1), and may further include other components shown in FIG. 1.

[0088] According to one embodiment, the display (421) may include a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0089] In one embodiment, when the display (421) is formed of one of a liquid crystal display (LCD), a digital mirror display (DMD), or a silicon liquid crystal display (SiLCD), the electronic device (401) may include a light source that irradiates light to a screen output area of ​​the display (421). In one embodiment, when the display (421) can generate light on its own, for example, when the electronic device (401) is formed of one of an organic light-emitting diode (OLED) or a micro LED, the electronic device (401) may provide a good quality XR content image to the user even without including a separate light source. In one embodiment, when the display (421) is implemented with an organic light-emitting diode (OLED) or a micro LED, a light source is unnecessary, and thus the electronic device (401) may be lightweight.

[0090] According to one embodiment, the display (421) may include a first transparent member (421a) and / or a second transparent member (421b). The user may use the electronic device (401) while wearing it on his or her face. The first transparent member (421a) and / or the second transparent member (421b) may be formed of a glass plate, a plastic plate, or a polymer, and may be manufactured to be transparent or translucent. According to one embodiment, the first transparent member (421a) may be arranged to face the user's left eye in the fourth direction (④), and the second transparent member (421b) may be arranged to face the user's right eye in the third direction (③). According to various embodiments, when the display (421) is transparent, it may be arranged at a position facing the user's eyes to form a display area.

[0091] According to one embodiment, the display (421) may include a lens including a transparent waveguide. The lens may serve to adjust the focus so that the screen (e.g., XR content image) output to the display (421) can be viewed by the user's eyes. For example, light emitted from the display panel may pass through the lens and be transmitted to the user through a waveguide formed within the lens. The lens may be configured as a Fresnel lens, a pancake lens, or a multi-channel lens.

[0092] An optical waveguide (e.g., a waveguide) may serve to transmit light generated by the display (421) to the user's eyes. The optical waveguide may be made of glass, plastic, or a polymer, and may include nano-patterns formed on a portion of the inner or outer surface, for example, a grating structure having a polygonal or curved shape. According to one embodiment, light incident on one end of the optical waveguide, that is, an output image of the display (421), may be propagated within the optical waveguide and provided to the user. In addition, an optical waveguide composed of a free-form prism may provide the incident light to the user through a reflective mirror. The optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). The optical waveguide may guide an image output from the display (421) to the user's eyes by using at least one diffractive element or reflective element included in the optical waveguide.

[0093] According to one embodiment, the diffractive element may include an input optical member / output optical member (not shown). For example, the input optical member may mean an input grating region, and the output optical member (not shown) may mean an output grating region. The input grating region may serve as an input terminal that diffracts (or reflects) light output from a light source (e.g., a micro LED) to transmit the light to a transparent member (e.g., a first transparent member (421a), a second transparent member (421b)) of the display area. The output grating region may serve as an outlet that diffracts (or reflects) light transmitted to a transparent member (e.g., a first transparent member, a second transparent member) of the optical waveguide to a user's eye.

[0094] In some embodiments, the reflective element may comprise a total internal reflection (TIR) ​​optical element or waveguide for total internal reflection. For example, total internal reflection may refer to a method of directing light such that light (e.g., a virtual image) entering through an input grating region is reflected substantially 100% from one surface (e.g., a specific surface) of the optical waveguide, thereby transmitting substantially 100% of the light to the output grating region at an angle of incidence.

[0095] In one embodiment, light emitted from the display (421) may be guided along an optical path through an input optical element into a waveguide. Light traveling within the optical waveguide may be guided toward the user's eyes through an output optical element. The display area may be determined based on the light emitted toward the user's eyes.

[0096] According to one embodiment, the electronic device (401) may include a plurality of cameras. For example, the cameras may include a first function camera (e.g., a recognition camera) (415) disposed in the second direction (②) of the main body (410), a second function camera (e.g., a shooting camera) (411, 412) disposed in the second direction (②), a third function camera (e.g., a gaze tracking camera) (428) disposed in the first direction (①), and / or a fourth function camera (e.g., a face recognition camera) (425, 426) disposed in the first direction (①), but may further include cameras for other functions not shown.

[0097] The first function camera (e.g., recognition camera) (415) can be used for the purpose of detecting user movement or recognizing user gestures. The first function camera (415) can support at least one of head tracking, hand detection and hand tracking, and spatial recognition. For example, the first function camera (415) mainly uses a GS (global shutter) camera, which has superior performance compared to an RS (rolling shutter) camera, to detect and track fine movements of hand movements and fingers, and can be configured as a stereo camera including two or more GS cameras for head tracking and spatial recognition. The first function camera (415) can perform spatial recognition for 6DoF and a SLAM (simultaneous localization and mapping) function to recognize information (e.g., location and / or direction) related to the surrounding space through depth shooting.

[0098] A second function camera (e.g., a shooting camera) (411, 412) can be used to capture the outside and generate an image or video corresponding to the outside and transmit it to a processor (e.g., a processor (120) of FIG. 1). The processor can display the image provided from the second function camera (411, 412) on a display (421). The second function camera (411, 412) may be referred to as, for example, HR (high resolution) or PV (photo video) and may include a high-resolution camera. For example, the second function camera (411, 412) may include, but is not limited to, a color camera equipped with functions for obtaining high-quality images, such as an AF (auto focus) function and an optical image stabilizer (OIS). The second function camera (411, 412) may also include a GS camera or an RS camera.

[0099] A third function camera (e.g., a gaze tracking camera) (428) may be positioned on the display (421) (or inside the main body) so that the camera lens faces the user's eyes when the user wears the electronic device (401). The third function camera (428) may be used for the purpose of detecting and tracking (ET: eye tracking) the pupil. The processor may track the movements of the user's left and right eyes in the images received from the third function camera (428) to determine the gaze direction. By tracking the position of the pupil in the images, the processor may ensure that the center of the XR content image displayed in the display area is positioned according to the direction in which the pupil is looking. As an example, a GS camera may be used as the third function camera (428) to detect the pupil and track the movement of the pupil. The third function cameras (428) may be installed respectively for the left and right eyes, and each camera having the same performance and specifications may be used.

[0100] A fourth functional camera (e.g., a camera for facial recognition) (425, 426) can be used to detect and track (FT: face tracking) the user's facial expression when the user wears the electronic device (401).

[0101] According to one embodiment, the electronic device (401) may include a lighting unit (e.g., LED) (not shown) as an auxiliary light for the cameras. As an example, the third function camera (425) may use the lighting included in the display so that the emitted light (e.g., IR LED of infrared wavelength) is directed toward the user's both eyes as an auxiliary light to facilitate gaze detection when tracking eye movements. As another example, the second function cameras (411, 412) may further include an illumination unit (e.g., flash) as an auxiliary light to supplement the ambient brightness when taking external pictures.

[0102] According to one embodiment, a depth sensor (or depth camera) (417) may be used to determine the distance to an object (e.g., an object), such as time of flight (TOF). Time of flight (TOF) is a technology that measures the distance to an object using a signal (e.g., near-infrared, ultrasound, or laser). After a signal is transmitted from a transmitter, a receiver measures the signal, and the distance to the object can be measured based on the flight time of the signal.

[0103] According to one embodiment, the touch sensor (413) may be arranged in the second direction (②) of the main body (410). For example, when a user wears the electronic device (401), the user's eyes may look in the first direction (①) of the main body. The touch sensor (413) may be implemented as a single type or a left / right separated type depending on the shape of the main body (410), but is not limited thereto. For example, when the touch sensor (413) is implemented as a left / right separated type as illustrated in FIG. 4A, when a user wears the electronic device (401), the first touch sensor (413a) may be arranged at the user's left eye position, such as in the fourth direction (④), and the second touch sensor (413b) may be arranged at the user's right eye position, such as in the third direction (③).

[0104] The touch sensor (413) can recognize a touch input using at least one of, for example, a capacitive, pressure-sensitive, infrared, or ultrasonic method. For example, the capacitive touch sensor (413) can recognize a physical touch (or contact) input or a hovering input (or proximity) of an external object. According to various embodiments, the electronic device (401) may also utilize a proximity sensor (not shown) to enable proximity recognition of an external object.

[0105] According to one embodiment, the touch sensor (413) has a two-dimensional surface and can transmit touch data (e.g., touch coordinates) of an external object (e.g., a user's finger) that comes into contact with the touch sensor (413) to a processor (e.g., the processor (120) of FIG. 1). The touch sensor (413) can detect a hovering input for an external object (e.g., a user's finger) that approaches within a first distance from the touch sensor (413), or detect a touch input that touches the touch sensor (413).

[0106] According to one embodiment, the touch sensor (413) may provide two-dimensional information about the point of contact as “touch data” to the processor (120) when an external object touches the touch sensor (413). The touch data may be described as a “touch mode.” The touch sensor (413) may provide hovering data about the time or location of hovering around the touch sensor (413) to the processor (120) when an external object is located within a first distance from the touch sensor (or in proximity, hovering above the touch sensor). The hovering data may be described as a “hovering mode / proximity mode.”

[0107] According to one embodiment, the electronic device (401) may obtain hovering data using at least one of a touch sensor (413), a proximity sensor (not shown), or / and a depth sensor (417) to generate information about a distance, location, or time point between the touch sensor (413) and an external object.

[0108] According to one embodiment, the interior of the main body (410) may include a processor (e.g., processor (120) of FIG. 1) and memory (e.g., memory (130) of FIG. 1).

[0109] Memory can store various instructions that can be executed by the processor. Instructions can include arithmetic and logical operations, data transfer, or control commands such as input / output that can be recognized by the processor. Memory can temporarily or permanently store various data, including volatile memory (e.g., volatile memory (132) of FIG. 1) and non-volatile memory (e.g., non-volatile memory (134) of FIG. 1).

[0110] The processor may be a configuration that is operatively, functionally, and / or electrically connected to each component of the electronic device (401) and can perform calculations or data processing related to control and / or communication of each component. The operations performed by the processor may be stored in memory and, when executed, executed by instructions that cause the processor to operate.

[0111] Hereinafter, the computational and data processing functions that the processor can implement on the electronic device (401) are not limited, but a series of operations related to the XR content service function will be described. The operations of the processor described below can be performed, for example, by executing instructions stored in memory.

[0112] According to one embodiment, the processor may generate a virtual object based on virtual information based on image information. The processor may output a virtual object related to an XR service together with background space information through the display (421). For example, the processor may capture an image related to a real space corresponding to the field of view of a user wearing the electronic device (401) through a second function camera (411, 412) to obtain image information or generate a virtual space for a virtual environment. For example, the processor may control the display (421) to display XR content (hereinafter referred to as an XR content screen) in which at least one virtual object is output so as to be overlapped in an area determined to be a field of view or a field of view (FoV) of the user.

[0113] According to one embodiment, the electronic device (401) may have a form factor for being worn on a user's head. The electronic device (401) may further include a strap and / or a wearable member for being secured on a body part of the user. The electronic device (401) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0114] FIG. 5 illustrates examples of construction of a virtual space, input from a user within the virtual space, and output to the user according to various embodiments.

[0115] An electronic device (e.g., an electronic device (101) of FIG. 1, an electronic device (201) of FIG. 2, an electronic device (301) of FIG. 3, an electronic device (401) of FIGS. 4A and 4B) (501) can obtain spatial information about a physical space where an object is located using a sensor. The spatial information can include a geographical location of the physical space where the object is located, a size of the space, an appearance of the space, a location of a physical object (551) arranged in the space, a size of the physical object (551), an appearance of the physical object (551), and illumination information. The appearance of the space and the physical object (551) can include at least one of a shape, a texture, or a color of the space and the physical object (551). The illumination information can include, for example, information about a light source that emits light acting within the physical space, and can include at least one of an intensity, a direction, or a color of the illumination. The aforementioned sensor can collect information for providing augmented reality. For example, referring to the augmented reality devices illustrated in FIGS. 2, 3, 4a, and 4b, the sensor may include a camera and a depth sensor. However, the present invention is not limited thereto, and the sensor may further include at least one of an infrared sensor, a depth sensor (e.g., a lidar sensor, a radar sensor, or a stereo camera), a gyro sensor, an acceleration sensor, or a geomagnetic sensor.

[0116] The electronic device (501) can collect spatial information over multiple time frames. For example, in each time frame, the electronic device (501) can collect spatial information about a portion of a scene within a sensing range (e.g., field of view (FOV)) of a sensor at the location of the electronic device (501) in physical space. By analyzing the spatial information of multiple time frames, the electronic device (501) can track changes in an object (e.g., movement of position or change of state) over time. The electronic device (501) can also obtain integrated spatial information (e.g., an image that spatially stitches scenes around the electronic device (501) in physical space) for the integrated sensing range of the multiple sensors by comprehensively analyzing the spatial information collected through the multiple sensors.

[0117] An electronic device (501) according to one embodiment can analyze a physical space into three-dimensional information by utilizing various input signals of sensors (e.g., sensing data from an RGB camera, an infrared sensor, a depth sensor, or a stereo camera). For example, the electronic device (501) can analyze at least one of the shape, size, and position of a physical space, and the shape, size, or position of a physical object (551).

[0118] For example, the electronic device (501) can detect an object captured within a scene corresponding to the camera's field of view using the camera's sensing data (e.g., a captured image). The electronic device (501) can determine a label of a physical object (551) from the camera's two-dimensional scene image (e.g., information indicating the classification of the object, including a value indicating a chair, a monitor, or a plant) and an area (e.g., a bounding box) occupied by the physical object (551) within the two-dimensional scene. Accordingly, the electronic device (501) can obtain two-dimensional scene information at a position viewed by the user (590). In addition, the electronic device (501) can also calculate the physical space location of the electronic device (501) based on the camera's sensing data.

[0119] The electronic device (501) can obtain location information of the user (590) and depth information of the actual space in the viewing direction using sensing data (e.g., depth data) of the depth sensor. The depth information is information indicating the distance from the depth sensor to each point and can be expressed in the form of a depth map. The electronic device (501) can analyze the distance of each pixel in the three-dimensional position viewed by the user (590).

[0120] The electronic device (501) can acquire information including a 3D point cloud and mesh using various sensing data. The electronic device (501) can analyze a physical space to acquire a surface, mesh, or 3D coordinate point cluster that constitutes the space. Based on the information acquired as described above, the electronic device (501) can acquire a 3D point cloud representing physical objects.

[0121] The electronic device (501) can analyze a physical space to obtain information including at least one of a three-dimensional position coordinate, a three-dimensional shape, or a three-dimensional size (e.g., a three-dimensional bounding box) of physical objects placed in the physical space.

[0122] Accordingly, the electronic device (501) can obtain information on a physical object detected in a three-dimensional space and semantic segmentation information on the three-dimensional space. The physical object information may include at least one of the position, appearance (e.g., shape, texture, and color), or size of the physical object (551) in the three-dimensional space. The semantic segmentation information is information that semantically divides the three-dimensional space into subspaces, and may include, for example, information indicating that the three-dimensional space is divided into an object and a background, and information indicating that the background is divided into a wall, a floor, and a ceiling. The electronic device (501) can obtain and store three-dimensional information (e.g., spatial information) on the physical object (551) and the physical space as described above. The electronic device (501) can store three-dimensional position information of the user (590) in the space together with the spatial information.

[0123] An electronic device (501) according to one embodiment can construct a virtual space (500) based on the physical location of the electronic device (501) and / or the user (590). The electronic device (501) can generate the virtual space (500) by referring to the spatial information described above. The electronic device (501) can generate a virtual space (500) having a scale substantially identical to that of the physical space based on the spatial information, and can place objects within the generated virtual space (500). The electronic device (501) can provide a complete virtual reality to the user (590) by outputting an image representing the entire physical space. The electronic device (501) can provide mixed reality (MR) or augmented reality (AR) by outputting an image representing a portion of the physical space. However, although the construction of a virtual space (500) based on spatial information acquired through analysis of the aforementioned physical space is described, the electronic device (501) may also construct a virtual space (500) regardless of the physical location of the user (590). In this specification, the virtual space (500) is a space corresponding to augmented reality or virtual reality, and may also be referred to as a metaverse space.

[0124] For example, the electronic device (501) may provide a virtual graphic representation that replaces at least a portion of a physical space. An electronic device (501) based on optical see-through may output a virtual graphic representation by overlaying the virtual graphic representation on a screen area corresponding to at least a portion of the space on a screen display unit. An electronic device (501) based on video see-through may output an image generated by replacing an image area corresponding to at least a portion of a space image corresponding to a physical space rendered based on spatial information with a virtual graphic representation. The electronic device (501) may replace at least a portion of a background in a physical space with a virtual graphic representation, but is not limited thereto. The electronic device (501) may also perform only additional placement of a virtual object (552) within a virtual space (500) based on spatial information without changing the background.

[0125] The electronic device (501) can place and output a virtual object (552) in a virtual space (500). The electronic device (501) can set an operation area of ​​the virtual object (552) in a space occupied by the virtual object (552) (e.g., a volume corresponding to the appearance of the virtual object (552). The operation area may indicate an area where manipulation of the virtual object (552) occurs. In addition, the electronic device (501) can output a physical object (551) by replacing it with the virtual object (552). The virtual object (552) corresponding to the physical object (551) may have a shape identical to or similar to that of the physical object (551). However, the present invention is not limited thereto, and the electronic device (501) may also set only an operation area in a space occupied by the physical object (551) or in a location corresponding to the physical object (551) without outputting a virtual object (552) that replaces the physical object (551). In other words, the electronic device (501) can transmit visual information representing a physical object (551) (e.g., light reflected from the physical object (551) or an image captured of the physical object (551)) to the user (590) without change, and set a manipulation area on the physical object (551). The manipulation area can be set to the same shape and volume as the space occupied by the virtual object (552) or the physical object (551), but is not limited thereto. The electronic device (501) can also set a manipulation area smaller than the space occupied by the virtual object (552) or the space occupied by the physical object (551).

[0126] According to one embodiment, the electronic device (501) may place a virtual object (not shown) representing the user (590) (e.g., an avatar object) within the virtual space (500). When the avatar object is provided in a first-person view, the electronic device (501) may visualize a graphical representation corresponding to a part (e.g., a hand, a torso, or a leg) of the avatar object to the user (590) through the aforementioned display (e.g., an optical see-through display or a video see-through display). However, the present invention is not limited thereto, and when the avatar object is provided in a third-person view, the electronic device (501) may also visualize a graphical representation corresponding to the entire shape (e.g., a back view) of the avatar object to the user (590) through the aforementioned display. The electronic device (501) may provide the user (590) with an experience integrated with the avatar object.

[0127] In addition, the electronic device (501) can provide the user (590) with an experience integrated with the avatar object by using the avatar object of another user who has entered the same virtual space (500). The electronic device (501) can receive feedback information that is the same or similar to feedback information (e.g., information based on at least one of visual, auditory, or tactile senses) provided to another electronic device (501) who has entered the same virtual space (500). For example, when an object is placed in a certain virtual space (500) and multiple users access the virtual space (500), the electronic devices (501) of the multiple users can receive feedback information (e.g., graphical representation, sound signal, or haptic feedback) of the same object placed in the virtual space (500) and provide the feedback information to each user (590).

[0128] The electronic device (501) can detect input to an avatar object of another electronic device (501) and can also receive feedback information from the avatar object of the other electronic device (501). The exchange of input and feedback for each virtual space (500) can be performed by a server (e.g., server (108) of FIG. 1). For example, a server (e.g., a server providing a metaverse space) can transmit input and feedback between an avatar object of a user (590) and an avatar object of another user between users (590). However, the present invention is not limited thereto, and the electronic device (501) can establish direct communication with another electronic device (501) without going through a server to provide input or receive feedback based on an avatar object.

[0129] For example, the electronic device (501) may determine that a physical object (551) corresponding to the selected manipulation area has been selected by the user (590) based on detecting a user input for selecting an manipulation area. The user's (590) input may include at least one of a gesture input using a part of the body (e.g., a hand or an eye), a motion input, an input using a separate virtual reality accessory device, a user's voice input, or a multi-modal input.

[0130] A gesture input is an input corresponding to a gesture identified based on, for example, tracking a body part (510) of a user (590). For example, the gesture input may include an input for pointing or selecting an object. The gesture input may include, for example, at least one of a gesture in which a body part (e.g., a hand) faces an object for a specified period of time or longer, a gesture in which a body part (e.g., a finger, an eye, a head) points to an object, or a gesture in which a body part and an object make spatial contact. A gesture in which an eye points to an object can be identified based on eye tracking. A gesture in which the head points to an object can be identified based on head tracking.

[0131] Tracking of a body part (510) of a user (590) may be performed primarily based on a camera of the electronic device (501), but is not limited thereto. The electronic device (501) may also track the body part (510) based on the cooperation of sensing data of a vision sensor (e.g., image data of a camera and depth data of a depth sensor) and information collected by an accessory device described below (e.g., controller tracking, finger tracking within the controller). Finger tracking may be performed by sensing the distance or contact between an individual finger and the controller based on a sensor built into the controller (e.g., an infrared sensor).

[0132] The virtual reality accessory device may include a ride-on device, a wearable device, a controller device (520), or other sensor-based device. The ride-on device is a device that a user (590) rides on and operates, and may include, for example, at least one of a treadmill-type device or a chair-type device. The wearable device is a manipulation device that is worn on at least a part of the user's (590) body, and may include, for example, at least one of a full-body and half-body suit-type controller, a vest-type controller, a shoe-type controller, a bag-type controller, a glove-type controller (e.g., a haptic glove), or a face mask-type controller. The controller device (520) may include, for example, an input device (e.g., a stick-type controller or a gun) that is operated by a hand, foot, toe, or other body part (510).

[0133] The electronic device (501) may establish direct communication with the accessory device to track at least one of the location and movement of the accessory device, but is not limited thereto. The electronic device (501) may also communicate with the accessory device via a base station for virtual reality.

[0134] For example, the electronic device (501) may determine that the virtual object (552) has been selected based on detecting an action of gazing at the virtual object (552) for a specified period of time or longer through the aforementioned eye gaze tracking technology. For example, the electronic device (501) may recognize a gesture indicating the virtual object (552) through hand tracking technology. The electronic device (501) may determine that the virtual object (552) has been selected based on whether the direction pointed by the tracked hand indicates the virtual object (552) for a specified period of time or longer, or whether the hand of the user (590) contacts or enters an area occupied by the virtual object (552) within the virtual space (500).

[0135] The user's voice input is an input corresponding to the user's voice acquired by the electronic device (501), and may include, for example, voice data sensed by an input module (e.g., a microphone) of the electronic device (501) (e.g., the input module (150) of FIG. 1) or received from an external electronic device of the electronic device (501). The electronic device (501) may determine that a physical object (551) or a virtual object (552) has been selected by analyzing the user's voice input. For example, the electronic device (501) may determine that at least one of the physical object (551) or the virtual object (552) corresponding to the detected keyword has been selected based on detecting a keyword indicating at least one of the physical object (551) or the virtual object (552) from the user's voice input.

[0136] The electronic device (501) may provide feedback as described below in response to the user (590) input described above.

[0137] The feedback may include visual feedback, auditory feedback, tactile feedback, olfactory feedback, and / or gustatory feedback. The feedback may be rendered by the server (108), the electronic device (101), and / or the external electronic device (102), as described above in FIG. 1.

[0138] Visual feedback may include outputting an image through a display (e.g., a transparent display or an opaque display) of the electronic device (501).

[0139] Auditory feedback may include outputting sound through a speaker of the electronic device (501).

[0140] Tactile feedback may include force feedback that simulates weight, shape, texture, dimension, and dynamics. For example, a haptic glove may include haptic elements (e.g., electrical muscles) that can simulate the sensation of touch by tensing and relaxing the body of a user (590). The haptic elements within the haptic glove may act as tendons. The haptic glove may provide haptic feedback to the entire hand of the user (590). The electronic device (501) may provide feedback indicating the shape, size, and stiffness of an object through the haptic glove. For example, the haptic glove may generate forces that mimic the shape, size, and stiffness of an object. The exoskeleton of the haptic glove (or suit-like device) may include sensors and finger motion measurement devices, and may transmit tactile information to the body by transmitting a force (e.g., an electromagnetic, direct current (DC) motor, or pneumatic-based force) that pulls a cable to the fingers of the user (590). Hardware providing tactile feedback may include sensors, actuators, power sources, and wireless transmission circuitry. Haptic gloves may operate by inflating and deflating inflatable air bladders on the glove surface.

[0141] The electronic device (501) may provide feedback to the user (590) based on the selection of an object within the virtual space (500). For example, the electronic device (501) may output a graphical representation indicating the selected object (e.g., an expression highlighting the selected object) through the display. For example, the electronic device (501) may output a sound (e.g., a voice) guiding the selected object through the speaker. For example, the electronic device (501) may provide the user (590) with a haptic movement that simulates the tactile sensation of the corresponding object by transmitting an electrical signal to a haptic-enabled accessory device (e.g., a haptic glove).

[0142] FIGS. 6A and 6B are diagrams illustrating examples of operations in which an electronic device according to various embodiments outputs first feedback and second feedback based on an object detected in a display area.

[0143] An electronic device (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIG. 4, electronic device (501) of FIG. 5) can be worn by a user.

[0144] An electronic device may display a display area (e.g., a first display area (600a) of FIG. 6A, a second display area (600b) of FIG. 6B). The display area may refer to, for example, an area displayed through a display of the electronic device. According to one embodiment, the display area may include an area determined to be a user's field of view (FOV). The electronic device may determine an area determined to be a user's field of view from among images acquired through a photographing camera (e.g., a third camera (245) of FIG. 2, a second functional camera (411, 412) of FIGS. 4A and 4B) as a display area, and display the determined display area through a display.

[0145] An electronic device can obtain input from a user. The input can include at least one of voice input, gesture input, motion input, multi-modal input, or input using a separate virtual reality accessory device. For example, in FIGS. 6A and 6B , the electronic device can obtain the voice input "What's the weather today?"

[0146] The electronic device can generate first feedback (661a, 661b) based on the user's input. As described in more detail later in FIG. 7, the electronic device can analyze the user's voice input to determine the user's intent and generate and / or output feedback based on the intent.

[0147] The first feedback (661a, 661b) may include feedback generated based on a user's input. According to one embodiment, the first feedback (661a, 661b) may be generated independently of an object detected in the display area, the surrounding physical space, and / or the virtual space. The first feedback (661a, 661b) may be generated identically even if the object detected in the first display area (600a) is different from the object detected in the second display area (600b). For example, the first feedback (661a, 661b) may be generated identically in each of the first display area (600a) where an air purifier (640a) is detected and the second display area (600b) where an air conditioner (650b) is detected. For example, in FIGS. 6a and 6b, the electronic device may generate the first feedback (661a, 661b) "Today's weather will be a high of 30 degrees, a low of 20 degrees, and it will be cloudy all day."

[0148] An electronic device can detect objects in at least one of a display area, a surrounding physical space, or a virtual space. The objects can include physical objects and / or virtual objects. For example, in FIG. 6A, the electronic device can detect a stand (610a), a window (620a), a monitor (630a), and an air purifier (640a) in a first display area (600a). In FIG. 6B, the electronic device can detect a stand (610b), a monitor (630b), and an air conditioner (650b) in a second display area (600b).

[0149] For example, an electronic device can detect an object in a display area by analyzing an image of the display area acquired through a vision sensor (e.g., a vision sensor including a camera for capturing images).

[0150] For example, an electronic device can detect an object in a surrounding physical space by analyzing an image captured of the surrounding physical space. The image captured of the surrounding physical space may include an image of a different area from the display area. For example, the surrounding physical space may be acquired by a rearview camera that captures the user's rearview. For example, the electronic device can detect an object that has established communication with the electronic device. The electronic device can detect the object based on the establishment of communication with the object, even if at least a portion of the object detected in the surrounding physical space is not visible in the display area (or the image captured of the surrounding physical space). For example, the electronic device can identify the surrounding physical space based on an image captured of the surrounding physical space and detect objects registered in the identified space. For example, the electronic device can store object registration data for each space, including information on registered objects corresponding to the corresponding space, in each of a first space (e.g., a kitchen), a second space (e.g., a living room), and a third space (e.g., a closet). An electronic device can detect objects (e.g., a refrigerator, a dishwasher, an oven, a microwave, an air fryer) registered in the identified space based on identifying the surrounding physical space as one of a plurality of spaces (e.g., a kitchen). The electronic device can detect both objects displayed in a display area and objects not displayed in the display area among the objects registered in the surrounding physical space. The electronic device can detect objects registered in the surrounding physical space even if the objects registered in the surrounding physical space are not displayed in the display area of ​​the electronic device.

[0151] For example, an electronic device can detect an object registered in a virtual space based on the virtual space it has entered. For example, the electronic device can detect a virtual object contained in the virtual space it has entered.

[0152] The electronic device may generate second feedback (662a-1, 662a-2, 662b). The second feedback (662a-1, 662a-2, 662b) may follow the first feedback (661a, 661b). The second feedback (662a-1, 662a-2, 662b) may be generated based on an object detected in at least one of the display area, the surrounding physical space, or the virtual space. For example, the second feedback (662a-1, 662a-2, 662b) may include at least one of feedback suggesting a control command for the detected object to the user, feedback displaying an interface for obtaining a control command for the detected object, feedback suggesting a user action for the detected object, feedback providing a preview of a control command and / or action for the detected object, or feedback providing additional information about the object.

[0153] According to one embodiment, the second feedback (662a-1, 662a-2, 662b) may be generated differently when an object detected in the first display area (600a) is different from an object detected in the second display area (600b). For example, the second feedback (662a-1, 662a-2, 662b) may be generated differently in the first display area (600a) where an air purifier is detected and in the second display area (600b) where an air conditioner is detected.

[0154] For example, in FIG. 6a, based on detecting an air purifier (640a) in the first display area (600a), the electronic device can generate second feedback (662a-1) “The fine dust is bad today. How about turning on the air purifier that is turned off?”. The electronic device can generate second feedback (662a-2) “The window is open even though the fine dust is bad today. Try closing the window.” In FIG. 6b, based on detecting an air conditioner in the second display area (600b), the electronic device can generate second feedback (662b) “The weather is hotter than yesterday. How about turning on the air conditioner?”

[0155] The electronic device can output first feedback (661a, 661b) and second feedback (662a-1, 662a-2, 662b). The electronic device can provide customized feedback based on objects visible to the user and / or located around the user by providing second feedback (662a-1, 662a-2, 662b) (e.g., additional feedback) generated based on the detected object, together with the first feedback (661a, 661b) (e.g., basic feedback) based on the user's input.

[0156] FIG. 7 is a drawing illustrating an example of a configuration of an electronic device according to various embodiments.

[0157] An electronic device (701) according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIGS. 4A and 4B, electronic device (501) of FIG. 5) can analyze a user's input and generate first feedback based on the user's input and second feedback based on a detected object.

[0158] An electronic device (701) according to one embodiment may include at least one of a front end (710), a natural language platform (720), a capsule database (730), a display region analysis module (740), an object detection module (750), a context control module (760), a relevance analysis module (770), or a feedback generation module (780).

[0159] The front end (710) can receive input (e.g., voice input) from a user. The front end (710) can transmit a response corresponding to the input.

[0160] According to one embodiment, the natural language platform (720) may include at least one of an automatic speech recognition module (ASR module) (721), a natural language understanding module (NLU module) (722), a planner module (723), a natural language generator module (NLG module) (724), or a text to speech module (TTS module) (725).

[0161] An automatic speech recognition module (721) of one embodiment can convert a user's voice input into text data. A natural language understanding module (722) of one embodiment can use the text data of the voice input to determine the user's intent. For example, the natural language understanding module (722) can perform syntactic analysis or semantic analysis to determine the user's intent. The natural language understanding module (722) of one embodiment can use linguistic features (e.g., grammatical elements) of morphemes or phrases to determine the meaning of words extracted from the voice input, and can match the meaning of the determined words to the intent to determine the user's intent.

[0162] In one embodiment, the planner module (723) can generate a plan using the intent and parameters determined by the natural language understanding module (722). According to one embodiment, the planner module (723) can determine a plurality of domains necessary to perform a task based on the determined intent. The planner module (723) can determine a plurality of operations included in each of the plurality of domains determined based on the intent. According to one embodiment, the planner module (723) can determine parameters necessary to execute the determined plurality of operations or result values ​​output by the execution of the plurality of operations. The parameters and result values ​​can be defined as concepts of a specified format (or class). Accordingly, the plan can include a plurality of operations and a plurality of concepts determined based on the user's intent. The planner module (723) can determine the relationship between the plurality of operations and the plurality of concepts in a stepwise (or hierarchical) manner. For example, the planner module (723) can determine the execution order of the plurality of operations determined based on the user's intent based on the plurality of concepts. In other words, the planner module (723) can determine the execution order of the multiple operations based on the parameters required for executing the multiple operations and the results output by executing the multiple operations. Accordingly, the planner module (723) can generate a plan including association information (e.g., ontology) between the multiple operations and the multiple concepts. The planner module (723) can generate the plan using information stored in the capsule database (730) in which a set of relationships between concepts and operations is stored.

[0163] The natural language generation module (724) of one embodiment can convert specified information into text format. The information converted into text format may be in the form of natural language utterances. The text-to-speech conversion module (725) of one embodiment can convert information in text format into information in speech format.

[0164] The capsule database (730) can store information about the relationships between multiple concepts and actions corresponding to multiple domains. According to one embodiment, a capsule can include multiple action objects (or action information) and concept objects (or concept information) included in a plan. According to one embodiment, the capsule database (730) can store multiple capsules in the form of a concept action network (CAN). According to one embodiment, the multiple capsules can be stored in a function registry included in the capsule database (730).

[0165] The capsule database (730) may include a strategy registry that stores strategy information necessary for determining a plan corresponding to an input. The strategy information may include reference information for determining a single plan when there are multiple plans corresponding to an input. According to one embodiment, the capsule database (730) may include a follow-up registry that stores information on follow-up actions for suggesting follow-up actions to a user in a given situation. The follow-up actions may include, for example, follow-up utterances. According to one embodiment, the capsule database (730) may include a layout registry that stores layout information of information output through the electronic device (701). According to one embodiment, the capsule database (730) may include a vocabulary registry that stores vocabulary information included in the capsule information. According to one embodiment, the capsule database (730) may include a dialog registry that stores information on a dialogue (or interaction) with a user. The capsule database (730) can update stored objects through a developer tool. The developer tool may include, for example, a function editor for updating action objects or concept objects. The developer tool may include a vocabulary editor for updating vocabulary. The developer tool may include a strategy editor for creating and registering strategies that determine plans. The developer tool may include a dialog editor for creating conversations with users.Developer tools may include a follow-up editor that enables follow-up goals and allows editing of follow-up utterances that provide hints. Follow-up goals may be determined based on the currently set goal, user preferences, or environmental conditions.

[0166] The display area analysis module (740) according to one embodiment may determine, for example, a display area and / or a viewing area. The viewing area may refer to a portion of an area (or display area) determined to be within the user's field of vision that is gazed upon by the user. According to one embodiment, the viewing area may be determined based on a gaze point corresponding to the user's gaze. For example, the viewing area may be determined as a circular area having a specified radius based on the gaze point corresponding to the user's gaze. For example, the viewing area may be an area inside an oval, a square, or a closed curve. However, the viewing area is not limited to being determined based on the user's gaze. According to one embodiment, the electronic device (701) may determine a portion of an area designated based on the display area as the user's viewing area.

[0167] For example, based on the electronic device (701) acquiring a user's input, the display area analysis module (740) can determine a display area and / or a viewing area corresponding to the acquisition time of the input (or a time interval including the acquisition time of the input).

[0168] An object detection module (750) according to one embodiment can detect an object in at least one of a display area, a surrounding physical space, or a virtual space. According to one embodiment, the object detection module (750) can detect an object in real time in at least one of the display area, a surrounding physical space, or a virtual space independently of a user's input. According to one embodiment, the object detection module (750) can detect an object in real time in the display area independently of a user's input, and detect an object in at least one of the surrounding physical space or a virtual space based on the user's input.

[0169] The context control module (760) according to one embodiment may determine whether to maintain and / or terminate a context corresponding to a user input. The context corresponding to the input may include a time interval from a first time point at which the user input is obtained to a subsequent second time point. The context corresponding to the input may refer to, for example, a time interval during which the electronic device (701) provides feedback (e.g., first feedback, second feedback) generated based on the input. The context control module (760) according to one embodiment may determine an end time point of the context. The context control module (760) may determine whether to maintain and / or terminate the context based on at least one of a subsequent input from the user, a time length from the first time point, a space in which the user is located, or a detected object.

[0170] The electronic device (701) can detect an object while a context corresponding to the input is maintained from the time when the user's input is acquired, and can stop detecting the object based on the termination of the context. Consequently, the electronic device (701) can generate and / or output a second feedback for the object detected while the context is maintained. The electronic device (701) may not generate and / or output the second feedback for the object detected after the context ends. The operation of stopping detection of an object due to the termination of the context is described in more detail below with reference to FIG. 8.

[0171] According to one embodiment, the relevance analysis module (770) can analyze the relevance between a user input and a detected object. According to one embodiment, the electronic device (701) can detect candidate objects and generate second feedback for objects among the detected candidate objects that have a high relevance to the user input. For example, the relevance analysis module (770) can calculate a relevance score for each of the user input and the detected candidate objects.

[0172] A feedback generation module (780) according to one embodiment can generate feedback including first feedback and second feedback. The first feedback can be generated based on a user's input. The second feedback can be generated based on the input and the detected object.

[0173] According to one embodiment, the feedback generation module (780) may generate the second feedback further based on the relevance received from the relevance analysis module (770). For example, the feedback generation module (780) may generate the second feedback for the object based on the input, each object, and the relevance score between the input and the object. The feedback generation module (780) may generate the second feedback for the object based on the relevance score exceeding a threshold score, and may not generate the second feedback for the object based on the relevance score being below the threshold score. Consequently, the electronic device (701) may generate and / or output the second feedback for the candidate object whose relevance score exceeds the threshold. The electronic device (701) may not generate and / or output the second feedback for the candidate object whose relevance score is below the threshold.

[0174] FIG. 8 is a flowchart illustrating an example of an operation of an electronic device outputting feedback according to various embodiments.

[0175] In the embodiments below, the operations may be performed sequentially as described, but these operations are not necessarily performed sequentially. For example, the order of the operations may be changed, and / or at least two operations may be performed in parallel.

[0176] According to one embodiment, operations (810) to (830) may be understood to be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIGS. 4A and 4B, electronic device (501) of FIG. 5, electronic device (701) of FIG. 7).

[0177] In an example embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIGS. 4A and 4B, electronic device (501) of FIG. 5, electronic device (701) of FIG. 7) may acquire a user's input and generate and output a first feedback for the user's input and a second feedback for the input and a detected object.

[0178] In operation (810), according to one embodiment, the electronic device may generate a first feedback for the input based on obtaining the user's input.

[0179] The first feedback may refer to feedback generated based on, for example, a user's input. The first feedback may be independent of the object detected in at least one of the display area, the surrounding physical space, or the virtual space. For example, if the user's input is the same even if the detected object is different, the first feedback may be the same.

[0180] In operation (820), according to one embodiment, the electronic device may generate a second feedback subsequent to the first feedback based on an object detected in at least one of the display area, the surrounding physical space, or the virtual space.

[0181] Secondary feedback may refer to feedback generated based on, for example, user input and detected objects. The second feedback may depend on objects detected in at least one of the display area, the surrounding physical space, or the virtual space.

[0182] In one embodiment, the first and second feedbacks may each correspond to different sentences. However, this is not limited to the first and second feedbacks, which may correspond to different parts of the same sentence. For example, the first feedback may be generated from the first part of a sentence, and the second feedback may be generated from the second part of the sentence.

[0183] According to one embodiment, an electronic device can detect an object associated with an input. The object may include at least one of a physical object or a virtual object. For example, the electronic device can detect a candidate object in at least one of a display area, a surrounding physical space, or a virtual space. The electronic device can calculate a relevance score between the input and each candidate object. The electronic device can determine the object based on the calculated relevance score.

[0184] A relevance score may indicate a relevance between an input and a corresponding candidate object. According to one embodiment, a relevance analysis module of an electronic device (e.g., the relevance analysis module 770 of FIG. 7) may calculate a relevance score between a user's input and a candidate object by applying a relevance analysis model to input data. The input data may include information about the user's input and information about the detected candidate object. The information about the user's input may include at least one of an intent, a domain, an action, a concept, or a parameter determined based on the user's input. The information about the candidate object may include at least one of an identifier of the candidate object or a category of the candidate object.

[0185] An electronic device may determine a candidate object as an object (e.g., an object related to the input) based on whether the candidate object's relevance score exceeds a threshold score. The electronic device may not determine a candidate object as an object based on whether the candidate object's relevance score is below the threshold score. For example, the electronic device may determine a candidate object as unrelated to the input based on whether the candidate object's relevance score is below the threshold score.

[0186] According to one embodiment, the electronic device may calculate a relevance score based on the user's line of sight. For example, the electronic device may apply a bonus point or a weight to the relevance score for a candidate object based on the viewing area. For example, if at least a portion of a first candidate object is included in the viewing area and not all of an area corresponding to a second candidate object is included in the viewing area, the electronic device may apply a bonus point or a weight of a greater value to the relevance score of the first candidate object than the relevance score of the second candidate object. In another example, if a portion of the first candidate object included in the viewing area is larger than a portion of the second candidate object included in the viewing area, the electronic device may apply a bonus point or a weight of a greater value to the relevance score of the first candidate object than the relevance score of the second candidate object.

[0187] For another example, the electronic device may apply a bonus or weight to the relevance score of a candidate object based on the distance from the gaze point to the candidate object. For example, if the first distance between the gaze point and the first candidate object (or the reference point of the first candidate object) is shorter than the second distance between the gaze point and the second candidate object (or the reference point of the second candidate object), the electronic device may apply a bonus or weight of a greater value to the relevance score of the first candidate object than to the relevance score of the second candidate object.

[0188] An electronic device according to one embodiment may detect an object based on a context corresponding to an input. For example, the electronic device may detect an object while the context corresponding to the input is maintained from a time point at which the input is acquired. The electronic device may stop detecting the object based on the termination of the context corresponding to the input. The maintenance and / or termination of the context corresponding to the input may be determined by a context control module of the electronic device (e.g., the context control module 760 of FIG. 7). The context corresponding to the input may include a period of time for providing feedback on the input after the time point at which the input is acquired. The maintenance and / or termination of the context corresponding to the input may be determined based on at least one of a user's motion, a subsequent input of the user, a length of time from a first time point, a space in which the user is located, or a detected object.

[0189] According to one embodiment, an electronic device may determine whether to maintain and / or terminate a context based on the space in which the user is located. For example, the electronic device may determine the space in which the user is located at the time of acquiring an input. The electronic device may determine that the context corresponding to the input is maintained while the user is located in the determined space. The electronic device may detect an object in the display area while the user is located in the determined space. The electronic device may determine that the context corresponding to the input is terminated based on the user leaving the determined space. The electronic device may stop detecting the object based on the user leaving the determined space.

[0190] According to one embodiment, an electronic device may determine whether to maintain and / or terminate a context based on a user's movement. For example, the electronic device may detect a user's motion corresponding to an input. The user's motion corresponding to the input may, for example, include a motion detected during a time interval including the time point at which the input is acquired. The electronic device may determine that the context corresponding to the input is maintained based on detecting the user's motion corresponding to the input. The electronic device may detect an object in the display area while detecting the motion. The electronic device may determine that the context corresponding to the input is terminated based on detecting the end of the user's motion corresponding to the input. The electronic device may stop detecting the object based on detecting the end of the motion.

[0191] For example, if a user utters "How's the weather today?" while walking, the electronic device can acquire the user's input while detecting the user's steps. The electronic device can determine that a context corresponding to the input is maintained while the steps are being taken. The electronic device can detect an object while detecting the steps. The electronic device can determine that the context corresponding to the input is maintained based on detecting the end of the steps. The electronic device can stop detecting the object based on detecting the end of the steps.

[0192] According to one embodiment, an electronic device may determine whether to maintain and / or terminate a context based on a detected object. The electronic device may detect an object associated with an input. The electronic device may determine that the context corresponding to the input is maintained based on detecting the object associated with the input within a threshold time from the time of acquiring the input. The electronic device may fail to detect an object associated with the input if the electronic device fails to detect the object in at least one of the display area, the surrounding physical space, or the virtual space and / or determines that all of the detected candidate objects are not associated with the input. The electronic device may determine that the context corresponding to the input is discontinued based on failing to detect an object associated with the input within a threshold time from the time of acquiring the input. The electronic device may discontinue detection of an object based on failing to detect an object associated with the input within a threshold time from the time of acquiring the input.

[0193] For example, when an electronic device receives a user's voice input "How is the weather today?", the electronic device may detect an air conditioner, an air purifier, a window, a flower pot, and a pencil as candidate objects in at least one of a display area, a surrounding physical space, and a virtual space. If the electronic device determines that the air conditioner, the air purifier, and the window among the candidate objects are related to the input, the electronic device may determine that the context corresponding to the input is maintained. If at least one of the display area, the surrounding physical space, or the virtual space changes due to a change (e.g., a user's movement, a change in the surrounding physical space, a change in the virtual space), the electronic device may re-detect the candidate objects (e.g., a tree, a pencil, and paper). If the electronic device determines that none of the detected candidate objects are related to the input, the electronic device may determine that the context corresponding to the input has ended. The electronic device may stop detecting objects based on the termination of the context.

[0194] According to one embodiment, an electronic device may determine whether to maintain and / or terminate a context based on a subsequent input from a user. For example, the electronic device may output first feedback based on the input and second feedback based on the detected object. The electronic device may obtain a subsequent input from the user following the input. The electronic device may determine whether the subsequent input is related to an object used to generate the second feedback. The electronic device may determine that the context corresponding to the input is maintained based on whether the subsequent input is related to an object used to generate the second feedback. The electronic device may determine that the context corresponding to the input is terminated based on whether the subsequent input is not related to an object used to generate the second feedback. For example, the electronic device may determine that the context corresponding to the input is terminated based on obtaining another input for an object other than the object. The electronic device may discontinue detection of the object based on obtaining another input for an object other than the object.

[0195] According to one embodiment, an electronic device may determine whether to maintain and / or terminate a context based on a user input. For example, the electronic device may determine whether to maintain and / or terminate a context based on an object corresponding to the user input. If the input includes a control command for an object, the electronic device may determine which object corresponds to the input. For example, the electronic device may obtain the user input "Turn on the air conditioner" while detecting an air conditioner in the display area. The electronic device may determine that the context corresponding to the input is maintained while detecting the air conditioner in the display area. The electronic device may determine that the context corresponding to the input is terminated based on detecting that the air conditioner has moved out of the display area.

[0196] For example, an electronic device may determine a threshold time for maintaining context based on a user's input. For example, if a pronoun is detected in the user's input, the electronic device may determine the threshold time for maintaining context as a first time period. If a pronoun is not detected in the user's input, the electronic device may determine the threshold time for maintaining context as a second time period that is longer than the first time period.

[0197] In operation (830), according to one embodiment, the electronic device can output first feedback and second feedback.

[0198] The feedback (e.g., first feedback, second feedback) may include at least one of visual feedback and auditory feedback. According to one embodiment, the electronic device may output the first feedback independently of the object. The electronic device may output the second feedback for an object related to the second feedback. For example, the electronic device may display the second feedback (e.g., visual feedback) generated based on the object in an area corresponding to each object. For example, the electronic device may play, for each object, auditory feedback (e.g., voice output) generated based on the object.

[0199] In one embodiment, the electronic device may output a second feedback based on a subsequent user input. For example, the electronic device may output a graphical representation of the object for which the second feedback was generated. The electronic device may generate the second feedback based on the object corresponding to the graphical representation based on the subsequent user input selecting the graphical representation.

[0200] For example, when the electronic device generates second feedback based on a first object, second feedback for a second object, and second feedback for a third object, the electronic device may output graphical representations for each of the first object, the second object, and the third object. The electronic device may output the second feedback corresponding to the selected graphical representation based on obtaining a subsequent input from the user selecting the graphical representation. Instead of directly outputting the generated second feedbacks, the electronic device may notify the user of the generation of the second feedback through the graphical representation and provide an interface through which the user can select an object for which he or she wants to output the second feedback, thereby preventing the user experience from being degraded or reduced due to the output of an excessive number of second feedbacks.

[0201] FIG. 9 is a diagram illustrating an operation of analyzing additional input when an electronic device obtains additional input according to various embodiments.

[0202] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIGS. 4A and 4B, electronic device (501) of FIG. 5, electronic device (701) of FIG. 7) may obtain additional input from a user and analyze the additional input based on at least one of the input, the first feedback, or the second feedback.

[0203] In FIG. 9, the electronic device can detect a first object (921), a second object (922), and a third object (923) in a display area (910). The electronic device can obtain an input “How is the weather today?” from a user.

[0204] The electronic device may output, based on the user's input, the first feedback (941) "Today's weather will be a high of 30 degrees, a low of 20 degrees, and cloudy all day."

[0205] The electronic device can output second feedback (e.g., first auditory feedback (942-1), second auditory feedback (942-2), and / or third auditory feedback (942-3)) based on each object. For example, the electronic device can play the first auditory feedback (942-1) “It’s hot, so why don’t you try turning on the air conditioner?” based on the first object (921) (e.g., an air conditioner) and / or display the first graphical representation (931) for turning on the first object (921). The electronic device can play the second auditory feedback (942-2) “The fine dust is bad, so why don’t you try turning on the air purifier?” based on the second object (922) (e.g., an air purifier) ​​and / or display the second graphical representation (932) for turning on the second object (922). The electronic device may play a third auditory feedback (942-3) based on a third object (923) (e.g., a light) such as "It's cloudy, why don't you turn on the light?" and / or display a third graphical representation (933) for a command to turn on the third object (923).

[0206] An electronic device may obtain additional input from a user. For example, the electronic device may obtain additional input from the user while maintaining the context corresponding to the input. The electronic device may analyze the additional input using at least one of the input, the first feedback (941), or the second feedback. By analyzing the additional input, the electronic device may determine the user's intention in response to the additional input.

[0207] For example, the electronic device may obtain an additional input “air conditioner” from the user. The electronic device may analyze the user’s intention based on the additional input “air conditioner” using at least one of the input “How is the weather today?”, the first feedback (941) (e.g., “Today’s weather will be a high of 30 degrees, a low of 20 degrees, and it will be cloudy all day”), or the second feedback (942) (e.g., “The weather is hot, so why don’t you try turning on the air conditioner?”, “The fine dust is bad, so why don’t you try turning on the air purifier?”, “The weather is cloudy, so why don’t you try turning on the lights?”, the first graphical representation (931), the second graphical representation (932), and the third graphical representation (933)). The electronic device may determine the user’s intention according to the additional input as turning on the air conditioner based on the second feedback “The weather is hot, so why don’t you try turning on the air conditioner?” and / or the first graphical representation (931). The electronic device can transmit a turn-on command to the air conditioner. The air conditioner can start operating based on receiving the turn-on command from the electronic device.

[0208] The electronic device may obtain additional input from the user after the context corresponding to the input has ended. The electronic device may exclude at least one of the input, the first feedback (941), and the second feedback from analyzing the additional input obtained after the context has ended. For example, if the electronic device obtains the additional input after the context has ended, the electronic device may not use the input, the first feedback (941), and the second feedback (942) to analyze the additional input.

[0209] For example, the electronic device may not use the input “How is the weather today?”, the first feedback (941) (e.g., “Today’s weather will be a high of 30 degrees, a low of 20 degrees, and it will be cloudy all day”), and the second feedback (e.g., “The weather is hot, so why don’t you try turning on the air conditioner?”, “The fine dust is bad, so why don’t you try turning on the air purifier?”, “The weather is cloudy, so why don’t you try turning on the lights?”, the first graphical representation (931), the second graphical representation (932), the third graphical representation (933)) to analyze the additional input based on acquiring the additional input “air conditioner” after the context has ended. Consequently, the electronic device may not determine the user’s intention as turning on the air conditioner based on the additional input “air conditioner”, unlike the case where the additional input is acquired while the context corresponding to the input is maintained.

[0210] FIGS. 10A and 10B are diagrams illustrating examples of operations in which an electronic device outputs an interface to an object according to various embodiments.

[0211] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIGS. 4A and 4B, electronic device (501) of FIG. 5, electronic device (701) of FIG. 7) can output an interface for a detected object.

[0212] In FIG. 10a, the electronic device can detect an object (e.g., an air conditioner (1020a)) in the display area (1010a). The electronic device can obtain an input from the user, “How is the weather today?”

[0213] The electronic device can output the first feedback (1011a) “Today’s weather is a high of 30 degrees and a low of 20 degrees” based on the user’s input.

[0214] The electronic device may output second feedback (1012a) based on an object. For example, the electronic device may output second feedback (1012a) based on an object (e.g., an air conditioner (1020a)): "Today's humidity is 95%. How about operating the air conditioner in dehumidification mode?"

[0215] According to one embodiment, an electronic device may generate and / or output an interface (1030a) for controlling an object. The interface (1030a) may include an interface (1030a) for obtaining a control input for an operation related to an input among operations of the object. For example, the electronic device may output an interface (1030a) for an operation related to an input among operations of the object based on the context corresponding to the input being maintained.

[0216] In one embodiment, an electronic device may determine an operation of an object based on at least one of an input, a first feedback, or a second feedback among candidate operations supported by the object. For example, the electronic device may determine an operation related to the input among the candidate operations. The electronic device may output an interface (1030a) for obtaining additional input for the determined operation. The electronic device may exclude information regarding operations different from the determined operation among the candidate operations from the interface (1030a).

[0217] For example, in FIG. 10a, candidate operations that can be supported by the air conditioner (1020a) may include, for example, one of a work start operation of a dehumidifying mode, a work start operation of a cooling mode, a work end operation, a desired temperature adjustment operation, a wind direction adjustment operation, a mode change operation between the dehumidifying mode and the cooling mode, or a reservation setting operation. The electronic device may determine the work start operation of the dehumidifying mode among the candidate operations for the air conditioner (1020a) as an operation associated with the input, based on at least one of an input (e.g., "How's the weather today?") or a second feedback (1012a) (e.g., "The humidity today is 95%. How about operating the air conditioner (1020a) in the dehumidifying mode?"). The electronic device may display an interface (1030a) for the work start operation of the dehumidifying mode in an area corresponding to the air conditioner (1020a). The electronic device may exclude information about other operations (e.g., operation start operation of cooling mode, operation end operation, desired temperature adjustment operation, wind direction adjustment operation, mode change operation between dehumidification mode and cooling mode, reservation setting operation) from the interface (1030a).

[0218] An electronic device according to one embodiment may output a concise interface (1030a) by displaying an interface (1030a) for a determined action (e.g., an action related to an input) and excluding other actions of an object (e.g., an action not related to an input) from the interface (1030a).

[0219] In this disclosure, the second feedback (1012a) and the interface (1030a) are primarily described separately, but are not limited thereto. The second feedback (1012a) may include an interface (1030a) for actions related to object input.

[0220] According to one embodiment, the electronic device may output an interface (1030b) for controlling an object independently of an input based on the termination of a context corresponding to the input. For example, in FIG. 10b, the electronic device may obtain an input "How's the weather today?" and determine that the context corresponding to the input "How's the weather today?" has terminated. The electronic device may detect an air conditioner (1020b) in a display area. The electronic device may display an interface (1030b) for controlling the air conditioner (1020b) independently of the input, the first feedback, and the second feedback. After the context has terminated, an interface (1030b) for both input-related and input-unrelated operations may be output.

[0221] FIG. 11 is a diagram illustrating an example of an operation of an electronic device according to various embodiments to output a preview of an object's operation.

[0222] An electronic device according to one embodiment (e.g., electronic device (101) of FIG. 1, electronic device (201) of FIG. 2, electronic device (301) of FIG. 3, electronic device (401) of FIGS. 4A and 4B, electronic device (501) of FIG. 5, electronic device (701) of FIG. 7) can output a preview (1130) of the motion of a detected object.

[0223] For example, in FIG. 11, the electronic device can detect an object (e.g., a light (1120)) in the display area (1110). The electronic device can obtain an input from the user, “How is the weather today?”

[0224] The electronic device can output the first feedback (1111) “Today’s weather is a high of 30 degrees and a low of 20 degrees” based on the user’s input.

[0225] The electronic device can output second feedback based on the object (e.g., auditory feedback (1112). The electronic device can play auditory feedback (1112) based on the light (1120) such as "It's cloudy, why don't you turn on the light?" and / or display a graphical representation (1113) for a command to turn on the light (1120).

[0226] According to one embodiment, the electronic device may determine an operation of an object based on at least one of an input, a first feedback (1111), or a second feedback among candidate operations supported by the object. For example, the electronic device may determine an operation related to the input among the candidate operations. The electronic device may output a preview (1130) of the object performing the determined operation.

[0227] For example, in FIG. 11, the electronic device may determine the turn-on operation of the light (1120) as an operation associated with the input, based on the input “How is the weather today?” or the second feedback (e.g., auditory feedback (1112) “It’s cloudy, so how about turning on the light (1120)?”, graphical representation (1113) for a command to turn on the light (1120)) among candidate operations (e.g., turn-on operation, turn-off operation, brightness adjustment operation, light color adjustment operation)) that can be supported by the light (1120). The electronic device may generate and / or output a preview (1130) of the light (1120) performing the turn-on operation. In FIG. 11, the electronic device may output a preview (1130) of at least a portion of the display area (1110) being brightened when the light (1120) performs the turn-on operation.

[0228] In this disclosure, secondary feedback (e.g., auditory feedback (1112), graphical representation (1113)) and a preview (1130) are primarily described, but are not limited thereto. Secondary feedback may include a preview (1130) of performing an action related to an input of an object.

[0229] According to one embodiment, the electronic device may output a preview (1130) of an object performing a determined action (e.g., an action related to the input) while a context corresponding to the input is maintained. The electronic device may stop outputting the preview (1130) of the object performing the determined action (e.g., an action related to the input) based on the termination of the context corresponding to the input.

[0230] In an example embodiment, an electronic device (101; 201; 301; 401; 501; 701) according to one embodiment may include a memory (130) having computer-executable instructions stored therein; and a processor (120) that accesses the memory (130) and executes the instructions. The above instructions, when executed, cause the processor (120) to generate a first feedback (661a; 661b; 941; 1011a) for the input based on obtaining a user input, and generate a second feedback (662a-1, 662a-2; 662b; 1012a) subsequent to the first feedback (661a; 661b; 941; 1011a) based on an object detected in at least one of a display area (600a; 600b; 910; 1010a; 1010b; 1110), a surrounding physical space, or a virtual space, and to generate the first feedback (661a; 661b; 941; 1011a) and the second feedback (662a-1, 662a-2; 662b; 1012a) can be caused to be printed.

[0231] In an example embodiment, the instructions, when executed, may cause the processor (120) to detect candidate objects in at least one of the display area (600a; 600b; 910; 1010a; 1010b; 1110), the surrounding physical space, or the virtual space, calculate a relevance score of the input and each candidate object, and determine the object based on the calculated relevance score.

[0232] In an example embodiment, the instructions, when executed, may cause the processor (120) to detect the object while a context corresponding to the input is maintained from a time point at which the input was obtained, and to stop detecting the object based on the termination of the context corresponding to the input.

[0233] In an example embodiment, the instructions, when executed, may cause the processor (120) to stop detecting the object based on obtaining other input about the object and other objects.

[0234] In an example embodiment, the instructions, when executed, may cause the processor (120) to determine a space in which the user is located at the time the input is acquired, detect the object in the display area (600a; 600b; 910; 1010a; 1010b; 1110) while the user is located in the determined space, and stop detecting the object based on the user leaving the determined space.

[0235] In an example embodiment, the instructions, when executed, may cause the processor (120) to detect an object in the display area (600a; 600b; 910; 1010a; 1010b; 1110) while detecting a motion of the user corresponding to the input, and to stop detecting the object based on detecting the end of the motion.

[0236] In an example embodiment, the instructions, when executed, may cause the processor (120) to cease detection of the object based on failure to detect the object associated with the input for a threshold time from the time the input was acquired.

[0237] In an example embodiment, the instructions, when executed, may cause the processor (120) to, based on obtaining additional input from the user, analyze the additional input using at least one of the input, the first feedback (661a; 661b; 941; 1011a), or the second feedback (662a-1, 662a-2; 662b; 1012a).

[0238] In an example embodiment, the instructions, when executed, may cause the processor (120) to determine an operation of the object based on at least one of the input, the first feedback (661a; 661b; 941; 1011a), or the second feedback (662a-1, 662a-2; 662b; 1012a) from among candidate operations supportable by the object, output an interface (1030a) for obtaining additional input for the determined operation, and exclude from the interface (1030a) information about operations other than the determined operation from among the candidate operations.

[0239] In an example embodiment, the instructions, when executed, may cause the processor (120) to determine an action of the object based on at least one of the input, the first feedback (661a; 661b; 941; 1011a), or the second feedback (662a-1, 662a-2; 662b; 1012a) from among candidate actions supportable by the object, and to output a preview (1130) of the object performing the determined action.

[0240] In an example embodiment, a method performed by an electronic device (101; 201; 301; 401; 501; 701) according to one embodiment comprises: generating a first feedback (661a; 661b; 941; 1011a) for an input based on obtaining a user input; generating a second feedback (662a-1, 662a-2; 662b; 1012a) subsequent to the first feedback (661a; 661b; 941; 1011a) based on an object detected in at least one of a display area (600a; 600b; 910; 1010a; 1010b; 1110), a surrounding physical space, or a virtual space; And it may include an operation of outputting the first feedback (661a; 661b; 941; 1011a) and the second feedback (662a-1, 662a-2; 662b; 1012a).

[0241] In an example embodiment, the operation of generating the second feedback may include: detecting a candidate object in at least one of the display area (600a; 600b; 910; 1010a; 1010b; 1110), the surrounding physical space, or the virtual space; calculating a relevance score between the input and each candidate object; and determining the object based on the calculated relevance score.

[0242] In an example embodiment, the operation of generating the second feedback may include: detecting the object while a context corresponding to the input is maintained from a time point at which the input is acquired; and stopping detection of the object based on the termination of the context corresponding to the input.

[0243] In an example embodiment, the act of generating the second feedback may include an act of ceasing detection of the object based on obtaining other input about the object and a different object.

[0244] In an example embodiment, the act of generating the second feedback may include: determining a space in which the user is located at the time the input is acquired; detecting the object in the display area (600a; 600b; 910; 1010a; 1010b; 1110) while the user is located in the determined space; and stopping detection of the object based on the user leaving the determined space.

[0245] In an example embodiment, the act of generating the second feedback may include: detecting the object in the display area (600a; 600b; 910; 1010a; 1010b; 1110) while detecting the motion based on detecting a motion of the user corresponding to the input; and stopping detection of the object based on detecting the end of the motion.

[0246] In an example embodiment, the act of generating the second feedback may include an act of ceasing detection of the object based on failure to detect the object associated with the input for a threshold time from the time the input was acquired.

[0247] In an example embodiment, the method may further include, based on obtaining additional input from the user, analyzing the additional input using at least one of the input, the first feedback (661a; 661b; 941; 1011a), or the second feedback (662a-1, 662a-2; 662b; 1012a), together with the additional input.

[0248] In an example embodiment, the method may further include: determining an operation of the object based on at least one of the input, the first feedback (661a; 661b; 941; 1011a), or the second feedback (662a-1, 662a-2; 662b; 1012a) among candidate operations supportable by the object; outputting an interface (1030a) for obtaining additional input for the determined operation; and excluding information about an operation different from the determined operation among the candidate operations from the interface (1030a).

[0249] In exemplary embodiments, electronic devices according to various embodiments disclosed herein may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the aforementioned devices.

[0250] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0251] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0252] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0253] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0254] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0255] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0256] Software may include a computer program, code, instructions, or a combination of one or more of these, and may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, or computer storage medium or device for interpretation by the processing device (e.g., at least one processor including processing circuitry) or for providing instructions or data to the processing device. The software may also be distributed over network-connected computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0257] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, and data structures, either singly or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter.

[0258] The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0259] While the present disclosure has been described and disclosed with reference to various exemplary embodiments, it will be understood that the various exemplary embodiments are intended to be illustrative and not limiting. It will be further understood that various changes in form and detail may be made by those skilled in the art without departing from the spirit and scope of the disclosure, including the appended claims and their equivalents. It will also be understood that any embodiment described herein can be utilized in conjunction with any other embodiment described herein.

Claims

1. In electronic devices (101; 201; 301; 401; 501; 701), A memory (130) in which computer-executable instructions are stored; and At least one processor (120) comprising a processing circuit Including, The above processor (120) Based on obtaining the user's input, generating a first feedback (661a; 661b; 941; 1011a) for said input, Generating a second feedback (662a-1, 662a-2; 662b; 1012a) subsequent to the first feedback (661a; 661b; 941; 1011a) based on an object detected in at least one of the display area (600a; 600b; 910; 1010a; 1010b; 1110), the surrounding physical space, or the virtual space, Output the first feedback (661a; 661b; 941; 1011a) and the second feedback (662a-1, 662a-2; 662b; 1012a) individually and / or collectively constituted, Electronic devices (101; 201; 301; 401; 501; 701).

2. In paragraph 1, At least one processor (120) above, Detecting a candidate object in at least one of the display area (600a; 600b; 910; 1010a; 1010b; 1110), the surrounding physical space, or the virtual space, Calculate the relevance score of the above input and each candidate object, Based on the above calculated relevance score, the detected object is determined. individually and / or collectively constituted, Electronic devices (101; 201; 301; 401; 501; 701).

3. In any one of paragraphs 1 and 2, At least one processor (120) above, Detecting the object while the context corresponding to the input is maintained from the time point at which the input is acquired, Stop detection of the object based on the termination of the context corresponding to the input. individually and / or collectively constituted, Electronic devices (101; 201; 301; 401; 501; 701).

4. In any one of paragraphs 1 to 3, At least one processor (120) above, Stop detection of the object based on obtaining other input for the object and other objects. individually and / or collectively constituted, Electronic devices (101; 201; 301; 401; 501; 701).

5. In any one of paragraphs 1 to 4, At least one processor (120) above, Determine the space where the user is located at the time the above input is obtained, While the user is positioned in the determined space, the object is detected in the display area (600a; 600b; 910; 1010a; 1010b; 1110). Based on the user leaving the determined space, detection of the object is stopped. individually and / or collectively constituted, Electronic devices (101; 201; 301; 401; 501; 701).

6. In any one of paragraphs 1 to 5, At least one processor (120) above, Based on detecting the user's motion corresponding to the input, detecting the object in the display area (600a; 600b; 910; 1010a; 1010b; 1110) while detecting the motion, Based on detecting the end of said movement, detection of said object is stopped. individually and / or collectively constituted, Electronic devices (101; 201; 301; 401; 501; 701).

7. In any one of paragraphs 1 to 6, At least one processor (120) above, Stop detection of the object based on failure to detect the object related to the input for a threshold time from the time of acquiring the input. individually and / or collectively constituted, Electronic devices (101; 201; 301; 401; 501; 701).

8. In any one of paragraphs 1 to 7, At least one processor (120) above, Based on obtaining additional input from said user, analyzing said additional input by further utilizing at least one of said input, said first feedback (661a; 661b; 941; 1011a), or said second feedback (662a-1, 662a-2; 662b; 1012a). individually and / or collectively constituted, Electronic devices (101; 201; 301; 401; 501; 701).

9. In any one of paragraphs 1 to 8, At least one processor (120) above, Determine an operation of the object based on at least one of the input, the first feedback (661a; 661b; 941; 1011a), or the second feedback (662a-1, 662a-2; 662b; 1012a) among candidate operations supported by the object, Output an interface (1030a) for obtaining additional input for the above-determined operation, Information about actions other than the determined action among the above candidate actions is excluded from the interface (1030a). individually and / or collectively constituted, Electronic devices (101; 201; 301; 401; 501; 701).

10. In any one of paragraphs 1 to 9, At least one processor (120) above, Determine an operation of the object based on at least one of the input, the first feedback (661a; 661b; 941; 1011a), or the second feedback (662a-1, 662a-2; 662b; 1012a) among candidate operations supported by the object, Output a preview (1130) of the above object performing the above determined action. individually and / or collectively constituted, Electronic devices (101; 201; 301; 401; 501; 701).

11. In a method performed by an electronic device (101; 201; 301; 401; 501; 701), An operation of generating a first feedback (661a; 661b; 941; 1011a) for said input based on obtaining a user input; An operation of generating a second feedback (662a-1, 662a-2; 662b; 1012a) subsequent to the first feedback (661a; 661b; 941; 1011a), based on an object detected in at least one of the display area (600a; 600b; 910; 1010a; 1010b; 1110), the surrounding physical space, or the virtual space; and An operation of outputting the first feedback (661a; 661b; 941; 1011a) and the second feedback (662a-1, 662a-2; 662b; 1012a) A method including:

12. In paragraph 11, The action of generating the second feedback is: An operation of detecting a candidate object in at least one of the display area (600a; 600b; 910; 1010a; 1010b; 1110), the surrounding physical space, or the virtual space; An operation for calculating a relevance score between the above input and each candidate object; and An operation for determining the detected object based on the calculated relevance score is included. method.

13. In any one of paragraphs 11 to 12, The action of generating the second feedback is: An operation of detecting the object while the context corresponding to the input is maintained from the time point at which the input is acquired; and An operation for stopping detection of the object based on the termination of the context corresponding to the input, method.

14. In any one of paragraphs 11 to 13, The action of generating the second feedback is: An operation of stopping detection of the object based on obtaining other input for the object and other objects, method.

15. A computer-readable recording medium storing one or more computer programs including commands for performing the method of claim 11.

Images