Wearable electronic device for hand tracking, operating method therefor, and storage medium

Abstract

According to one embodiment, a wearable electronic device may comprise: a housing; a camera disposed on a first surface of the housing and configured to capture an image in front of the wearable electronic device; a display; at least one processor; and a memory for storing instructions. According to one embodiment, the instructions may be configured to, when executed individually or collectively by the at least one processor, instruct the wearable electronic device to: perform hand tracking for tracking a first hand entering a field of view of the camera; identify whether the shape of the first hand is a left hand shape or a right hand shape; identify whether the movement direction of the first hand corresponds to a direction in which access is allowed for a hand having the identified shape; and display a first object corresponding to the first hand in a virtual reality space provided through the display if the movement direction of the first hand corresponds to the direction in which access is allowed for the hand having the identified shape.

Description

Wearable electronic device for hand tracking, method of operation thereof, and storage medium

[0001] One embodiment disclosed in this document relates to a wearable electronic device for hand tracking, a method of operation thereof, and a storage medium.

[0002] As the variety of services and additional functions provided through electronic devices such as smartphones gradually increases, various applications executable on these devices are being developed. Furthermore, the hardware and / or software aspects of electronic devices are also continuously evolving.

[0003] Recently, in line with consumer trends that prioritize design, the design of graphic objects—visual elements such as wallpapers, icons, and widgets displayed on screens—is being given significant consideration in the development of electronic devices. Furthermore, beyond developing electronic devices that take graphic object design into account, technological development aimed at implementing content display (or playback, output) more naturally can be crucial for enhancing user sensibility, immersion, and / or realism.

[0004] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure.

[0005] According to one embodiment, a wearable electronic device may include a housing, a camera disposed on a first surface of the housing and configured to capture an image from the front of the wearable electronic device, a display, at least one processor, and a memory for storing instructions.

[0006] According to one embodiment, the instructions may be configured to cause the wearable electronic device to perform hand tracking that tracks a first hand entering the field of view of the camera when executed individually or collectively by the at least one processor.

[0007] According to one embodiment, the instructions may be configured to cause the wearable electronic device to determine whether the shape of the first hand is a left-hand shape or a right-hand shape when executed individually or collectively by the at least one processor.

[0008] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device may be configured to determine whether the direction of movement of the first hand corresponds to a direction in which access is allowed to the hand having the identified shape.

[0009] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device may be configured to display a first object corresponding to the first hand in a virtual reality space provided through the display when the direction of movement of the first hand corresponds to a direction in which access to the hand having the identified shape is permitted.

[0010] According to one embodiment, a method for hand tracking in a wearable electronic device may include an operation of performing hand tracking to track a first hand entering the field of view of a camera of the wearable electronic device.

[0011] According to one embodiment, the method may include an operation to determine whether the shape of the first hand is a left-hand shape or a right-hand shape.

[0012] According to one embodiment, the method may include an operation to determine whether the direction of movement of the first hand corresponds to a direction in which access is allowed to the hand having the confirmed shape.

[0013] According to one embodiment, the method may include an operation of displaying a first object corresponding to the first hand in a virtual reality space when the direction of movement of the first hand corresponds to a direction in which access to the hand having the identified shape is allowed.

[0014] According to one embodiment, in a non-transient storage medium storing instructions, the instructions are configured to cause the wearable electronic device to perform at least one operation when executed individually or collectively by at least one processor of the wearable electronic device, wherein the at least one operation may include performing hand tracking to track a first hand entering the field of view of the camera of the wearable electronic device.

[0015] According to one embodiment, the at least one operation may include an operation to determine whether the shape of the first hand is a left-hand shape or a right-hand shape.

[0016] According to one embodiment, the at least one operation may include an operation to determine whether the direction of movement of the first hand corresponds to a direction in which access is allowed to the hand having the identified shape.

[0017] According to one embodiment, the at least one operation may include an operation of displaying a first object corresponding to the first hand in a virtual reality space when the direction of movement of the first hand corresponds to a direction in which access to the hand having the identified shape is allowed.

[0018] FIG. 1 is a block diagram of an electronic device in a network environment according to one embodiment.

[0019] FIG. 2 is a drawing showing the external appearance of a wearable electronic device according to one embodiment.

[0020] FIG. 3a is a drawing showing the front view of a wearable electronic device according to one embodiment.

[0021] FIG. 3b is a drawing showing the rear side of a wearable electronic device according to one embodiment.

[0022] FIG. 4 is a diagram showing the structure of a display and an eye-tracking camera of a wearable electronic device according to one embodiment.

[0023] FIG. 5 is an example diagram showing a case in which a hand of another person other than the user's hand is recognized according to one embodiment.

[0024] FIG. 6 is an internal block diagram of a wearable electronic device according to one embodiment.

[0025] FIG. 7 is an operation flowchart of a wearable electronic device for hand tracking according to one embodiment.

[0026] FIG. 8 is an example diagram illustrating the direction of movement of a hand entering the field of view of a camera according to one embodiment.

[0027] FIG. 9a is a detailed operation flowchart of a wearable electronic device according to one embodiment.

[0028] FIG. 9b is an example diagram illustrating hand tracking for a hand other than the user's hand according to one embodiment.

[0029] FIG. 10 is an example diagram illustrating a case in which the direction of movement of a user's hand and the direction of movement of another person's hand are the same according to one embodiment.

[0030] FIG. 11 is an example diagram showing a guide requesting user confirmation according to one embodiment.

[0031] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0032] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to one embodiment. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through the server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0033] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0034] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0035] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0036] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0037] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0038] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0039] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0040] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0041] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0042] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0043] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0044] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0045] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0046] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0047] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0048] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0049] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0050] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0051] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0052] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0053] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0054] Before describing the configuration of the electronic device and the method of operation of the electronic device with reference to the drawings below, the term "virtual object" in this disclosure may be described as a collective term for an object representing an execution screen of one or more applications displayed in a virtual reality space, an object included in an execution screen, an object related to a menu or function, an object for at least one input interface, or various other objects displayed. Furthermore, the virtual object described in this disclosure means that it includes at least one of an object representing an execution screen of one or more applications displayed in a virtual reality space, an object included in an execution screen, an object related to a menu or function, an object for at least one input interface, or various other objects displayed. Here, although the object is described in the singular for convenience of explanation, the singular form of the noun corresponding to the object may include one or multiple instances.

[0055] The term virtual reality space described in this disclosure may refer to a virtual space provided in a virtual environment using various virtual reality technology methods (e.g., eXtended Reality (XR), virtual reality (VR), or augmented reality (AR)).

[0056] FIG. 2 is a drawing showing the external appearance of a wearable electronic device according to one embodiment.

[0057] Referring to FIG. 2, a wearable electronic device (200) according to one embodiment (e.g., the electronic device (101) of FIG. 1) may include at least one of a light output module (211), a display member (201), a camera module (or camera, camera device) (250), and a speaker (261). In the wearable electronic device (200) of FIG. 2, the display member (201) may be of the glass type, which is a transparent member.

[0058] According to one embodiment, the light output module (211) (e.g., the display module (160) of FIG. 1) may include a light source capable of outputting an image and a lens that guides the image to a display member (201). According to one embodiment, the light output module (211) may include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0059] According to one embodiment, the display member (201) (e.g., the display module (160) of FIG. 1) may include an optical waveguide (e.g., a waveguide). According to one embodiment, an output image of the optical output module (211) incident on one end of the optical waveguide may propagate within the optical waveguide and be provided to the user. According to one embodiment, the optical waveguide may include at least one diffractive element (e.g., a DOE (diffractive optical element), a HOE (holographic optical element)) or a reflective element (e.g., a reflective mirror). For example, the optical waveguide may guide the output image of the optical output module (211) to the user's eye using at least one diffractive element or reflective element.

[0060] According to one embodiment, the camera module (250) (e.g., the camera module (180) of FIG. 1) can capture still images and / or videos. According to one embodiment, the camera module (250) can be placed within a lens frame and around a display member (201).

[0061] According to one embodiment, the camera module (250) may include one or more first cameras (251), one or more second cameras (253), and / or one or more third cameras (255).

[0062] Images obtained through one or more first cameras (251) according to one embodiment may be used to detect and track the user's pupils. For example, images obtained through one or more first cameras (251) may be used to track the direction of the user's gaze. One or more first cameras (251) according to one embodiment may be GS cameras. One or more first cameras (251) according to one embodiment may correspond to the left eye and the right eye, respectively, and the performance of each of the one or more first cameras (251) may be substantially the same. According to one embodiment, one or more first cameras (251) may periodically or non-periodically transmit information related to the trajectory of the user's eyes or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1).

[0063] According to one embodiment, one or more second cameras (253) can capture external images and may be high-resolution cameras. According to one embodiment, one or more second cameras (253) may perform an auto-focusing (AF) function or a shake correction function. According to one embodiment, one or more second cameras (253) may be GS cameras or RS (rolling shutter) cameras.

[0064] According to one embodiment, images acquired through one or more third cameras (255) may be used for user hand detection and hand tracking, user gestures (e.g., hand movements), user head tracking, and / or spatial recognition. One or more third cameras (255) according to one embodiment may be global shutter (GS) cameras. One or more third cameras (255) according to one embodiment may perform simultaneous localization and mapping (SLAM) operations through depth imaging. One or more third cameras (255) according to one embodiment may perform spatial recognition for 6 degrees of freedom (DoF).

[0065] According to one embodiment, at least one of the first camera (251) to the third camera (255) may be replaced with a sensor module (or sensor circuit) (e.g., a LiDAR sensor). For example, the sensor module may include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0066] According to one embodiment, the speaker (261) (e.g., the acoustic output module (155) of FIG. 1) can output an acoustic signal (e.g., sound and / or virtual vibration sound). Although the speaker (261) is described as being configured in a member that is mounted on the user's ear when the wearable electronic device (200) is worn in FIG. 2, it is not limited thereto and may be configured in other locations depending on the implementation of the wearable electronic device (200).

[0067] FIG. 3a is a drawing showing the front of a wearable electronic device according to one embodiment, and FIG. 3b is a drawing showing the rear of a wearable electronic device according to one embodiment.

[0068] Referring to FIGS. 3a and 3b, in one embodiment, camera modules (311, 312, 313, 314, 315, 316) and / or depth sensors (317)) for acquiring information related to the surrounding environment of a wearable electronic device (300) (e.g., electronic device (101) of FIG. 1) may be disposed on a first surface (310) of a housing. The wearable electronic device (300) in FIGS. 3a and 3b is a head-mounted display (HMD) device and may be an augmented reality (AR) device, a virtual reality (VR) device, a mixed reality (MR) device, or an extended reality (XR) device.

[0069] In one embodiment, camera modules (or camera circuits) (311, 312) can acquire images related to the surrounding environment of the wearable electronic device (300).

[0070] In one embodiment, camera modules (313, 314, 315, 316) can acquire images while the wearable electronic device (300) is worn by a user. The camera modules (313, 314, 315, 316) can be used for hand detection, tracking, and user gesture (e.g., hand movements) recognition. The camera modules (313, 314, 315, 316) can be used for 3DoF, 6DoF head tracking, location (space, environment) recognition, and / or movement recognition. In one embodiment, camera modules (311, 312) may be used for hand detection and tracking, and user gestures.

[0071] In one embodiment, the depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for determining the distance to an object, such as time of flight (TOF). In place of or additionally to the depth sensor (317), camera modules (313, 314, 315, 316) may determine the distance to an object.

[0072] According to one embodiment, camera modules (325, 326) for face recognition and / or a display (321) (e.g., a display for the left eye and a display for the right eye) (and / or a lens) may be disposed on the second surface (320) of the housing.

[0073] In one embodiment, the face recognition camera modules (325, 326) adjacent to the display (321) may be used to recognize the user's face or to recognize and / or track both of the user's eyes.

[0074] In one embodiment, the display (321) (and / or lens) may be disposed on a second surface (320) of the wearable electronic device (300). In one embodiment, the wearable electronic device (300) may not include camera modules (315, 316) among a plurality of camera modules (313, 314, 315, 316). Although not illustrated in FIG. 3a and 3b, the wearable electronic device (300) may further include at least one of the configurations illustrated in FIG. 2.

[0075] FIG. 4 is a diagram showing the structure of a display and an eye-tracking camera of a wearable electronic device according to one embodiment.

[0076] A wearable electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIG. 3) (400) according to one embodiment may include a display (421), an input optical member (422), a display optical waveguide (423), an output optical member (424), an eye-tracking camera (410), a first splitter (441), an eye-tracking optical waveguide (442), and / or a second splitter (443).

[0077] In a wearable electronic device (400) according to one embodiment, the display (421) may be the display (321) shown in FIG. 3b (e.g., a display for the left eye and a display for the right eye). Light output from the display (421) passes through an input optical member (422) and is incident on a display optical waveguide (423), and may be output through an output optical member (424) after passing through the display optical waveguide (423). The light output from the output optical member (424) can be seen by the user's eye (430). According to one embodiment, light output from the display (421) is output through the output optical member (424), and the shape of an object is seen by the user's eye (430) by the light output through the output optical member (424), which may mean displaying an object on the display. Light output from a display (421) according to one embodiment is output through an output optical member (424), and controlling the display (421) so that the shape of an object is visible to the user's eye (430) by the light output through the output optical member (424) may mean controlling to display an object. Light (432) reflected from the user's eye (430) passes through a first splitter (441) and is incident on an eye-tracking optical waveguide (442), and may pass through the eye-tracking optical waveguide (442) and be output to an eye-tracking camera (410) through a second splitter (443). According to one embodiment, the eye-tracking camera (410) may be one or more first cameras (251) shown in FIG. 2.

[0078] FIG. 5 is an example diagram showing a case in which a hand of another person other than the user's hand is recognized according to one embodiment.

[0079] According to one embodiment, the wearable electronic device may be worn on a user's body (e.g., head) and may be implemented as, for example, augmented reality glasses, a video see-through (VST) device, or a head-mount display (HMD) device. However, this is just one example, and the wearable electronic device may be implemented as various devices.

[0080] In addition, wearable electronic devices can be referred to as HMD devices, AR (augment reality) devices, VR (virtual reality) devices, MR (mixed reality) devices, or XR (eXtened Reality) devices that can be worn on a user's head. Since touch input is impossible for these wearable electronic devices due to their nature, hand interaction, which utilizes the user's three-dimensional pose and hand gesture as a means of input, can be important as an input interface.

[0081] For example, when a user activates a wearable electronic device (e.g., the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3a and FIG. 3b) while wearing it on their head, the wearable electronic device can generate a scene in a virtual reality space and provide the generated scene to the user.

[0082] As illustrated in 500a of FIG. 5, when a user's hand (510, 520) is positioned within the field of view (or shooting area) of the camera of the wearable electronic device, an object (511, 521) corresponding to each of the user's hands (510, 520) may be displayed in the virtual reality space as illustrated in 500b. However, when another person's hand (530) other than the user's hand (510, 520) is positioned within the field of view of the camera of the wearable electronic device, the other person's hand (530) is recognized as not being the user's hand as illustrated in 500c, and an object (531) corresponding to the recognized other person's hand (530) may be displayed. In this way, the wearable electronic device only recognizes the hand within the field of view of the camera, so interference caused by the movement of another person's hand may occur.

[0083] According to one embodiment, a wearable electronic device for hand tracking, a method of operation thereof, and a storage medium may be provided so as to identify another person's hand other than the user's hand within the camera's field of view in order to minimize the influence of a third party other than the user wearing the wearable electronic device, and to limit unintended input by the user.

[0084] According to one embodiment, by limiting unintended input by the user, incorrect input of the wearable electronic device can be prevented and usability can be improved.

[0085] FIG. 6 is an internal block diagram of a wearable electronic device according to one embodiment.

[0086] According to one embodiment, the wearable electronic device (601) may include a memory (630) (e.g., memory (130) of FIG. 1), at least one processor (620) (e.g., processor (120) of FIG. 1), a display (660) (e.g., display (160) of FIG. 1), and a camera (e.g., camera module (180) of FIG. 1). Not limited thereto, the wearable electronic device (601) may further include at least one sensor and other components necessary to perform at least one operation. According to one embodiment, the wearable electronic device (601) may be implemented identically or similarly to the electronic device (101) of FIG. 1. According to one embodiment, the processor (620) including a processing circuit may control the overall operation of the wearable electronic device (601).

[0087] According to one embodiment, the processor (620) can display at least one virtual object in a virtual reality space through the display (620) (e.g., the display (321) of FIG. 3b) while the user is wearing the wearable electronic device (601). According to one embodiment, the electronic device (200) can perform at least one of the following actions on the virtual object displayed in the virtual reality space through hand interaction: execution (or activation), moving the display position, selection, removal (or deactivation), or other actions.

[0088] According to one embodiment, the display (660) may display an object representing at least one input interface (e.g., a user's hand) under the control of the processor (620). The display (660) may display a virtual object under the control of the processor (620). For example, the virtual object may include a default menu or a universal menu provided by the wearable electronic device (601). The universal menu may include a menu that can execute an application installed on the wearable electronic device (601). Not limited thereto, the universal menu may also include a menu that provides various other functions.

[0089] According to one embodiment, the camera (680) may include one or more first cameras (251), one or more second cameras (253), and / or one or more third cameras (255) as in FIG. 2.

[0090] According to one embodiment, one or more first cameras (251) may represent eye-tracking cameras that track a user's gaze. The eye-tracking cameras may include infrared (IR) cameras. For example, the eye-tracking cameras may be placed on a second surface of the housing (e.g., 320 in FIG. 3b). For example, the eye-tracking cameras may be implemented identically or similarly to the camera modules (325, 326) of FIG. 3b. However, the number and arrangement structure of the eye-tracking cameras included in the wearable electronic device (601) may be identically or similarly to the camera modules (325, 326) of FIG. 3b or different from each other.

[0091] According to one embodiment, one or more third cameras (255) can capture the field of view (FOV) (or field of view, shooting area) of the third cameras (255). One or more third cameras (255) may be placed on a first surface of the housing (e.g., 310 in FIG. 3a). According to one embodiment, one or more third cameras (255) can acquire images related to the surrounding environment of the wearable electronic device (601). Images acquired by capturing the field of view of the third cameras (255) may be used for user hand detection and hand tracking, user gestures (e.g., hand movements), user head tracking, and / or spatial recognition.

[0092] For example, one or more third cameras (255) may be implemented identically or similarly to the camera modules (311, 312) of FIG. 3a. However, the number and arrangement structure of one or more third cameras (255) included in the wearable electronic device (601) may be identically or similarly to the camera modules (311, 312) of FIG. 3a or different from each other.

[0093] According to one embodiment, one or more third cameras (255) are positioned above the user's hand when the user wears the wearable electronic device (601) on their head, so that the user's hand within the shooting area can be photographed to obtain an image including the hand.

[0094] According to one embodiment, when the wearable electronic device (601) is activated (or turned on) as a user wears it on their head, the processor (620) can provide (or output, display) a scene of the virtual reality space through the display (660). According to one embodiment, as the wearable electronic device (601) is activated, the processor (620) can activate (or drive) the camera (680) to photograph a subject within the shooting area. Accordingly, the processor (620) can detect (or identify) a hand from the acquired image and start hand tracking.

[0095] According to one embodiment, the processor (620) can detect a hand entering (or appearing) the field of view (or shooting area) of the camera (680) from the acquired image when acquiring an image through the camera (680).

[0096] According to one embodiment, a single input function may be provided to restrict unintended input by the user other than the user's hands. According to one embodiment, the single input may be an input by either of the user's two hands.

[0097] According to one embodiment, a multi-input function may be provided to enable controlling one or more objects displayed in a virtual reality space with both hands. According to one embodiment, the multi-input may include input by each of the user's two hands. Additionally, the multi-input may include input by the hands of an external user (another person) permitted by the user, in addition to input by the user's hands.

[0098] According to one embodiment, when a multi-input function is set, the direction of the hand is set relative to the user, and a function to restrict hand tracking for hands that move differently from the set direction can be provided. In this way, it is possible to identify whether it is the user's hand or another person's hand based on the direction of hand movement.

[0099] According to one embodiment, when a hand of another person, other than the user's hand, moves in the same direction as a set direction, hand tracking of the other person's hand is allowed depending on whether the user allows it, thereby providing a function that enables control by another person's hand permitted by the user, in addition to the user's hand.

[0100] According to one embodiment, the processor (620) can determine whether a single input or multiple input is set while performing hand tracking for a first hand that has entered the field of view of the camera (680). For example, if a single input is set, the processor (620) can start hand tracking for one hand that has entered the field of view of the camera (680). On the other hand, if multiple inputs are set, the processor (620) can start hand tracking for each of the two hands that have entered the field of view of the camera (680).

[0101] Additionally, the processor (620) can determine whether the shape of the first hand entering the field of view has the shape of a left hand or a right hand in order to determine whether the first hand entering the field of view is a left hand or a right hand. In one embodiment, the case of determining whether it is a left hand back shape or a right hand back shape based on the shape of the hand with the back facing upward will be explained as an example, but this is for convenience of explanation and can be applied in the same way to distinguish the shape of the hand with the palm facing upward as a left palm shape or a right palm shape.

[0102] The processor (620) can determine the direction of movement of the first hand through hand tracking after confirming the shape of the first hand. The processor (620) can determine whether the direction of movement of the first hand corresponds to a direction in which access is permitted for a hand having the confirmed shape. For example, since the user is wearing the wearable electronic device (601) on their head, the user's left hand may move from left to right when entering the shooting area of ​​the camera (680). Conversely, the user's right hand may move from right to left when entering the shooting area of ​​the camera (680). Therefore, if the direction of movement of the user's left hand corresponds to a direction from right to left rather than a direction from left to right, the processor (620) can determine that the hand that entered the shooting area of ​​the camera (680) is the hand of a third party, not the user.

[0103] According to one embodiment, the processor (620) may display a first object corresponding to the first hand in a virtual reality space provided through the display (660) when the direction of movement of the first hand corresponds to a direction in which access is permitted to the hand having the identified shape. For example, the first object may be a hand-shaped object representing the first hand.

[0104] According to one embodiment, if the direction of movement of the first hand does not correspond to the direction in which access is permitted for the hand of the identified shape, the processor (620) may determine that it is the hand of a third party other than the user and stop hand tracking that tracks the first hand. The processor (620) may output a first guide to guide the user to enter the field of view of the camera (680). For example, the processor (620) may output a guide through a display (660) or a speaker (not shown) to induce hand tracking of the user's hand, which is not a third party.

[0105] According to one embodiment, if the direction of movement of the first hand does not correspond to the direction in which access is permitted for the hand of the identified shape, the processor (620) may output a second guide to guide the user to verify whether the first hand is the user's hand in a virtual reality space. While outputting the second guide, the processor (620) may stop hand tracking that tracks the first hand in response to identifying that the first hand is not the user's hand upon receiving a preset gesture or voice input. For example, if it is determined that it is the hand of a third party, the user may be prompted to verify by outputting a guide to verify whether it is the user's hand. For example, if the user receives a preset gesture of shaking their head or a voice command saying 'no' while outputting the second guide, the processor (620) may consider the first hand to correspond to the hand of a third party and stop hand tracking that tracks the first hand. On the other hand, for example, if the user receives a preset gesture of nodding or a voice command of 'yes' while outputting the second guide, even if the first hand corresponds to a third person's hand, since the user has allowed it, the first object corresponding to the first hand may be displayed in the virtual reality space.

[0106] According to one embodiment, if the direction of movement of the first hand does not correspond to the direction in which access is allowed for the hand of the identified shape, the processor (620) may output a first guide to guide the user to enter the field of view of the camera. For example, if it is determined that the hand is that of a third party, the processor (620) may output a first guide to ensure that the user's hand is detected within the shooting area.

[0107] The processor (620) can perform hand tracking to track a second hand entering the field of view of the camera (680) while outputting the first guide, thereby determining whether the shape of the second hand is the shape of the left hand or the shape of the right hand. The processor (620) can determine whether the direction of movement of the second hand corresponds to the direction in which access is permitted to the hand of the confirmed shape, and if the direction of movement of the second hand corresponds to the direction in which access is permitted to the hand of the confirmed shape, it can display a second object corresponding to the second hand in the virtual reality space. For example, if the direction of movement of the second hand is the same as the direction of movement set for the shape of the hand, it is considered to be the user's hand, and the processor (620) can display a second object corresponding to the second hand that is the user's hand.

[0108] According to one embodiment, the processor (620) can perform hand tracking to track a second hand entering the field of view of the camera (680) while displaying a first object corresponding to the first hand in a virtual reality space when multi-input is set. The processor (620) can determine whether the second hand is the shape of the left hand or the shape of the right hand, and can determine whether the shape of the first hand and the shape of the second hand are the same by comparing the shape of the first hand and the shape of the second hand. If the shape of the first hand and the shape of the second hand are the same, for example, while an object corresponding to the user's left hand is being displayed, the second hand corresponding to the left hand being hand-tracked may be the hand of a third party, so the processor (620) can stop the hand tracking to track the second hand. On the other hand, if the shape of the first hand and the shape of the second hand are different, the processor (620) can display a first object and a second object corresponding to both hands of the user in a virtual reality space.

[0109] Meanwhile, the processor (620) may output a second guide to determine whether the second hand is the user's hand in the virtual reality space when the direction of movement of the second hand corresponds to the direction in which access to the hand of the confirmed shape is allowed. According to one embodiment, the user's gaze may be used as an input to determine whether it is the user's hand. Accordingly, the processor (620) may determine the direction of the user's gaze tracked through the gaze tracking camera, and, in response to confirming that the second hand is the user's hand based on the direction of the user's gaze, display a second object corresponding to the second hand together with the first object in the virtual reality space.

[0110] According to one embodiment, the processor (620) may display a second object corresponding to the second hand together with the first object in the virtual reality space in response to the reception of a preset gesture or voice input while outputting the second guide. For example, as a preset gesture or voice confirming that the user's hand is correct is input, the processor (620) may display the first object and the second object corresponding to both of the user's hands in the virtual reality space. For example, an example of the display of the guide is as illustrated in FIG. 11. FIG. 11 is an example diagram showing the display of a guide requesting user confirmation according to one embodiment.

[0111] According to one embodiment, the processor (620) may perform hand tracking to track a third hand entering the field of view of the camera (680) while displaying the first object and the second object in the virtual reality space. For example, if another hand is hand-tracked while displaying objects corresponding to both hands of the user in the virtual reality space, the processor (620) may output a third guide to request user confirmation regarding whether to allow this. Based on the user confirmation, the processor (620) may stop the hand tracking to track the third hand. For example, if another hand is hand-tracked while displaying objects corresponding to both hands of the user, the processor (620) may stop the hand-tracking operation of the other hand if the user does not allow it.

[0112] According to one embodiment as described above, the processor (620) can display an object corresponding to the tracked hand in the virtual reality space by hand tracking at least one of the user's two hands, and can perform control over one or more objects displayed in the virtual reality space by the tracked hand.

[0113] According to one embodiment, a wearable electronic device (200, 300, 601) may include a housing, a camera (680) disposed on a first surface of the housing and configured to capture an image from the front of the wearable electronic device, a display (660), at least one processor (620), and a memory (630) for storing instructions.

[0114] According to one embodiment, the instructions may be configured to cause the wearable electronic device to perform hand tracking that tracks a first hand entering the field of view of the camera when executed individually or collectively by the at least one processor.

[0115] According to one embodiment, the instructions may be configured to cause the wearable electronic device to determine whether the shape of the first hand is a left-hand shape or a right-hand shape when executed individually or collectively by the at least one processor.

[0116] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device may be configured to determine whether the direction of movement of the first hand corresponds to a direction in which access is allowed to the hand having the identified shape.

[0117] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device may be configured to display a first object corresponding to the first hand in a virtual reality space provided through the display when the direction of movement of the first hand corresponds to a direction in which access to the hand having the identified shape is permitted.

[0118] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device may be configured to stop hand tracking tracking the first hand and output a first guide to guide the user to enter the field of view of the camera when the direction of movement of the first hand does not correspond to the direction in which access to the hand of the identified shape is allowed.

[0119] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device may be configured to output a second guide to guide the user to verify whether the first hand is the user's hand in the virtual reality space when the direction of movement of the first hand does not correspond to the direction in which access to the hand of the identified shape is allowed, and to stop hand tracking that tracks the first hand in response to identifying that the first hand is not the user's hand upon receiving a preset gesture or voice input while outputting the second guide.

[0120] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device may be configured to output a first guide to guide the user to enter the field of view of the camera when the direction of movement of the first hand does not correspond to the direction in which access to the hand of the identified shape is allowed, perform hand tracking to track the second hand entering the field of view of the camera while outputting the first guide, check whether the second hand is the left hand shape or the right hand shape, check whether the direction of movement of the second hand corresponds to the direction in which access to the hand of the identified shape is allowed, and display a second object corresponding to the second hand in the virtual reality space when the direction of movement of the second hand corresponds to the direction in which access to the hand of the identified shape is allowed.

[0121] According to one embodiment, the instructions may be configured such that, when executed individually or collectively by the at least one processor, the wearable electronic device performs hand tracking to track a second hand entering the field of view of the camera while displaying a first object corresponding to the first hand in the virtual reality space, checks whether the second hand is the shape of the left hand or the shape of the right hand, compares the shape of the first hand with the shape of the second hand, and stops the hand tracking to track the second hand in correspondence with the shape of the first hand and the shape of the second hand being identical.

[0122] According to one embodiment, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device may be configured to check whether the direction of movement of the second hand corresponds to a direction in which access to the hand of the identified shape is permitted, in correspondence with the shape of the first hand and the shape of the second hand being different from each other, and if the direction of movement of the second hand corresponds to a direction in which access to the hand of the identified shape is permitted, to display a second object corresponding to the second hand together with the first object in the virtual reality space.

[0123] According to one embodiment, the device further includes an eye-tracking camera disposed on a second surface of the housing and configured to track the user's gaze, and the instructions, when executed individually or collectively by the at least one processor, may be configured to output a second guide to determine whether the second hand is the user's hand in the virtual reality space when the direction of movement of the second hand corresponds to a direction in which access to the hand of the identified shape is permitted, and to determine the direction of the user's gaze tracked through the eye-tracking camera, and to display a second object corresponding to the second hand together with the first object in the virtual reality space in response to determining that the second hand is the user's hand based on the direction of the user's gaze.

[0124] According to one embodiment, the instructions may be configured such that, when executed individually or collectively by the at least one processor, the wearable electronic device displays a second object corresponding to the second hand together with the first object in the virtual reality space in response to the reception of a preset gesture or voice input while outputting the second guide.

[0125] According to one embodiment, the instructions may be configured such that, when executed individually or collectively by the at least one processor, the wearable electronic device performs hand tracking to track a third hand entering the field of view of the camera while displaying the first object and the second object in the virtual reality space, outputs a third guide to request user confirmation in the virtual reality space, and stops the hand tracking to track the third hand based on the user confirmation.

[0126] According to one embodiment, the instructions may be configured such that, when executed individually or collectively by the at least one processor, the wearable electronic device displays an object corresponding to the tracked hand in the virtual reality space by hand tracking at least one of the user's two hands, and performs control over one or more objects displayed in the virtual reality space by the tracked hand.

[0127] FIG. 7 is a flowchart of the operation of a wearable electronic device for hand tracking according to one embodiment. Referring to FIG. 7, the operation method may include operations 705 through 720. Each operation of the operation method of FIG. 7 may be performed by a wearable electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3, or the wearable electronic device (601) of FIG. 6)) and at least one processor of the wearable electronic device (e.g., the processor (120) of FIG. 1, the processor (620) of FIG. 6). In one embodiment, at least one of operations 705 through 720 may be omitted, the order of some operations may be changed, or other operations may be added.

[0128] In operation 705, the wearable electronic device (601) can perform hand tracking to track a first hand entering the field of view of a camera (e.g., the third camera (255) of FIG. 2).

[0129] In operation 710, the wearable electronic device (601) can determine whether the shape of the first hand is a left-hand shape or a right-hand shape.

[0130] In operation 715, the wearable electronic device (601) can determine whether the direction of movement of the first hand corresponds to the direction in which access is allowed to the hand having the confirmed shape.

[0131] In operation 720, the wearable electronic device (601) may display a first object corresponding to the first hand in a virtual reality space through a display (e.g., display member (201) of FIG. 2, display (421) of FIG. 3b, display (421) of FIG. 4, or display (660) of FIG. 6) when the direction of movement of the first hand corresponds to the direction in which access is allowed to the hand having the identified shape.

[0132] According to one embodiment, the wearable electronic device (601) may stop hand tracking that tracks the first hand if the direction of movement of the first hand does not correspond to the direction in which access is allowed to the hand of the identified shape. The wearable electronic device (601) may output a first guide to guide the user to enter the field of view of the camera.

[0133] According to one embodiment, the wearable electronic device (601) may output a second guide to guide the user to verify whether the shape of the first hand in the virtual reality space is the user's hand when the direction of movement of the first hand does not correspond to the direction in which access to the identified shape of the hand is allowed. The wearable electronic device (601) may stop hand tracking that tracks the first hand in response to identifying that the first hand is not the user's hand upon receiving a preset gesture or voice input while outputting the second guide.

[0134] According to one embodiment, the wearable electronic device (601) may output a first guide to guide the user to enter the field of view of the camera when the direction of movement of the first hand does not correspond to the direction in which access is permitted to the hand of the identified shape. The wearable electronic device (601) may perform hand tracking to track the second hand entering the field of view of the camera while outputting the first guide. The wearable electronic device (601) may check whether the shape of the second hand is the left hand shape or the right hand shape. The wearable electronic device (601) may check whether the direction of movement of the second hand corresponds to the direction in which access is permitted to the hand of the identified shape. If the direction of movement of the second hand corresponds to the direction in which access is permitted to the hand of the identified shape, the wearable electronic device (601) may display a second object corresponding to the second hand in the virtual reality space.

[0135] According to one embodiment, the wearable electronic device (601) can perform hand tracking to track a second hand entering the field of view of the camera while displaying a first object corresponding to the first hand in the virtual reality space. The wearable electronic device (601) can determine whether the shape of the second hand is the shape of the left hand or the shape of the right hand. The wearable electronic device (601) can compare the shape of the first hand with the shape of the second hand. The wearable electronic device (601) can stop hand tracking to track the second hand in correspondence with the shape of the first hand being identical to the shape of the second hand.

[0136] According to one embodiment, the wearable electronic device (601) can determine whether the direction of movement of the second hand corresponds to the direction in which access is permitted to the hand of the confirmed shape, in correspondence with the shape of the first hand and the shape of the second hand being different from each other. If the direction of movement of the second hand corresponds to the direction in which access is permitted to the hand of the confirmed shape, the wearable electronic device (601) can display a second object corresponding to the second hand together with the first object in the virtual reality space.

[0137] According to one embodiment, the wearable electronic device (601) may output a second guide to determine whether the second hand is the user's hand in the virtual reality space when the direction of movement of the second hand corresponds to the direction in which access is allowed to the hand of the confirmed shape. The wearable electronic device (601) may determine the direction of the user's gaze tracked through the eye-tracking camera of the wearable electronic device. In response to determining that the second hand is the user's hand based on the direction of the user's gaze, the wearable electronic device (601) may display a second object corresponding to the second hand together with the first object in the virtual reality space.

[0138] According to one embodiment, the wearable electronic device (601) may display a second object corresponding to the second hand together with the first object in the virtual reality space in response to the reception of a preset gesture or voice input while outputting the second guide.

[0139] According to one embodiment, the wearable electronic device (601) may perform hand tracking to track a third hand entering the field of view of the camera while displaying the first object and the second object in the virtual reality space. The wearable electronic device (601) may output a third guide to request user verification in the virtual reality space. The wearable electronic device (601) may stop the hand tracking to track the third hand based on the user verification.

[0140] FIG. 8 is an example diagram illustrating the direction of movement of a hand entering the field of view of a camera according to one embodiment.

[0141] Referring to FIG. 8, since the user is wearing a wearable electronic device (601) on their head, when the user's left hand (810) enters the camera's field of view (or shooting area) (800), the direction of movement of the hand-tracked hand shape (811) may be in the form of moving from left to right (820). If the direction of movement of the hand-tracked hand shape (812) is in the form of moving from right to left (830), it may not be the user's hand.

[0142] Therefore, if the direction of movement of the user's left hand corresponds to the direction of movement from right to left rather than the direction of movement from left to right, the wearable electronic device (601) may determine that the hand that has entered the shooting area of ​​the camera (680) is the hand of a third party other than the user. According to one embodiment, if the wearable electronic device (601) determines that the hand is that of a third party other than the user, it may stop hand tracking of the third party's hand. Additionally, if the wearable electronic device (601) determines that the hand is that of a third party other than the user, it may request user confirmation by providing a notification to the user asking whether to stop hand tracking of the third party's hand.

[0143] FIG. 9a is a detailed operation flowchart of a wearable electronic device according to one embodiment. To aid in understanding the explanation of FIG. 9a, the explanation will be described with reference to FIG. 9b. FIG. 9b is an example diagram illustrating hand tracking for a hand other than the user's hand according to one embodiment.

[0144] Referring to FIG. 9a, in operation 905, the wearable electronic device (601) may start hand tracking to track a first hand entering the camera's field of view. In operation 910, the wearable electronic device (601) may check whether a single input or multiple input is set. For example, controlling one or more objects displayed in a virtual reality space with one hand may correspond to a single input, and controlling said objects with both hands may correspond to a multiple input. The single input or multiple input may be set via a menu according to user selection.

[0145] According to one embodiment, a single input may be an input by either of the user's two hands. According to one embodiment, a multi-input may include inputs by each of the user's two hands. Additionally, the multi-input may include inputs by an external user (another person) in addition to inputs by the user's hands.

[0146] In response to identifying that a single input is set, the wearable electronic device (601) may stop hand tracking tracking a second hand other than the first hand in operation 915. For example, if one of the user's two hands is recognized, the wearable electronic device (601) may stop hand tracking for the other hand so that the other hand is not additionally recognized. Additionally, for example, if one of the user's two hands is recognized, hand tracking for the external user's hand may stop so that the external user's hand is not additionally recognized.

[0147] On the other hand, in response to identifying that multi-input is set, the wearable electronic device (601) may, in operation 920, start hand tracking to track a second hand in addition to the first hand. According to one embodiment, the wearable electronic device (601) may output a guide to guide the user to confirm whether to allow hand tracking to track the second hand.

[0148] For example, in the case of multi-input corresponding to input by each of the user's two hands, even if hand tracking is performed while the first hand is recognized, if the user does not want hand tracking for the second hand, the user's selection can be received through the guide above.

[0149] For example, in the case of multi-input corresponding to input by another person's hand in addition to input by the user's hand, if the user allows input by another person, the user's choice can be received through the above guide to determine whether to allow hand tracking that tracks another person's hand in addition to the user's hand.

[0150] The wearable electronic device (601) can check whether the user allows hand tracking that tracks the second hand in the 925 operation. For example, the wearable electronic device (601) can display the guide and receive the user's selection regarding permission. In response to the user confirming permission, the wearable electronic device (601) can display a first object corresponding to the first hand and a second object corresponding to the second hand in a virtual reality space in the 930 operation.

[0151] For example, referring to FIG. 9b, in the case of multi-input corresponding to input by each of the user's two hands, objects (960, 970) corresponding to the first and second hands (e.g., both of the user's hands) may be displayed in the virtual space reality (e.g., HMD screen) (950).

[0152] On the other hand, in the case of multi-input corresponding to input by another person's hand in addition to input by the user's hand, an object (960 or 970) corresponding to the first hand (e.g., one of the user's two hands) and an object (980) corresponding to the second hand (e.g., another person's hand) may be displayed in the virtual space reality (e.g., HMD screen) (950).

[0153] Meanwhile, the wearable electronic device (601) can display a first object corresponding to the first hand in a virtual reality space in 935 operation in response to the user confirming that hand tracking tracking the second hand is not allowed.

[0154] For example, referring to FIG. 9b, in the case of multi-input corresponding to input by each of the user's two hands, an object (960 or 970) corresponding to either the first or second hand (e.g., both of the user's hands) may be displayed in the virtual space reality (e.g., HMD screen) (950).

[0155] FIG. 10 is an example diagram illustrating a case in which the direction of movement of a user's hand and the direction of movement of another person's hand are the same according to one embodiment.

[0156] As illustrated in FIG. 10, in the case of a third party (e.g., a parent) (1010) other than the user (1000) wearing the wearable electronic device (601), the other person's hand (1011, 1012) may enter the field of view (or shooting area) of the camera (680) in the same direction as the user's hand (1001, 1002). Objects (1011, 1012) representing the hands that have entered the field of view of the camera (680) may be displayed through a virtual reality space (e.g., an HMD screen) (1020). Additionally, the virtual reality space (1020) may be mirrored through an external electronic device connected to the wearable electronic device (601), allowing the other person to view the mirrored virtual reality space through the external electronic device. In this case, whether control by the other person's hand is allowed may be determined according to user settings. For example, based on at least one of the user's gaze, preset gesture, or voice input, the wearable electronic device (601) may receive an input that allows or does not allow control by another person's hand. For example, control of the wearable electronic device (601) may be possible even when a young child or a user with limited mobility is wearing the wearable electronic device (601).

[0157] FIG. 11 is an example diagram showing a guide requesting user confirmation according to one embodiment.

[0158] Referring to FIG. 11, when objects (1110, 1120) corresponding to both hands that have entered the field of view of the camera (680) in a virtual reality space (1100) are displayed, the wearable electronic device (601) may output (or display) a notification (1130) to check whether the user's hands are correct. The wearable electronic device (601) may receive confirmation input from the user based on at least one of the user's gaze, a preset gesture, or voice input. FIG. 11 illustrates a case where a message asking whether the user is correct is output, but a notification such as an indicator (e.g., an arrow) that induces the user's hands to enter the field of view of the camera (680) may also be output.

[0159] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0160] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0161] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0162] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0163] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or an application store (e.g., Play Store). TM It can be distributed online (e.g., downloaded or uploaded) through ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0164] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0165] According to one embodiment, in a non-transient storage medium storing instructions, the instructions are configured to cause the wearable electronic device to perform at least one operation when executed individually or collectively by at least one processor of the wearable electronic device, wherein the at least one operation may include performing hand tracking to track a first hand entering the field of view of the camera of the wearable electronic device.

[0166] According to one embodiment, the at least one operation may include an operation to determine whether the shape of the first hand is a left-hand shape or a right-hand shape.

[0167] According to one embodiment, the at least one operation may include an operation to determine whether the direction of movement of the first hand corresponds to a direction in which access is allowed to the hand having the identified shape.

[0168] According to one embodiment, the at least one operation may include an operation of displaying a first object corresponding to the first hand in a virtual reality space when the direction of movement of the first hand corresponds to a direction in which access to the hand having the identified shape is allowed.

Claims

1. In a wearable electronic device (200, 300, 601), Housing; A camera disposed on the first surface of the housing and configured to capture an image from the front of the wearable electronic device; display; At least one processor; and It includes a memory (310) for storing instructions, When the above instructions are executed individually or collectively by the at least one processor, the wearable electronic device, Hand tracking is performed to track a first hand entering the field of view of the above camera, and Check whether the shape of the first hand is a left-hand shape or a right-hand shape, and Check whether the movement direction of the first hand corresponds to the direction in which access is permitted to the hand having the confirmed shape, and An electronic device configured to display a first object corresponding to the first hand in a virtual reality space provided through the display when the direction of movement of the first hand corresponds to a direction in which access to the hand having the identified shape is permitted.

2. In paragraph 1, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device, If the direction of movement of the first hand does not correspond to the direction in which access is permitted to the hand of the confirmed shape, hand tracking of the first hand is stopped, and An electronic device configured to output a first guide to guide the user to enter the field of view of the camera.

3. In claim 1 or 2, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device, If the movement direction of the first hand does not correspond to the direction in which access is permitted to the hand of the confirmed shape, a second guide is output to guide the user to verify whether the first hand is the user's hand in the virtual reality space, and An electronic device configured to stop hand tracking of the first hand in response to the reception of a preset gesture or voice input while outputting the second guide, in response to identifying that the first hand is not the user's hand.

4. In any one of claims 1 to 3, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device, If the movement direction of the first hand does not correspond to the direction in which access is permitted for the hand of the confirmed shape, a first guide is output to guide the user to enter the field of view of the camera, and Hand tracking is performed to track a second hand entering the field of view of the camera while outputting the first guide, and Check whether the shape of the second hand is the shape of the left hand or the shape of the right hand, Check whether the movement direction of the second hand corresponds to the direction in which access is permitted to the hand of the confirmed shape, and An electronic device configured to display a second object corresponding to the second hand in the virtual reality space when the direction of movement of the second hand corresponds to the direction in which access to the hand of the identified shape is permitted.

5. In any one of paragraphs 1 through 4, When the above instructions are executed individually or collectively by the at least one processor, the wearable electronic device, Hand tracking is performed to track a second hand entering the field of view of the camera while displaying a first object corresponding to the first hand in the virtual reality space, and Check whether the shape of the second hand is the shape of the left hand or the shape of the right hand, Compare the shape of the first hand and the shape of the second hand, An electronic device configured to stop hand tracking that tracks the second hand in correspondence with the shape of the first hand and the shape of the second hand being identical.

6. In any one of claims 1 to 5, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device, In correspondence with the fact that the shape of the first hand and the shape of the second hand are different from each other, check whether the direction of movement of the second hand corresponds to the direction in which access is permitted to the hand of the confirmed shape, and An electronic device configured to display a second object corresponding to the second hand together with the first object in the virtual reality space when the direction of movement of the second hand corresponds to the direction in which access to the hand of the identified shape is permitted.

7. In any one of claims 1 to 6, further comprising an eye-tracking camera disposed on the second surface of the housing and configured to track the user's gaze, When the above instructions are executed individually or collectively by the at least one processor, the wearable electronic device, If the movement direction of the second hand corresponds to the direction in which access is permitted to the hand of the confirmed shape, a second guide is output to verify whether the second hand is the user's hand in the virtual reality space, and Check the direction of the user's gaze tracked through the eye-tracking camera, and An electronic device configured to display a second object corresponding to the second hand together with the first object in the virtual reality space, in response to confirming that the second hand is the user's hand based on the user's gaze direction.

8. In any one of claims 1 to 7, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device, An electronic device configured to display a second object corresponding to the second hand together with the first object in the virtual reality space in response to the reception of a preset gesture or voice input while outputting the second guide.

9. In any one of claims 1 through 8, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device, While displaying the first object and the second object in the virtual reality space, hand tracking is performed to track a third hand entering the field of view of the camera, Output a third guide for requesting user verification in the above virtual reality space, and An electronic device configured to stop hand tracking tracking the third hand based on the above user verification.

10. In any one of claims 1 to 9, when the instructions are executed individually or collectively by the at least one processor, the wearable electronic device, By hand-tracking at least one of the user's two hands, an object corresponding to the tracked hand is displayed in the virtual reality space, and An electronic device configured to perform control over one or more objects displayed in the virtual reality space by the tracked hand.

11. A method for hand tracking in a wearable electronic device (200, 300, 601), An operation to perform hand tracking that tracks a first hand entering the field of view of the camera of the above-mentioned wearable electronic device; An action to determine whether the shape of the first hand is a left-hand shape or a right-hand shape; An operation to determine whether the movement direction of the first hand corresponds to a direction in which access is permitted to the hand having the confirmed shape; and A method for hand tracking, comprising the operation of displaying a first object corresponding to the first hand in a virtual reality space when the direction of movement of the first hand corresponds to a direction in which access to the hand having the confirmed shape is allowed.

12. In Paragraph 11, An action of stopping hand tracking that tracks the first hand when the direction of movement of the first hand does not correspond to the direction in which access is permitted to the hand of the confirmed shape; and A method for hand tracking, further comprising the action of outputting a first guide to guide the user to enter the field of view of the camera.

13. In Paragraph 11 or 12, If the movement direction of the first hand does not correspond to the direction in which access is permitted to the hand of the confirmed shape, the operation of outputting a second guide to guide the user to confirm whether the first hand is the user's hand in the virtual reality space; and A method for hand tracking, further comprising the action of stopping hand tracking that tracks the first hand in response to identifying that the first hand is not the user's hand upon receiving a preset gesture or voice input while outputting the second guide.

14. In any one of paragraphs 11 through 13, If the movement direction of the first hand does not correspond to the direction in which access is permitted for the hand of the confirmed shape, an operation to output a first guide to guide the user to enter the field of view of the camera; An operation to perform hand tracking that tracks a second hand entering the field of view of the camera while outputting the first guide; An action to determine whether the shape of the second hand is the shape of the left hand or the shape of the right hand; An operation to determine whether the movement direction of the second hand corresponds to the direction in which access is permitted to the hand of the confirmed shape; and A method for hand tracking, further comprising the operation of displaying a second object corresponding to the second hand in the virtual reality space when the direction of movement of the second hand corresponds to the direction in which access to the hand of the identified shape is permitted.

15. In a non-transient storage medium storing instructions, said instructions are configured to cause the wearable electronic device to perform at least one operation when executed individually or collectively by at least one processor (620) of the wearable electronic device (200, 300, 601), said at least one operation being, said at least one operation An operation to perform hand tracking that tracks a first hand entering the field of view of the camera of the above-mentioned wearable electronic device; An action to determine whether the shape of the first hand is a left-hand shape or a right-hand shape; An operation to determine whether the movement direction of the first hand corresponds to a direction in which access is permitted to the hand having the confirmed shape; and A storage medium comprising an operation to display a first object corresponding to the first hand in a virtual reality space when the direction of movement of the first hand corresponds to a direction in which access is permitted to the hand having the confirmed shape.

Images