Wearable device for moving virtual object using gesture, and method therefor
The wearable device effectively addresses the challenge of maintaining virtual object movement by tracking user gestures and adjusting object positions within augmented reality environments, ensuring continuous interaction even when the hand is out of view, thus enhancing user experience.
Patent Information
- Application Number
- PCT/KR2024/001240
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-12
- Filing Date
- 2024-01-25
- Publication Date
- 2024-11-14
AI Technical Summary
Current wearable devices struggle to seamlessly integrate user gestures for moving virtual objects within augmented reality environments, particularly in maintaining object movement and position when the user's hand moves out of the camera's field of view, leading to disruptions and loss of functionality.
A wearable device equipped with a camera and processor that identifies body parts within its field of view, tracks their movement, and adjusts virtual object positions accordingly, ensuring continuous interaction by maintaining the object's last known position when the hand moves out of view and re-establishing the gesture when it re-enters the field of view.
This solution provides a seamless user experience by ensuring virtual objects follow the user's gestures without interruption, even when the hand is out of the camera's view, allowing for uninterrupted interaction and maintaining the intended functionality.
Smart Images

Figure KR2024001240_14112024_PF_FP_ABST
Abstract
Description
Wearable device and method for moving a virtual object using gestures
[0001] The present disclosure relates to a wearable device and method for moving a virtual object using gestures.
[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these electronic devices may be AR glasses and / or head-mounted devices (HMDs).
[0003] According to an embodiment, a wearable device may include a camera, a display, and at least one processor including a processing circuit. The at least one processor may be configured, individually and / or collectively, to identify a body part within a field-of-view (FoV) of the camera while displaying a virtual object within a virtual space via the display. The at least one processor may be configured, individually and / or collectively, to control the display to display the virtual object moving along a path of the body part moving within the FoV based on identifying the body part in a designated pose for movement of the virtual object. The at least one processor may be configured, individually and / or collectively, to control the display to display the virtual object at a second location within the virtual space corresponding to the first location based on identifying, using the camera, that the body part is moving outside the FoV through a first location at an edge of the FoV while moving the virtual object along the path. At least one processor may be configured, individually and / or collectively, to control the display to display the virtual object, based on identifying the body part of the designated pose that has been moved into the FoV, from the second location, and moving along the path of the body part within the FoV.
[0004] In one embodiment, a method of a wearable device may include an operation of identifying a body part within a field-of-view (FoV) of a camera of the wearable device based on a video acquired from a camera of the wearable device while displaying a virtual object within a virtual space through a display of the wearable device. The method may include an operation of displaying the virtual object moving along a path of the body part moving within the FoV based on the identification of the body part in a specified posture for movement of the virtual object. The method may include an operation of displaying the virtual object at a second location within the virtual space corresponding to the first location based on the identification, from the video, that the body part is moving out of the FoV through a first location of an edge of the FoV while moving the virtual object along the path. The method may include an action of displaying the virtual object, which is moved from the second location and moves along a path of the body part within the FoV, based on identifying the body part of the specified posture that has moved into the FoV after displaying the virtual object at the second location.
[0005] In one embodiment, a wearable device may include a camera, a display, and at least one processor including a processing circuit. The at least one processor may be configured, individually and / or collectively, to identify a body part that moves into a field-of-view (FoV) of the camera at a first timing based on the camera. The at least one processor may be configured, individually and / or collectively, to identify, based on identifying that a posture of the body part moved into the FoV at the first timing corresponds to a designated posture, at least one virtual object that was associated with the body part at a second timing prior to the first timing. The at least one processor may be configured, individually and / or collectively, to control the display to display the at least one virtual object based on a position of the body part within the FoV within the display.
[0006] In one embodiment, a method of a wearable device may include an operation of identifying a body part that moves into a FoV of a camera of the wearable device at a first timing based on a camera of the wearable device. The method may include an operation of identifying at least one virtual object that was connected to the body part at a second timing prior to the first timing based on identifying that a posture of the body part that moved into the FoV at the first timing corresponds to a designated posture. The method may include an operation of displaying at least one virtual object within a display of the wearable device based on a position of the body part within the FoV.
[0007] The above-described and other aspects, features, and advantages of some embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:
[0008] FIGS. 1A and 1B illustrate examples of screens displayed through a display of a wearable device according to one embodiment;
[0009] FIG. 2 illustrates an exemplary configuration of a block diagram of a wearable device according to one embodiment;
[0010] FIG. 3 is a flowchart illustrating exemplary operations of a wearable device according to one embodiment;
[0011] FIG. 4 is a diagram illustrating an exemplary operation performed by a wearable device to identify a body part;
[0012] FIG. 5 is a diagram illustrating exemplary operations performed by a wearable device to obtain information about a body part;
[0013] FIGS. 6A, 6B and 6C are diagrams illustrating exemplary operations of a wearable device performed based on movement of a body part;
[0014] FIG. 7a, FIG. 7b and FIG. 7c are diagrams illustrating exemplary operations of a wearable device performed based on movement of a body part;
[0015] FIG. 8 is a flowchart illustrating exemplary operations of a wearable device according to one embodiment;
[0016] FIGS. 9A, 9B and 9C are diagrams illustrating exemplary operations of a wearable device performed based on movement of body parts;
[0017] FIG. 10 is a diagram illustrating an exemplary operation of a wearable device performed based on movement of body parts;
[0018] FIG. 11A, FIG. 11B and FIG. 11C are diagrams illustrating exemplary operations of a wearable device performed based on movement of body parts;
[0019] FIG. 12A and FIG. 12B are diagrams illustrating exemplary operations of a wearable device performed based on movement of body parts;
[0020] FIG. 13a, FIG. 13b and FIG. 13c are diagrams illustrating exemplary operations of a wearable device performed based on movement of body parts;
[0021] FIGS. 14a, 14b and 14c are diagrams illustrating different states of a screen displayed by a wearable device based on the position of a body part relative to the field-of-view (FoV) of a camera;
[0022] FIG. 15 is a flowchart illustrating exemplary operations of a wearable device according to one embodiment;
[0023] FIG. 16A is a perspective view of an exemplary wearable device according to one embodiment;
[0024] FIG. 16B is a perspective view illustrating one or more hardware elements arranged within a wearable device according to one embodiment; and
[0025] FIGS. 17A and 17B are perspective views illustrating an example of the appearance of a wearable device according to one embodiment.
[0026] Hereinafter, various embodiments of the present disclosure are described in more detail with reference to the attached drawings.
[0027] The various embodiments of the present disclosure and the terminology used therein are not intended to limit the technology described in this document to a specific embodiment, but should be understood to include various modifications, equivalents, and / or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expression may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B", "at least one of A and / or B", "A, B, or C", or "at least one of A, B, and / or C" may include all possible combinations of the items listed together. Expressions such as "first", "second", "first", or "second" may modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), said component may be directly connected to said other component, or may be connected via another component (e.g., a third component).
[0028] The term "module" as used in this document includes a unit composed of hardware, and may be used interchangeably with terms such as "component" or "circuit." A module may be an integrally composed component, or a minimal unit or portion thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).
[0029] FIGS. 1A and 1B are diagrams of examples of screens displayed through a display of a wearable device (101), according to one embodiment. The wearable device (101) may include a head-mounted display (HMD) that is wearable on the head of a user (110). Although the external appearance of the wearable device (101) in the form of glasses is illustrated, the present disclosure is not limited thereto. An example of one or more hardware components included in the wearable device (101) is described as a non-limiting example with reference to FIG. 2. An example of the structure of the wearable device (101) that is wearable on the head of a user (110) is described in more detail with reference to FIGS. 16A and 16B and / or 17A and 17B. The wearable device (101) may be referred to as an electronic device (e.g., the electronic device (101-1) of FIGS. 7A to 7C). For example, the electronic device may be combined with an accessory to be attached to a user's head to form an HMD. The wearable device (101) may display the real world, a virtual world, or both the real world and the virtual world together.
[0030] According to one embodiment, a wearable device (101) may perform functions related to video see-through (VST) and / or virtual reality (VR). Referring to FIGS. 1A and 1B , when a user (110) wears the wearable device (101), the wearable device (101) may include a housing that covers the eyes of the user (110). In the state, the wearable device (101) may include a display disposed on a first side of the housing facing the eyes. The wearable device (101) may include a camera disposed on a second side opposite the first side. Using the camera, the wearable device (101) may acquire images and / or videos representing ambient light. The wearable device (101) can output the image and / or video within the display disposed on the first surface, thereby allowing the user (110) to recognize the ambient light through the display. The displaying area (or active area) of the display disposed on the first surface can be formed by one or more pixels included in the display. The wearable device (101) can synthesize a virtual object into the image and / or video output through the display, thereby allowing the user (110) to recognize the virtual object together with a real object recognized by the ambient light.
[0031] According to one embodiment, a wearable device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). As shown in FIGS. 1A and 1B , when a user (110) wears the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the eye of the user (110). The wearable device (101) may combine ambient light passing through the lens with light emitted from a display of the wearable device (101). A display area of the display may be formed within the lens through which the ambient light passes. Since the wearable device (101) combines the ambient light and the light emitted from the display, the user (110) can see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).
[0032] Referring to FIGS. 1A and 1B , according to one embodiment, a wearable device (101) can identify a body part, such as a hand (112), based on one or more cameras (e.g., a 3D camera) and / or sensors (e.g., a depth sensor, a proximity sensor, a time-of-flight (ToF) sensor). Referring to FIG. 1A , the wearable device (101) can identify a body part included in an external space that is sensible (e.g., able to be sensed) by the one or more cameras and / or sensors using data from the one or more cameras and / or sensors. Referring to FIG. 1A, the wearable device (101) can identify a hand (112) within a field-of-view (FoV) (120) of a camera positioned toward a direction (df) (e.g., a front direction of the wearable device (101). The FoV may refer to, for example, an external space detectable by the camera. The wearable device (101) identifying the hand (112) may include an operation of identifying at least one of a pose of the hand (112) within the FoV (120), a position of the hand (112) within the FoV (120), and / or a type of gesture corresponding to the pose, based on an image and / or video of the camera. An example of an operation of the wearable device (101) performed to identify the hand (112) is described in more detail with reference to FIGS. 4 and 5. Below, the operation of the wearable device (101) for identifying the hand (112) of the user (110) is described, but the present disclosure is not limited thereto.
[0033] Referring to FIGS. 1A and 1B , an exemplary screen (130) displayed by a wearable device (101) according to one embodiment is illustrated. The screen (130) may include a display area in which a virtual space, a real world, a mixed reality, or an extended reality is displayed through the display of the wearable device (101). For example, the wearable device (101) may display a three-dimensional virtual space surrounding the user's (110) avatar, having an arbitrary shape such as a sphere, a hemisphere, or a cylinder, with the user's (110) avatar as the center point. The screen (130) may be an area currently being displayed through the display of the wearable device (101) in the virtual space. When the user (110) turns his / her gaze, the wearable device (101) may display an area or an external space different from the area currently being displayed in the virtual space. That is, the external space is an area included in the virtual space, and at least a portion of the external space may include an area different from the currently displayed area. On the screen (130), for example, within the virtual space, the wearable device (101) may display one or more virtual objects to the user (110). Referring to FIGS. 1A and 1B , a virtual object (140) having a three-dimensional cylinder shape is exemplarily illustrated, but the present disclosure is not limited thereto. The virtual object (140) displayed within the screen (130) may include, for example, a window provided by a software application (e.g., a web browser and / or a gallery) executed by the wearable device (101). The window may have a two-dimensional rectangular shape, such as, for example, an activity of the Android operating system and / or, for example, a window (or pop-up window) of the Windows operating system. The virtual object (140) may include a two-dimensional and / or three-dimensional widget (e.g., a weather widget) provided by the software application.The present disclosure is not limited thereto, and the virtual object (140) may include any graphical object that can be displayed within the screen (130).
[0034] According to one embodiment, the wearable device (101) can substantially simultaneously display images including virtual objects (140) positioned at different locations to each of the user's (110) two eyes. The difference (or distance) between the locations of the virtual objects (140) commonly included in the images may be referred to as binocular parallax. By utilizing the binocular parallax, the wearable device (101) can adjust the distance perception of the virtual object (140) recognized by the user (110) wearing the wearable device (101). For example, the greater the binocular parallax of the virtual object (140), the more the user (110) viewing the virtual object (140) through the screen (130) can perceive that the virtual object (140) is positioned at a closer distance from the user (110).
[0035] According to one embodiment, the wearable device (101) can identify a body part (e.g., a hand (112)) within the FoV (120) of the camera of the wearable device (101) while displaying a virtual object (140) through the display. Based on identifying the body part within the FoV (120), the wearable device (101) can identify a posture of the body part and / or a position of the body part within the FoV (120). Based on the posture and / or the position, the wearable device (101) can change the display of the virtual object (140) displayed on the screen (130). For example, based on identifying a body part in a designated posture for movement of the virtual object (140), the wearable device (101) can display the virtual object (140) moving along a path of the body part moving within the FoV (120).
[0036] Referring to FIGS. 1A and 1B , the wearable device (101) can identify a posture of a hand (112) included in the FoV (120) of the camera using a camera. In the exemplary states of FIGS. 1A and 1B , based on identifying a hand (112) including a fingertip of an index finger in contact with a fingertip of a thumb, the wearable device (101) can identify a posture of the hand (112) corresponding to a pinch gesture. The present disclosure is not limited thereto, and the posture of the hand (112) corresponding to the pinch gesture can be identified based on the thumb and the index finger having a ring shape. In one embodiment, the wearable device (101) can identify an input indicating a movement of a virtual object (140) based on the posture of a body part (e.g., a hand (112)). For example, based on identifying a hand (112) having the posture of the pinch gesture illustrated in FIGS. 1A and 1B, the wearable device (101) can identify the input. For example, based on identifying that a position (pf1) of the hand (112) having the posture of the pinch gesture within the FoV (120) corresponds to a position (pv1) of a virtual object (140) within the screen (130), for example, within a virtual space, the wearable device (101) can identify the input. For example, a designated posture for moving the virtual object (140) may include the posture of the hand (112) illustrated in FIGS. 1A and 1B.
[0037] Referring to FIGS. 1A and 1B, in response to an input that is identified based on a hand (112) of a pinch gesture and indicates movement of a virtual object (140), the wearable device (101) can move the virtual object (140) along a path (114) of the hand (112). Referring to FIGS. 1A and 1B, the wearable device (101) can move the virtual object (140) within the screen (130) along a path (141) corresponding to the path (114) of the hand (112). For example, a wearable device (101) that identifies a hand (112) moving from a position (pf1) to a position (pf2) within the FoV (120) along a path (114) can move a virtual object (140) from a position (pv1) to a position (pv2) within a screen (130), for example, within a virtual space. In FIGS. 1A and 1B , the path (114) of the hand (112) is illustrated as if it were on a two-dimensional plane for convenience, but each of the positions (pf1), (pf2), (pf3), (pf4), (pf5), and (pf6) on the path (114) may have three-dimensional coordinate values (x, y, z). In addition, the format (for example, two-dimensional coordinates or three-dimensional coordinates) of the positions (pf1 to pf6) on the path (114) may be determined depending on the format (for example, two-dimensional plane or three-dimensional space) displayed on the screen (130). For example, when the screen (130) is a virtual space, the path (114) or positions (pf1 to pf6) of the hand (112) may have three-dimensional coordinate values, and the position of the object (140) in the virtual space may be displayed according to the path (114) of the hand (112). In addition, referring to FIGS. 1A and 1B, according to one embodiment, the wearable device (101) may display a virtual object (142) representing the hand (112). The virtual object (142) may be referred to as an avatar of the user (110). The virtual object (142) has a shape corresponding to the posture of the hand (112) and can be displayed based on a position in the virtual space corresponding to the position of the hand (112).Referring to FIG. 1A, a virtual object (142) having a shape of being held with a pinch gesture or a hand within a virtual space is exemplarily illustrated. In addition, instead of being displayed in the shape of a user's (110) hand, a shape of a pointer, a ray, a stick, or various tools may be displayed.
[0038] In one embodiment, the range (e.g., FoV (120)) over which the wearable device (101) can detect a body part using one or more cameras and / or sensors may be finite. In the exemplary state of FIGS. 1A and 1B where the user (110) wears the wearable device (101), the range (or area) over which the user (110) can move the hand (112) may be different from the FoV (120) of the camera of the wearable device (101). For example, a hand (112) moving along a path (114) may move out of the FoV (120) at a position (pf3) at the edge of the FoV (120). In a state where a virtual object (140) is moved along a path (114), the wearable device (101) can use a camera to identify that a body part (e.g., a hand (112)) is moved out of the FoV (120) through a position (pf3) of an edge of the FoV (120). Based on identifying that a body part (e.g., a hand (112)) is moved out of the FoV (120) through a position (pf3) of an edge of the FoV (120), the wearable device (101) can display a virtual object (140) at a position (pv3) of a display corresponding to the position (pf3).
[0039] In one embodiment, after the hand (112) is moved out of the FoV (120), the wearable device (101) may at least temporarily cease using the camera to identify the position and / or pose of the hand (112). Referring to FIG. 1A, while the hand (112) is moved along the path (114) from a position (pf3) at the edge of the FoV (120) to a position (pf4) outside the FoV (120), the wearable device (101), which acquires images and / or videos inside the FoV (120), may not be able to identify the hand (112). While the hand (112) is moved out of the FoV (120), the wearable device (101) may display a virtual object (140) based on the last position within the FoV (120) at which the hand (112) was identified. Within the exemplary state of FIG. 1A, the wearable device (101) can display a virtual object (140) at a location (pv3) within the screen (130) corresponding to the last location (pf3) within the FoV (120) that identified the hand (112). For example, while the hand (112) moves along the path (114) from a location (pf3) at the edge of the FoV (120) to a location (pf4) outside the FoV (120), the wearable device (101) can maintain the location of the virtual object (140) within the screen (130) at the location (pv3).
[0040] In one embodiment, after tracking of a hand (112) is stopped as it moves out of the FoV (120), the wearable device (101) can, based on identifying the hand (112) as moving into the FoV (120), re-execute a function that was being executed prior to stopping tracking of the hand (112). In the exemplary state of FIG. 1A, the wearable device (101) that identifies the hand (112) as moving into the FoV (120) via the location (pf5) of the edge of the FoV (120) along the path (114) can re-execute a function (e.g., a function of moving a virtual object (140) based on the path (114) of the hand (112)) that was being executed by the hand (112) prior to it moving out of the FoV (120) via the location (pf3). Because the above function is re-executed, the wearable device (101) can compensate for unintentional interruption of the function caused by the hand (112) moving out of the FoV (120).
[0041] Referring to FIG. 1A, after displaying a virtual object (140) at a location (pv3) based on a hand (112) before being moved out of the FoV (120), the wearable device (101) may identify a body part (e.g., hand (112)) in a specified posture (e.g., posture of a pinch gesture) that has been moved into the FoV (120), and display a virtual object (140) that has been moved from the location (pv3) and has been moved along a path (114) of the body part within the FoV (120). For example, the wearable device (101) that has identified a hand (112) in a posture of a pinch gesture that has been moved into the FoV (120) may change the location (pv3) of the virtual object (140) within the screen (130) based on the path (114) of the hand (112). For example, a wearable device (101) that identifies a hand (112) moving from a position (pf5) at the edge of the FoV (120) along a path (114) to a position (pf6) within the FoV (120) can move a virtual object (140) from a position (pv5) corresponding to the position (pf5) to a position (pv6) within the screen (130) along the path (114).
[0042] Referring to FIG. 1B, the posture of the hand (112) can change from a pinch gesture to another posture (e.g., a posture with all fingers spread out) different from the pinch gesture at a position (pf4) outside the FoV (120). When the hand (112) with the different posture moves into the FoV (120) through a position (pf5) of the edge of the FoV (120), the wearable device (101) can identify the different posture of the hand (112). Based on identifying the different posture from the pinch gesture of the hand (112) before it is moved out of the FoV (120), the wearable device (101) can not change the virtual object (140) displayed according to the position (pv3) of the hand (112) before it is moved out of the FoV (120) according to the position of the hand (112). For example, while a hand (112) having a different posture than a pinch gesture moves from a position (pf5) at the edge of the FoV (120) to a position (pf6) within the FoV (120), the wearable device (101) can independently display a virtual object (142) (e.g., an avatar representing the hand of the user (110)) that moves from a position (pv5) within the screen (130) corresponding to the position (pf5) to a position (pv6) within the screen (130) corresponding to the position (pf6).
[0043] As described above, according to one embodiment, the wearable device (101) can identify a gesture of a user (110) performed independently of a range detectable by a camera and / or sensor of the wearable device (101) (e.g., FoV (120)). The gesture can include a gesture performed by a hand (112) moving along a path (114). For example, the wearable device (101) that has identified a path (114) of a hand (112) that is discontinuously recognized as it moves out of the FoV (120) can continuously execute a function (e.g., a function of moving a virtual object (140)) that was performed by the hand (112) based on whether the posture of the hand (112) is maintained in a designated posture.
[0044] Hereinafter, with reference to FIG. 2, one or more hardwares included in the wearable device (101) and / or software applications executed by the wearable device (101) to perform the operations of the wearable device (101) described above with reference to FIGS. 1A and 1B are described in more detail.
[0045] FIG. 2 is a block diagram illustrating an exemplary configuration of a wearable device (101) according to one embodiment. The wearable device (101) of FIGS. 1A and 1B may be an example of the wearable device (101) described with reference to FIG. 2.
[0046] According to one embodiment, a wearable device (101) may include at least one of a processor (210) (e.g., including a processing circuit), a memory (215), a display (220), a camera (225), a sensor (230), and / or a communication circuit (240). The processor (210), the memory (215), the display (220), the camera (225), the sensor (230), and the communication circuit (240) may be electronically and / or operably coupled with each other by electronic components such as a communication bus (202). Hereinafter, operably coupled hardware may refer to a direct connection or an indirect connection established between the hardwares, either wired or wireless, such that a second hardware is controlled by a first hardware among the hardwares. Although illustrated based on different blocks, the present disclosure is not limited thereto, and some of the hardware of FIG. 2 (e.g., at least a portion of the processor (210), the memory (215), and the communication circuit (240)) may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of hardware included in the wearable device (101) is not limited to that illustrated in FIG. 2. For example, the wearable device (101) may include only some of the hardware illustrated in FIG. 2.
[0047] In one embodiment, the processor (210) of the wearable device (101) may include hardware for processing data based on one or more instructions. The hardware for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). The processor (210) may have a single-core processor structure, or a multi-core processor structure, such as a dual core, a quad core, or a hexa core. The processor (210) according to one embodiment of the present disclosure may include various processing circuits and / or a plurality of processors. For example, as used herein, including in the claims, the term "processor" may encompass various processing circuits, including at least one processor, wherein one or more of the at least one processors may be configured, individually and / or collectively, in a distributed manner, to perform the various functions described herein. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform several functions, these terms may, by way of non-limiting example, cover situations where one processor performs some of the recited functions, other processor(s) perform other of the recited functions, and may also cover situations where a single processor performs all of the recited functions. Additionally, the at least one processor may encompass a combination of processors, e.g., in a distributed manner, that perform the various functions recited / disclosed.At least one processor can execute program instructions to accomplish or perform various functions.
[0048] In one embodiment, the memory (215) of the wearable device (101) may include hardware for storing data and / or instructions input and / or output to the processor (210) of the wearable device (101). The memory (215) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, solid state drive (SSD), and embedded multi media card (eMMC).
[0049] In one embodiment, the display (220) of the wearable device (101) provides visualized information (e.g., the screen (130) of FIGS. 1A to 1B, FIGS. 4, FIG. 5, FIGS. 6A to 6C, FIGS. 7A to 7C, FIGS. 9A to 9C, FIGS. 10, FIGS. 11A to 11C, FIGS. 12A to 12B, FIGS. 13A to 13C, FIGS. 14A to 14C) to a user (e.g., the user (110) of FIGS. 1A to 1B, FIGS. 4, FIGS. 5, screen (620) of FIGS. 6A to 6C, screen (720) of FIGS. 7A to 7C, screen (920) of FIGS. 9A to 9C, screen (1020) of FIG. 10, and screen (11A to 11C)) to the user. The display (220) can output a screen (1120), a screen (1220) of FIGS. 12A to 12B, a screen (1320) of FIGS. 13A to 13C, and a screen (1410) of FIGS. 14A to 14C). For example, the display (220) can be controlled by a processor (210) including a circuit such as a GPU (graphic processing unit) to output visualized information to a user. The display (220) can include a flexible display, a flat panel display (FPD), and / or electronic paper. The flexible display can have a structure that is deformable by an external force. The FPD can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs). The display (220) of FIG. 2 may include at least one display (1650, 1750), which will be described in more detail with reference to FIGS. 16a and 16b, and / or FIGS. 17a and 17b.
[0050] In one embodiment, the camera (225) of the wearable device (101) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The plurality of optical sensors included in the camera (225) may be arranged in the form of a two-dimensional array. The camera (225) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously to generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (225) may refer to, for example, one (a) two-dimensional frame data acquired from the camera (225). For example, video data captured using the camera (225) may refer to, for example, a sequence of a plurality of two-dimensional frame data acquired from the camera (225) according to a frame rate. The camera (225) may be positioned toward the direction in which the camera (225) receives light, and may further include a flash light for outputting light toward the direction.
[0051] According to one embodiment, a sensor (230) of a wearable device (101) may generate electrical information that may be processed by a processor (210) and / or a memory (215) of the wearable device (101) from non-electronic information related to the wearable device (101). The information may be referred to as sensor data. The sensor (230) may include a global positioning system (GPS) sensor, an image sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor for detecting a geographic location of the wearable device (101), and an inertial measurement unit (IMU) for detecting a physical motion of the wearable device (101). In one embodiment, the IMU may include at least one of an acceleration sensor, a gyro sensor, and a gravity sensor. Using the IMU, the processor (210) of the wearable device (101) can identify the motion of the wearable device (101) based on six degrees of freedom (DoF). The motion of the wearable device (101) based on six degrees of freedom can include translation and rotation (e.g., roll, pitch, and yaw) of the wearable device (101) in three mutually perpendicular axes (e.g., x-axis, y-axis, and z-axis).
[0052] In one embodiment, the communication circuit (240) of the wearable device (101) may include hardware components for supporting transmission and / or reception of electrical signals between the wearable device (101) and an external electronic device (e.g., a remote controller (242)). The communication circuit (240) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (240) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G NR (new radio), and / or 6G.
[0053] Referring to FIG. 2, according to one embodiment, a wearable device (101) may receive a signal for executing at least one function supported by the wearable device (101) from a remote controller (242) connected via a communication circuit (240). For example, the remote controller (242) may include one or more buttons, a joystick, and / or a touch sensor for receiving user input. Based on the signal, the processor (210) of the wearable device (101) may execute the at least one function.
[0054] Although not illustrated, according to one embodiment, the wearable device (101) may include an output means for outputting information in a form other than a visualized form. For example, the wearable device (101) may include a speaker for outputting an acoustic signal. For example, the wearable device (101) may include a motor for providing haptic feedback based on vibration.
[0055] Referring to FIG. 2, according to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (210) of the wearable device (101) may be stored in the memory (215) of the wearable device (101). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or an application. Hereinafter, the fact that an application is installed in an electronic device (e.g., the wearable device (101)) may refer to, for example, that one or more instructions provided in the form of an application are stored in the memory (215), and that the one or more applications are stored in a format executable by the processor of the electronic device (e.g., a file having an extension designated by the operating system of the wearable device (101)). According to one embodiment, the wearable device (101) may execute one or more instructions stored in the memory (215) to perform, for example, the operations of FIG. 3, FIG. 8, and / or FIG. 15.
[0056] Referring to FIG. 2, programs installed in a wearable device (101) may be classified into one of different layers, including an application layer (260), a framework layer (270), and / or a hardware abstraction layer (HAL) (250), based on the target. For example, within the hardware abstraction layer (250), programs (e.g., drivers) designed to target the hardware of the wearable device (101) (e.g., a display (220), a camera (225), a sensor (230), and / or a communication circuit (240)) may be classified. For example, within the framework layer (270), programs designed to target at least one of the hardware abstraction layer (250) and / or the application layer (260) (e.g., an eye tracker (271), a gesture tracker (272), a motion tracker (273), an external space recognizer (274), and / or a virtual space manager (275)) may be classified. Programs classified within the framework layer (270) may provide an executable API (application programming interface) based on other programs.
[0057] Referring to FIG. 2, programs designed to target a user (e.g., the user (110) of FIGS. 1A to 1B) controlling a wearable device (101) may be classified within the application layer (260). For example, a program classified into the application layer (260) may include at least one of an application (261) for a VR-based game, an application (262) for playing and / or streaming video, an application (263) for viewing media content (e.g., images and / or videos) of memory (215), or an application (264) for call connection. The present disclosure is not limited thereto. For example, a program classified into the application layer (260) may cause execution of a function supported by programs classified into the framework layer (270) based on a call of an API.
[0058] Referring to FIG. 2, according to one embodiment, the processor (210) may process information related to the gaze of a user wearing the wearable device (101) based on the execution of the gaze tracker (271) within the framework layer (270). For example, the processor (210) may obtain an image including the user's eyes from a first camera positioned toward the user's eyes while the wearable device (101) is worn by the user. Based on the position and / or direction of the pupil included in the image, the processor (210) may identify the direction of the user's gaze.
[0059] Referring to FIG. 2, according to one embodiment, the processor (210) may identify a motion of a designated body part, including a hand, based on the execution of a gesture tracker (272) within the framework layer (270). For example, the processor (210) may obtain an image and / or video including the body part from a second camera. Based on the motion and / or posture of the designated body part, as represented by the image and / or video, the processor (210) may identify a gesture performed by the designated body part.
[0060] Referring to FIG. 2, according to one embodiment, the processor (210) may identify the motion of the wearable device (101) based on the execution of the motion tracker (273) within the framework layer (270). When the wearable device (101) is worn by a user, the motion of the wearable device (101) may be related to the motion of the user's head. For example, the processor (210) may identify an orientation of the wearable device (101) that substantially matches the orientation of the head. The processor (210) may identify the motion of the wearable device (101) based on sensor data of a sensor (230) including an IMU.
[0061] Referring to FIG. 2, according to one embodiment, the processor (210) may obtain information about an external space including a wearable device (101) based on the execution of an external space recognizer (274) within a framework layer (270). The information about the external space may include the shape and / or position of one or more external objects (e.g., a wall structure) included in the external space. The information about the external space may include a positional relationship of one or more external objects with respect to the wearable device (101). Based on the execution of the external space recognizer (274), the processor (210) of the wearable device (101) may obtain or generate the information representing the external space in three dimensions.
[0062] Referring to FIG. 2, according to one embodiment, the processor (210) may perform rendering for a virtual space based on the execution of the virtual space manager (275) within the framework layer (270). While the virtual space manager (275) is running, the processor (210) may identify a virtual space including at least one virtual object provided from at least one software application within the application layer (260). In one embodiment, based on the external space recognizer (274) and / or the virtual space manager (275), the processor (210) may identify a virtual space mapped to an external space. The processor (210) may determine a reference position within the virtual space based on a position and / or direction of the wearable device (101) in the external space, which is identified based on data from the sensor (230). The reference position may correspond to a position of the wearable device (101) in the virtual space. In one embodiment, the processor (210) may perform simultaneous localization and mapping (SLAM) to recognize external space and the location of the wearable device (101) within the external space based on the reference position. Based on the SLAM, the processor (210) may create or acquire a virtual space including information about the external space.
[0063] According to one embodiment, the processor (210) of the wearable device (101) can identify a body part (e.g., a hand (112) of FIGS. 1A and 1B) of a user wearing the wearable device (101) using the camera (225). Even if the body part moves out of the FoV (e.g., the FoV (120) of FIGS. 1A and 1B) of the camera (225), the processor (210) can re-execute a function that was executed before the body part was moved out of the FoV based on the posture of the body part moved into the FoV. For example, if the posture of the body part moved into the FoV corresponds to the posture of the body part before it was moved out of the FoV, the processor (210) can execute a function corresponding to the posture of the body part moved into the FoV based on a state in which the function was executed based on the body part before it was moved out of the FoV.
[0064] For example, as described in more detail with reference to FIGS. 3 through 7C, the processor (210) may, when moving a virtual object displayed on the display (220) by a body part in a specified posture, move the virtual object again based on the body part that has moved back into the FoV, even if the body part has moved out of the FoV. For example, as described in more detail with reference to FIGS. 8 through 10, the processor (210) may, when adjusting the size of the virtual object by one or more body parts, adjust the size of the virtual object again based on the one or more body parts that have moved back into the FoV, even if the one or more body parts have moved out of the FoV. The present disclosure is not limited thereto, and as described in more detail with reference to FIGS. 11A through 15, the processor (210) may, based on the posture of the body part that has moved into the FoV, display a virtual object that has been hidden by a body part that has moved back into the FoV.
[0065] Below, with reference to FIG. 3, the operation performed by the wearable device (101) of FIG. 2 is described in more detail.
[0066] FIG. 3 is a flowchart illustrating exemplary operations of a wearable device according to one embodiment. The wearable device (101) of FIG. 1A and FIG. 1B, FIG. 2, and / or the processor (210) of FIG. 2 may perform the operations described with reference to FIG. 3.
[0067] Referring to FIG. 3, in operation (310), a processor of a wearable device according to one embodiment may change the position of a virtual object displayed within a display based on identifying a gesture indicating movement of a virtual object using a camera. The camera of operation (310) may include camera (225) of FIG. 2. While operations of a wearable device and / or processor based on a camera are described, the present disclosure is not limited thereto. For example, the processor of the wearable device may identify the gesture of operation (310) using a depth sensor included in the wearable device.
[0068] In one embodiment, the processor can identify a body part (e.g., a hand (112) of FIGS. 1A and 1B) moving within a FoV of a camera (e.g., a FoV (120) of FIGS. 1A and 1B) using images and / or videos acquired from the camera. The processor can identify the body part by performing object recognition on the images and / or videos. For example, the processor can identify a visual object corresponding to the body part within the images and / or videos based on the object recognition. Based on a pose of the body part identified based on the visual object (e.g., the pose of the pinch gesture illustrated in FIGS. 1A and 1B), the processor can identify the gesture of the operation (310). The present disclosure is not limited thereto, and the processor can identify the gesture of the operation (310) based on a software application executed by the processor. For example, a gesture indicating movement of a virtual object may correspond to a posture defined by the software application.
[0069] In one embodiment, a processor that identifies a gesture indicating movement of a virtual object of action (310) can change the position of a virtual object (e.g., a virtual object (140) of FIGS. 1A-1B) within a screen displayed on a display (e.g., a screen (130) of FIGS. 1A-1B). The processor can change the position of the virtual object within the virtual space based on the gesture of action (310). The processor can change the position of the virtual object within the display based on the position within the FoV of the body part performing the gesture of action (310). The relationship between the FoV of the camera and the screen displayed on the display is described in more detail with reference to FIG. 4.
[0070] Referring to FIG. 3, in operation (320), a processor of a wearable device according to an embodiment may identify whether a body part performing a gesture moves out of the FoV of a camera. Before the body part moves out of the FoV, or while the body part is moving within the FoV (320-No), the processor may change the position of the virtual object based on operation (310). While the body part is moving within the FoV, the processor may change the position of the virtual object based on a path (e.g., path (114) of FIG. 1A) of the body part performing the gesture of operation (310). While moving the virtual object (140) of FIG. 1A based on operation (310), the processor may move the virtual object (140) along a path (141).
[0071] Referring to FIG. 3, based on identifying a body part that has moved out of the FoV of the camera based on operation (320) (320-Yes), the processor of the wearable device may perform operation (330). In operation (330), the processor may set a position of a virtual object in the virtual space based on a first position of the body part identified by the camera within the FoV. The processor may determine a position of the virtual object based on the first position of the body part within the FoV before it has moved out of the FoV. The processor may place the virtual object at a position in the virtual space corresponding to the first position. For example, a virtual object that was being moved within the display and / or virtual space based on operation (310) may be placed at the first position of the body part last identified by the camera based on operation (330).
[0072] In one embodiment, the processor may set the position of the virtual object within the virtual space based on the relative positional relationship between the wearable device and the first location. For example, the processor may set the position of the virtual object within the virtual space based on the positional relationship between the first location and the wearable device at a time when the body part is moved outside the FoV. In the example, when a user wearing the wearable device moves, the processor may move the virtual object within the virtual space using the positional relationship. In the example, the user who views the virtual object moved within the virtual space based on the positional relationship may perceive that the virtual object follows the user. For example, the position of the virtual object set based on operation (330) may be interlocked with the position of the wearable device.
[0073] Referring to FIG. 3, after setting the position of the virtual object based on the first position of the operation (330), the processor can obtain information about the external space using the camera. In operation (340) of FIG. 3, the processor of the wearable device according to one embodiment can identify a body part that moves into the FoV of the camera. Before identifying the body part within the FoV (340-No), or if the body part is not identified within the FoV, the processor can maintain the position of the virtual object within the virtual space set based on operation (330). Based on identifying the body part that has moved into the FoV of the camera (340-Yes), the processor can perform operation (350). For example, when the hand (112) of FIG. 1A moves into the FoV (120) from the position (pf5) of the edge of the FoV (120), the processor can perform operation (350).
[0074] Referring to FIG. 3 , in operation (350), a processor of a wearable device according to one embodiment may identify a gesture indicating movement of a virtual object based on a body part moved into the FoV. For example, the processor may identify whether the identified body part using a camera corresponds to (or matches) a body part that was moved into the FoV. For example, the processor may identify whether the posture of the body part moved into the FoV corresponds to the gesture of operation (310). If the gesture of operation (350) is not identified (350-No), or before identifying the gesture of operation (350), the processor may perform operation (360). For example, if the posture of the body part moved into the FoV does not correspond to the gesture of operation (310), the processor may perform operation (360). Based on identifying a gesture of action (350) from a body part moved into the FoV (350-example), the processor can perform an action (370).
[0075] Referring to FIG. 3, in operation (360), according to one embodiment, a processor of a wearable device may determine a location of a virtual object in a virtual space based on a first location. The processor may determine a last location of the virtual object that was changed based on operation (310) as the location of the virtual object. The processor may determine a location of the virtual object in a virtual space independently of a body part moved into the FoV based on identifying a gesture that is different from the gesture of operation (350). The processor, which placed the virtual object at the first location linked to the location of the wearable device based on operation (330), may perform operation (360) to disable the link between the location of the virtual object in a virtual space based on the first location and the location of the wearable device. For example, based on operation (360), the processor may set the location of the virtual object in a virtual space to coordinate values in the virtual space corresponding to the first location.
[0076] Referring to FIG. 3 , in operation (370), a processor of a wearable device according to one embodiment may display a virtual object within a display based on a second position within the FoV of a body part moved into the FoV. The processor may resume changing the position of the virtual object based on the second position, similar to operation (310). Based on operation (370), the processor may move a virtual object positioned at a position within the virtual space corresponding to the first position to the second position of the body part moved into the FoV. For example, the processor changing the position of the virtual object based on operation (370) may be performed similarly to operation (310).
[0077] As described above, according to one embodiment, the processor of the wearable device can identify, based on the re-entry of the body part into the FoV, whether the first posture (or first gesture) of the body part corresponds to a second posture (or second gesture) of the body part that was identified prior to the re-entry. If the first posture corresponds to the second posture, the processor can re-execute a function that was executed based on the second posture (e.g., a function of changing the position of the virtual object of operation (310). For example, the processor can re-execute the function that was interrupted due to a mismatch between the range of motion of the body part and the FoV.
[0078] Hereinafter, exemplary operations of a wearable device for identifying gestures performed by a body part, such as operations (310, 340, 350) of FIG. 3, are described in more detail with reference to FIG. 4 and / or FIG. 5.
[0079] FIG. 4 is a diagram illustrating exemplary operations performed by a wearable device (101) to identify a body part. The wearable device (101) of FIGS. 1A and 1B, FIG. 2, and / or the processor (210) of FIG. 2 may perform the operations described with reference to FIG. 4. The operations of the wearable device (101) described with reference to FIG. 4 may be related to at least one of the operations of FIG. 3.
[0080] Referring to FIG. 4, an exemplary state (401) of a wearable device (101) that identifies a body part (e.g., a hand (112)) within a FoV (120) is illustrated. Using images and / or videos acquired from a camera (e.g., camera (225) of FIG. 2) of the wearable device (101), the wearable device (101) can identify the body part (e.g., the hand (112)) within the FoV (120). For example, the wearable device (101) can identify a portion (410) corresponding to the hand (112) within the image acquired from the camera. In one embodiment, the wearable device (101) can identify a gesture performed by the hand (112) using the portion (410) of the image corresponding to the hand (112).
[0081] According to one embodiment, the wearable device (101) can identify the posture of the hand (112) based on one or more feature points included in the portion (410). The present disclosure is not limited thereto, and the wearable device (101) can identify the posture of the hand (112) using a neural network trained to identify the hand (112). The neural network can have a structure based on, for example, but not limited to, a convolutional neural network (CNN).
[0082] Referring to FIG. 4, portions (411, 412, 413) of an image corresponding to a hand (112) in different poses are exemplarily illustrated. When an image including a portion (411) is identified, the wearable device (101) can identify a pinch gesture exhibited by the thumb and index finger of the hand (112) connected in a ring shape. When an image including a portion (412) is identified, the wearable device (101) can identify a pointing gesture exhibited by an extended index finger and folded other fingers. When an image including a portion (412) is identified, the wearable device (101) can identify a clenched gesture exhibited by five folded fingers.
[0083] In one embodiment, a wearable device (101) that identifies a hand (112) within the FoV (120) may execute a function corresponding to a gesture of the hand (112). The function may include a function for moving a virtual object (e.g., the virtual object (140) of FIGS. 1A and 1B) as described with reference to FIGS. 1A and 1B . A pair of a gesture of the hand (112) and a function matching the gesture may be set by a software application executed by the wearable device (101). Based on identifying a gesture of grasping an object (e.g., a gesture of a hand (112) identified by parts (411, 412, 413)) or pointing to an object (e.g., a gesture of a hand (112) identified by parts (412)), the wearable device (101) can execute a function for moving a virtual object.
[0084] According to one embodiment, the wearable device (101), while executing a function for moving a virtual object, can adjust the position of the virtual object based on a reference point of a body part (e.g., a hand (112)) within the FoV (120). The wearable device (101) that has identified an image including a portion (411) can adjust the position of the virtual object based on the positions (a1) of the fingertips of the hands (112) that are in contact with each other within the image. The wearable device (101) that has identified an image including a portion (412) can adjust the position of the virtual object based on the positions (a2) of the fingertips of the spread fingers within the image. The wearable device (101) that has identified an image including a portion (413) can adjust the position of the virtual object based on the positions (a3) within the image corresponding to the center of the hand (112).
[0085] According to one embodiment, a wearable device (101) may display a screen (130) based on AR, VR, and / or VST. The size of the screen (130) displayed through the display of the wearable device (101) (e.g., the display (220) of FIG. 2) may be controlled by a software application executed by a processor of the wearable device (101) (e.g., the processor (210) of FIG. 2). For example, the size of the screen (130) may be controlled independently from the size of the FoV (120) of the camera of the wearable device (101). Referring to FIG. 4, an exemplary state (401) of the FoV (120) formed toward a direction (df) corresponding to the front of a user (110) wearing the wearable device (101) and the screen (130) displayed to the user (110) is illustrated.
[0086] In an exemplary state (401) of FIG. 4, the wearable device (101) can display a screen (130) having a size smaller than the size of the FoV (120) to the user (110). When the wearable device (101) moves a virtual object within the screen (130) based on the position of the hand (112) within the FoV (120), the wearable device (101) can identify a position within the screen (130) corresponding to the position of the hand (112) based on the relationship between the FoV (120) and the screen (130). While providing a user experience based on VST, the wearable device (101) can display an image acquired through a camera on the screen (130). Based on the image for FoV (120), a user (110) wearing a wearable device (101) can recognize the external environment in the direction (df) (e.g., forward). The wearable device (101) can synthesize a virtual object on the image and display a screen (130) in which the virtual object and the external environment are mixed to the user (110).
[0087] In one embodiment, within a screen (130) where a virtual object and an external environment are mixed, the wearable device (101) can display a virtual object (e.g., a virtual object (142) of FIG. 1) overlapping a hand (112) visible to the user (110) through the image. The wearable device (101), which has identified an input for moving the virtual object based on the hand (112), can move the virtual object based on the position of the hand (112) visible through the screen (130). For example, while the hand (112) is moved within the FoV (120), the wearable device (101) can adjust the position of the virtual object within the screen (130) based on the position of the hand (112) within the FoV (120). While the hand (112) is moved within the region (r1) where the FoV (120) and the screen (130) overlap each other, the user (110) wearing the wearable device (101) can simultaneously view the hand (112) and the virtual object moving based on the hand (112) through the screen (130). While the hand (112) is moved within the region (r2) of the FoV (120) outside the screen (130), the user (110) wearing the wearable device (101) cannot view the hand (112) through the screen (130), but can view the virtual object moving based on the hand (112). While the hand (112) is moved in an area (r3) outside the FoV (120), the user (110) wearing the wearable device (101) cannot see the hand (112) and can see that the virtual object moved by the hand (112) is stationary.
[0088] According to one embodiment, the wearable device (101) can re-execute a function that was performed by a gesture of the hand (112) based on the re-entry of the hand (112) that has moved into an area (r3) outside the FoV (120). The wearable device (101) can identify, based on the posture and / or gesture of the hand (112) identified using the image of the camera, whether the hand (112) that has moved into the FoV (120) corresponds to a body part that has moved out of the FoV (120).
[0089] Hereinafter, with reference to FIG. 5, the operation performed by the wearable device (101) that identifies the hand (112) moved into the FoV (120) is described in more detail.
[0090] FIG. 5 is a diagram illustrating exemplary operations performed by a wearable device (101) to obtain information about a body part. The wearable device (101) of FIGS. 1A and 1B, FIG. 2, and / or the processor (210) of FIG. 2 may perform the operations described with reference to FIG. 5. The operations of the wearable device (101) described with reference to FIG. 5 may be related to at least one of the operations of FIG. 3 (e.g., operation (340) of FIG. 3).
[0091] Referring to FIG. 5, an embodiment of a wearable device (101) that identifies a body part (e.g., a hand (112)) based on a camera (e.g., a camera (225) of FIG. 2) is illustrated. The wearable device (101) can obtain information about an area in which the body part can move using the camera. For example, the wearable device (101) can identify or measure a range (or area) in which the hand (112) can move using an image and / or video of the FoV (120) of the camera. For example, the wearable device (101) can obtain information for distinguishing the hand (112) using the image and / or video. The wearable device (101) can obtain the information based on the shape, skin color, size of the hand (112) and / or one or more accessories attached to the hand (112).
[0092] In one embodiment, the information acquired by the wearable device (101) may be used to identify the identity of body parts (e.g., hands (112)) identified in different time intervals. For example, the wearable device (101), which acquired information about a hand (112) that moved within the FoV (120) in a first time interval, may use the information to identify a body part that moves within the FoV (120) in a second time interval after the first time interval, to identify whether the body part and the hand (112) match each other.
[0093] Referring to FIG. 5, the wearable device (101) that has acquired information about the hand (112) of the user (110) can identify the hand (112) of the user (110) based on the information when the hands of different users included in the FoV (120) are identified. For example, the wearable device (101) can identify an area in which the hand (112) identified from the information can move. The wearable device (101) can determine one hand included in the area among the plurality of hands as the hand (112) of the user (110). The wearable device (101) can identify the hand (112) of the user (110) among the plurality of hands based on the shape, skin color, and / or size of the hand (112) identified from the information within the area. The wearable device (101) can identify the hand (112) of the user (110) among the plurality of hands based on one or more accessories attached to the hand (112) identified from the information within the area.
[0094] Referring to FIG. 5, an exemplary state of a wearable device (101) identifying a hand (112) is illustrated. According to one embodiment, the wearable device (101) may output a message to a user (110) for obtaining information about an area of the FoV (120) in which the hand (112) can move. The message may include designated text for obtaining an image and / or video of the area in which the hand (112) can move, such as, for example, “Rotate your arm greatly.” While outputting the message, the wearable device (101) may use the image and / or video of the FoV (120) to identify a position (ph1) of the hand (112) of the user (110) and a path and / or range of the hand (112) moving along one of the directions (dh1, dh2, dh3, dh4). The wearable device (101) can identify a hand (112) displayed at a position (ih1) corresponding to the position (ph1) in the image and / or the video and moving along directions (id1, id2, id3, id4) corresponding to each of the directions (dh1, dh2, dh3, dh4). Based on the image and / or the video, the wearable device (101) can identify a position (ph1) of the hand (112) and / or positions (j1, j2) of one or more joints connected to the hand (112). The one or more joints may be included in an arm of the user (110).
[0095] Referring to FIG. 5, according to one embodiment, the wearable device (101) can identify a range (510) within the FoV (120) in which the hand (112) of the user (110) can move based on the movement of the hand (112) within the FoV (120). Based on identifying the hand (112) moving from outside the FoV (120) to the range (510) of the FoV (120), the wearable device (101) can determine the identified hand (112) as the hand (112) of the user (110). According to one embodiment, the wearable device (101) can identify whether the hand (112) identified based on the range (510) of the FoV (120) is the hand (112) of the user (110) based on the direction (iv1) of the hand (112) with respect to the arm. For example, the wearable device (101) can identify whether the hand (112) included in the image is the hand (112) of the user (110) by comparing the direction (iv1) identified from the image for the FoV (120) and the direction (av) of the hand (112) with respect to the arm of the user (110) within the external space.
[0096] As described above, according to one embodiment, the wearable device (101) can determine whether a body part identified from an image and / or video for the FoV (120) corresponds to a body part that has moved out of the FoV (120). The wearable device (101) can use information about an area (e.g., range (510)) in which the body part can move to identify whether the body part within the FoV (120) is a designated body part for identification of a gesture (e.g., a hand (112) of a user (110)). Within the exemplary state of FIG. 5, the wearable device (101) that has identified the hand (112) of the user (110) can identify an input for controlling at least one virtual object displayed through the display of the wearable device (101) (e.g., the display (220) of FIG. 2) based on the posture and / or gesture of the hand (112).
[0097] Hereinafter, exemplary operations of the wearable device (101) related to the path of the hand (112) identified based on the FoV (120) are described in more detail with reference to FIGS. 6a, 6b and 6c and / or FIGS. 7a, 7b and 7c.
[0098] FIGS. 6A, 6B, and 6C are diagrams illustrating exemplary operations of a wearable device (101) performed based on movement of a body part. The wearable device (101) of FIGS. 1A and 1B, FIG. 2, and / or the processor (210) of FIG. 2 may perform the operations described with reference to FIGS. 6A, 6B, and 6C. The operations of the wearable device (101) described with reference to FIGS. 6A, 6B, and 6C may be related to at least one of the operations of FIG. 3.
[0099] Referring to FIGS. 6A, 6B, and 6C, exemplary states (601, 602, 603) of a wearable device (101) are illustrated in which a hand (112) of a user (110) is identified within a FoV (610) of one or more cameras (e.g., camera (225) of FIG. 2) positioned toward one direction of the wearable device (101) (e.g., toward the front of the wearable device (101)). In one embodiment, the wearable device (101) can identify a body part, such as a hand (112), using an image and / or video corresponding to the FoV (120).
[0100] Within the state (601) of FIG. 6A, the wearable device (101) can display a screen (620) including a virtual object (630) on a display (e.g., display (220) of FIG. 2). When the wearable device (101) displays a screen (620) based on VST, the wearable device (101) can display a screen (620) including an image for the FoV (120) and all of the virtual objects (630). When the wearable device (101) displays a screen (620) based on VR, the wearable device (101) can display a screen (620) representing a virtual environment that completely obscures the external space.
[0101] Within the state (601) of FIG. 6a, the wearable device (101) can identify a gesture of a hand (112) indicating movement of a virtual object (630) within a screen (620). The wearable device (101) can identify a pinch gesture of a hand (112) performed within the FoV (610) based on the operations described above with reference to FIGS. 4 and 5 . Although the operation of the wearable device (101) based on the pinch gesture is described, the present disclosure is not limited thereto. For example, the wearable device (101) can perform the operations of FIGS. 6a, 6b, and 6c based on identifying a pointing gesture. For example, based on identifying a hand (112) including one finger extended toward a location (fp1) within the FoV (610), the wearable device (101) can identify the pinch gesture performed by the hand (112). The wearable device (101) can identify a location (ip1) within the screen (620) corresponding to the location (fp1) within the FoV (610) corresponding to the fingertip of the extended finger. Based on identifying a location (ip1) superimposed on a virtual object (630), the wearable device (101) can identify an input indicating movement of the virtual object (630).
[0102] Referring to FIG. 6A, within a state (601) in which an input indicating movement of a virtual object (630) based on a pinch gesture is identified, the wearable device (101) can identify a path (640) along which a hand (112) moves within the FoV (610). Based on the path (640), the wearable device (101) can change the position of the virtual object (630) within the display. Within the state (601) of FIG. 6A, the wearable device (101) can move the virtual object (630) along a path (642) corresponding to the path (640) of the hand (112), based on identifying the hand (112) that is moved from a position (fp1) along the path (640) and maintains the pinch gesture. For example, the wearable device (101) can move a virtual object (630) from a location (ip1) within the screen (620) along a path (642). The shape of the path (642) along which the virtual object (630) moves within the screen (620) can be matched to the path (640) along which the hand (112) moves within the FoV (610). For example, the wearable device (101) can provide a user experience in which the virtual object (630) is connected to the hand (112) that moves within the FoV (610).
[0103] In state (601) of FIG. 6A, the wearable device (101) may stop the movement of the virtual object (630) based on the release of the pinch gesture performed by the hand (112). For example, if the posture of the hand (112) being moved within the FoV (610) changes to a posture different from the posture of the pinch gesture, the wearable device (101) may stop moving the virtual object (630) based on the position and / or path of the hand (112) within the FoV (610). In the example, the position of the virtual object (630) within the screen (620) may be fixed to the position of the hand (112) within the FoV (610) at the time when the posture of the hand (112) changes to the different posture.
[0104] Referring to FIG. 6A, as the hand (112) moves out of the FoV (610) along the path (640), the wearable device (101) can display a virtual object (630) based on the position (ip2) of the edge of the FoV (610). The position (ip2) within the FoV (610) corresponding to the position of the virtual object (630) within the screen (620) can correspond to the position of the hand (112) within the FoV (610) that was last identified before the hand (112) was moved out of the FoV (610). For example, the wearable device (101) can display the virtual object (630) on the screen (620) based on the position of the hand (112) within the FoV (610) at the timing when the hand (112) is moved out of the FoV (610). The wearable device (101) can display a virtual object (630) based on the location (ip2) of the intersection of the path (640) of the hand (112) and the edge of the FoV (610).
[0105] According to one embodiment, the wearable device (101) may display a virtual object (630) at a location connected to the external space within a state (601) in which the virtual object (630) is displayed based on a virtual space mapped to the external space. As the location of the virtual object (630) is linked to the external space, the wearable device (101) may move the virtual object (630) within the screen (620) similarly to being fixed to a specific location in the external space. Within the state (601), based on identifying a hand (112) having a posture of a pinch gesture, the wearable device (101) may disconnect the connection between the virtual object (630) and the external space. For example, based on a hand (112) moving along a path (640), the wearable device (101) may display a virtual object (630) moving along a path (642). In a state (601) of displaying a virtual object (630) based on the position (ip2) of the edge of the FoV (610) by the hand (112) moved outside the FoV (610), the wearable device (101) can display the virtual object (630) independently of the position of the virtual object (630) with respect to the external space (e.g., the absolute position of the virtual object (630) with respect to the external space). For example, the wearable device (101) can display the virtual object (630) based on the position of the virtual object (630) within the screen (620) (e.g., the relative position of the virtual object (630) with respect to the wearable device (101).
[0106] Referring to FIG. 6B, in a state (602) after the state (601) of FIG. 6A, the wearable device (101) can identify a body part (e.g., a hand (112)) that moves into the FoV (610) of the camera. For example, a user (110) can move a hand (112) in a pinch gesture posture from a location (fp3) outside the FoV (610) into the FoV (610) along a path (650). At the timing when the hand (112) moves into the FoV (610) through a location (fp4) of the edge of the FoV (610) along the path (650), the wearable device (101) can identify the hand (112) based on an image and / or video of the FoV (610). At a timing when moving into the FoV (610), within a state (602) in which a hand (112) of a designated posture indicating movement of a virtual object (630) is identified, the wearable device (101) can identify a virtual object (630) that was connected to the hand (112) prior to the timing. For example, the wearable device (101) can identify a virtual object (630) that was moved by the hand (112) in a state (601) prior to the state (602).
[0107] In the state (602) of FIG. 6b, the wearable device (101) that has identified a virtual object (630) moved by the hand (112) can display the virtual object (630) based on the position of the hand (112) within the FoV (610). For example, the wearable device (101) can display the virtual object (630) that is moved within the screen (620) based on the path of the hand (112) within the FoV (610). For example, the wearable device (101) that has identified the hand (112) that has moved within the FoV (610) from the position (fp4) along the path (650) can move the virtual object (630) to the position (ip4) within the screen (620) corresponding to the position (fp4). The wearable device (101) can move a virtual object (630) moved to a location (ip4) along a path (652) within a screen (620) corresponding to the path (650) based on identifying a path (650) within the FoV (610) of the hand (112). Referring to FIG. 6B, while the hand (112) sequentially passes through locations (fp4, fp5, fp6) on the path (650), the wearable device (101) can move the virtual object (630) along locations (ip4, ip5, ip6) within the screen (620) corresponding to each of the locations (fp4, fp5, fp6).
[0108] Although the operation of the wearable device (101) for a hand (112) once moved outside the FoV (610) has been described based on the states (601, 602) of FIGS. 6A and 6B, the present disclosure is not limited thereto. For example, if the hand (112) is repeatedly moved from the edge of the FoV (610) (e.g., a gesture of waving the hand (112)), the wearable device (101) may continue to execute a function corresponding to the posture and / or gesture of the hand (112) based on whether the posture and / or gesture of the hand (112) is maintained.
[0109] Referring to FIG. 6C, in a state (603) after the state (601) of FIG. 6A, the wearable device (101) can identify a body part that moves into the FoV (610) of the camera. In a state (603) after a hand (112) having a pinch gesture posture moves out of the FoV (610), the wearable device (101) can identify a hand (112) having a posture different from the posture of the pinch gesture. For example, when a user (110) moves a fully extended hand (112) from a location (fp7) outside the FoV (610) into the FoV (610) along a path (660), the wearable device (101) can execute a different function corresponding to the pinch gesture based on the posture of the hand (112).
[0110] For example, at a timing when a hand (112) is identified moving into the FoV (610) through a position (fp8) of the edge of the FoV (610) along a path (660), the wearable device (101) can display a virtual object representing the hand (112) at a position (ip8) within the screen (620) corresponding to the position (fp8). While the hand (112) is moving from the position (fp8) to the position (fp9) along the path (660), the wearable device (101) can move the virtual object displayed at the position (ip8) to the position (ip9) within the screen (620) corresponding to the position (fp9) along a path (662) corresponding to the path (660). Because the wearable device (101) executes a different function than the function corresponding to the pinch gesture, the wearable device (101) may not move the virtual object (630) displayed at the location (ip2) within the screen (620) based on the path (660) of the hand (112).
[0111] Referring to FIG. 6c, when re-entry of a hand (112) moved out of the FoV (610) is identified, and the posture of the hand (112) is different from the posture of the hand (112) before being moved out of the FoV (610), the wearable device (101) may determine the position of the virtual object (630) as the position (ip2) to which the hand (112) was moved before being moved out of the FoV (610). The present disclosure is not limited thereto, and the wearable device (101) may determine the position of the virtual object (630) as the position of the virtual object (630) before being moved by the hand (112) before being moved out of the FoV (610). When the posture of the hand (112) is different from the designated posture for movement of the virtual object (630), the wearable device (101) may display the virtual object (630) in conjunction with an external space. Within the state (603) of FIG. 6c, the wearable device (101) can display a virtual object (630) based on a location within the external space corresponding to a location (ip2) within the screen (620).
[0112] As described above, the wearable device (101) that tracks the hand (112) within the FoV (610) can continuously execute the function that was executed by the tracking of the hand (112) based on the re-entry of the hand (112) into the FoV (610) after the tracking of the hand (112) is stopped as the hand (112) moves out of the FoV (610). Although the wearable device (101) in the form of an HMD wearable on the head of a user (110) is described as an example, the present disclosure is not limited thereto. Hereinafter, an operation of tracking a body part to move a virtual object (630) is described based on an electronic device having a different form factor from the HMD.
[0113] FIGS. 7A, 7B, and 7C are diagrams illustrating exemplary operations of an electronic device (101-1) performed based on movement of a body part. The electronic device (101-1) of FIGS. 7A, 7B, and 7C may perform the operations of the wearable device (101) described with reference to FIGS. 1A to 6C. For example, the operations of the electronic device (101-1) described with reference to FIGS. 7A, 7B, and 7C may be related to at least one of the operations of the wearable device of FIG. 3. The electronic device (101-1) described with reference to FIGS. 7A, 7B, and 7C may include one or more hardwares described with reference to FIG. 2. The electronic device (101-1) described with reference to FIGS. 7a, 7b and 7c can execute one or more programs described with reference to FIG. 2.
[0114] Referring to FIGS. 7A, 7B, and 7C, exemplary states (701, 702, 703) of an electronic device (101-1) that identify a hand (112) of a user (110) within a FoV (710) of one or more cameras (e.g., camera (225) of FIG. 2) disposed on one side of the electronic device (101-1) opposite to the other side of the electronic device (101-1) (e.g., rear side of the electronic device (101-1)) on which a display (220) is disposed are illustrated. According to one embodiment, the electronic device (101-1) can identify a body part, such as a hand (112), using an image and / or video corresponding to the FoV (710).
[0115] Within the state (701) of FIG. 7A, the electronic device (101-1) may display a screen (720) including one or more icons on the display (220). Referring to FIG. 7A, icons included in the screen (720) are represented as rectangles filled with consecutive slashes. The numbers written within the icons are written to indicate the icons for convenience of explanation and do not limit the present disclosure. The form of the icons that the electronic device (101-1) may display through the screen (720) is not limited to the exemplary forms of FIGS. 7A, 7B, and 7C. The electronic device (101-1) may display icons based on images (e.g., raster images and / or vector images) set by a software application installed in a memory (e.g., memory (215) of FIG. 2).
[0116] According to one embodiment, the electronic device (101-1) may arrange a plurality of icons based on a grid formed within the screen (720). Referring to FIG. 7A, in an exemplary state (701) displaying eight icons, the electronic device (101-1) may arrange the eight icons based on a grid including three rows formed along a first direction (e.g., vertical direction) of the screen (720) and four columns formed along a second direction (e.g., horizontal direction) within the screen (720) perpendicular to the first direction. The number of rows and / or columns of the grid is exemplary and does not limit the present disclosure. Referring to FIG. 7A, the electronic device (101-1) may arrange the eight icons using twelve slots formed within the screen (720) based on the exemplary grid. Among the above slots, a slot in which no icon is placed (e.g., an empty slot) may be depicted as a square with a dashed line in the screen (720). Since the electronic device (101-1) displays icons based on a grid, in the second direction in the screen (720), the spacing (e.g., horizontal spacing) of the icons (e.g., icons including numbers 1, 2, 3, and 4) may be the same. Similarly, in the first direction in the screen (720), the spacing (e.g., vertical spacing) of the icons (e.g., icons including numbers 1, 5, and 7) may be the same.
[0117] Within state (701) of FIG. 7A, the electronic device (101-1) can identify a gesture of a hand (112) indicating movement of an icon within a screen (720). The electronic device (101-1) can identify a pointing gesture of a hand (112) performed within a FoV (710) based on the operations described above with reference to FIGS. 4 and 5 . For example, based on identifying a hand (112) including a finger extended toward a location (fp1) within the FoV (710), the electronic device (101-1) can identify an input indicating movement of an icon (e.g., an icon including a number 8) within the screen (720) corresponding to the location (fp1). The electronic device (101-1) can identify an input indicating movement of an icon (e.g., an icon including the number 8) displayed at a location (dp1) within the screen (720) corresponding to the location (fp1). The electronic device (101-1) that has identified the input can change the location of the icon corresponding to the input within the screen (720) based on the path (730) of the hand (112) moving within the FoV (710).
[0118] Referring to FIG. 7a, the electronic device (101-1) that identifies a hand (112) moving along a path (730) within the FoV (710) can move an icon corresponding to the hand (112) along a path (732) within the screen (720) corresponding to the path (730). As the hand (112) moves out of the FoV (710) along the path (730), the electronic device (101-1) can display the icon based on the position (fp2) of the edge of the FoV (710). Referring to FIG. 7a, the electronic device (101-1) can display the icon that has been moved along the path (732) at a position (dp2) within the screen (720) corresponding to the position (fp2). After the hand (112) moves out of the FoV (710) through the position (fp2) at the edge of the FoV (710), the electronic device (101-1) may stop tracking or monitoring the hand (112). For example, while the hand (112) moves from the position (fp2) to the position (fp3) outside the FoV (710), the electronic device (101-1) may not move the icon corresponding to the hand (112).
[0119] Referring to FIG. 7A, after the hand (112) is moved out of the FoV (710), the location (dp2) at which the icon that was moved in response to the hand (112) is displayed corresponds to the location (fp2) of the edge of the FoV (710) and may not correspond to the grid and / or slots formed for aligning the icon within the screen (720). For example, the electronic device (101-1), at the timing of identifying the hand (112) moving out of the FoV (710), may fix the location of the icon that is moved in response to the hand (112) to the location (dp2) within the screen (720) corresponding to the location (fp2) of the hand (112) identified at the timing. After displaying the icon at the location (dp2), the electronic device (101-1) may move the icon again based on the re-entry of the hand (112) into the FoV (710).
[0120] In the state (702) of FIG. 7b after the state (701) of FIG. 7a, the user (110) can move a hand (112) having a posture of a pointing gesture into the FoV (710) along a path (740). For example, the hand (112) can be moved from a position (fp4) outside the FoV (710) to a position (fp5) of the edge of the FoV (710) into the FoV (710). At the timing of identifying the hand (112) at the position (fp5) of the edge of the FoV (710), the electronic device (101-1) can identify an input to resume movement of an icon (e.g., an icon including a number 8) that was moved in the state (701) prior to the state (702) based on the posture of the hand (112). For example, if the gesture indicated by the hand (112) identified at the above timing matches the gesture (e.g., pointing gesture) identified in the state (701), the electronic device (101-1) can resume movement of the icon.
[0121] Referring to FIG. 7B, the electronic device (101-1), which has identified a hand (112) that has moved into the FoV (710) through a position (fp5) with a posture of a pointing gesture, can move an icon displayed at a position (dp2) within the screen (720) along a path (740) of the hand (112) within the FoV (710). For example, the electronic device (101-1) can move an icon along a path (742) within the screen (720) corresponding to the path (740). While the hand (112) moves from a position (fp5) to a position (fp6) along the path (740), the electronic device (101-1) can move an icon including a number 8 from a position (dp5) to a position (dp6) along the path (742).
[0122] According to one embodiment, the electronic device (101-1) can identify a hand (112) that has changed to a different posture from the pointing gesture within a state (702) of moving an icon based on a hand (112) having a posture of a pointing gesture. The electronic device (101-1) that has identified the hand (112) that has changed to the different posture can stop moving the icon. The electronic device (101-1) that has identified the hand (112) that has changed to the different posture can determine the location of the icon based on a grid formed within a screen (720). Referring to FIG. 7B, when the posture of the hand (112) at the position (fp6) is changed to a posture different from the posture of the pointing gesture, the electronic device (101-1) can move the icon displayed at the position (dp6) within the screen (720) corresponding to the position (fp6) to an empty slot (744) adjacent to the icon among the slots within the screen (720) formed by the grid. Based on displaying the icon based on the empty slot (744), the electronic device (101-1) can complete the movement of the icon using the hand (112).
[0123] Referring to FIG. 7C, in a state (703) after the state (701) of FIG. 7A, the user (110) can move a hand (112) having a posture of a gesture different from the pointing gesture into the FoV (710). Based on identifying the hand (112) moving from a position (fp7) outside the FoV (710) to a position (fp8) of the edge of the FoV (710) into the FoV (710), the electronic device (101-1) can compare the posture of the hand (112) with a designated posture indicating movement of the icon. Based on identifying the hand (112) having a posture different from the designated posture, the electronic device (101-1) can move the position of the icon within the screen (720) to the position of the icon before the state (701). Referring to FIG. 7C, independently of the hand (112) moving from the position (fp8) of the edge of the FoV (710) to the position (fp9) along the path (750), the electronic device (101-1) can move an icon displayed at the position (dp2) within the screen (720) along the path (754). In one embodiment, the wearable device (101) can display a pointer (e.g., a cursor) moving from the position (dp5) to the position (dp9) along the path (752) corresponding to the path (750) within the screen (720) based on identifying the hand (112) moving from the position (fp8) to the position (fp9) along the path (750). Based on the icon moving along the path (754), the electronic device (101-1) can move the icon to a position prior to the state (701) where the gesture indicating the movement of the icon was identified.
[0124] Although the operation of the electronic device (101-1) and / or the wearable device (101) for moving an icon and / or a virtual object (630) based on a pointing gesture is described in FIGS. 6A, 6B, and 6C, and FIGS. 7A, 7B, and 7C, the present disclosure is not limited thereto. For example, the electronic device (101-1) may identify a gesture for moving an icon within a screen (720) based on a pinch gesture and / or a clench gesture, as described with reference to FIG. 4 .
[0125] As described above, the electronic device (101-1) tracking the hand (112) within the FoV (710) can re-track the hand (112) based on the re-entry of the hand (112) into the FoV (710) after the tracking of the hand (112) was interrupted by the finite FoV (710). The electronic device (101-1) can use the pose and / or gesture of the hand (112) to re-execute the function that was executed before the tracking of the hand (112) was interrupted. Although the operation of the electronic device (101-1) and / or the wearable device (101) to re-execute the movement of an icon (e.g., an icon including the number 8 of FIGS. 7A to 7C) and / or a virtual object (e.g., a virtual object (630) of FIGS. 6A to 6C) based on the re-entry of the hand (112) into the FoV (710) has been described, the present disclosure is not limited thereto. Hereinafter, the operation of the electronic device (101-1) and / or the wearable device (101) to re-execute the function of adjusting the size of a virtual object will be described in more detail with reference to FIG. 8.
[0126] FIG. 8 is a flowchart illustrating exemplary operations of a wearable device according to one embodiment. The wearable device (101) of FIGS. 1A and 2 and / or the processor (210) of FIG. 2 may perform the operations described with reference to FIG. 8. The electronic device (101-1) described with reference to FIGS. 7A, 7B, and 7C may perform the operations described with reference to FIG. 8. The operations of the wearable device described with reference to FIG. 8 may be related to, or performed similarly to, at least one of the operations of FIG. 3.
[0127] Referring to FIG. 8, in operation (810), a processor of a wearable device according to one embodiment may change the size of a virtual object displayed in a display based on identifying a gesture for adjusting the size of the virtual object using a camera and / or an external electronic device. The camera of operation (810) may include the camera (225) of FIG. 2. The external electronic device of operation (810) may include the remote controller (242) of FIG. 2. The wearable device may identify the gesture of operation (810) using a sensor (e.g., a depth sensor) different from the camera. The gesture performed to change the size of the virtual object may be performed using both hands of the user, as described in more detail below with reference to FIGS. 9A, 9B, and 9C. A gesture performed to change the size of a virtual object may be identified by both the user's hand and the external electronic device held by the user's other hand, as described in more detail below with reference to FIG. 10. For example, a gesture for adjusting the size of a virtual object may require movement of both external objects (e.g., the hand and / or the external electronic device) identifiable by the wearable device. The gesture for adjusting the size of the virtual object of operation (810) may be set by a software application executed by the processor.
[0128] Referring to FIG. 8, within operation (820), a processor of a wearable device according to an embodiment may identify whether a body part performing a gesture moves out of the FoV of a camera. The processor may perform operation (820) similar to operation (320) of FIG. 3. For example, before the body part performing the gesture moves out of the FoV or while it is moving within the FoV (820-No), the processor may change the size of the virtual object based on operation (810). Within a state where the body part moving out of the FoV is identified (820-Yes), the processor may perform operation (830). After the body part of operation (820) moves out of the FoV, the processor may perform operation (830).
[0129] Referring to FIG. 8, in operation (830), a processor of a wearable device according to an embodiment may set a size of a virtual object based on a first size corresponding to a first position within the FoV of a body part identified by a camera. The first size of operation (830) may be related to a position within the FoV of the body part that was last identified before being moved out of the FoV. With the size of the virtual object set to the first size based on the body part before being moved out of the FoV, the processor may perform operation (840).
[0130] Referring to FIG. 8, in operation (840), a processor of a wearable device according to an embodiment may identify a body part moving into the FoV of a camera. Prior to identifying the body part moving into the FoV (840-No), the processor may continue displaying a virtual object based on the first size of operation (830). Based on identifying the body part moving into the FoV (840-Yes), the processor may perform operation (850). The processor may perform operation (840) similarly to operation (340) of FIG. 3.
[0131] Referring to FIG. 8, in operation (850), a processor of a wearable device according to an embodiment may identify a gesture for adjusting the size of a virtual object based on a body part moved into the FoV. The gesture of operation (850) may match the gesture of operation (810). For example, the processor may identify whether a gesture performed by a body part moved into the FoV corresponds to the gesture of operation (810). If the gesture of operation (850) is not identified (850-No), or before identifying the gesture of operation (850), the processor may perform operation (860) of FIG. 8. For example, if a gesture different from the gesture of operation (850) is identified, the processor may perform operation (860). Based on identifying a gesture (850) from a body part moved into the FoV (850-example), the processor can perform operation (870). The processor can perform operation (850) similarly to operation (350) of FIG. 3.
[0132] Referring to FIG. 8, in operation (860), the processor of the wearable device according to one embodiment may determine the size of the virtual object as the first size of operation (830). The processor may determine the size of the virtual object, which was changed based on operation (810), as the size of the virtual object.
[0133] Referring to FIG. 8, in operation (870), a processor of a wearable device according to one embodiment may adjust the size of a virtual object to a second size different from the first size based on a second position within the FoV of a body part moved into the FoV. Similar to operation (810), the processor may change the size of the virtual object based on the second position. For example, the processor may resume changing the size of the virtual object after operation (830) based on operation (870).
[0134] As described above, according to one embodiment, the processor of the wearable device may, based on the re-entry of a body part into the FoV, re-execute a function previously performed by the body part prior to the re-entry. For example, the processor may re-execute a function for changing the size of a virtual object displayed on the display.
[0135] Hereinafter, exemplary operations of a wearable device for changing the size of a virtual object based on the operation of FIG. 8 are described in more detail with reference to FIGS. 9A, 9B, and 9C and / or 10.
[0136] FIGS. 9A, 9B, and 9C are diagrams illustrating exemplary operations of a wearable device (101) performed based on movements of body parts. The wearable device (101) of FIGS. 1A and 1B, FIG. 2, and / or the processor (210) of FIG. 2 may perform the operations described with reference to FIGS. 9A and 9C. The operations of the wearable device (101) described with reference to FIGS. 9A, 9B, and 9C may be related to at least one of the operations of FIG. 8.
[0137] Referring to FIGS. 9A, 9B, and 9C, exemplary states (901, 902, 903) of a wearable device (101) that have identified hands (112-1, 112-2) of a user (110) within a FoV (910) of one or more cameras (e.g., camera (225) of FIG. 2) are illustrated. In one embodiment, the wearable device (101) may identify the hands (112-1, 112-2) using images and / or videos corresponding to the FoV (910). Within state (901) of FIG. 9A, the wearable device (101) may display a screen (920) including a virtual object (925) on a display (e.g., display (220) of FIG. 2). Referring to FIGS. 9a, 9b and 9c, virtual objects (925-1, 925-2, 925-3) in different states are shown, switched by inputs for changing the size of the virtual object (925) represented by hands (112-1, 112-2).
[0138] Within the state (901) of FIG. 9a, the wearable device (101) can identify an input indicating a change in the size of a virtual object (925) based on the hands (112-1, 112-2) moving within the FoV (910). The wearable device (101) can identify a pinch gesture of the hands (112-1, 112-2) performed within the FoV (910) based on the operations described above with reference to FIGS. 4 and 5. For example, based on identifying a pinch gesture of hands (112-1, 112-2) placed at positions (fr1, fl1) in the FoV (910) corresponding to different positions (ir1, il1) of a virtual object (925-1) in a first state in the screen (920), the wearable device (101) can change the size of the virtual object (925) according to the distance between the hands (112-1, 112-2) maintaining the pinch gesture. The wearable device (101) that changes the size of the virtual object (925) in response to the hands (112-1, 112-2) having the pinch gesture can provide a user experience, such as grabbing the virtual object (925) based on the pinch gesture, to a user (110) wearing the wearable device (101).
[0139] In the exemplary state (901) of FIG. 9A, while a hand (112-1) having a pinch gesture posture is moved along a path (931) and a hand (112-2) having a pinch gesture posture is moved along a path (932), the wearable device (101) can enlarge or reduce a virtual object (925) based on the distance between the hands (112-1, 112-2). For example, when a hand (112-1) having a pinch gesture moves from a position (fr1) to a position (fr2) within the FoV (910) along a path (931), and a hand (112-2) having a pinch gesture moves from a position (fl1) to a position (fl2) within the FoV (910) along a path (932), the wearable device (101) can enlarge the virtual object (925) based on the distance between the positions (fr2, fl2). For example, the wearable device (101) can move a point of the virtual object (925) corresponding to the hand (112-1) along a path (933) within the screen (920) corresponding to the path (931), and can move a point of the virtual object (925) corresponding to the hand (112-2) along a path (934) within the screen (920) corresponding to the path (932). For example, the wearable device (101) can display an enlarged second state virtual object (925-2) based on positions (ir1, ir2) within the screen (920) corresponding to each of the positions (fr2, fl2).
[0140] Referring to FIG. 9A, in a state (901) of changing the size of a virtual object (925) based on the hands (112-1, 112-2) of a pinch gesture, the wearable device (101) may stop changing the size of the virtual object (925) based on whether the posture of at least one of the hands (112-1, 112-2) changes to a posture different from the posture of the pinch gesture. For example, in response to identifying that the pinch gesture performed by at least one of the hands (112-1, 112-2) is released, the wearable device (101) may stop changing the size of the virtual object (925) based on the position and / or path within the FoV (910) of the hands (112-1, 112-2). In the above example, the size of the virtual object (925) can be fixed based on the positions of the hands (112-1, 112-2) within the FoV (910) at the timing at which the release of the pinch gesture is identified.
[0141] Referring to FIG. 9A, based on at least one of the hands (112-1, 112-2) performing the pinch gesture to change the size of the virtual object (925) being moved out of the FoV (910), the wearable device (101) can at least temporarily stop changing the size of the virtual object (925). For example, based on the hand (112-1) moving along the path (931) being moved out of the FoV (910) (e.g., to the location (fr3)) through the location (fr2) of the edge of the FoV (910), the wearable device (101) can stop changing the size of the virtual object (925). For example, after the hand (112-1) moves out of the FoV (910) through the location (fr2), the wearable device (101) can continue to display the virtual object (925) based on the location (ir2) within the screen (920) corresponding to the location (fr2).
[0142] According to one embodiment, the wearable device (101) may re-execute the function of changing the size of the virtual object (925) based on the re-entry of the hand (112-1) into the FoV (910) after the state (901) in which the hand (112-1) has moved out of the FoV (910). Referring to FIG. 9B, in the state (902) after the state (901), the user (110) may move the hand (112-1) from the position (fr4) outside the FoV (910) into the FoV (910) through the position (fr5) of the edge of the FoV (910) along the path (941). The wearable device (101) can identify the re-entry of the hand (112-1) into the FoV (910) from the timing at which the hand (112-1) moving along the path (941) moves into the FoV (910) through the position (fr5) of the edge of the FoV (910). The wearable device (101) that has identified the re-entry of the hand (112-1) into the FoV (910) can re-execute the function for changing the size of the virtual object (925) based on whether the posture of the hand (112-1) has a designated posture (e.g., a posture of a pinch gesture) for changing the size of the virtual object (925). Within the exemplary state (902) of FIG. 9b, based on identifying a hand (112-1) having a posture of a pinch gesture moved into the FoV (910), the wearable device (101) can change the size of a virtual object (925) using the path (941) of the hand (112-1).
[0143] Within the exemplary state (902) of FIG. 9B, the wearable device (101) can change the size of a virtual object (925) within the screen (920) by using the positions and / or paths of the hands (112-1, 112-2) moving within the FoV (910). Within the state (902), the wearable device (101) can change the size of the virtual object (925) along a path (942) within the screen (920) corresponding to the path (941) of the hand (112-1) within the FoV (910). A wearable device (101) displaying a virtual object (925) having a size based on a position (ir2) can display the virtual object (925) using a position (ir5) within the screen (920) corresponding to a position (fr5) of an edge of the FoV (910) of the hand (112-1) and a position (il2) within the screen (920) corresponding to a position (fl2) within the FoV (910) of the hand (112-2) maintaining the pinch gesture. At the timing when the hand (112-1) moves to the position (fr6) within the FoV (910) along the path (941), the wearable device (101) can display a third state virtual object (925-3) having a changed size based on the positions (ir6, il2) within the screen (920) corresponding to the positions (fr6, fl2) of the hands (112-1, 112-2).
[0144] Referring to FIG. 9c, in a state (903) after a state (901), a user (110) can move a hand (112-1) having a different posture from the posture of the pinch gesture from a location (fr7) outside the FoV (910) along a path (951) into the FoV (910). The wearable device (101) that identifies the hand (112-1) at a location (fr8) of the edge of the FoV (910) can change the size of the virtual object (925) and perform other functions based on the different posture of the hand (112-1). For example, even though the posture of the hand (112-2) maintains a pinch gesture, a wearable device (101) that identifies a hand (112-1) having a posture different from the posture of the pinch gesture may not change the size of a virtual object (925) based on a path (951) within the FoV (910) of the hand (112-1) (e.g., a path (951) connecting locations (fr8, fr9)). For example, the virtual object (925) may not be enlarged or reduced by a path (952) within the screen (920) corresponding to the path (951) (e.g., a path (952) connecting locations (ir8, ir9) within the screen (920). For example, the wearable device (101) may determine that the function of changing the size of the virtual object (925) is completed at the point in time when the hand (112-1) has moved out of the FoV (910).
[0145] As described above, the wearable device (101) that tracks the hands (112-1, 112-2) within the FoV (910) may temporarily suspend a function that was executed based on a gesture of the hands (112-1, 112-2) when at least one of the hands (112-1, 112-2) moves out of the FoV (910). When at least one hand that was moved out of the FoV (910) moves into the FoV (910) with a designated posture indicating the gesture, the wearable device (101) may re-execute the function that was temporarily suspended (e.g., state (902) of FIG. 9b after state (901) of FIG. 9a). If at least one hand that was moved out of the FoV (910) is moved into the FoV (910) in a posture different from the specified posture, the wearable device (101) can complete the function that was temporarily stopped.
[0146] Below, the operation of a wearable device (101) that changes the size of a virtual object (925) based on a combination of an external electronic device such as a remote controller and a hand is described.
[0147] FIG. 10 is a diagram illustrating exemplary operations of a wearable device (101) performed based on the movement of body parts. The wearable device (101) of FIGS. 1A and 1B, FIG. 2, and / or the processor (210) of FIG. 2 may perform the operations described with reference to FIG. 10. The operations of the wearable device (101) described with reference to FIG. 10 may be related to at least one of the operations of FIG. 8.
[0148] Referring to FIG. 10, an exemplary state of a wearable device (101) that identifies an input for changing the size of a virtual object (1030) within a screen (1020) using one or more cameras (e.g., camera (225) of FIG. 2) and a communication circuit (e.g., communication circuit (240) of FIG. 2) is illustrated. The wearable device (101) can identify a hand (112-1) included in the FoV (1010) formed by the one or more cameras. The wearable device (101) can transmit or receive a wireless signal to or from a remote controller (242) via the communication circuit. Referring to FIG. 10, the wearable device (101) can display a screen (1020) including a virtual object (1030). Referring to FIG. 10, virtual objects (1030-1, 1030-2, 1030-3) in different states are shown, switched by inputs for changing the size of the virtual object (1030) indicated by the hand (112-1) and the remote controller (242).
[0149] Within the exemplary state of FIG. 10, the wearable device (101) can identify an input indicating a change in the size of a virtual object (1030) based on a signal received via a hand (112-1) moving within the FoV (1010) and a remote controller (242). The wearable device (101) can identify a pinch gesture of a hand (112-1) performed within the FoV (1010) based on the operations described above with reference to FIGS. 4 and 5. For example, while identifying a pinch gesture of a hand (112-1) placed at a location (fr1) in the FoV (1010) corresponding to a location (ir1) of a virtual object (1030), the wearable device (101) may identify an input for changing the size of the virtual object (1030) based on receiving a signal for notifying the pressing of a designated button of the remote controller (242) placed at a location (rc1) corresponding to the location (vc1) of the virtual object (1030). The designated button may include a button formed on the remote controller (242) for receiving an input indicating selection of the virtual object (1030), such as a click button.
[0150] Referring to FIG. 10, while a hand (112-1) having a pinch gesture posture is moved along a path (1051) and a remote controller (242) is moved along a path (1041) by a hand (112-2), the wearable device (101) can perform at least one of movement, enlargement and / or reduction of a virtual object (1030) based on the positions of the remote controller (242) and the hand (112-1). For example, when a hand (112-1) is moved from a position (fr1) to a position (fr2) along a path (1051) and a remote controller (242) is moved from a position (rc1) to a position (rc2) along a path (1052), the wearable device (101) can display a virtual object (1030-2) having a size based on positions (vc2, ir2) within the screen (1020) corresponding to the positions (fr2, rc2).
[0151] In one embodiment, within the exemplary state of FIG. 10 where the size of the virtual object (1030) is changed using the hand (112-1) and the remote controller (242), the wearable device (101) may stop changing the size of the virtual object (1030) based on at least one of the release of the pinch gesture based on the posture of the hand (112-1) and / or a signal received from the remote controller (242). For example, if the posture of the hand (112-1) is changed to a different posture than the posture of the pinch gesture, the wearable device (101) may stop changing the size of the virtual object (1030). For example, when receiving a signal from the remote controller (242) indicating that the pressing of a designated button by the hand (112-2) is released, the wearable device (101) may stop changing the size of the virtual object (1030).
[0152] Referring to FIG. 10, based on the hand (112-1) performing the pinch gesture to change the size of the virtual object (1030) moving out of the FoV (1010), the wearable device (101) can at least temporarily stop changing the size of the virtual object (1030). For example, based on the hand (112-1) moving along the path (1051) moving out of the FoV (1010) through the position (fr2) of the edge of the FoV (1010), the wearable device (101) can stop changing the size of the virtual object (1030). For example, after the hand (112-1) is moved out of the FoV (1010) through the position (fr2) of the edge of the FoV (1010), the wearable device (101) can maintain displaying the virtual object (1030) based on the position (ir2) within the screen (1020) corresponding to the position (fr2). For example, while the hand (112-1) is moved from the position (fr2) of the edge of the FoV (1010) to the position (fr3) along the path (1051), the wearable device (101) can maintain the virtual object (1030) in a state having a size based on the position (ir2) within the screen (1020).
[0153] In one embodiment, the wearable device (101) may re-execute the function of changing the size of the virtual object (1030) based on the re-entry of the hand (112-1) into the FoV (1010) after the hand (112-1) has moved out of the FoV (1010) through a location (fr2) of the edge of the FoV (1010). For example, the wearable device (101) may identify the re-entry of the hand (112-1) based on identifying the hand (112-1) moving into the FoV (1010) through a location (fr5) of the edge of the FoV (1010) from a location (fr4) outside the FoV (1010) along a path (1061). The wearable device (101) that has identified the re-entry of the hand (112-1) can re-execute the function for changing the size of the virtual object (1030) based on whether the posture of the hand (112-1) has a designated posture (e.g., a posture of a pinch gesture) for changing the size of the virtual object (1030). For example, the wearable device (101) that has identified the hand (112-1) that has the posture of a pinch gesture moved into the FoV (1010) can change the size of the virtual object (1030) based on the hand (112-1) moving within the FoV (1010) along the path (1061) from the position (fr5) of the edge of the FoV (1010). For example, when a hand (112-1) is moved from a position (fr5) to a position (fr6) within the FoV (1010) along a path (1061), and a remote controller (242) is moved from a position (rc2) to a position (rc6) along a path (1041), the wearable device (101) can display a virtual object (1030-3) having a size based on positions (ir6, vc6) corresponding to each of the positions (fr6, rc6) within the screen (1020).For example, the wearable device (101) can change the size of a virtual object based on a path (1062) within the screen (1020) corresponding to a path (1061) of the hand (112-1) and / or a path (1041) identified by the remote controller (242).
[0154] According to one embodiment, the wearable device (101) can identify a hand (112-1) that moves into the FoV (1010) from a location (fr7) outside the FoV (1010) along a path (1071) after the hand (112-1) moves out of the FoV (1010) through a location (fr2) of the edge of the FoV (1010). Based on identifying a hand (112-1) having a different gesture posture than a pinch gesture for changing a virtual object (1030), the wearable device (101) can perform another function different from changing the size of the virtual object (1030). For example, even though a signal indicating the pressing of a designated button from a remote controller (242) is received, a wearable device (101) that identifies a hand (112-1) having a posture different from that of a pinch gesture may stop the hand (112-1) from changing the size of a virtual object (1030) along a path (1071) within the FoV (1010). For example, even though the hand (112-1) having a posture different from that of the pinch gesture moves from a position (fr8) to a position (fr9) along the path (1071), the wearable device (101) may maintain displaying a virtual object (1030-2) having a size based on a position (ir2) within the screen (1020) corresponding to the position (fr2).
[0155] Hereinafter, the operation of the wearable device (101) that identifies the speed and / or velocity of the hand (112-1) moving at the edge of the FoV (1010) is described in more detail with reference to FIGS. 11a, 11b and 11c, and / or FIGS. 12a and 12b.
[0156] FIGS. 11A, 11B, and 11C illustrate an example of an operation of a wearable device (101) performed based on the movement of body parts. The wearable device (101) of FIGS. 1A to 1B, FIG. 2, and / or the processor (210) of FIG. 2 may perform the operations described with reference to FIGS. 11A, 11B, and 11C.
[0157] Referring to FIGS. 11A, 11B, and 11C, exemplary states (1101, 1102, 1103) of a wearable device (101) that have identified hands (112-1, 112-2) of a user (110) within a FoV (1110) of one or more cameras (e.g., camera (225) of FIG. 2) are illustrated. In one embodiment, the wearable device (101) may identify the hands (112-1, 112-2) using images and / or videos corresponding to the FoV (1110). Within state (1101) of FIG. 11A, the wearable device (101) may display a screen (1120) including a virtual object (1130) on a display (e.g., display (220) of FIG. 2). Referring to FIGS. 11A, 11B, and 11C, exemplary states (1101, 1102, 1103) of a wearable device (101) displaying a virtual object (1130) having the shape of a slingshot are illustrated. The virtual object (1130) having the shape of a slingshot may be provided from a software application executed by the wearable device (101).
[0158] Within the state (1101) of FIG. 11a, the wearable device (101) can identify an input indicating control of a virtual object (1130) based on the hands (112-1, 112-2) moving within the FoV (1110). The wearable device (101) can identify a pinch gesture of the hands (112-1, 112-2) performed within the FoV (1110) based on the operations described above with reference to FIGS. 4 and 5. For example, based on identifying a pinch gesture of hands (112-1, 112-2) placed at positions (fr1, fl1) in the FoV (1110) corresponding to positions (ir1, il1) on a virtual object (1130) in a screen (1120), the wearable device (101) can change the display of the virtual object (1130) according to the distance between the hands (112-1, 112-2) maintaining the pinch gesture. Within the exemplary state (1101) of FIG. 11a, based on identifying that the position (fl1) of the hand (112-2) within the FoV (1110) corresponds to the position (il1) of the part (1131) representing the handle of the slingshot, and the position (fr1) of the hand (112-1) within the FoV (1110) corresponds to the position (ir1) of the part (1132) representing the elastic body of the slingshot, the wearable device (101) may initiate changing the display of the virtual object (1130) according to the distance between the hands (112-1, 112-2).
[0159] In an exemplary state (1101) of FIG. 11A, while a hand (112-1) having a pinch gesture posture is moved along a path (1141), the wearable device (101) can change the display of the virtual object (1130) based on the distance between the hands (112-1, 112-2). For example, when the hand (112-1) having the pinch gesture is moved from a location (fr1) to a location (fr2) within the FoV (1110) along the path (1141), the wearable device (101) can extend a portion (1132) of the virtual object (1130) corresponding to the elastic body based on the distance between the locations (fr2, fl1). For example, the wearable device (101) can move along the path (1142) to a position (ir1) corresponding to the hand (112-1) in the above-mentioned section (1132).
[0160] Referring to FIG. 11A, in a state (1101) of changing the display of a virtual object (1130) based on the hands (112-1, 112-2) of a pinch gesture, the wearable device (101) may execute a function based on the virtual object (1130) having a changed shape based on whether the posture of at least one of the hands (112-1, 112-2) changes to a posture different from the posture of the pinch gesture. For example, in response to identifying that the pinch gesture performed by the hand (112-1) is released, the wearable device (101) may execute a function based on the changed shape of the virtual object (1130). For example, the wearable device (101) may execute a pitching function related to the virtual object (1130).
[0161] Referring to FIG. 11A, based on at least one of the hands (112-1, 112-2) performing a pinch gesture to change the size of the virtual object (1130) being moved out of the FoV (1110), the wearable device (101) can at least temporarily stop changing the display of the virtual object (1130). For example, based on the hand (112-1) moving along the path (1141) being moved out of the FoV (1110) through the position (fr2) of the edge of the FoV (1110), the wearable device (101) can stop changing a portion (1132) of the virtual object (1130) along the path (1141) of the hand (112-1). For example, after the hand (112-1) is moved out of the FoV (1110) through the location (fr2), the wearable device (101) can continue to display a portion (1132) of the virtual object (1130) based on the location (ir2) within the screen (1120) corresponding to the location (fr2).
[0162] In one embodiment, the wearable device (101) can change the display of the virtual object (1130) based on the speed and / or velocity of the hand (112-1) as it moves out of the FoV (1110) through a location (fr2) of the edge of the FoV (1110). For example, the wearable device (101) can change the display of the virtual object (1130) based on the speed of the hand (112-1) as it was moved within the FoV (1110). In the above example, even though tracking of the hand (112-1) is discontinued as the hand (112-1) moves out of the FoV (1110), the wearable device (101) can continue to extend a portion (1132) of the virtual object (1130) based on the speed. Additionally, the wearable device (101) can continuously extend a portion (1132) of a virtual object (1130) based on at least one of the path of the hand (112-1) before it moves out of the FoV (1110), information about an area in which the hand (112-1) can move, and the movement speed of the hand (112-1).
[0163] Referring to FIG. 11A, after the hand (112-1) is moved out of the FoV (1110), the wearable device (101) may display a visual object (1171) to guide the hand (112-1) to move into the FoV (1110). For example, the wearable device (101) may display a visual object (1171) including specified text (e.g., “Fire in 3 seconds”) indicating that a specified function (e.g., projection function) associated with a virtual object (1130) will be executed if the hand (112-1) is not moved into the FoV (1110) for a specified period of time. The present disclosure is not limited thereto, and the wearable device (101) may guide the user (110) to move the hand (112-1) into the FoV (1110) based on a visual effect such as changing the color of the virtual object (1130) or blinking.
[0164] According to one embodiment, the wearable device (101) may re-execute the function of changing the display of the virtual object (1130) based on the re-entry of the hand (112-1) into the FoV (1110) after the state (1101) in which the hand (112-1) has moved out of the FoV (1110). Referring to FIG. 11B, in the state (1102) after the state (1101), the user (110) may move the hand (112-1) from the position (fr3) outside the FoV (1110) into the FoV (1110) through the position (fr4) of the edge of the FoV (1110) along the path (1151). The wearable device (101) can identify the re-entry of the hand (112-1) into the FoV (1110) from the timing at which the hand (112-1) moving along the path (1151) moves into the FoV (1110) through the position (fr4) of the edge of the FoV (1110). The wearable device (101) that has identified the re-entry of the hand (112-1) into the FoV (1110) can re-execute a function for changing the display of the virtual object (1130) based on whether the posture of the hand (112-1) has a designated posture (e.g., a posture of a pinch gesture) for changing the size of the virtual object (1130). Within the exemplary state (1102) of FIG. 11b, based on identifying a hand (112-1) having a posture of a pinch gesture moved into the FoV (1110), the wearable device (101) can change the size of a virtual object (1130) using the path (1151) of the hand (112-1).
[0165] Within the exemplary state (1102) of FIG. 11B, the wearable device (101) can change the display of a portion (1132) of a virtual object (1130) within the screen (1120) by using the position and / or path of the hand (112-1) moving within the FoV (1110). Within the state (1102), the wearable device (101) can change the portion (1132) of the virtual object (1130) along a path (1152) within the screen (1120) corresponding to the path (1151) of the hand (112-1) within the FoV (1110). The wearable device (101) can change the display of a portion (1132) along a path (1152) within the screen (1120) corresponding to the path (1151) from a position (ir4) within the screen (1120) corresponding to the position (fr4) of the edge of the FoV (1110) of the hand (112-1).
[0166] In the exemplary state (1102) of FIG. 11B, the wearable device (101) can identify that the posture of the hand (112-1) moved to the position (fr5) within the FoV (1110) has changed to a posture different from the pinch gesture (e.g., a posture with fingers spread out more than a specified number). Based on identifying the hand (112-1) changed to a posture different from the pinch gesture, the wearable device (101) can execute a projection function of a virtual object (1130) based on the position (fr5). For example, the wearable device (101) can execute a projection function for the virtual object (1130) based on a position (ir5) within the screen (1120) corresponding to the position (fr5).
[0167] Referring to FIG. 11c, in a state (1103) after a state (1101), a user (110) can move a hand (112-1) having a different posture from the posture of the pinch gesture from a location (fr6) outside the FoV (1110) along a path (1161) into the FoV (1110). The wearable device (101) that identifies the hand (112-1) at a location (fr7) of the edge of the FoV (1110) can change the display of a virtual object (1130) and perform other functions based on the different posture of the hand (112-1). For example, even though the posture of the hand (112-2) maintains a pinch gesture, a wearable device (101) that identifies a hand (112-1) having a posture different from the posture of the pinch gesture may not change the display of a portion (1132) of a virtual object (1130) based on a path (1161) within the FoV (1110) of the hand (112-1). For example, the wearable device (101) may display a visual object (1172) to guide an input related to the virtual object (1130). The visual object (1171) may include, for example, designated text such as “Aim again.” In state (1103) of FIG. 11c, even though the hand (112-1) moves along the path (1161) to a location (fr8) within the FoV (1110), the wearable device (101) may not change the display of the virtual object (1130) based on the location (fr8).
[0168] As described above, according to one embodiment, the wearable device (101) can execute a function related to a virtual object (1130) provided from a software application based on the position of the hand (112-1) within the FoV (1110). Even if the hand (112-1) moves out of the FoV (1110), the wearable device (101) can execute the function again based on the re-entry of the hand (112-1) into the FoV (1110).
[0169] FIGS. 12A and 12B are diagrams illustrating exemplary operations of a wearable device (101) performed based on the movement of body parts. The wearable device (101) of FIGS. 1A to 1B and FIG. 2 and / or the processor (210) of FIG. 2 may perform the operations described with reference to FIGS. 12A and 12B.
[0170] Referring to FIGS. 12A and 12B , exemplary states (1201, 1202) of a wearable device (101) that have identified a hand (112-1) of a user (110) within a FoV (1210) of one or more cameras (e.g., camera (225) of FIG. 2 ) are illustrated. In one embodiment, the wearable device (101) may identify the hand (112-1) using an image and / or video corresponding to the FoV (1210). Within state (1201) of FIG. 12A , the wearable device (101) may display a screen (1220) including virtual objects (1231, 1232, 1233) on a display (e.g., display (220) of FIG. 2 ). The wearable device (101) can display virtual objects (1231, 1232, 1233) included in a virtual space mapped with an external space. The positions of the virtual objects (1231, 1232, 1233) within the screen (1220) can be linked to the external space based on the virtual space mapped with the external space. For example, the wearable device (101) can change the positions of the virtual objects (1231, 1232, 1233) within the screen (1220) based on the rotation and / or movement of the head of a user (110) wearing the wearable device (101).
[0171] Within the state (1201) of FIG. 12a, the wearable device (101) can identify an input indicating movement of a virtual object (1231) based on the hand (112-1) moving within the FoV (1210). The wearable device (101) can identify a pointing gesture of the hand (112-1) performed within the FoV (1210) based on the operations described above with reference to FIGS. 4 and 5. For example, based on identifying a pointing gesture of a hand (112-1) placed at a position (fr1) within the FoV (1210) corresponding to a position (v1) of a virtual object (1231) within the screen (1220), the wearable device (101) can change the position of the virtual object (1231) within the screen (1220) based on a path (1241) within the FoV (1210) of the hand (112-1) maintaining the pointing gesture. Within the exemplary state (1201) of FIG. 12A, the wearable device (101) can identify a hand (112-1) moving from a position (fr1) within the FoV (1210) along a path (1241) to a position (fr2), and can move the position of a virtual object (1231) within the screen (1220) from a position (v1) corresponding to the position (fr1) along a path (1242) to a position (v2).
[0172] Referring to FIG. 12A, in a state (1201) of changing the position of a virtual object (1231) based on a hand (112-1) of a pointing gesture, the wearable device (101) may stop changing the position of the virtual object (1231) based on whether the posture of the hand (112-1) changes to a posture different from the posture of the pointing gesture. For example, in response to identifying that the pointing gesture performed by the hand (112-1) is released, the wearable device (101) may stop moving the virtual object (1231) based on a path (1241) within the FoV (1210) of the hand (112-1).
[0173] Referring to FIG. 12A, tracking of the hand (112-1) may be stopped based on the hand (112-1) performing the pointing gesture to move the virtual object (1231) moving out of the FoV (1210). When tracking of the hand (112-1) is stopped, the wearable device (101) may move the virtual object (1231) based on the speed of the hand (112-1) that was moved within the FoV (1210). For example, at a position (fr2) of the edge of the FoV (1210), the wearable device (101) may move the virtual object (1231) using the first speed based on identifying the hand (112-1) moving out of the FoV (1210) at a first speed. Referring to FIG. 12A, independently of the hand (112-1) moving from a position (fr2) to a position (fr3) outside the FoV (1210), the wearable device (101) can move a virtual object (1231) displayed at a position (v2) within the screen (1220) corresponding to the position (fr2) based on the speed of the hand (112-1) at the position (fr2). The position (v3) of the virtual object (1231) moved by the wearable device (101) can be a position moved by the speed, independently of the position (fr2).
[0174] In one embodiment, after the hand (112-1) is moved out of the FoV (1210), the wearable device (101) can move the virtual object (1231) using the speed of the hand (112-1) identified within the FoV (1210) based on a specified period and / or distance. For example, the wearable device (101) can move the virtual object (1231) based on the external space in which the hand (112-1) can move. For example, the wearable device (101) can move the virtual object (1231) within a portion of the virtual space corresponding to the external space in which the hand (112-1) can move, based on the virtual space mapped with the external space.
[0175] Referring to FIG. 12B, after the hand (112-1) has moved out of the FoV (1210), a state (1202) is illustrated in which the user (110) has rotated his / her head in a direction different from the direction of the head in the state (1201). Based on identifying the rotation of the head, the wearable device (101) can display a screen (1220) based on the direction of the head. Virtual objects (1232, 1233) can be displayed based on positions mapped to external space. The virtual object (1231) that was moved by the hand (112-1) before being moved out of the FoV (1210) can be displayed at the position (v3) moved by the speed of the hand (112-1) before being moved out of the FoV (1210). The present disclosure is not limited thereto, and the wearable device (101) can maintain movement of a virtual object (1231) within the screen (1220) based on the speed.
[0176] According to one embodiment, the wearable device (101) may re-execute the function of moving the virtual object (1231) based on the re-entry of the hand (112-1) into the FoV (1210) after the state (1201) in which the hand (112-1) has moved out of the FoV (1210). Referring to FIG. 12B, in the state (1202) after the state (1201), the user (110) may move the hand (112-1) from the position (fr4) outside the FoV (1210) into the FoV (1210) through the position (fr5) of the edge of the FoV (1210) along the path (1251). The wearable device (101) can identify the re-entry of the hand (112-1) into the FoV (1210) from the timing at which the hand (112-1) moving along the path (1251) moves into the FoV (1210) through the position (fr5) of the edge of the FoV (1210).
[0177] Referring to FIG. 12b, the wearable device (101) that has identified the re-entry of the hand (112-1) into the FoV (1210) can re-execute the function for moving the virtual object (1231) based on whether the posture of the hand (112-1) has a posture for moving the virtual object (1231) (e.g., a posture of a pointing gesture). In the exemplary state (1202) of FIG. 12b, based on identifying the hand (112-1) having the posture of a pointing gesture moved into the FoV (1210), the wearable device (101) can move the virtual object (1231) using the path (1251) of the hand (112-1). For example, the wearable device (101) may move a virtual object (1232) from a position (ir5) in the screen (1220) corresponding to the position (fr5) along a path (1251) to a position (ir6) in the screen (1220) corresponding to the position (fr6) along a path (1252) corresponding to the path (1251) based on identifying a hand (112-1) moving from a position (fr5) in the FoV (1210) to a position (fr6).
[0178] As described above, according to one embodiment, the wearable device (101) can move the virtual object (1231) based on the speed of the hand (112-1) even after the hand (112-1) has moved out of the FoV (1210). Since the virtual object (1231) is moved based on the speed, the wearable device (101) can display the virtual object (1231) based on the location to which the hand (112-1) is expected to move even if tracking of the hand (112-1) moved out of the FoV (1210) is stopped.
[0179] Hereinafter, with reference to FIGS. 13a, 13b and 13c, 14a, 14b and 14c and / or 15, the operation of a wearable device (101) for executing a function for storing a virtual object corresponding to a hand (112-1) based on the hand (112-1) moving out of the FoV (1210) is described in more detail.
[0180] FIGS. 13A, 13B, and 13C are diagrams illustrating exemplary operations of a wearable device (101) performed based on movements of body parts. The wearable device (101) of FIGS. 1A to 1B and FIG. 2 and / or the processor (210) of FIG. 2 may perform the operations described with reference to FIGS. 13A, 13B, and 13C.
[0181] Referring to FIGS. 13A, 13B, and 13C, exemplary states (1301, 1302, 1303) of a wearable device (101) that has identified a hand (112-1) of a user (110) within a FoV (1310) of one or more cameras (e.g., camera (225) of FIG. 2) are illustrated. In one embodiment, the wearable device (101) may identify the hand (112-1) using an image and / or video corresponding to the FoV (1310). Within state (1301) of FIG. 13A, the wearable device (101) may display a screen (1320) including a virtual object (1330) on a display (e.g., display (220) of FIG. 2).
[0182] Within the state (1301) of FIG. 13a, the wearable device (101) can identify an input indicating a change in the size of a virtual object (1330) based on the hand (112-1) moving within the FoV (1310). The wearable device (101) can identify a pointing gesture of the hand (112-1) performed within the FoV (1310) based on the operations described above with reference to FIGS. 4 and 5. For example, based on identifying a pointing gesture of a hand (112-1) placed at a position (fr1) within the FoV (1310) corresponding to a position (ip1) within a virtual object (1330) within a screen (1320), the wearable device (101) can move the virtual object (1330) along a path (1341) of the hand (112-1) maintaining the pointing gesture. The wearable device (101) that moves the virtual object (1330) in response to the hand (112-1) making the pointing gesture can provide a user experience, such as dragging the virtual object (1330) by the hand (112-1), to a user (110) wearing the wearable device (101).
[0183] In an exemplary state (1301) of FIG. 13A, while a hand (112-1) having a posture of a pointing gesture is moved along a path (1341), the wearable device (101) can move a virtual object (1330) based on a path (1342) within a screen (1320) corresponding to the path (1341). For example, based on the path (1341) of the hand (112-1) having the pointing gesture, the wearable device (101) can move a virtual object (1330) displayed at a location (ip1) within the screen (1320) to a location (ip2) along a path (1342) corresponding to the path (1341).
[0184] Referring to FIG. 13A, in a state (1301) of moving a virtual object (1330) based on a hand (112-1) of a pointing gesture, the wearable device (101) may stop moving the virtual object (1330) based on whether the posture of the hand (112-1) changes to a posture different from the posture of the pointing gesture. For example, in response to identifying that the pointing gesture performed by the hand (112-1) is released, the wearable device (101) may stop moving the position of the virtual object (1330) based on the position and / or path (1341) within the FoV (1310) of the hand (112-1). In the above example, the position of the virtual object (1330) can be fixed based on the position within the FoV (1310) of the hand (112-1) at the timing at which the release of the pointing gesture is identified.
[0185] Referring to FIG. 13A, based on the hand (112-1) performing the pointing gesture to change the size of the virtual object (1330) moving out of the FoV (1310), the wearable device (101) may store information related to the virtual object (1330) in the memory of the wearable device (101) (e.g., the memory (215) of FIG. 2). In response to identifying the hand (112-1) moving out of the FoV (1310), the wearable device (101) may stop displaying the virtual object (1330) corresponding to the hand (112-1).
[0186] In one embodiment, the function of the wearable device (101) to store information related to the virtual object (1330) in the memory may correspond to a function (e.g., a copy function and / or a cut function) for storing the virtual object (1330), such as a clipboard (or inventory). The clipboard corresponds to a designated area of the memory of the wearable device (101) and may be formed by the wearable device (101) to support copying and / or moving of information by the user (110). The wearable device (101) may store information about a graphical object displayed three-dimensionally in the screen (1320), such as the virtual object (1330), in the memory based on the clipboard. The present disclosure is not limited thereto, and the wearable device (101) may store multimedia content such as images and / or videos, contact text, and / or links for executing software applications (e.g., text in the format of a uniform resource indicator (URI)) in the memory based on the clipboard.
[0187] Referring to FIG. 13A, the wearable device (101) may stop displaying a virtual object (1320) based on identifying a hand (112-1) moving out of the FoV (1310) and display a visual object (1350) to indicate that information about the virtual object (1320) has been stored in memory. In a state (1301) in which the hand (112-1) moving out of the FoV (1310) along the path (1341) is identified, the wearable device (101) may remove a virtual object (1330) displayed based on a position (ip2) corresponding to an edge of the FoV (1310) overlapping the path (1341) from the screen (1320). A visual object (1350) containing designated text (e.g., “saved”) to indicate that the information has been stored in the memory is illustratively shown, but the present disclosure is not limited thereto.
[0188] According to one embodiment, the wearable device (101) may, after a state (1301) in which the hand (112-1) has moved out of the FoV (1310), re-execute a function related to a virtual object (1330) stored in the memory based on the re-entry of the hand (112-1) into the FoV (1310). Referring to FIG. 13B, within a state (1302) after the state (1301), the user (110) may move the hand (112-1) at a location (fp3) outside the FoV (1310) into the FoV (1310) along a path (1343). The wearable device (101) can identify the re-entry of the hand (112-1) into the FoV (1310) from the timing at which the hand (112-1) moving along the path (1341) moves into the FoV (1310). The wearable device (101) that has identified the re-entry of the hand (112-1) into the FoV (1310) can identify whether the posture of the hand (112-1) has a designated posture for calling a clipboard (or inventory) in which information about a virtual object (1330) is stored.
[0189] Referring to FIG. 13B, based on identifying a hand (112-1) having a posture of a pointing gesture, the wearable device (101) may execute a function to display at least one virtual object (e.g., a virtual object (1330) stored in the state (1301) of FIG. 13A) based on information stored in the memory. In one embodiment of FIG. 13B, an action of a user (110) moving the hand (112-1) having the pointing gesture from outside the FoV (1310) into the FoV (1310) may correspond to an input indicating the display of information stored in the memory. In response to the input, the wearable device (101) may display at least one virtual object corresponding to the information on the screen (1320).
[0190] Referring to FIG. 13b, based on identifying a hand (112-1) moving from outside the FoV (1310) into the FoV (1310) along a path (1343), the wearable device (101) can display a list of information stored in the memory based on the clipboard within the screen (1320). Having identified the hand (112-1) moving into the FoV (1310) along the path (1343), the wearable device (101) can display a virtual object (e.g., the virtual object (142) of FIG. 1) moving along a path (1344) corresponding to the path (1343). The list can include one or more virtual objects arranged in a circular shape. Referring to FIG. 13b, in an exemplary state (1302) in which four virtual objects (1330, 1331, 1332, 1333) are stored based on a clipboard, the wearable device (101) may display the four virtual objects (1330, 1331, 1332, 1333) in the shape of a circle within the screen (1320). The center of the circle may correspond to a position (ip5) within the screen (1320) corresponding to a position (fp5) within the FoV (1310) of the hand (112-1) moved within the FoV (1310).
[0191] Although an embodiment of displaying virtual objects (1330, 1331, 1332, 1333) stored in a wearable device (101) based on the shape of a circle is illustrated, the present disclosure is not limited thereto. The wearable device (101) may display a virtual object corresponding to information that was last stored in the memory based on the clipboard (e.g., a virtual object (1330) stored in the memory within the state (1301) of FIG. 13A prior to the state (1302) of FIG. 13B). The wearable device (101) may display information corresponding to a software application displayed through the screen (1320) among the information stored in the memory based on the clipboard.
[0192] Referring to FIG. 13B, in a state (1302) in which a list of information stored in a memory based on a clipboard is displayed on a screen (1320), the wearable device (101) may stop displaying the list based on the position, path, and / or posture of the hand (112-1) within the FoV (1310). For example, the wearable device (101) may stop displaying the list based on identifying the hand (112-1) changing to a posture different from the posture of the pointing gesture in an area (1350) including the center of virtual objects (1330, 1331, 1332, 1333) arranged in the shape of a circle.
[0193] Referring to FIG. 13b, in a state (1302) where a list of information stored in a memory based on a clipboard is displayed on a screen (1320), the wearable device (101) can display at least one of virtual objects (1330, 1331, 1332, 1333) included in the list based on the position, path, and / or posture within the FoV (1310) of the hand (112-1). Referring to FIG. 13c, the wearable device (101) that identifies a hand (112-1) that has moved from a position (fp5) to a position (fp6) within the FoV (1310) along a path (1345) can determine that the hand (112-1) has moved from a position (ip5) to a position (ip6) within the screen (1320) where a virtual object (1330) is displayed along a path (1346) within the screen (1320) corresponding to the path (1345).
[0194] In state (1303) of FIG. 13c, based on identifying that the posture of the hand (112-1) placed at the position (fp6) within the FoV (1310) has changed to a posture different from the posture of the pointing gesture, the wearable device (101) can display a virtual object (1330) corresponding to the position (fp6) within the screen (1320). The position (fp6) can correspond to a position (ip6) within an area (1360) related to the virtual object (1330) within the screen (1320). For example, the wearable device (101) that identifies that the hand (112-1) has changed to a posture in which all fingers are spread out can switch to state (1304) that displays the virtual object (1330) within the screen (1320). Referring to FIG. 13c, within the state (1304), the wearable device (101) can display a virtual object (1330) at a location (ip6) within the screen (1320) corresponding to the location (fp6).
[0195] Although the operation of the wearable device (101) based on a hand (112-1) entering the FoV (1310) through an edge corresponding to the top of the FoV (1310) among the edges of the FoV (1310) has been described, the present disclosure is not limited thereto.
[0196] Hereinafter, with reference to FIGS. 14a, 14b and 14c, the operation of a wearable device (101) for adding one or more virtual objects to a screen (1320) based on a hand (112-1) moving from outside the FoV (1310) into the FoV (1310) is described.
[0197] FIGS. 14A, 14B, and 14C are diagrams illustrating different exemplary states of a screen displayed by a wearable device (101) based on the position of a body part with respect to a field-of-view (FoV) of a camera. The wearable device (101) of FIGS. 1A and 1B, FIG. 2, and / or the processor (210) of FIG. 2 may perform the operations described with reference to FIGS. 14A, 14B, and 14C.
[0198] Referring to FIGS. 14A-14C , exemplary states (1401, 1402, 1403) of a wearable device (101) that have identified hands (112-1, 112-2) of a user (110) within a FoV of one or more cameras (e.g., camera (225) of FIG. 2 ) are illustrated. In one embodiment, the wearable device (101) can identify the hands (112-1, 112-2) using images and / or videos corresponding to the FoV.
[0199] In state (1401) of FIG. 14A, the wearable device (101) can display a screen (1410) including virtual objects (1431, 1432) on a display (e.g., display (220) of FIG. 2). The virtual objects (1431, 1432) can be displayed based on a software application (e.g., a game application) executed by the wearable device (101). The wearable device (101) can display virtual objects (1421, 1422) corresponding to each of the hands (112-1, 112-2) within the screen (1410). For example, the wearable device (101) can display the virtual object (1421) corresponding to the hand (112-1) based on the position of the hand (112-1) within the FoV. For example, the wearable device (101) can display a virtual object (1422) corresponding to the hand (112-2) based on the position within the FoV of the hand (112-2).
[0200] Within the state (1401) of FIG. 14A, the wearable device (101) can change the display of virtual objects (1421, 1422, 1431, 1432) displayed on the screen (1410) based on gestures performed by each of the hands (112-1, 112-2). For example, the wearable device (101) that identifies a hand (112-2) having a posture of a clenching gesture at a position (hl1) within the FoV can display a virtual object (1422) having a shape similar to that of grabbing a virtual object (1432) displayed at a position (vl1) within the screen (1410) corresponding to the position (hl1). For example, a wearable device (101) that identifies a hand (112-1) having a posture of a clenching gesture at a location (hr1) within the FoV can display a virtual object (1421) having a shape similar to that of grabbing a virtual object (1431) displayed at a location (vr1) corresponding to the location (hr1).
[0201] Within the state (1401) of FIG. 14a, the wearable device (101) can identify an input indicating movement of a virtual object (1431) based on a hand (112-1) moving within the FoV. Based on the operations described above with reference to FIGS. 4 and 5, the wearable device (101) that has identified a clenching gesture of the hand (112-1) performed within the FoV can move virtual objects (1421, 1431) along the path (1441) of the hand (112-1) that maintains the clenching gesture. A wearable device (101) that moves virtual objects (1421, 1431) in response to a hand (112-1) making a clenching gesture can provide a user experience, such as moving a virtual object (1431) by the hand (112-1), to a user (110) wearing the wearable device (101).
[0202] In an exemplary state (1401) of FIG. 14A, while a hand (112-1) having a posture of a clenching gesture is moved along a path (1441), the wearable device (101) can move virtual objects (1421, 1431) based on a path (1442) within a screen (1420) corresponding to the path (1441). For example, based on the path (1441) of the hand (112-1) having the clenching gesture, the wearable device (101) can move virtual objects (1421, 1431) displayed at a location (vr1) within the screen (1420) to a location (vr2) along a path (1442) corresponding to the path (1441).
[0203] Referring to FIG. 14A, in a state (1401) of moving virtual objects (1421, 1431) based on a hand (112-1) of a clench gesture, the wearable device (101) may stop moving the virtual objects (1421, 1431) based on whether the posture of the hand (112-1) changes to a posture different from the posture of the clench gesture. For example, in response to identifying that the clench gesture performed by the hand (112-1) is released, the wearable device (101) may stop moving the position of the virtual object (1431) based on the position and / or path (1441) within the FoV of the hand (112-1). In the above example, the position of the virtual object (1431) can be fixed based on the position within the FoV of the hand (112-1) at the timing when the release of the clench gesture is identified. In the above example, the virtual object (1421) for indicating the position of the hand (112-1) can be moved based on the position of the hand (112-1) by the wearable device (101) even after the release of the clench gesture.
[0204] Referring to FIG. 14A, based on the hand (112-1) performing the clench gesture to change the size of the virtual object (1431) moving out of the FoV, the wearable device (101) may store information related to the virtual object (1431) in the memory of the wearable device (101) (e.g., the memory (215) of FIG. 2). In response to identifying the hand (112-1) moving out of the FoV through the location (hr2) overlapping the edge of the FoV, the wearable device (101) may stop displaying the virtual object (1431).
[0205] According to one embodiment, the wearable device (101) can re-execute a function related to a virtual object (1431) stored in the memory based on the re-entry of the hand (112-1) into the FoV after the state (1401) of FIG. 14a in which the hand (112-1) moved out of the FoV. Referring to FIG. 14b, in the state (1402) after the state (1401), the user (110) can move the hand (112-1) at the location (hr3) outside the FoV into the FoV along the path (1443). The wearable device (101) can identify the re-entry of the hand (112-1) into the FoV from the timing at which the hand (112-1) moved along the path (1443) moves into the FoV. The wearable device (101) that has identified the re-entry of the hand (112-1) into the FoV can identify whether the posture of the hand (112-1) has a designated posture for resuming display of the virtual object (1431).
[0206] Referring to FIG. 14B, based on identifying a hand (112-1) having a posture of a clenching gesture, the wearable device (101) can execute a function to resume display of a virtual object (1431). In one embodiment of FIG. 14B, an action of a user (110) moving the hand (112-1) having the clenching gesture from outside the FoV into the FoV can correspond to an input to resume display of the virtual object (1431). In response to the input, the wearable device (101) can display the virtual object (1431) on the screen (1410).
[0207] Referring to FIG. 14b, based on identifying a hand (112-1) moving from outside the FoV into the FoV along a path (1443), the wearable device (101) can display a virtual object (1431) within the screen (1420). Based on a path (1444) corresponding to the path (1443) of the hand (112-1) moving into the FoV, the wearable device (101) can display a virtual object (1421) and a virtual object (1431) corresponding to the hand (112-1). When the hand (112-1) moves from a position (hr3) outside the FoV along the path (1443) to a position (hr5) passing through a position (hr4) at the edge of the FoV, the wearable device (101) can display virtual objects (1421, 1431) that move from a position (vr4) within the screen (1410) corresponding to the position (hr4) to a position (vr5) within the screen (1410) corresponding to the position (hr5) along the path (1444).
[0208] Referring to FIG. 14c, in a state (1403) after the state (1401) in which the display of the virtual object (1431) is stopped based on FIG. 14a, the wearable device (101) may keep stopping the display of the virtual object (1431) if the posture of the hand (112-1) moving into the FoV is different from the designated posture for resuming the display of the virtual object (1431). For example, based on the hand (112-1) moving into the FoV in a posture different from the posture for a clenching gesture (e.g., a posture with all fingers spread out), the wearable device (101) may not display the virtual object (1431) independently of the path (1445) of the hand (112-1). In a state (1403) where the virtual object (1431) is not displayed, the wearable device (101) can display a virtual object (1421) corresponding to the hand (112-1) within the screen (1410). For example, the wearable device (101) can move the virtual object (1421) along a path (1446) within the screen (1410) corresponding to the path (1445) of the hand (112-1) within the screen (1410). For example, when a hand (112-1) moves from a location (hr6) outside the FoV along a path (1445) to a location (hr8) at the edge of the FoV, the wearable device (101) can display a virtual object (1421) that moves from a location (vr7) within the screen (1410) corresponding to the location (hr7) along a path (1446) to a location (vr8) within the screen (1410) corresponding to the location (hr8).
[0209] Hereinafter, the operation of the wearable device (101) described with reference to FIG. 15, FIG. 12a and FIG. 12b and / or FIG. 14a, FIG. 14b and FIG. 14c is described.
[0210] FIG. 15 is a flowchart illustrating exemplary operations of a wearable device according to one embodiment. The wearable device (101) of FIG. 1A and FIG. 1B, FIG. 2, and / or the processor (210) of FIG. 2 may perform the operations of the wearable device described with reference to FIG. 15. The operations of the wearable device described with reference to FIG. 15 may be related to, or performed similarly to, at least one of the operations of FIG. 3 and / or FIG. 8.
[0211] Referring to FIG. 15, in operation (1510), a processor of a wearable device according to an embodiment may change the position of a virtual object displayed in a display based on identifying a gesture indicating movement of a virtual object using a camera. The camera of operation (1510) may include the camera (225) of FIG. 2. The display of operation (1510) may include the display (220) of FIG. 2. The wearable device may change the position of the virtual object based on a path of a body part (e.g., hand (112-1) of FIGS. 13A to 13C and / or FIGS. 14A to 14C) that maintains the gesture.
[0212] Referring to FIG. 15, within operation (1520), a processor of a wearable device according to an embodiment may identify whether a body part performing a gesture moves out of the FoV of a camera. The processor may perform operation (1520) similar to operation (320) of FIG. 3 and / or operation (820) of FIG. 8. For example, before the body part performing the gesture moves out of the FoV or while it is moving within the FoV (1520—No), the processor may change the position of the virtual object based on operation (1510). Within a state where the body part moving out of the FoV is identified (1520—Yes), the processor may perform operation (1530).
[0213] Referring to FIG. 15, in operation (1530), according to one embodiment, a processor of a wearable device may stop displaying a virtual object within the display. The processor may store information for resuming display of the virtual object within a memory (e.g., memory (215) of FIG. 2). After stopping display of the virtual object based on a body part moving out of the FoV, the processor may identify the re-entry of the body part.
[0214] Referring to FIG. 15, within operation (1540), a processor of a wearable device according to an embodiment may identify a body part moving into the FoV of a camera. Prior to identifying the body part moving into the FoV (1540 - No), the processor may not display a virtual object based on operation (1530). Based on identifying the body part moving into the FoV (1540 - Yes), the processor may perform operation (1550). The processor may perform operation (1540) of FIG. 15 similarly to operation (340) of FIG. 3 and / or operation (840) of FIG. 8.
[0215] Referring to FIG. 15, in operation (1550), a processor of a wearable device according to an embodiment may identify a gesture indicating a display of a virtual object based on a body part moved into the FoV. The gesture of operation (1550) may match the gesture of operation (1510). For example, the processor may identify whether a gesture performed by a body part moved into the FoV corresponds to the gesture of operation (1510). If the gesture of operation (1550) is not identified (1550-No), or prior to identifying the gesture of operation (1550), the processor may continue to limit the display of the virtual object based on operation (1530). Based on identifying the gesture of operation (1550) (1550-Yes), the processor may perform operation (1560).
[0216] Referring to FIG. 15, in operation (1560), according to one embodiment, a processor of a wearable device may display a panel for resuming display of a virtual object within the display. The panel may include a list including one or more virtual objects, as described with reference to FIG. 13B. Based on the panel, the processor may identify an input for resuming display of the virtual object of operation (1530). Based on the input, the processor may display the virtual object.
[0217] As described above, in one embodiment, the processor of a wearable device can execute a function for controlling a virtual object using the finite range of a camera and / or sensor for detecting a body part, such as a hand. The function can be re-executed based on the re-entry of a body part that has moved out of the range.
[0218] Hereinafter, with reference to FIGS. 16A and 16B and / or FIGS. 17A and 17B, an example of a form factor of a wearable device (101) according to one embodiment is described.
[0219] FIG. 16A is a perspective view illustrating an exemplary wearable device according to one embodiment. FIG. 16B is a perspective view illustrating an exemplary configuration of one or more hardware elements disposed within a wearable device (1600) according to one embodiment. The wearable device (101) of FIGS. 1A and 1B, and / or 2 may include the wearable device (1600) of FIGS. 16A and 16B. As shown in FIG. 16A, the wearable device (1600) according to one embodiment may include at least one display (1650) and a frame supporting the at least one display (1650).
[0220] According to one embodiment, a wearable device (1600) can be worn on a part of a user's body. The wearable device (1600) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines AR and VR to a user wearing the wearable device (1600). For example, the wearable device (1600) can output a virtual reality image to the user through at least one display (1650) in response to a user's designated gesture acquired through the motion recognition camera (1640-2) of FIG. 16B.
[0221] According to one embodiment, at least one display (1650) within a wearable device (1600) may provide visual information to a user. The at least one display (1650) may include the display (220) of FIG. 2. For example, the at least one display (1650) may include a transparent or translucent lens (1609) (e.g., a lens (1609-1) corresponding to the user's left eye and a lens (1609-2) corresponding to the user's right eye). The at least one display (1650) may include a first display (1650-1) and / or a second display (1650-2) spaced apart from the first display (1650-1). For example, the first display (1650-1) and the second display (1650-2) may be positioned at positions corresponding to the user's left eye and right eye, respectively.
[0222] Referring to FIG. 16B, at least one display (1650) may form a display area on a lens to provide a user wearing the wearable device (1600) with visual information contained in external light passing through the lens, along with other visual information distinct from the visual information. The lens may be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. The display area formed by at least one display (1650) may be formed on a second surface (1632) among a first surface (1631) and a second surface (1632) of the lens. When a user wears the wearable device (1600), external light may be incident on the first surface (1631) and transmitted through the second surface (1632) to the user. As another example, at least one display (1650) may display a virtual reality image to be combined with a real screen transmitted via external light. The virtual reality image output from the at least one display (1650) may be transmitted to the user's eyes via one or more hardware included in the wearable device (1600) (e.g., optical devices (1682, 1684) and / or at least one waveguide (1633, 1634)).
[0223] According to one embodiment, a wearable device (1600) may include waveguides (1633, 1634) that diffract light transmitted from at least one display (1650) and relayed by optical devices (1682, 1684) and transmit the diffracted light to a user. The waveguides (1633, 1634) may be formed based on at least one of glass, plastic, or polymer. Nanopatterns may be formed on at least a portion of the exterior or interior of the waveguides (1633, 1634). The nanopatterns may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the waveguides (1633, 1634) may be propagated to the other end of the waveguides (1633, 1634) by the nanopatterns. The waveguides (1633, 1634) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, the waveguides (1633, 1634) may be arranged within the wearable device (1600) to guide a screen displayed by at least one display (1650) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) occurring within the waveguides (1633, 1634).
[0224] According to one embodiment, a wearable device (1600) may analyze an object included in a real image collected through a capturing camera (1640-1), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (1650). The virtual object may include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (1600) may analyze the object based on a multi-camera, such as a stereo camera. For the object analysis, the wearable device (1600) may execute time-of-flight (ToF) and / or simultaneous localization and mapping (SLAM) supported by the multi-camera. A user wearing the wearable device (1600) may view an image displayed on at least one display (1650).
[0225] According to one embodiment, the frame may be formed as a physical structure that allows the wearable device (1600) to be worn on the user's body. According to one embodiment, the frame may be configured so that, when the user wears the wearable device (1600), the first display (1650-1) and the second display (1650-2) can be positioned corresponding to the user's left and right eyes. The frame may support at least one display (1650). For example, the frame may support the first display (1650-1) and the second display (1650-2) to be positioned corresponding to the user's left and right eyes.
[0226] Referring to FIG. 16A, the frame may include an area (1620) that at least partially contacts a portion of the user's body when the user wears the wearable device (1600). For example, the area (1620) of the frame that contacts a portion of the user's body may include an area that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (1600) makes contact with. According to one embodiment, the frame may include a nose pad (1610) that contacts a portion of the user's body. When the wearable device (1600) is worn by the user, the nose pad (1610) may contact a portion of the user's nose. The frame may include a first temple (1604) and a second temple (1605) that contact another portion of the user's body that is distinct from the portion of the user's body.
[0227] In one embodiment, the frame may include a first rim (1601) surrounding at least a portion of a first display (1650-1), a second rim (1602) surrounding at least a portion of a second display (1650-2), a bridge (1603) disposed between the first rim (1601) and the second rim (1602), a first pad (1611) disposed along a portion of an edge of the first rim (1601) from one end of the bridge (1603), a second pad (1612) disposed along a portion of an edge of the second rim (1602) from the other end of the bridge (1603), a first temple (1604) extending from the first rim (1601) and secured to a portion of an ear of the wearer, and a second temple (1605) extending from the second rim (1602) and secured to a portion of an ear opposite the ear. The first pad (1611) and the second pad (1612) may be in contact with a portion of the user's nose, and the first temple (1604) and the second temple (1605) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (1604, 1605) may be rotatably connected to the rim via hinge units (1606, 1607) of FIG. 16B. The first temple (1604) may be rotatably connected to the first rim (1601) via a first hinge unit (1606) disposed between the first rim (1601) and the first temple (1604). The second temple (1605) may be rotatably connected to the second rim (1602) via a second hinge unit (1607) disposed between the second rim (1602) and the second temple (1605). In one embodiment, the wearable device (1600) may use a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame to identify an external object (e.g., a user's fingertip) touching the frame and / or a gesture performed by the external object.
[0228] According to one embodiment, a wearable device (1600) may include hardwares that perform various functions (e.g., hardwares described above based on the block diagram of FIG. 2). For example, the hardwares may include a battery module (1670), an antenna module (1675), optical devices (1682, 1684), speakers (1692-1, 1692-2), microphones (1694-1, 1694-2, 1694-3), a light-emitting module (not shown), and / or a printed circuit board (1690). The various hardwares may be arranged within a frame.
[0229] According to one embodiment, microphones (1694-1, 1694-2, 1694-3) of a wearable device (1600) may be arranged on at least a portion of a frame to acquire sound signals. A first microphone (1694-1) arranged on a nose pad (1610), a second microphone (1694-2) arranged on a second rim (1602), and a third microphone (1694-3) arranged on a first rim (1601) are illustrated in FIG. 16B , but the number and arrangement of the microphones (1694) are not limited to the embodiment of FIG. 16B . When the number of microphones (1694) included in the wearable device (1600) is two or more, the wearable device (1600) may identify the direction of a sound signal by using a plurality of microphones arranged on different portions of the frame.
[0230] According to one embodiment, the optical devices (1682, 1684) can transmit a virtual object transmitted from at least one display (1650) to the wave guides (1633, 1634). For example, the optical devices (1682, 1684) can be projectors. The optical devices (1682, 1684) can be positioned adjacent to at least one display (1650) or can be included within at least one display (1650) as a part of the at least one display (1650). The first optical device (1682) can correspond to the first display (1650-1), and the second optical device (1684) can correspond to the second display (1650-2). The first optical device (1682) can transmit light output from the first display (1650-1) to the first waveguide (1633), and the second optical device (1684) can transmit light output from the second display (1650-2) to the second waveguide (1634).
[0231] In one embodiment, the camera (1640) may include an eye tracking camera (ET CAM) (1640-1), a motion recognition camera (1640-2), and / or a recording camera (1640-3). The recording camera (1640-3), the eye tracking camera (1640-1), and the motion recognition camera (1640-2) may be positioned at different locations on the frame and may perform different functions. The recording camera (1640-3), the eye tracking camera (1640-1), and the motion recognition camera (1640-2) may be an example of the camera (225) of FIG. 2. The eye tracking camera (1640-1) may output data representing the gaze of a user wearing the wearable device (1600). For example, the wearable device (1600) can detect the gaze from an image including the user's pupils, obtained through the gaze tracking camera (1640-1). Although an example in which the gaze tracking camera (1640-1) is positioned toward the user's right eye is illustrated in FIG. 16B, the present disclosure is not limited thereto, and the gaze tracking camera (1640-1) may be positioned solely toward the user's left eye, or toward both eyes.
[0232] In one embodiment, the capturing camera (1640-3) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (1650). The at least one display (1650) can display a single image in which information about the actual image or background including the image of the specific object acquired using the capturing camera is superimposed on a virtual image provided through optical devices (1682, 1684). In one embodiment, the capturing camera can be placed on a bridge (1603) disposed between the first rim (1601) and the second rim (1602).
[0233] In one embodiment, the gaze tracking camera (1640-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (1600) and matching the user's gaze with visual information provided to at least one display (1650). For example, when the wearable device (1600) looks straight ahead, the wearable device (1600) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (1650). The gaze tracking camera (1640-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (1640-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (1640-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (1640-1) may be positioned within the first rim (1601) and / or the second rim (1602) to face the direction in which the user wearing the wearable device (1600) is positioned.
[0234] In one embodiment, the gesture recognition camera (1640-2) can provide a specific event on a screen provided on at least one display (1650) by recognizing the movement of the user's entire body, such as the user's torso, hands, or face, or a part of the body. The gesture recognition camera (1640-2) can recognize the user's gesture, obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (1650). The processor can identify the signal corresponding to the gesture and perform a designated function based on the identification. In one embodiment, the gesture recognition camera (1640-2) can be disposed on the first rim (1601) and / or the second rim (1602).
[0235] In one embodiment, the camera (1640) included in the wearable device (1600) is not limited to the gaze tracking camera (1640-1) and the motion recognition camera (1640-2) described above. For example, the wearable device (1600) can identify an external object included in the FoV using a photographing camera (1640-3) positioned toward the user's FoV. The wearable device (1600) can identify an external object based on a sensor for identifying the distance between the wearable device (1600) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (1640) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (1600) may include a camera (1640) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (1600).
[0236] Although not shown, in one embodiment, the wearable device (1600) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being photographed using the camera (1640). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame and hinge units (1606, 1607).
[0237] According to one embodiment, the battery module (1670) may supply power to the electronic components of the wearable device (1600). In one embodiment, the battery module (1670) may be disposed within the first temple (1604) and / or the second temple (1605). For example, the battery module (1670) may be a plurality of battery modules (1670). The plurality of battery modules (1670) may be disposed within each of the first temple (1604) and the second temple (1605). In one embodiment, the battery module (1670) may be disposed at an end of the first temple (1604) and / or the second temple (1605).
[0238] In one embodiment, the antenna module (1675) can transmit signals or power to, or receive signals or power from, the outside of the wearable device (1600). The antenna module (1675) can be electrically and / or operatively connected to communication circuitry within the wearable device (1600) (e.g., communication circuitry (240) of FIG. 2 ). In one embodiment, the antenna module (1675) can be positioned within the first temple (1604) and / or the second temple (1605). For example, the antenna module (1675) can be positioned proximate one surface of the first temple (1604) and / or the second temple (1605).
[0239] In one embodiment, the speakers (1692-1, 1692-2) may output audio signals to the outside of the wearable device (1600). The audio output module may be referred to as a speaker. In one embodiment, the speakers (1692-1, 1692-2) may be positioned within the first temple (1604) and / or the second temple (1605) so as to be positioned adjacent to the ears of a user wearing the wearable device (1600). For example, the wearable device (1600) may include a second speaker (1692-2) positioned within the first temple (1604) and thus adjacent to the user's left ear, and a first speaker (1692-1) positioned within the second temple (1605) and thus adjacent to the user's right ear.
[0240] In one embodiment, the light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state to visually provide a user with information regarding a specific state of the wearable device (1600). For example, when the wearable device (1600) requires charging, the wearable device (1600) may repeatedly emit red light at a designated time. In one embodiment, the light-emitting module may be disposed on the first rim (1601) and / or the second rim (1602).
[0241] Referring to FIG. 16B, according to one embodiment, a wearable device (1600) may include a printed circuit board (PCB) (1690). The PCB (1690) may be included in at least one of the first temple (1604) or the second temple (1605). The PCB (1690) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (1600) (e.g., hardwares illustrated by the blocks described above with reference to FIG. 2) may be disposed on the PCB (1690). The wearable device (1600) may include a flexible PCB (FPCB) for interconnecting the hardwares.
[0242] According to one embodiment, a wearable device (1600) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (1600) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (1600). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (1600) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (1600) based on the IMU.
[0243] Figures 17a and 17b are perspective views illustrating exemplary exterior appearances of a wearable device (1700) according to one embodiment. The wearable device (101) of Figures 1a and 1b and 2 may include the wearable device (1700) of Figures 17a and 17b. An example of the exterior appearance of a first side (1710) of a housing of the wearable device (1700) according to one embodiment is illustrated in Figure 17a, and an example of the exterior appearance of a second side (1720) opposite to the first side (1710) may be illustrated in Figure 17b.
[0244] Referring to FIG. 17A, a first surface (1710) of a wearable device (1700) according to one embodiment may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (1700) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (1604) and / or the second temple (1605) of FIGS. 16A and 16B). A first display (1750-1) for outputting an image to a left eye among the user's two eyes, and a second display (1750-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (1710). The wearable device (1700) may further include a rubber or silicone packing formed on the first surface (1710) to prevent / reduce interference from light (e.g., ambient light) that is different from the light emitted from the first display (1750-1) and the second display (1750-2).
[0245] According to one embodiment, a wearable device (1700) may include cameras (1740-3, 1740-4) for photographing and / or tracking both eyes of a user adjacent to each of the first display (1750-1) and the second display (1750-2). The cameras (1740-3, 1740-4) may be referred to as ET cameras. According to one embodiment, a wearable device (1700) may include cameras (1740-1, 1740-2) for photographing and / or recognizing a face of a user. The cameras (1740-1, 1740-2) may be referred to as FT cameras.
[0246] Referring to FIG. 17B, a camera (e.g., cameras (1740-5, 1740-6, 1740-7, 1740-8, 1740-9, 1740-10)) and / or a sensor (e.g., a depth sensor (1730)) for obtaining information related to the external environment of the wearable device (1700) may be disposed on a second surface (1720) opposite to the first surface (1710) of FIG. 17A. For example, the cameras (1740-5, 1740-6, 1740-7, 1740-8, 1740-9, 1740-10) may be disposed on the second surface (1720) to recognize external objects different from the wearable device (1700). For example, using cameras (1740-9, 1740-10), the wearable device (1700) can acquire images and / or media to be transmitted to each of the user's eyes. The camera (1740-9) can be positioned on a second surface (1720) of the wearable device (1700) to acquire a frame to be displayed through a second display (1750-2) corresponding to the right eye among the two eyes. The camera (1740-10) can be positioned on a second surface (1720) of the wearable device (1700) to acquire a frame to be displayed through a first display (1750-1) corresponding to the left eye among the two eyes.
[0247] In one embodiment, the wearable device (1700) may include a depth sensor (1730) disposed on the second surface (1720) to identify a distance between the wearable device (1700) and an external object. Using the depth sensor (1730), the wearable device (1700) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (1700).
[0248] Although not shown, a microphone may be placed on the second side (1720) of the wearable device (1700) to acquire sound output from an external object. The number of microphones may be one or more, depending on the embodiment.
[0249] In one embodiment, a method may be required to identify re-entry of a body part into a range detectable by a camera and / or sensor of a wearable device after the body part has moved out of the range. As described above, according to one embodiment, a wearable device (e.g., wearable device (101) of FIGS. 1A-1B and 2 and / or electronic device (101-1) of FIGS. 7A-7C) may include at least one processor (e.g., processor (210) of FIG. 2) including a camera (e.g., camera (225) of FIG. 2), a display (e.g., display (220) of FIG. 2), and processing circuitry. At least one processor, individually and / or collectively, displays a virtual object (e.g., a virtual object (140) of FIGS. 1A to 1B, a virtual object (630) of FIGS. 6A to 6C, a virtual object (925) of FIGS. 9A to 9C, a virtual object (1030) of FIG. 10, a virtual object (1130) of FIGS. 11A to 11C, a virtual object (1231) of FIGS. 12A to 12B, a virtual object (1330) of FIGS. 13A to 13C, virtual objects (1421, 1422, 1431, 1432) of FIGS. 14A to 14C) in a virtual space via the display, while the field-of-view (FoV) of the camera (e.g., a FoV (120) of FIGS. 1A to 1B, 4 to 5, 6A to 6C) It can be configured to identify a body part within the FoV (610), the FoV (710) of FIGS. 7A to 7C, the FoV (910) of FIGS. 9A to 9C, the FoV (1010) of FIG. 10, the FoV (1110) of FIGS. 11A to 11C, the FoV (1210) of FIGS. 12A to 12B, and the FoV (1310) of FIGS. 13A to 13C).At least one processor may be configured, individually and / or collectively, to control a display to display the virtual object moving along a path (e.g., path (642) of FIG. 6A) of the body part moving within the FoV based on identifying the body part in a designated pose for movement of the virtual object. At least one processor may be configured, individually and / or collectively, to control the display to display the virtual object at a second location within the virtual space corresponding to the first location based on identifying, using the camera, that the body part is moving outside the FoV through a first location at an edge of the FoV within the state of moving the virtual object along the path. At least one processor may be configured, individually and / or collectively, to control the display to display the virtual object, based on identifying the body part of the designated pose that has moved into the FoV after displaying the virtual object at the second location, and moving along a path of the body part within the FoV (e.g., path (652) of FIG. 6B ). In one embodiment, the wearable device may re-execute a function associated with the virtual object that was matched to the body part based on the body part moving out of the FoV (or a range detectable by a sensor) of the camera, and then re-entering the FoV of the body part.
[0250] For example, at least one processor may be configured, individually and / or collectively, to determine, based on displaying the virtual object at the second location, whether the body part identified using the camera that has moved into the FoV corresponds to the body part that has moved out of the FoV, using information about the area in which the body part that has moved out of the FoV can move.
[0251] For example, at least one processor may be configured, individually and / or collectively, to disconnect the connection between the virtual object and the external space based on identifying that the body part in the designated pose moves into the FoV while displaying the virtual object connected to the external space of the area currently displayed through the display in the virtual space. The at least one processor may be configured, individually and / or collectively, to maintain the second position of the virtual object within the display independently of movement of the wearable device from the currently displayed area to the external space while the body part moves out of the FoV.
[0252] For example, at least one processor may be configured, individually and / or collectively, to change the position of the virtual object from the second position to a third position in the external space based on at least one of information about an area in which the body part can move and a movement speed of the body part, based on identifying, within the FoV, a body part having a posture different from the designated posture, based on displaying the virtual object at the second position.
[0253] For example, the wearable device may further include a memory (e.g., memory (215) of FIG. 2). At least one processor may be configured, individually and / or collectively, to stop displaying the virtual object at the second location based on identifying, within the FoV, a body part having a posture different from the designated posture based on displaying the virtual object at the second location. The at least one processor may be configured, individually and / or collectively, to store information about the virtual object in the memory.
[0254] For example, at least one processor may be configured, individually and / or collectively, to display the virtual object corresponding to the information on the display based on identifying an input representing a representation of the information stored in the memory based on a pose of the body part moving within the FoV.
[0255] For example, at least one processor may be configured, individually and / or collectively, to move the virtual object using the first velocity based on identifying that the body part is moving outside the FoV at the first location at the first velocity.
[0256] For example, at least one processor may be configured, individually and / or collectively, to identify the specified posture based on an application being executed by the at least one processor.
[0257] As described above, according to one embodiment, a method of a wearable device may include an operation of identifying a body part within a field-of-view (FoV) of a camera of the wearable device based on a video acquired from a camera of the wearable device while displaying a virtual object within a virtual space through a display of the wearable device. The method may include an operation of displaying the virtual object moving along a path of the body part moving within the FoV based on the identification of the body part in a designated posture for movement of the virtual object. The method may include an operation of displaying the virtual object at a second location within the virtual space corresponding to the first location based on the identification, from the video, that the body part is moving out of the FoV through a first location of an edge of the FoV while moving the virtual object along the path. The method may include an action of displaying the virtual object, based on identifying the body part of the specified posture that has moved into the FoV, from the second location and moving along a path of the body part within the FoV, based on displaying the virtual object at the second location.
[0258] For example, the action of displaying the virtual object moved from the second location may include an action of determining whether the body part moved into the FoV identified using the camera corresponds to the body part moved out of the FoV, using information about an area in which the body part that was moved out of the FoV can move, based on displaying the virtual object at the second location.
[0259] For example, the identifying action may include an action of disconnecting the connection between the virtual object and the external space based on identifying that the body part in the designated posture moves into the FoV while displaying the virtual object connected to the external space of the area currently displayed through the display in the virtual space. The method may include an action of maintaining the second position of the virtual object within the display independently of movement of the wearable device from the currently displayed area to the external space while the body part is moved out of the FoV.
[0260] For example, the method may include an operation of changing the position of the virtual object from the second position to a third position in external space based on at least one of a path before the body part moves out of the FoV, information about an area in which the body part can move, and a moving speed of the body part, based on identifying the body part within the FoV, which has a posture different from the designated posture, based on displaying the virtual object at the second position.
[0261] For example, the method may include an action of stopping displaying the virtual object at the second location based on identifying, within the FoV, a body part having a posture different from the designated posture based on displaying the virtual object at the second location. The method may include an action of storing information about the virtual object in a memory of the wearable device.
[0262] For example, the method may include an action of displaying the virtual object corresponding to the information on the display based on identifying an input representing a representation of the information stored in the memory based on a posture of the body part moving within the FoV.
[0263] For example, the act of displaying the virtual object at the second location may include an act of moving the virtual object using the first speed based on identifying that the body part is moving outside the FoV at the first speed at the first location.
[0264] For example, the identifying action may include an action of identifying the specified posture based on an application executed by a processor of the wearable device.
[0265] According to one embodiment, a wearable device as described above may include a camera, a display, and at least one processor including a processing circuit. The at least one processor may be configured, individually and / or collectively, to identify a body part that moves into a field-of-view (FoV) of the camera at a first timing based on the camera. The at least one processor may be configured, individually and / or collectively, to identify, based on identifying that a posture of the body part moved into the FoV at the first timing corresponds to a designated posture, at least one virtual object that was associated with the body part at a second timing prior to the first timing. The at least one processor may be configured, individually and / or collectively, to control the display to display the at least one virtual object based on a position of the body part within the FoV.
[0266] For example, at least one processor may be configured, individually and / or collectively, to use information about an area in which the body part can move to identify whether the posture of the body part corresponds to the designated posture.
[0267] For example, at least one processor may be configured, individually and / or collectively, to display the at least one virtual object at an edge of the FoV to which the body part has moved based on identifying the body part that has moved out of the FoV at the second timing.
[0268] For example, at least one processor may be configured, individually and / or collectively, to identify a path along which the body part moves within the FoV based on the camera. At least one processor may be configured, individually and / or collectively, to change a position of the at least one virtual object within the display based on the path.
[0269] According to one embodiment, a method of a wearable device as described above may include an operation of identifying a body part that moves into a FoV of a camera of the wearable device at a first timing based on a camera of the wearable device. The method may include an operation of identifying at least one virtual object that was connected to the body part at a second timing prior to the first timing based on identifying that a posture of the body part that moved into the FoV at the first timing corresponds to a designated posture. The method may include an operation of displaying at least one virtual object within a display of the wearable device based on a position of the body part within the FoV.
[0270] For example, the identifying action may include an action of identifying whether the posture of the body part corresponds to a specified posture by using information about the area in which the body part can move.
[0271] For example, the displaying action may include an action of displaying at least one virtual object at an edge of the FoV to which the body part has moved, based on identifying the body part that has moved out of the FoV at the second timing.
[0272] For example, the displaying action may include an action of identifying a path along which the body part moves within the FoV based on the camera. The method may include an action of changing the position of the at least one virtual object within the display based on the path.
[0273] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the present disclosure may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.
[0274] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.
[0275] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.
[0276] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.
[0277] It should be understood that any embodiment(s) disclosed herein may be utilized in combination with any other embodiment(s) disclosed herein.
Claims
1. In wearable devices, camera; display; Memory for storing instructions; and At least one processor comprising a processing circuit, individually and / or collectively, the wearable device, While displaying a virtual object in a virtual space through the above display, a body part within the field-of-view (FoV) of the camera is identified; Based on identifying the body part in the designated posture for movement of the virtual object, displaying the virtual object moving along the path of the body part moving within the FoV; In a state of moving the virtual object along the path, using the camera, based on identifying that the body part is moved out of the FoV through a first position of the edge of the FoV, displaying the virtual object at a second position in the virtual space corresponding to the first position; and Based on displaying the virtual object at the second location, and identifying the body part of the designated posture moved into the FoV, causing the virtual object to be displayed, moved from the second location, and moved along the path of the body part within the FoV. Wearable devices.
2. In claim 1, the at least one processor, individually and / or collectively, the wearable device, Based on the display of the virtual object at the second location, using information about the area in which the body part that has moved out of the FoV can move, determining whether the body part that has moved into the FoV identified using the camera corresponds to the body part that has moved out of the FoV is configured to cause. Wearable devices.
3. In claim 1, the at least one processor, individually and / or collectively, the wearable device, While displaying the virtual object connected to the external space of the area currently displayed through the display in the virtual space, based on identifying that the body part of the designated posture moves into the FoV, disconnecting the connection between the virtual object and the external space; configured to cause said second position of said virtual object to be maintained within said display, independently of movement of said wearable device from said currently displayed area to said external space, while said body part is moved out of said FoV; Wearable devices.
4. In claim 3, the at least one processor, individually and / or collectively, the wearable device, Based on displaying the virtual object at the second location, identifying the body part having a posture different from the designated posture within the FoV, and causing the location of the virtual object to be changed from the second location to a third location in the external space based on at least one of a path before the body part is moved out of the FoV, information about an area in which the body part can move, and a moving speed of the body part. Wearable devices.
5. In claim 3, The at least one processor, individually and / or collectively, causes the wearable device to: Based on displaying the virtual object at the second location, stopping displaying the virtual object at the second location based on identifying, within the FoV, the body part having a posture different from the designated posture; and configured to cause information about said virtual object to be stored in said memory, Wearable devices.
6. In claim 5, the at least one processor, individually and / or collectively, the wearable device, Based on the posture of the body part moving into the FoV, and based on identifying an input representing a representation of the information stored in the memory, causing the display to display the virtual object corresponding to the information. Wearable devices.
7. In claim 1, the at least one processor, individually and / or collectively, the wearable device, At the first location, based on identifying that the body part is moving out of the FoV at a first speed, causing the virtual object to move using the first speed. Wearable devices.
8. In claim 1, the at least one processor, individually and / or collectively, the wearable device, configured to cause, based on an application executed by said processor, to identify said specified posture; Wearable devices.
9. In the method of a wearable device, An operation of identifying a body part within the field-of-view (FoV) of a camera of the wearable device based on a video acquired from the camera of the wearable device while displaying a virtual object within a virtual space through the display of the wearable device; An action of displaying the virtual object moving along a path of the body part moving within the FoV based on identifying the body part in a designated posture for movement of the virtual object; An operation of displaying the virtual object at a second location in the virtual space corresponding to the first location, based on identifying that the body part has moved out of the FoV through a first location of the edge of the FoV from the video, within a state of moving the virtual object along the path; and Based on displaying the virtual object at the second location, identifying the body part of the designated posture moved into the FoV, including an action of displaying the virtual object moved from the second location and moving along the path of the body part within the FoV. method.
10. In claim 9, the action of displaying the virtual object moved from the second position is: Based on displaying the virtual object at the second location, an operation of determining whether the body part moved into the FoV identified using the camera corresponds to the body part moved out of the FoV by using information about the area in which the body part moved out of the FoV can move. method.
11. In claim 9, the identifying action comprises: An action of disconnecting the connection between the virtual object and the external space based on identifying that the body part of the designated posture moves into the FoV while displaying the virtual object connected to the external space of the area currently displayed through the display in the virtual space; An action of maintaining the second position of the virtual object within the display, independently of movement of the wearable device from the currently displayed area to the external space, while the body part is moved out of the FoV. method.
12. In claim 11, Based on displaying the virtual object at the second location, an operation of changing the location of the virtual object from the second location to a third location in the external space based on at least one of: a path before the body part is moved out of the FoV, information about an area in which the body part can move, and a moving speed of the body part, based on identifying the body part within the FoV having a different posture from the designated posture. method.
13. In claim 11, An action of stopping displaying the virtual object at the second location based on identifying, within the FoV, a body part having a posture different from the designated posture based on displaying the virtual object at the second location; and Comprising an operation of storing information about the virtual object in the memory of the wearable device, method.
14. In claim 13, Based on the posture of the body part moving into the FoV, an operation of displaying the virtual object corresponding to the information on the display is included, based on identifying an input representing a display of the information stored in the memory. method.
15. In claim 9, the action of displaying the virtual object at the second location is: In the first location, based on identifying that the body part is moving out of the FoV at a first speed, the motion of moving the virtual object using the first speed is included. method.
Citation Information
Patent Citations
Information processing apparatus, information processing method and recording medium
JP2020042369A
Information processing device, information processing method, and program
JP2021007020A
Program, information storage medium, and image generation system
JP5148660B2
Apparatus and method for interactive 3D display
KR1020170096420A
Hand motion and orientation-aware buttons and grabbable objects in mixed reality
US20200225757A1
