Wearable device, method, and computer-readable storage medium for touch input in 3D space

Abstract

This wearable device comprises: a memory storing instructions and including one or more storage mediums; a plurality of cameras; a display; and at least one processor including a processing circuit. The instructions, when executed individually or collectively by the at least one processor, cause the wearable device to: provide a 3D space via the display; display a visual object according to first depth data while providing at least a portion of the 3D space via the display; identify second depth data by changing the first depth data by using reference data; set a touch recognition area for the visual object related to the 3D space by using the second depth data; while the visual object is displayed in the 3D space, use images acquired via the plurality of cameras to identify that the position of an input means corresponds to the touch recognition area; and recognize the correspondence as a touch input for the visual object.

Description

Wearable device, method, and computer-readable storage medium for touch input in 3D space

[0001] The present disclosure relates to a wearable device, method, and computer-readable storage medium for input within a 3D space.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these electronic devices may be AR glasses and / or head-mounted devices (HMDs).

[0003] The above information may be provided as background art to aid in understanding the present disclosure.

[0004] No claim or determination is made as to whether any of the above is applicable as prior art to the present disclosure.

[0005] A wearable device is described. The wearable device may include a plurality of cameras positioned toward the eyes of a user wearing the wearable device. The wearable device may include a display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide a three-dimensional (3D) space through the display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a visual object according to first depth data while providing at least a portion of the 3D space through the display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify second depth data by modifying the first depth data using reference data. The reference data may be used to apply a display location of the visual object recognized by a user of the wearable device to touch input recognition. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to set a touch recognition area for the visual object relative to the touch recognition area within the 3D space using the second depth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, using images acquired through the plurality of cameras while the visual object is displayed within the 3D space, that a location of an input means corresponds to the touch recognition area. The instructions may cause the wearable device to recognize the correspondence as a touch input to the visual object.

[0006] A method is described. The method may be performed in a wearable device including a plurality of cameras positioned toward the eyes of a user wearing the wearable device and a display. The method may include an operation of providing a three-dimensional (3D) space through the display. The method may include an operation of displaying a visual object according to first depth data while providing at least a portion of the 3D space through the display. The method may include an operation of identifying second depth data by modifying the first depth data using reference data. The reference data may be used to apply a display position of the visual object recognized by a user of the wearable device to touch input recognition. The method may include an operation of setting a touch recognition area for the visual object related to the 3D space using the second depth data. The method may include an operation of identifying that a position of an input means corresponds to the touch recognition area using images acquired through the plurality of cameras while the visual object is displayed in the 3D space. The method may include an action of recognizing the correspondence as a touch input to the visual object.

[0007] A non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by a wearable device including a display and a plurality of cameras positioned toward the eyes of a user wearing the wearable device, cause the electronic device to provide a three-dimensional (3D) space through the display. The one or more programs may include instructions that, when executed by the electronic device, cause the wearable device to display a visual object according to first depth data while providing at least a portion of the 3D space through the display. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify second depth data by modifying the first depth data using reference data. The reference data may be used to apply a display location of the visual object recognized by a user of the wearable device to touch input recognition. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to set a touch recognition area for the visual object associated with the 3D space using the second depth data. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify, using images acquired through the plurality of cameras while the visual object is displayed within the 3D space, that a location of an input means corresponds to the touch recognition area.The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to recognize the correspondence as a touch input to the visual object.

[0008] A wearable device is described. The wearable device may include a plurality of cameras positioned toward the eyes of a user wearing the wearable device. The wearable device may include a display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide a three-dimensional (3D) space (401) through the display (110). The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a visual object based on depth data while providing at least a portion of the 3D space through the display (110). The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify an input means moved relative to the visual object using images acquired through the plurality of cameras while the visual object is displayed in the 3D space. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine a position at which a direction of movement of the input means changes with respect to depth data based on the identification. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain reference data for applying a display position of the visual object recognized by a user of the wearable device to touch input recognition using the determined position.

[0009] A method is described. The method may be performed in a wearable device including a plurality of cameras positioned toward the eyes of a user wearing the wearable device and a display. The method may include an operation of providing a three-dimensional (3D) space (401) through the display (110). The method may include an operation of displaying a visual object according to depth data while providing at least a portion of the 3D space through the display (110). The method may include an operation of identifying an input means moved with respect to the visual object using images acquired through the plurality of cameras while the visual object is displayed in the 3D space. The method may include an operation of determining a position at which a direction of movement of the input means changes with respect to the depth data based on the identification. The method may include an operation of obtaining reference data for applying a display position of the visual object recognized by a user of the wearable device to touch input recognition using the determined position.

[0010] A non-transitory computer-readable storage medium is described. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by a wearable device including a display and a plurality of cameras positioned toward the eyes of a user wearing the wearable device, cause the electronic device to provide a three-dimensional (3D) space (401) through the display (110). The one or more programs may include instructions that, when executed by the electronic device, cause the wearable device to display a visual object based on depth data while providing at least a portion of the 3D space through the display (110). The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify an input means moved with respect to a visual object using images acquired through the plurality of cameras while the visual object is displayed in the 3D space. The one or more programs may include instructions that cause the wearable device to determine a location at which the direction of movement of the input means changes with respect to the depth data based on the identification. The one or more programs may include instructions that cause the wearable device to obtain reference data for applying the display location of the visual object recognized by the user of the wearable device to touch input recognition using the determined location.

[0011] Figure 1 illustrates an example of touch input received for a visual object within 3D space.

[0012] Figure 2 is a simplified block diagram of an exemplary wearable device.

[0013] FIG. 3 is a flowchart illustrating exemplary operations of a wearable device that acquires reference data for determining the location of a touch recognition area that recognizes a touch input to a visual object according to a change in the direction of movement of an input means to a visual object.

[0014] Figure 4 illustrates an example in which the direction of movement for depth data of an input means is changed.

[0015] Figures 5a, 5b, and 5c illustrate exemplary methods for obtaining reference data for determining the location of a touch recognition area that recognizes input to a visual object.

[0016] FIG. 6 is a flowchart illustrating exemplary operations of a wearable device for identifying whether an input means is positioned within a reference spatial range of a visual object to obtain reference data.

[0017] Figure 7 illustrates an example of an input means positioned within the reference spatial range of a visual object.

[0018] FIG. 8 is a flowchart illustrating exemplary operations of a wearable device for identifying whether depth data representing a determined position of an input means to obtain reference data corresponds to depth data representing a position of a visual object.

[0019] Figure 9 illustrates an example of depth data indicating the position of a visual object and depth data indicating the determined position of another input means.

[0020] FIG. 10 is a flowchart illustrating exemplary operations of a wearable device that recognizes touch input to a visual object using second depth data offset from first depth data.

[0021] Figure 11 illustrates an example of a touch recognition area that recognizes touch input to a visual object.

[0022] Figures 12a and 12b illustrate examples of changing the reference data applied according to depth data indicating the position of a visual object.

[0023] Figure 13 illustrates an example of updating reference data using a touch recognition area associated with a visual object.

[0024] FIG. 14 is a block diagram of an electronic device within a network environment according to various embodiments.

[0025] FIG. 15a illustrates an example of a perspective view of a wearable device.

[0026] FIG. 15b illustrates an example of one or more hardware devices arranged within a wearable device.

[0027] Figures 16a and 16b illustrate examples of the appearance of a wearable device.

[0028] Figure 1 illustrates an example of touch input received for a visual object within 3D space.

[0029] Referring to FIG. 1, a wearable device (101) may include a head-mounted display (HMD) that can be worn on the head of a user (102). The wearable device (101) may be described as a head-mounted display (HMD) device, a headgear electronic device, a glasses-type (or goggle-type) electronic device, a video see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. An example of the structure of a wearable device (101) that can be worn on the head of a user is described with reference to FIGS. 15A, 15B, 16A, and / or 16B.

[0030] For example, the wearable device (101) may be available to provide augmented reality (AR) and / or mixed reality (MR).

[0031] As a non-limiting example, a first light from outside the wearable device (101) may penetrate the display (110) of the wearable device (101). For example, the wearable device (101) may provide an augmented reality and / or mixed reality to a user wearing the wearable device (101) defined using the first light penetrating the display (110) and the second light emitted from the display (110) according to an image generated by a processor of the wearable device (101).

[0032] As a non-limiting example, the wearable device (101) may include one or more cameras configured to receive the first light from outside the wearable device (101). For example, the wearable device (101) may provide an augmented reality and / or mixed reality to a user wearing the wearable device (101) defined using one or more first images acquired via the one or more cameras (e.g., representing scenes of an environment surrounding the wearable device (101)) and one or more second images generated by a processor of the wearable device (101) without the use of the one or more cameras (e.g., including one or more virtual objects).

[0033] For example, the wearable device (101) can display a visual object (104) within a three-dimensional (3D) space. For example, the visual object (104) can be associated with a touch recognition area capable of recognizing a user's touch input. For example, the visual object (104) can be associated with the touch recognition area to provide a function corresponding to the visual object (104) in response to the user's touch input.

[0034] As a non-limiting example, the visual object (104) may correspond to a virtual object within the extended reality. As a non-limiting example, the visual object (104) may correspond to an external object within the extended reality (e.g., a real object located around the wearable device (101). For example, the external object may be identified using images acquired through a plurality of cameras (220).

[0035] For example, the wearable device (101) may include multiple cameras. For example, at least some of the multiple cameras may have a field of view (FOV) corresponding to the field of view (FOV) of the eyes of a user wearing the wearable device. For example, the multiple cameras may be used to acquire images.

[0036] For example, the wearable device (101) can identify the input means (107) using images acquired through multiple cameras. For example, the wearable device (101) can identify that the input means (107) is in contact with the touch recognition area associated with the visual object (104) using images acquired through multiple cameras.

[0037] For example, the wearable device (101) may display a pointer (or cursor) corresponding to the input means through the display (110). As a non-limiting example, the shape of the pointer may be the same as the shape of the input means. As a non-limiting example, the shape of the pointer may be different from the shape of the input means. As a non-limiting example, the pointer may be an arrow. As a non-limiting example, the pointer may have a shape corresponding to the user's hand.

[0038] As a non-limiting example, the input means (107) may correspond to an external object. As a non-limiting example, the input means (107) may correspond to at least a part of the body of the user (102). As a non-limiting example, the input means (107) may correspond to a stylus pen available in conjunction with the hand of the user (102) or the wearable device (101).

[0039] For example, a user (102) can control an input means (107). For example, the input means (107) can be moved for a touch input to a visual object (104). For example, the user (102) can bring the input means (107) into contact with a touch recognition area (e.g., associated with the visual object (104)) for recognizing a touch input to the visual object (104) for a touch input to the visual object (104).

[0040] As a non-limiting example, the touch input may include an input of tapping a touch-sensitive area that recognizes a touch input to a visual object (e.g., a single tap input). As a non-limiting example, the touch input may include an input of tapping a touch-sensitive area that recognizes an input to a visual object more than once (e.g., a multiple tap input). As a non-limiting example, the touch input may include an input of contacting a touch-sensitive area that recognizes an input to a visual object for a reference time interval (e.g., a touch-hold input). As a non-limiting example, the touch input may include an input that connects touch-sensitive areas that recognize an input to multiple visual objects (e.g., a drag input). However, the present invention is not limited thereto.

[0041] For example, there may be a state (100) in which the input means (107) is offset in a direction toward the user (102) from the depth data (106) indicating the position of the visual object (104). For example, the offset state (100) may be described as a state in which the input means (107) is spaced in a direction toward the user (102) from the depth data (106) indicating the position of the visual object (104). For example, the state (100) may be described as a state in which the visual object (104) displayed according to the depth data (106) is recognized by the user (102) as being displayed according to the depth data (108).

[0042] For example, within a state (100), a user (102) may perceive a visual object (104) as being at a location corresponding to depth data (108) that is different from depth data (106). For example, within a state (100), a touch input to a visual object (104) caused by a user (102) may be identified (or recognized) by the wearable device (101) at a timing that is not intended by the user (102) due to a difference (109) between depth data (106) representing a location within a 3D space (103) in which a visual object (104) is displayed and depth data (108) representing a location of the visual object (104) within a 3D space (103) perceived by the user (102).

[0043] For example, within the state (100), the wearable device (101) may recognize a touch input to a visual object (104) at a second timing that is later than a first timing predicted by the user (102). For example, within the state (100), the user (102) may perceive a difference between the first timing and the second timing as a delay in response to the touch input. For example, since the difference between the first timing and the second timing being perceived as a delay in response to the touch input causes inconvenience to the user, a method for recognizing a touch input based on depth data (108) representing a position of the visual object (104) within the 3D space (103) perceived by the user (102) may be required within the wearable device (101).

[0044] For example, there may be a state (111) in which the input means (107) is offset in the opposite direction to the user (102) from the depth data (106) indicating the location of the visual object (104). For example, the state (111) may be described as a state in which the visual object (104) displayed according to the depth data (106) is recognized by the user (102) as being displayed according to the depth data (113).

[0045] For example, within a state (111), a user (102) may perceive that a visual object (104) is located at a location corresponding to depth data (113) that is different from depth data (106). For example, within a state (111), a touch input to a visual object (104) caused by the user (102) may be identified (or recognized) by the wearable device (101) at a timing that is not intended by the user (102) due to a difference (114) between depth data (106) representing a location within a 3D space (103) in which a visual object (104) is displayed and depth data (113) representing a location of the visual object (104) within a 3D space (103) perceived by the user (102).

[0046] For example, within a state (111), due to a difference (114) between depth data (106) indicating a position of a visual object (104) within a displayed 3D space (103) and depth data (113) indicating a position of the visual object (104) within a 3D space (103) recognized by a user (102), the depth data (113) indicating a position of the input means (107) may be deeper than the depth data (106) indicating a position of the visual object (104) at a point in time when the movement direction of the input means (107) varies according to the depth data. For example, within a state (111), causing the user (102) to press the visual object (104) more deeply due to the difference (114) may cause discomfort to the user (102).

[0047] For example, within a state (111), the wearable device (101) may cause the user (102) to press harder on the visual object (104) due to a difference (114) between the depth data (106) representing the position of the visual object (104) within the displayed 3D space (103) and the depth data (113) representing the position of the visual object (104) within the 3D space (103) recognized by the user (102). For example, within a state (111), causing the user (102) to press harder on the visual object (104) due to the difference (114) may cause discomfort to the user (102).

[0048] For example, within state (111), the wearable device (101) may recognize a touch input to a visual object (104) at a second timing that is earlier than a third timing predicted by the user (102). For example, within state (111), the user (102) may recognize the difference between the third timing and the second timing as a response to a touch input prior to causing the touch input. For example, since recognizing the difference between the first timing and the second timing as a response to a touch input prior to causing the touch input causes discomfort to the user (102), a method for recognizing a touch input based on depth data (113) representing the position of the visual object (104) within the 3D space (103) recognized by the user (102) may be required within the wearable device (101).

[0049] For example, a method for recognizing a touch input to a visual object (104) based at least in part on depth data (108) or depth data (113) representing a location of the visual object (104) within a 3D space (103) perceived by a user (102) may be executed within a wearable device (101). For example, the wearable device (101) may include components for executing such a method. The components of the wearable device (101) are exemplified within the description of FIG. 2.

[0050] Figure 2 is a simplified block diagram of an exemplary wearable device.

[0051] Referring to FIG. 2, the wearable device (101) may be described as a wearable device that can be worn on a user's head. The wearable device (101) may include at least a portion of the electronic device (1402) of FIG. 14, or may correspond to at least a portion of the electronic device (1402) of FIG. 14. The wearable device (101) may include at least one processor (200), a memory (210), a plurality of cameras (220), and a display (110).

[0052] At least one processor (200) may include processing circuitry. For example, at least one processor (200) may include a central processing unit (CPU) (e.g., including processing circuitry). For example, at least one processor (200) may include a graphic processing unit (GPU) (e.g., including processing circuitry) and a neural processing unit (NPU) (e.g., including processing circuitry). For example, at least one processor (200) may be configured to control a memory (210), a plurality of cameras (220), and a display (110). At least one processor (200) may be configured to individually or collectively execute instructions stored in the memory (210) to cause the wearable device (101) to perform at least some of the operations illustrated in the description of FIG. 1. At least one processor (200) may be configured to execute instructions stored in the memory (210) to cause the wearable device (101) to perform at least some of the operations illustrated in the descriptions of FIGS. 3 through 13.

[0053] The memory (210) may include one or more storage media. For example, the memory (210) may store various data used by at least one component of the wearable device (101) (e.g., at least one processor (200) and / or multiple cameras (220)). For example, the data may include input data or output data for software and commands related thereto. The memory (210) may include volatile memory or non-volatile memory.

[0054] The plurality of cameras (220) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate an electrical signal representing the color and / or brightness of light. For example, the plurality of cameras (220) may be described as image sensors. For example, each of the plurality of cameras (220) may be available to acquire an image of the environment surrounding the wearable device (101). For example, at least some of the plurality of cameras (220) may have a field of view (FOV) corresponding to the FOV of a user's eye. For example, the FOV of some of the plurality of cameras (220) may be different from the FOV of other some of the plurality of cameras (220). For example, the plurality of cameras (220) may be used to identify an input means positioned around the wearable device (101).

[0055] The display (110) may be configured to display visual information, visual data, images, and / or user interfaces. For example, the display (110) may be used to display visual objects within 3D space. For example, the display (110) may be used to display input means.

[0056] The wearable device (101) illustrated in the description of FIG. 2 can execute at least some of the operations illustrated in the description of FIGS. 3 to 13. For example, the operations illustrated in the description of FIGS. 3 to 13 can be caused by (or within) the wearable device (101) under the control of at least one processor (200).

[0057] FIG. 3 is a flowchart illustrating exemplary operations of a wearable device that acquires reference data for determining the location of a touch recognition area that recognizes a touch input to a visual object according to a change in the direction of movement of an input means to a visual object.

[0058] Referring to FIG. 3, in operation 300, at least one processor (200) may display a visual object through the display (110). For example, at least one processor (200) may display a visual object according to depth data within a 3D space. As a non-limiting example, when a position within the 3D space is defined by an x-axis coordinate, a y-axis coordinate, and a z-axis coordinate, the depth data used for displaying the visual object may represent a z-axis coordinate of the visual object. As a non-limiting example, the depth data may represent a z-axis coordinate of a representative position within an area or space where the visual object is displayed.

[0059] For example, the visual object may be configured to execute a function in response to a touch input received for the visual object. For example, at least one processor (200) may execute the function mapped to the visual object before the touch input is generated in response to a touch input for the visual object.

[0060] As a non-limiting example, the visual object (104) may correspond to a virtual object within extended reality (XR). As a non-limiting example, the visual object (104) may correspond to an external object within extended reality (XR). For example, the external object may be identified using images acquired through a plurality of cameras (220).

[0061] For example, a visual object may be displayed through the display (110) to allow a user of the wearable device (101) to perform calibration of a location where a touch input to the visual object is identified. For example, the calibration may be performed to obtain depth data representing a location of a visual object recognized by a user of the wearable device (101). For example, the calibration may be performed to obtain depth data representing a location of an input means at a time (or timing) when a user of the wearable device (101) causes a touch input to the visual object. For example, the calibration may be performed to reduce a difference between a time when a user of the wearable device (101) recognizes that a touch input has been made to a visual object and a time when an input to the visual object is identified by the wearable device (101).

[0062] Actions 310 to 340 below may be performed while a visual object is displayed within 3D space.

[0063] In operation 310, at least one processor (200) can acquire images through a plurality of cameras (220). For example, at least one processor (200) can identify an input means using the images.

[0064] As a non-limiting example, the input means may correspond to an external object. As a non-limiting example, the input means may include a body part (e.g., a finger) of a user of the wearable device (101). As a non-limiting example, the input means may include a stylus pen that is available in conjunction with the wearable device (101).

[0065] For example, a user of a wearable device (101) can control an input means. For example, a user of a wearable device (101) can move the input means to provide touch input to a visual object.

[0066] In operation 320, at least one processor (200) can identify whether the direction in which the input means is moved changes with respect to the depth data by using images acquired through the plurality of cameras (220).

[0067] The direction in which the input means moves is changed with respect to the depth data as exemplified in the description of Fig. 4.

[0068] Figure 4 illustrates an example in which the direction of movement for depth data of an input means is changed.

[0069] Referring to FIG. 4, at least one processor (200) can display a visual object (402) via a display (110). For example, at least one processor (200) can display a visual object (402) within a 3D space (401). For example, at least one processor (200) can display a visual object (402) based on depth data (404) within the 3D space (401).

[0070] For example, the wearable device (101) can display a pointer (or cursor) corresponding to an input means through the display (110).

[0071] For example, there may be a state (400) in which a user (102) of a wearable device (101) moves an input means (405) in a direction (408) toward a visual object (402).

[0072] For example, at least one processor (200) may identify a direction in which the input means (405) moves by using images acquired through a plurality of cameras (220). For example, within a state (400), a user (102) of the wearable device (101) may move the input means (405) to cause a touch input to a visual object (402). For example, the input means (405) may be moved in a direction (408) with respect to the visual object (402).

[0073] For example, the state (400) may be a state in which no touch input is caused to the visual object (402). For example, the state (400) may be described as a state before a touch input is caused to the visual object (402). For example, within the state (400), at least one processor (200) may bypass determining where the direction (408) of movement of the input means (405) changes with respect to the depth data. For example, within the state (400), at least one processor (200) may maintain identifying whether the direction (408) in which the input means (405) moves changes with respect to the depth data. For example, the state (400) may correspond to the direction (408) in which the input means (405) moves being maintained with respect to the depth data in operation 320 of FIG. 3.

[0074] For example, the direction (408) in which the input means (405) moves with respect to the visual object (402) may be changed with respect to the depth data. For example, depending on the intention of the user (102) for a touch input with respect to the visual object (402), the state (400) may be changed to the state (410). As a non-limiting example, the change from the state (400) to the state (410) may be recognized by at least one processor (200) using images acquired through a plurality of cameras (220).

[0075] For example, before (or immediately before) a state (410) changed from a state (400), the user (102) may at least temporarily stop the movement of the input means (405) at a point in time when the user (102) recognizes that a touch input has been caused to the visual object (402). For example, depending on the recognition of the user (102) of the wearable device (101) that a touch input has been caused to the visual object (402), the direction of movement of the input means (405) may be changed from direction (408) to direction (413), such as the state (410) changed from the state (400).

[0076] For example, at least one processor (200) may identify whether the direction of movement of the input means (405) changes relative to the depth data to determine (or identify) (or recognize) (or confirm) (or monitor) when the user (102) recognizes that a touch input has occurred with respect to the visual object (402).

[0077] Referring again to FIG. 3, at operation 330, at least one processor (200) can determine a position in which the direction in which the input means is moved has changed relative to the depth data.

[0078] For example, the location where the direction of movement of the input means changes relative to the depth data may be the location where the user of the wearable device perceives that a touch input has occurred relative to a visual object. For example, the location where the direction of movement of the input means changes relative to the depth data may be different from the location of the visual object.

[0079] Referring back to FIG. 4, within the state (410), the position at which the direction in which the input means (405) moves with respect to the visual object (402) changes with respect to the depth data may be the position of the input means (405) at the time when the user of the wearable device (101) recognizes that an input has been made with respect to the visual object. For example, within the state (410), the input means (405) may be positioned within the 3D space (401) according to the depth data (411). For example, there may be a difference (412) between the depth data (411) indicating the position of the input means (405) and the depth data (404) indicating the position of the visual object (402). For example, the difference (412) may be the difference between the depth data (404) indicating the position of the visual object (402) and the depth data (411) indicating the position of the visual object (402) recognized by the user (102).

[0080] For example, at least one processor (200) may recognize a touch input to a visual object (402) at a fifth timing that is later than a fourth timing predicted by the user (102). For example, the user (102) may recognize the difference between the fourth timing and the fifth timing as a delay in the response to the touch input. For example, at least one processor (200) may recognize a touch input to a visual object (104) at a fifth timing that is earlier than a sixth timing predicted by the user (102). For example, the user (102) may recognize the difference between the sixth timing and the fifth timing as a response to the touch input before causing the touch input.

[0081] For example, at least one processor (200) may determine depth data representing the location of a visual object (402) recognized by the user (102) based on determining a changed location relative to the depth data in which the input means (405) is moved.

[0082] Referring back to FIG. 3, at operation 340, at least one processor (200) may acquire reference data using a position in which the direction in which the input means is moved has changed with respect to the depth data. For example, the reference data may be determined based on a difference between depth data indicating a position in which the direction in which the input means is moved has changed with respect to the depth data and depth data indicating a position in which a visual object is displayed within 3D space. For example, the reference data may be determined based on a difference between depth data indicating a position of a visual object and depth data indicating a position of the visual object recognized by a user of the wearable device (101).

[0083] For example, the reference data may be the result of a user calibration of the wearable device (101). For example, the reference data may be applied to depth data indicating the location of a visual object. For example, the reference data may be used to adjust the location of a touch recognition area that recognizes touch input to a visual object to a different location distinct from the display location of the visual object.

[0084] For example, the above calibration can be performed in various types. The types of the above calibration are exemplified in the descriptions of FIGS. 5a, 5b, and 5c.

[0085] Figures 5a, 5b, and 5c illustrate exemplary methods for obtaining reference data for determining the location of a touch recognition area that recognizes input to a visual object.

[0086] Referring to FIG. 5A, at least one processor (200) can display a plurality of visual objects (501) through a display (110). For example, the plurality of visual objects (501) can be displayed within a 3D space.

[0087] For example, the plurality of visual objects (501) may have different sizes. For example, the plurality of visual objects (501) may be displayed according to different depth data within 3D space.

[0088] For example, at least one processor (200) may display guidance (502) to input touches on a plurality of visual objects (501) through the display (110). For example, a user of the wearable device (101) may cause touch inputs on the plurality of visual objects (501) according to the guidance (502).

[0089] For example, at least one processor (200) may display a pointer (or cursor) corresponding to an input means through the display (110).

[0090] For example, a user of a wearable device (101) may cause a touch input to a visual object (500), which is one of a plurality of visual objects (501), by moving the input means (503). For example, at least one processor (200) may determine a position of the input means (503) at a time when the user of the wearable device (101) causes a touch input to the visual object (500). For example, the time when the touch input is caused to the visual object (500) may be a time when the direction in which the input means (503) is moved changes with respect to the depth data. For example, the position of the input means (503) at a time when the touch input is caused to the visual object (500) may be a position of the input means at a time when the direction in which the input means (503) is moved changes with respect to the depth data. As a non-limiting example, the position of the input means (503) at the time of causing a touch input on the visual object (500) may be determined using a separate sensor of a stylus pen available in conjunction with the wearable device (101). However, the present invention is not limited thereto. For example, at least one processor (200) may obtain reference data using the determined position of the input means (503). For example, the reference data may be obtained based on depth data for the determined position of the input means (503).

[0091] As a non-limiting example, the reference data may vary depending on the size and / or depth data of the plurality of visual objects (501). For example, at least one processor (200) may obtain the reference data based on the size and / or depth data of the plurality of visual objects (501). For example, the reference data may be the result of calibration performed related to the location at which the user of the wearable device (101) recognizes the visual object.

[0092] Referring to FIG. 5B, at least one processor (200) can display a plurality of visual objects (506) via the display (110). For example, the plurality of visual objects (506) can be displayed within a 3D space.

[0093] As a non-limiting example, the plurality of visual objects (506) may be circular. For example, the plurality of visual objects (506) may have various sizes. For example, the plurality of visual objects (506) may be displayed according to various depth data within 3D space.

[0094] For example, at least one processor (200) may display guidance (507) to grip a plurality of visual objects (506) via the display (110). For example, a user of the wearable device (101) may cause a touch input to the plurality of visual objects (506) according to the guidance (507). As a non-limiting example, the touch input may include an input of gripping a visual object.

[0095] For example, at least one processor (200) may display a pointer (or cursor) corresponding to an input means through the display (110).

[0096] As a non-limiting example, the input means (508) may correspond to the hand of a user of the wearable device (101). As a non-limiting example, the input means (508) may correspond to the finger of a user of the wearable device (101).

[0097] For example, a user of a wearable device (101) can cause a touch input to a visual object (505), which is one of a plurality of visual objects (506), by moving an input means (503). For example, a user of a wearable device (101) can cause a touch input to a visual object (505) by gripping the visual object (505) with an input means (508) corresponding to a finger of the user. For example, at least one processor (200) can determine the size of the visual object (505) recognized by the user of the wearable device (101) at the time when the user of the wearable device (101) causes a touch input to the visual object (505). For example, the time when the touch input is caused to the visual object (505) may be the time when the direction in which the input means (508) is moved changes with respect to the depth data. For example, the position of the input means (508) at the time when a touch input is caused to the visual object (505) may be the position of the input means at the time when the direction in which the input means (508) moves changes with respect to the depth data. As a non-limiting example, the position of the input means (508) at the time when a touch input is caused to the visual object (505) may be determined using a separate sensor of a stylus pen that is available in conjunction with the wearable device (101). However, the present invention is not limited thereto. For example, at least one processor (200) may obtain reference data using the position of the input means (508) that represents the size of the visual object (505).

[0098] As a non-limiting example, the reference data may vary depending on the size and / or depth data of the plurality of visual objects (506). For example, at least one processor (200) may obtain the reference data according to the size and / or depth data of the plurality of visual objects (506). For example, the reference data may be the result of performing calibration related to the size of the visual object (505) recognized by the user of the wearable device (101).

[0099] Referring to FIG. 5c, at least one processor (200) can display a plurality of visual objects (509) via the display (110). For example, the plurality of visual objects (509) can be displayed within a 3D space.

[0100] As a non-limiting example, the plurality of visual objects (509) may be cylindrical in shape. For example, the plurality of visual objects (509) may have different sizes. For example, the plurality of visual objects (509) may have different heights. For example, the height of a visual object (510) may be different from the height of a visual object (511). For example, the plurality of visual objects (509) may be displayed according to different depth data within a 3D space.

[0101] For example, at least one processor (200) may display guidance (507) to connect the top surfaces of a plurality of visual objects (509) via the display (110). For example, a user of the wearable device (101) may cause a touch input to connect the top surfaces of the plurality of visual objects (509) according to the guidance (507).

[0102] For example, at least one processor (200) may display a pointer (or cursor) corresponding to an input means through the display (110).

[0103] For example, a user of a wearable device (101) can cause a touch input on the upper surface of a visual object (510), which is one of a plurality of visual objects (509), by moving the input means (513). For example, a user of a wearable device (101) can move the input means (513) from the upper surface of a visual object (510) to the upper surface of a visual object (511). For example, a user of a wearable device (101) can move the input means (513) to connect the upper surface of a visual object (510) and the upper surface of a visual object (511).

[0104] For example, a user of a wearable device (101) can move the input means (513) from the upper surface of a visual object (511) to the upper surface of a visual object (514). For example, a user of a wearable device (101) can move the input means (513) to connect the upper surface of a visual object (511) and the upper surface of a visual object (514).

[0105] For example, at least one processor (200) may obtain a curve (515) intended by a user to connect the upper surfaces of a plurality of visual objects (509) by the input means (513). For example, the curve (515) may be a path (e.g., a path recognized by the user) of the input means (513) moved to connect the upper surfaces of the plurality of visual objects (509).

[0106] For example, the height of the visual object (511) may be lower than the heights of the visual object (510) and the visual object (514). For example, since the height of the visual object (511) is lower than the heights of the visual object (510) and the visual object (514), the curve (515) may have an inflection point (516) at the upper surface of the visual object (511).

[0107] For example, at least one processor (200) may determine the height of the visual object (511) recognized by the user of the wearable device (101) at a time when the user of the wearable device (101) causes a touch input to the upper surface of the visual object (511). For example, the time when the touch input is caused to the visual object (511) may be a time when the direction in which the input means (513) moves changes with respect to the depth data. For example, the position of the input means (513) at a time when the touch input is caused to the visual object (511) may be a position of the input means at a time when the direction in which the input means (513) moves changes with respect to the depth data. As a non-limiting example, the position of the input means (513) at a time when the touch input is caused to the visual object (511) may be determined using a separate sensor of a stylus pen that is available in conjunction with the wearable device (101). However, the present invention is not limited thereto. For example, the location of the inflection point (516) may represent the height of a visual object (511) recognized by a user of the wearable device. For example, at least one processor (200) may obtain reference data using the location of the inflection point (516).

[0108] As a non-limiting example, the reference data may vary depending on the size and / or depth data of the plurality of visual objects (509). For example, at least one processor (200) may acquire the reference data according to the size and / or depth data of the plurality of visual objects (509). For example, the reference data may be the result of performing calibration related to the height of the visual object (511) recognized by the user of the wearable device (101).

[0109] For example, at least one processor (200) may, when performing the calibration, acquire a position in which the direction in which the input means moves is changed relative to the depth data. For example, the result of the calibration (or reference data) identified within a state in which the position in which the direction in which the input means moves is changed relative to the depth data is relatively far from the visual object may have an error. For example, to reduce such an error, a reference space range may be utilized within the calibration. For example, at least one processor (200) may perform a process for acquiring the reference data by identifying whether the input means is positioned within the reference space range of the visual object. This operation is exemplified in the description of FIG. 6.

[0110] FIG. 6 is a flowchart illustrating exemplary operations of a wearable device for identifying whether an input means is positioned within a reference spatial range of a visual object to obtain reference data.

[0111] Referring to FIG. 6, in operation 600, at least one processor (200) may acquire images through a plurality of cameras (220). For example, at least one processor (200) may identify an input means using the images.

[0112] For example, a user of a wearable device (101) may control an input means. For example, a user of the wearable device (101) may move the input means to input touch input to a visual object. For example, the input means may be moved relative to the visual object.

[0113] In operation 610, at least one processor (200) can identify whether the input means is positioned within a reference spatial range according to depth data indicating the position of the visual object.

[0114] Whether the input means is positioned within the reference spatial range according to the depth data of the visual object is exemplified in the description of Fig. 7.

[0115] Figure 7 illustrates an example of an input means positioned within the reference spatial range of a visual object.

[0116] Referring to FIG. 7, at least one processor (200) can display a visual object (402) via a display (110). For example, at least one processor (200) can display a visual object (402) within a 3D space (401). For example, at least one processor (200) can display a visual object (402) based on depth data (404) within the 3D space (401).

[0117] For example, a reference spatial range (701) may be defined for a visual object (402). For example, the reference spatial range (701) may be defined based on depth data (404) indicating a location of the visual object (402). For example, at least one processor (200) may identify whether the input means (405) is located within the reference spatial range (701). For example, the reference spatial range (701) may be a spatial range for identifying whether the input means (405) is located relatively close to the visual object (402). For example, the reference spatial range (701) may include a spatial range that extends in a direction toward the user with respect to the visual object (402). For example, the reference spatial range (701) may include a spatial range that extends in an opposite direction toward the user with respect to the visual object (402).

[0118] For example, at least one processor (200) may display a pointer (or cursor) corresponding to an input means through the display (110).

[0119] For example, a user (102) of a wearable device (101) may move an input means (405) for touch input with respect to a visual object (402). For example, the input means (405) may be moved in a direction with respect to the visual object (402).

[0120] For example, there may be a state (700) where the input means (405) is located outside the reference spatial range (701) of (or for) the visual object (402).

[0121] For example, in state (700), the input means (405) can be positioned according to depth data (702) within the 3D space (401). For example, the depth data (702) indicating the position of the input means (405) and the depth data (404) indicating the position of the visual object (402) can be different. For example, the difference (703) between the depth data (702) and the depth data (404) can exceed the difference between the depth data (404) and the depth data of the reference space range (701).

[0122] For example, a change in the direction of movement of the input means (405) that occurs outside the reference spatial range (701) of the visual object (402) may cause the movement of the input means (405) to be independent of the visual object (402). For example, in state (700), the input means (405) may be moved independently of the visual object (402).

[0123] For example, in state (700), at least one processor (200) may bypass identifying whether the direction of movement of the input means (405) changes with respect to the depth data. For example, at least one processor (200) may maintain identifying whether the input means (405) is located within a reference spatial range (701) of the visual object (402). For example, state (700) may correspond to the input means (405) being located outside the reference spatial range (701) according to the depth data (404) of the visual object (402) in operation 610 of FIG. 6.

[0124] For example, there may be a state (704) in which the input means (405) is positioned within the reference space range (701) of the visual object (402).

[0125] For example, in state (704), the input means (405) can be positioned according to depth data (705) within the 3D space (401). For example, the depth data (705) indicating the position of the input means (405) and the depth data (404) indicating the position of the visual object (402) can be different. For example, the difference (706) between the depth data (705) and the depth data (404) can be less than or equal to the difference between the depth data (404) indicating the position of the visual object (402) and the depth data of the reference space range (701).

[0126] For example, a change in the direction of movement of the input means (405) caused within the reference space range (701) may be related to the movement of the input means (405) with respect to a visual object (402). For example, the point in time at which the direction of movement of the input means (405) caused within the reference space range (701) is changed may be a point in time at which the user (102) of the wearable device (101) recognizes that a touch input has been made with respect to the visual object (402).

[0127] For example, at least one processor (200) can obtain reference data by using a position where the direction of movement of the input means (405) is changed with respect to depth data within a reference space range (701).

[0128] Referring again to FIG. 6 , at operation 620, at least one processor (200) may identify whether the direction in which the input means is moved changes with respect to the depth data based on whether the input means is positioned within the reference spatial range of the visual object. For example, operation 620 may correspond to operation 320 of FIG. 3 .

[0129] For example, depth data indicating the position of a visual object may correspond to depth data indicating a position in which the direction in which the input means is moved has changed relative to the depth data. For example, a user of a wearable device (101) may recognize that a visual object is located at a position corresponding to the depth data indicating the position of the visual object.

[0130] For example, if the depth data indicating the changed position relative to the depth data in the direction in which the input means is moved corresponds to the depth data indicating the position of the visual object, calibration of the position at which the touch input for the visual object is identified may not need to be performed. For example, if the depth data indicating the changed position relative to the depth data in the direction in which the input means is moved corresponds to the depth data indicating the position of the visual object, at least one processor (200) may not need to acquire reference data. The correspondence of the depth data indicating the changed position relative to the depth data in the direction in which the input means is moved with the depth data indicating the position of the visual object is exemplified in the description of FIG. 8.

[0131] FIG. 8 is a flowchart illustrating exemplary operations of a wearable device for identifying whether depth data representing a determined position of an input means to obtain reference data corresponds to depth data representing a position of a visual object.

[0132] Referring to FIG. 8, in operation 800, at least one processor (200) can determine a location where the direction in which the input means moves changes with respect to depth data using images acquired through a plurality of cameras (220).

[0133] For example, the location where the direction in which the input means moves is changed with respect to the depth data may be a location where the user of the wearable device (101) recognizes that a touch input has been made with respect to a visual object. For example, the location where the direction in which the input means moves is changed with respect to the depth data may be different from the depth data indicating the location of the visual object.

[0134] In operation 810, at least one processor (200) can determine whether depth data indicating a position of a visual object and depth data indicating a position in which the direction in which the input means is moved are different with respect to the depth data.

[0135] For example, depth data indicating a change in the direction in which the input means is moved relative to the depth data may differ from depth data indicating the position of a visual object. For example, depth data indicating a change in the direction in which the input means is moved relative to the depth data may correspond to depth data indicating the position of a visual object.

[0136] Whether the depth data indicating the changed position relative to the depth data in the direction in which the input means is moved corresponds to the depth data indicating the position of the visual object is exemplified in the description of Fig. 9.

[0137] Figure 9 illustrates an example of depth data indicating the position of a visual object and depth data indicating the determined position of another input means.

[0138] Referring to FIG. 9, at least one processor (200) can display a visual object (402) through a display (110). For example, at least one processor (200) can display a visual object (402) within a 3D space (401). For example, at least one processor (200) can display a visual object (402) based on depth data (404) within the 3D space (401).

[0139] For example, at least one processor (200) can identify an input means (107) using images acquired through multiple cameras. For example, the wearable device (101) can display a pointer (or cursor) corresponding to the input means through the display (110).

[0140] For example, a user (102) of a wearable device (101) may move an input means (405) to cause a touch input to a visual object (402). For example, the input means (405) may be moved in a direction relative to the visual object (402).

[0141] For example, at least one processor (200) can determine a position at which the direction in which the input means (405) moves with respect to the visual object (402) changes with respect to the depth data. For example, the determined position can be a position of the input means (405) at a point in time when the user (102) of the wearable device (101) recognizes that a touch input has been made with respect to the visual object (402).

[0142] For example, there may be a state (900) in which depth data (901) indicating a position in which the direction in which the input means (405) moves with respect to the visual object (402) changes with respect to the depth data is different from depth data (404) indicating the position of the visual object (402).

[0143] For example, in state (900), the position at which the direction in which the input means (405) moves with respect to the visual object (402) changes with respect to the depth data may be located in the depth data (901) within the 3D space (401). For example, the depth data (901) indicating the position at which the direction in which the input means (405) moves with respect to the visual object (402) changes with respect to the depth data may be different from the depth data (404) indicating the position of the visual object (402).

[0144] For example, a state (900) can be described as a state in which a visual object (402) displayed according to depth data (404) is recognized by a user (102) as being located in depth data (901).

[0145] For example, within a state (900), a user (102) may perceive that a visual object (402) is at a location corresponding to depth data (901) that is different from depth data (404). For example, within a state (900), a touch input to a visual object (402) caused by the user (102) may be identified (or recognized) by the wearable device (101) at a timing that is not intended by the user (102) due to a difference (902) between depth data (404) representing a location within a 3D space (401) in which a visual object (402) is displayed and depth data (901) representing a location of the visual object (402) within a 3D space (103) perceived by the user (102).

[0146] For example, within state (900), at least one processor (200) may recognize a touch input to a visual object (402) at an eighth timing that is later than a seventh timing predicted by the user (102). For example, within state (900), the user (102) may recognize the difference between the seventh timing and the eighth timing as a delay in the response to the touch input.

[0147] For example, in state (900), at least one processor (200) may need to perform calibration of a location where a touch input to a visual object is identified. For example, in state (900), at least one processor (200) may need to obtain reference data using depth data (901) indicating a location where the direction in which the input means (405) is moved changes relative to the depth data.

[0148] For example, FIG. 9 illustrates a state (900) where depth data (901) indicating a position where the direction in which the input means (405) moves relative to the visual object (402) changes relative to the depth data is offset in a first direction toward the user from the depth data (404) indicating the position of the visual object (402), but this is merely exemplary. For example, the depth data (901) may also be offset in a second direction (e.g., opposite the first direction) from the user from the depth data (404) indicating the position of the visual object (402).

[0149] For example, a state (900) in which depth data (901) is offset in a first direction toward the user from depth data (404) indicating the location of a visual object (402) may include a state in which depth data (901) is spaced in a first direction toward the user from depth data (404) indicating the location of a visual object (402).

[0150] For example, there may be a state (903) in which depth data (904) indicating a position in which the direction in which the input means (405) moves changes with respect to the depth data substantially corresponds to depth data (404) indicating the position of the visual object (402).

[0151] For example, in state (903), the position at which the direction in which the input means (405) moves changes with respect to the depth data may be located in the depth data (904) within the 3D space (401). For example, the depth data (904) indicating the position at which the direction in which the input means (405) moves changes with respect to the depth data may substantially correspond to the depth data (404) indicating the position of the visual object (402).

[0152] For example, depth data (904) indicating a position where the direction in which the input means (405) moves changes with respect to the depth data may be located within a reference range from the depth data (404) indicating the position of the visual object (402). For example, by positioning the depth data (904) within a reference range from the depth data (404), the depth data (904) may substantially correspond to the depth data (404).

[0153] For example, the difference between the depth data (904) and the depth data (404) may be less than a reference value. For example, since the difference between the depth data (904) and the depth data (404) is less than a reference value, the depth data (904) may substantially correspond to the depth data (404). For example, the state (903) may be described as a state in which a visual object (402) displayed according to the depth data (404) is recognized by the user (102) as being located in the depth data (904).

[0154] For example, within state (903), the user (102) may perceive that the visual object (402) is at a location corresponding to the depth data (404). For example, within state (900), a touch input to the visual object (402) caused by the user (102) may be identified (or recognized) by the wearable device (101) at a timing intended by the user (102).

[0155] For example, within state (903), at least one processor (200) can recognize a touch input to a visual object (402) at the same timing as predicted by the user (102).

[0156] For example, in state (903), at least one processor (200) may not need to perform calibration of a location where a touch input to a visual object is identified. For example, in state (903), at least one processor (200) may not need to obtain reference data using depth data (904) indicating a location where the direction in which the input means (405) is moved changes relative to the depth data.

[0157] For example, state (903) may correspond to the fact that depth data (404) indicating the position of the visual object (402) in operation 810 of FIG. 8 and depth data (904) indicating the position in which the direction in which the input means (405) moves is changed with respect to the depth data are the same.

[0158] Referring again to FIG. 8, at operation 820, at least one processor (200) may acquire reference data using a position in which the direction in which the input means is moved has changed with respect to the depth data. For example, the reference data may be data for recognizing a touch input for a visual object occurring within a 3D space.

[0159] For example, the reference data may be the result of a user calibration of the wearable device (101). For example, the reference data may be applied to depth data indicating the location of a visual object. For example, the reference data may be used to adjust the location of a touch recognition area that recognizes touch input to a visual object to a different location distinct from the display location of the visual object.

[0160] In operation 830, at least one processor (200) may refrain from (or skip, delay, bypass, or not acquire) acquiring reference data using a location where the direction in which the input means is moved has changed with respect to the depth data. For example, at least one processor (200) may not need to perform calibration of a location where a touch input to a visual object is identified. For example, at least one processor (200) may not need to acquire reference data using depth data indicating a location where the direction in which the input means is moved has changed with respect to the depth data.

[0161] For example, a touch recognition area associated with a visual object can be offset from depth data representing the location of the visual object using the results (or reference data) of a calibration of the location where touch input to the visual object is identified.

[0162] For example, offsetting from depth data indicating the location of a visual object using the result (or reference data) of the calibration may include applying (or subtracting, or merging) the result (or reference data) of the calibration to a touch recognition area associated with the visual object. For example, throughout the description below, the offsetting operation may include applying (or subtracting, or merging) the result (or reference data) of the calibration.

[0163] Recognition of touch input to a visual object using second depth data offset from first depth data is illustrated in the description of FIG. 10.

[0164] FIG. 10 is a flowchart illustrating exemplary operations of a wearable device that recognizes touch input to a visual object using second depth data offset from first depth data.

[0165] Referring to FIG. 10, in operation 1000, at least one processor (200) may display a visual object through a display (110). For example, at least one processor (200) may display a visual object according to first depth data within a 3D space.

[0166] For example, a visual object may be associated with a touch-sensitive area that can recognize a user's touch input. For example, a visual object may be mapped to provide a predetermined function based on the user's input.

[0167] In operation 1010, at least one processor (200) can identify second depth data by offsetting first depth data using reference data.

[0168] For example, the reference data may be used to apply the display location of a visual object recognized by the user of the wearable device (101) to touch input recognition. For example, the reference data may be data personalized to the user of the wearable device (101). For example, the reference data may be the result of calibration performed by the user of the location where a touch input for a visual object is identified.

[0169] For example, the second depth data may be depth data indicating a location where the direction of movement of the input means changes with respect to the depth data. For example, the second depth data may be depth data indicating a location of a visual object recognized by the user.

[0170] In operation 1020, at least one processor (200) may set a touch recognition area using the second depth data. For example, the touch recognition area may be an area associated with a visual object. For example, the touch recognition area may be positioned within 3D space according to the second depth data.

[0171] In operation 1030, at least one processor (200) may identify that the input means has contacted an area corresponding to second depth data offset from the visual object. For example, the visual object (104) may be associated with the touch recognition area to provide a function corresponding to the visual object (104) in response to a user's touch input.

[0172] For example, a touch recognition area set to recognize a touch input for a visual object may be determined using reference data acquired through calibration. For example, a touch recognition area associated with a visual object may be an area corresponding to second depth data offset from first depth data indicating the location of the visual object.

[0173] In operation 1040, at least one processor (200) can recognize an input means contacting a touch recognition area associated with a visual object as a touch input for the visual object.

[0174] As a non-limiting example, the touch input may include an input of tapping a touch-sensitive area that recognizes a touch input to a visual object (e.g., a single tap input). As a non-limiting example, the touch input may include an input of tapping a touch-sensitive area that recognizes an input to a visual object more than once (e.g., a multiple tap input). As a non-limiting example, the touch input may include an input of contacting a touch-sensitive area that recognizes an input to a visual object for a reference time interval (e.g., a touch-hold input). As a non-limiting example, the touch input may include an input that connects touch-sensitive areas that recognize an input to multiple visual objects (e.g., a drag input). However, the present invention is not limited thereto.

[0175] The input means contacting the touch recognition area corresponding to the second depth data offset from the visual object is exemplified in the description of Fig. 11.

[0176] Figure 11 illustrates an example of a touch recognition area that recognizes touch input to a visual object.

[0177] Referring to FIG. 11, at least one processor (200) can display a visual object (1101) through a display (110). For example, at least one processor (200) can display a visual object (1101) within a 3D space (401). For example, at least one processor (200) can display a visual object (1101) based on depth data (1102) within the 3D space (401).

[0178] For example, at least one processor (200) can identify the input means (107) using images acquired through multiple cameras (220). For example, at least one processor (200) can display a pointer (or cursor) corresponding to the input means through the display (110). For example, a user (102) of the wearable device (101) can move the input means (405) to make a touch input to a visual object (1101). For example, the input means (405) can be moved in the direction of the visual object (1101).

[0179] For example, there may be a state (1100) where a touch recognition area (1103) associated with a visual object (1101) is an area corresponding to depth data (1104) offset in a first direction from the user (102) from the visual object (1101).

[0180] For example, in state (1100), at least one processor (200) can use depth data (1102) of a visual object (1101) as reference data to identify depth data (1104) offset from the depth data (1102). For example, at least one processor (200) can use the depth data (1104) to set a touch recognition area (1103) associated with the visual object (1101) within the 3D space (401).

[0181] For example, in state (1100), a touch recognition area (1103) associated with a visual object (1101) may be an area corresponding to depth data (1104) within a 3D space (401). For example, state (1100) may be described as a state in which the result (or reference data) of performing calibration of a location where a touch input for a visual object is identified is applied to depth data (1102) indicating the location of the visual object (1101).

[0182] For example, depth data (1104) representing a location of a touch recognition area (1103) associated with a visual object (1101) may have a difference (1105) from depth data (1102) representing a location of the visual object (1101). For example, in a state (1100), the user (102) may recognize that the visual object (1101) is at a location corresponding to depth data (1104) that is different from the depth data (1102). For example, within the state (1100), the difference (1105) may be a difference between depth data (1102) representing a location of the visual object (1101) within a 3D space (401) in which the visual object (1101) is displayed and depth data representing a location of the visual object (1101) within a 3D space (401) recognized by the user (102).

[0183] For example, in state (1100), the user (102) may recognize that a touch input has occurred with respect to a visual object (1101). For example, in state (1100), at least one processor (200) may identify that the input means (405) has contacted a touch recognition area (1103) associated with the visual object (1101). For example, a touch input to the visual object (1102) caused by the user (102) may be identified (or recognized) by at least one processor (200) at a timing intended by the user (102).

[0184] For example, there may be a state (1106) where a touch recognition area (1107) associated with a visual object (1101) corresponds to depth data (1108) offset in a second direction (e.g., opposite to the first direction) from the user (102) away from the visual object (1101).

[0185] For example, in state (1106), at least one processor (200) can identify offset depth data (1108) using reference data in depth data (1102) of a visual object (1101). For example, at least one processor (200) can use the depth data (1108) to set a touch recognition area (1107) associated with the visual object (1101) within the 3D space (401).

[0186] For example, in state (1106), the touch recognition area (1107) associated with the visual object (1101) may be an area corresponding to depth data (1108) within the 3D space (401). For example, state (1106) may be described as a state in which the result (or reference data) of performing calibration of a location where a touch input for the visual object is identified is applied to depth data (1102) indicating the location of the visual object (1101).

[0187] For example, depth data (1108) representing a location of a touch recognition area (1107) associated with a visual object (1101) may have a difference (1109) from depth data (1102) representing a location of the visual object (1101). For example, in a state (1106), the user (102) may perceive that the visual object (1101) is at a location corresponding to depth data (1108) that is different from the depth data (1102). For example, within the state (1106), the difference (1109) may be a difference between depth data (1102) representing a location of the visual object (1101) within the 3D space (401) in which the visual object (1101) is displayed and depth data representing a location of the visual object (1101) within the 3D space (401) recognized by the user (102).

[0188] For example, in state (1106), the user (102) may recognize that a touch input has occurred with respect to a visual object (1101). For example, in state (1106), at least one processor (200) may identify that the input means (405) has contacted a touch recognition area (1107) associated with the visual object (1101). For example, a touch input to the visual object (1101) caused by the user (102) may be identified (or recognized) by at least one processor (200) at a timing intended by the user (102).

[0189] As a non-limiting example, the visual object (1101) may be recognized by the user (102) as a three-dimensional shape. For example, even if the visual object (1101) is recognized by the user (102) as a three-dimensional shape, the visual object (1101) may be displayed as an area according to depth data (1102) on the display (110) of the wearable device (101). For example, even if the visual object (1101) is recognized by the user (102) as a three-dimensional shape, within the state (1100), the touch recognition area (1103) associated with the visual object (1101) may be an area corresponding to depth data (1104) within the 3D space (401). For example, even if a visual object (1101) is recognized as a three-dimensional shape by the user (102), within the state (1106), the touch recognition area (1107) associated with the visual object (1101) may be an area corresponding to depth data (1108) within the 3D space (401).

[0190] As a non-limiting example, there may be a reference space range (not shown) within which a touch input to a visual object (1101) can be recognized. As a non-limiting example, the reference space range (not shown) may be formed according to depth data (1102) indicating the location of the visual object (1101). As a non-limiting example, the reference space range (not shown) may include the visual object (1101). As a non-limiting example, the boundary of the reference space range (not shown) may include a touch recognition area (1103) (or a touch recognition area (1107)).

[0191] As a non-limiting example, at least one processor (200) may recognize a touch input to a visual object (1101) based on the location where the movement direction of the input means (405) changes with respect to the depth data being located within the reference spatial range (not shown). However, the present invention is not limited thereto.

[0192] For example, at least one processor (200) can display a plurality of visual objects through the display (110). For example, each of the plurality of visual objects can be positioned according to different depth data within the 3D space. For example, at least one processor (200) can apply the same reference data to the depth data indicating the positions of the visual objects according to the different depth data. For example, at least one processor (200) can apply different reference data to the depth data indicating the positions of the visual objects according to the different depth data. Changing the reference data applied according to the depth data indicating the positions of the visual objects is exemplified in the descriptions of FIGS. 12A and 12B.

[0193] Figures 12a and 12b illustrate examples of changing the reference data applied according to depth data indicating the position of a visual object.

[0194] Referring to FIGS. 12A and 12B, at least one processor (200) can display a first visual object (1101) through a display (110). For example, at least one processor (200) can display the first visual object (1101) within a 3D space (401). For example, at least one processor (200) can display the first visual object (1101) according to first depth data (1102) within the 3D space (401).

[0195] For example, at least one processor (200) can display a second visual object (1205) different from the first visual object (1101) through the display (110). For example, at least one processor (200) can display the second visual object (1205) within a 3D space (401). For example, at least one processor (200) can display the second visual object (1205) according to second depth data (1207) different from the first depth data (1102) within the 3D space (401).

[0196] For example, at least one processor (200) can identify the input means (405) using images acquired through multiple cameras. For example, at least one processor (200) can display a pointer (or cursor) corresponding to the input means through the display (110). For example, a user (102) of the wearable device (101) can move the input means (405) to cause a touch input to the first visual object (1101) or the second visual object (1205). For example, the input means (405) can be moved in a direction relative to the first visual object (1101) or the second visual object (1205).

[0197] Referring to FIG. 12a, there may be a state (1200) in which a touch recognition area (1201) associated with a first visual object (1101) corresponds to third depth data (1202) offset from first depth data (1102) indicating the position of the first visual object (1101) using first reference data.

[0198] For example, in state (1200), at least one processor (200) can identify third depth data (1202) offset from first reference data in first depth data (1102) of a first visual object (1101). For example, at least one processor (200) can use the third depth data (1202) to set a touch recognition area (1201) associated with the first visual object (1101) within the 3D space (401).

[0199] For example, in state (1200), the touch recognition area (1201) associated with the first visual object (1101) may be an area corresponding to third depth data (1202) within the 3D space (401). For example, the third depth data (1202) indicating the position of the touch recognition area (1201) associated with the first visual object (1101) may have a difference (1203) from the first depth data (1102) indicating the position of the first visual object (1101). For example, the difference (1203) between the third depth data (1202) and the first depth data (1102) may be determined using the first reference data. For example, the state (1200) may be described as a state in which the first reference data is applied to the first depth data (1102) indicating the position of the first visual object (1101).

[0200] For example, there may be a state (1204) in which a touch recognition area (1208) associated with a second visual object (1205) corresponds to fourth depth data (1209) offset from second depth data (1207) indicating the location of the second visual object (1205) using the first reference data.

[0201] For example, in state (1204), at least one processor (200) can identify fourth depth data (1209) offset from the first reference data in second depth data (1207) of the second visual object (1205). For example, at least one processor (200) can use the fourth depth data (1209) to set a touch recognition area (1208) associated with the second visual object (1205) within the 3D space (401).

[0202] For example, the first reference data may be applied to the second depth data (1207). For example, the first reference data may be applied to the depth data indicating the position of the visual object, regardless of whether the depth data indicating the position of the visual object is different.

[0203] For example, in state (1204), the touch recognition area (1208) associated with the second visual object (1205) may be an area corresponding to the fourth depth data (1209) within the 3D space (401). For example, the fourth depth data (1209) indicating the location of the touch recognition area (1208) associated with the second visual object (1205) may have a difference (1210) from the second depth data (1207) indicating the location of the second visual object (1205). For example, the difference (1210) between the fourth depth data (1209) and the second depth data (1207) may be determined using the first reference data. For example, the state (1204) may be described as a state in which the first reference data is applied to the second depth data (1207) indicating the location of the second visual object (1205).

[0204] For example, the difference (1210) between the fourth depth data (1209) and the second depth data (1207) in state (1204) may correspond to the difference (1203) between the third depth data (1202) and the first depth data (1102) in state (1200).

[0205] For example, the user (102) can, independently of the depth data indicating the position of the visual object within the 3D space (401), recognize that the visual object is located at a position corresponding to depth data offset by the same depth data from the depth data indicating the position of the visual object. For example, at least one processor (200) can, independently of the depth data indicating the position of the visual object, identify offset depth data using the same reference data for each of the depth data indicating the positions of the plurality of visual objects. For example, at least one processor (200) can, independently of the depth data indicating the position of the visual object, set touch recognition areas associated with the plurality of visual objects using the depth data that are each offset by the same depth data from the depth data indicating the positions of the plurality of visual objects.

[0206] Referring to FIG. 12B, there may be a state (1200) in which a touch recognition area (1103) associated with a first visual object (1101) corresponds to third depth data (1202) offset from first depth data (1102) indicating a location of the first visual object (1101) using first reference data. For example, the state (1200) of FIG. 12B may correspond to the state (1200) of FIG. 12A.

[0207] For example, there may be a state (1211) in which a touch recognition area (1212) associated with a second visual object (1205) corresponds to fifth depth data (1213) offset from second depth data (1207) indicating the location of the second visual object (1205) using second reference data. For example, the second reference data may be different from the first reference data. For example, the second reference data may be reference data acquired by performing a calibration different from the calibration performed to acquire the first reference data.

[0208] For example, the first reference data may be data obtained by performing calibration on a visual object located in depth data within a first reference depth range that includes the first depth data. For example, the second reference data may be data obtained by performing calibration on a visual object located in depth data within a second reference depth range that includes the second depth data. For example, the second reference depth range may be distinguished from the first depth range.

[0209] For example, in state (1211), at least one processor (200) can identify fifth depth data (1213) offset from second reference data in second depth data (1207) of a second visual object (1205). For example, at least one processor (200) can use the fifth depth data (1213) to set a touch recognition area (1212) associated with the second visual object (1205) within the 3D space (401).

[0210] For example, second reference data may be applied to second depth data (1207) indicating the location of a second visual object (1205). For example, the second reference data may be applied to depth data indicating the location of a visual object located within the depth data within the second reference depth range.

[0211] For example, in state (1211), a touch recognition area (1212) associated with a second visual object (1205) may be located at fifth depth data (1213) within the 3D space (401). For example, the fifth depth data (1213) indicating the location of the touch recognition area (1212) associated with the second visual object (1205) may have a difference (1214) from second depth data (1207) indicating the location of the second visual object (1205). For example, the difference (1214) between the fifth depth data (1213) and the second depth data (1207) may be determined using second reference data. For example, state (1211) may be described as a state in which second reference data is applied to the second depth data (1207) indicating the location of the second visual object (1205).

[0212] For example, the difference (1214) between the fifth depth data (1213) and the second depth data (1207) in state (1211) may be different from the difference (1203) between the third depth data (1202) and the first depth data (1102) in state (1200).

[0213] For example, the user (102) may perceive that a visual object is offset by different depth data from the depth data representing the position of the visual object within the 3D space (401) depending on whether the depth data representing the position of the visual object is different. For example, the user (102) may perceive that a visual object is offset by depth data proportional to the depth data representing the position of the visual object from the depth data representing the position of the visual object within the 3D space (401). For example, at least one processor (200) may apply different reference data to different plurality of visual objects depending on whether the depth data representing the position of the visual object is different.

[0214] As a non-limiting example, the visual object (1101) (or visual object (1205)) may be recognized by the user (102) as a three-dimensional shape. For example, even if the visual object (1101) is recognized by the user (102) as a three-dimensional shape, the visual object (1101) may be displayed as an area according to depth data (1102) on the display (110) of the wearable device (101). For example, even if the visual object (1205) is recognized by the user (102) as a three-dimensional shape, the visual object (1205) may be displayed as an area according to depth data (1207) on the display (110) of the wearable device (101).

[0215] For example, even if a visual object (1101) is recognized as a three-dimensional shape by a user (102), within a state (1200), a touch recognition area (1201) related to the visual object (1101) may be an area corresponding to depth data (1202) within a 3D space (401). For example, even if a visual object (1205) is recognized as a three-dimensional shape by a user (102), within a state (1204), a touch recognition area (1208) related to the visual object (1205) may be an area corresponding to depth data (1209) within a 3D space (401). For example, even if a visual object (1205) is recognized as a three-dimensional shape by the user (102), within the state (1211), the touch recognition area (1212) associated with the visual object (1205) may be an area corresponding to depth data (1213) within the 3D space (401).

[0216] As a non-limiting example, there may be a reference space range (not shown) within which a touch input to a visual object (1101) can be recognized. As a non-limiting example, the reference space range (not shown) may be formed according to depth data (1102) indicating the location of the visual object (1101). As a non-limiting example, the reference space range (not shown) may include the visual object (1101). As a non-limiting example, the boundary of the reference space range (not shown) may include a touch recognition area (1201).

[0217] As a non-limiting example, at least one processor (200) may recognize a touch input to a visual object (1101) based on the location where the movement direction of the input means (405) changes with respect to the depth data being located within the reference spatial range (not shown). However, the present invention is not limited thereto.

[0218] As a non-limiting example, there may be a reference space range (not shown) within which a touch input to a visual object (1205) can be recognized. As a non-limiting example, the reference space range (not shown) may be formed according to depth data (1207) indicating the location of the visual object (1205). As a non-limiting example, the reference space range (not shown) may include the visual object (1205). As a non-limiting example, the boundary of the reference space range (not shown) may include a touch recognition area (1208) (or a touch recognition area (1212)).

[0219] As a non-limiting example, at least one processor (200) may recognize a touch input to a visual object (1205) based on the location where the movement direction of the input means (405) changes with respect to the depth data being located within the reference spatial range (not shown). However, the present invention is not limited thereto.

[0220] As a non-limiting example, at least one processor (200) may identify depth data that is relatively offset by a large amount using other reference data, as the depth data indicating the location of the visual object is relatively large. For example, at least one processor (200) may use the depth data that is relatively offset by a large amount to set a touch recognition area associated with the visual object within the 3D space (401).

[0221] As a non-limiting example, at least one processor (200) may use different reference data to identify depth data that is offset relative to the depth data indicating the position of the visual object. For example, at least one processor (200) may use depth data that is offset relative to the depth data indicating the position of the visual object to establish a touch recognition area associated with the visual object within the 3D space (401).

[0222] For example, although FIGS. 12A and 12B illustrate a state in which a touch recognition area associated with a visual object (104) is offset in a first direction toward the user from depth data indicating the location of the visual object, this is merely exemplary. For example, the depth data may also be offset in a second direction (e.g., opposite the first direction) from the user from depth data indicating the location of the visual object.

[0223] Referring again to FIG. 10, at operation 1050, at least one processor (200) may provide a function mapped to a visual object based on recognizing a touch input to the visual object.

[0224] For example, a function mapped to a visual object may include various functions assigned to the visual object. As a non-limiting example, if the visual object is an application icon, the visual object may be mapped with a function for executing the application. As a non-limiting example, if the visual object is a pop-up window requesting a selection from a user of the wearable device (101), the visual object may be mapped with a function for a selection selected by the user of the wearable device (101). As a non-limiting example, the visual object may include text for conveying information. As a non-limiting example, the visual object may be mapped with a function described in the text. As a non-limiting example, at least one processor (200) may map another visual object to be displayed based on recognizing a touch input to the visual object. As a non-limiting example, the visual object may be mapped to provide a function based on an input from a virtual touch screen. As a non-limiting example, the visual object may be mapped to provide a function based on an input from a virtual input device. However, the present invention is not limited thereto.

[0225] For example, there may be a difference in the depth data indicating the location of a visual object recognized by a user of the wearable device (101). For example, the difference may occur as the user of the wearable device (101) uses the wearable device (101). For example, the result of the calibration (or depth data) is applied to the depth data indicating the location of a visual object, and then adjustments may be required in the depth data indicating the location of a touch recognition area associated with the visual object. Updating the reference data using the touch recognition area associated with the visual object is exemplified in the description of FIG. 13.

[0226] Figure 13 illustrates an example of updating reference data using a touch recognition area associated with a visual object.

[0227] Referring to FIG. 13, at least one processor (200) can display a visual object (1101) through a display (110). For example, at least one processor (200) can display a visual object (1101) within a 3D space (401). For example, at least one processor (200) can display a visual object (1101) based on depth data (1102) within the 3D space (401).

[0228] For example, the wearable device (101) may display a pointer (or cursor) corresponding to the input means through the display (110). For example, a user (102) of the wearable device (101) may move the input means (405) to cause a touch input to a visual object (1101). For example, the input means (405) may be moved in the direction of the visual object (1101).

[0229] For example, at least one processor (200) can identify an input means (405) using images acquired through multiple cameras (220).

[0230] For example, at least one processor (200) can identify depth data (1302) offset from depth data (1102) using reference data in depth data (1102) indicating a location of a visual object (1101). For example, at least one processor (200) can use depth data (1302) to set a touch recognition area (1301) associated with the visual object (1101) within a 3D space (401). For example, the touch recognition area (1301) associated with the visual object (1101) can be located at the depth data (1302) within the 3D space (401). For example, the depth data (1302) indicating a location of the touch recognition area (1301) associated with the visual object (1101) can have a difference (1303) from the depth data (1102) indicating a location of the visual object (1101). For example, the difference (1303) may be the difference between depth data (1102) representing the position of the visual object (1101) within the displayed 3D space (401) and depth data representing the position of the visual object (1101) within the 3D space (401) perceived by the user (102) at the time the calibration was performed.

[0231] For example, depth data indicating the location of a visual object (1101) within a 3D space (401) recognized by a user (102) may be changed. For example, at least one processor (200) may receive an input for re-establishing the reference data. For example, at least one processor (200) may receive the input from a user (102).

[0232] For example, at least one processor (200) may determine, based on the input, a position at which the direction in which the input means (405) moves with respect to the visual object (402) changes with respect to the depth data. For example, the determined position may be a position of the input means (405) at a point in time when the user (102) of the wearable device (101) recognizes that a touch input has been made with respect to the visual object (1101).

[0233] For example, there may be a state (1300) where depth data (1304) indicating a position where the direction in which the input means (405) moves with respect to the visual object (1101) changes with respect to the depth data is offset from the depth data (1302) in a first direction with respect to the user.

[0234] For example, state (1300) can be described as a state in which at least one processor (200) receives the input. For example, in state (1300), the location in which the direction in which the input means (405) moves is changed with respect to the depth data can be located at the depth data (1304) within the 3D space (401).

[0235] For example, depth data (1304) indicating a location where the direction in which the input means (405) is moved changes with respect to the depth data may have a difference (1305) from depth data (1302) indicating a location of a touch recognition area (1301) associated with a visual object (1101). For example, in state (1300), the user (102) may recognize that the visual object (1101) is at a location corresponding to depth data (1304) that is different from the depth data (1102). For example, within state (1300), the difference (1305) may be a difference between depth data (1302) indicating a location of the visual object (1101) within the 3D space (401) recognized by the user (102) at the time when calibration was performed and depth data indicating a location of the visual object (1101) within the 3D space (401) recognized by the user (102).

[0236] For example, in state (1300), at least one processor (200) may need to at least partially reset the reference data using depth data (1304) indicating a location at which the direction in which the input means (405) is moved changes relative to the depth data. For example, in state (1300), at least one processor (200) may need to reset the location of a touch recognition area (1301) associated with a visual object (1101).

[0237] For example, in state (1300), at least one processor (200) can use the depth data (1102) of the visual object (1101) as reference data to re-identify depth data (1304) offset from the depth data (1102). For example, at least one processor (200) can use the depth data (1304) to re-establish a touch recognition area associated with the visual object (1101) within the 3D space (401).

[0238] For example, in state (1300), at least one processor (200) can change (or update, or refine) a touch recognition area (1301) associated with a visual object (1101).

[0239] For example, there may be a state (1306) where depth data (1304) indicating a position where the direction in which the input means (405) moves with respect to the visual object (1101) changes with respect to the depth data is offset from the depth data (1302) in a second direction (e.g., opposite to the first direction) with respect to the user.

[0240] For example, state (1306) can be described as a state in which at least one processor (200) receives the input. For example, in state (1306), the location in which the direction in which the input means (405) moves is changed with respect to the depth data can be located at the depth data (1307) within the 3D space (401).

[0241] For example, depth data (1307) indicating a location where the direction in which the input means (405) is moved changes with respect to the depth data may have a difference (1308) from depth data (1302) indicating a location of a touch recognition area (1301) associated with a visual object (1101). For example, in a state (1306), the user (102) may recognize that the visual object (1101) is at a location corresponding to depth data (1307) that is different from the depth data (1102). For example, in a state (1306), the difference (1305) may be a difference between depth data (1302) indicating a location of the visual object (1101) within the 3D space (401) recognized by the user (102) at the time when calibration was performed and depth data indicating a location of the visual object (1101) within the 3D space (401) recognized by the user (102).

[0242] For example, at state (1306), at least one processor (200) may need to at least partially reset the reference data using depth data (1307) indicating a location where the direction in which the input means (405) is moved changes relative to the depth data. For example, at state (1306), at least one processor (200) may need to reset the location of the touch recognition area (1301) relative to the visual object (1101).

[0243] For example, in state (1306), at least one processor (200) can use the depth data (1102) of the visual object (1101) as reference data to re-identify depth data (1307) offset from the depth data (1102). For example, at least one processor (200) can use the depth data (1307) to re-establish a touch recognition area associated with the visual object (1101) within the 3D space (401).

[0244] For example, in state (1306), at least one processor (200) can change (or update, or refine) a touch recognition area (1301) associated with a visual object (1101).

[0245] As a non-limiting example, the visual object (1101) may be recognized by the user (102) as a three-dimensional shape. For example, even if the visual object (1101) is recognized by the user (102) as a three-dimensional shape, the visual object (1101) may be displayed as an area according to depth data (1102) on the display (110) of the wearable device (101). For example, even if the visual object (1101) is recognized by the user (102) as a three-dimensional shape, within state (1300) or state (1306), the touch recognition area (1301) associated with the visual object (1101) may be an area corresponding to depth data (1302) within the 3D space (401).

[0246] As a non-limiting example, there may be a reference space range (not shown) within which a touch input to a visual object (1101) can be recognized. As a non-limiting example, the reference space range (not shown) may be formed according to depth data (1102) indicating the location of the visual object (1101). As a non-limiting example, the reference space range (not shown) may include the visual object (1101). As a non-limiting example, the boundary of the reference space range (not shown) may include a touch recognition area (1301).

[0247] As a non-limiting example, at least one processor (200) may recognize a touch input to a visual object (1101) based on the location where the movement direction of the input means (405) changes with respect to the depth data being located within the reference spatial range (not shown). However, the present invention is not limited thereto.

[0248] For example, although FIG. 13 illustrates a state in which a touch recognition area associated with a visual object (104) is offset in a first direction toward the user from depth data indicating the location of the visual object, this is merely exemplary. For example, the depth data may also be offset in a second direction (e.g., opposite the first direction) from the user from depth data indicating the location of the visual object.

[0249] FIG. 14 is a block diagram of an electronic device within a network environment according to various embodiments.

[0250] Referring to FIG. 14, in a network environment (1400), an electronic device (1401) may communicate with an electronic device (1402) via a first network (1498) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (1404) or a server (1408) via a second network (1499) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (1401) may communicate with the electronic device (1404) via the server (1408). According to one embodiment, the electronic device (1401) may include a processor (1420), a memory (1430), an input module (1450), an audio output module (1455), a display module (1460), an audio module (1470), a sensor module (1476), an interface (1477), a connection terminal (1478), a haptic module (1479), a camera module (1480), a power management module (1488), a battery (1489), a communication module (1490), a subscriber identification module (1496), or an antenna module (1497). In some embodiments, the electronic device (1401) may omit at least one of these components (e.g., the connection terminal (1478)), or may have one or more other components added. In some embodiments, some of these components (e.g., sensor module (1476), camera module (1480), or antenna module (1497)) may be integrated into a single component (e.g., display module (1460)).

[0251] The processor (1420) may, for example, execute software (e.g., a program (1440)) to control at least one other component (e.g., a hardware or software component) of the electronic device (1401) connected to the processor (1420) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1420) may store commands or data received from other components (e.g., a sensor module (1476) or a communication module (1490)) in a volatile memory (1432), process the commands or data stored in the volatile memory (1432), and store result data in a non-volatile memory (1434). According to one embodiment, the processor (1420) may include a main processor (1421) (e.g., a central processing unit or an application processor) or an auxiliary processor (1423) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1421). For example, when the electronic device (1401) includes the main processor (1421) and the auxiliary processor (1423), the auxiliary processor (1423) may be configured to use less power than the main processor (1421) or to be specialized for a given function. The auxiliary processor (1423) may be implemented separately from the main processor (1421) or as a part thereof.

[0252] The auxiliary processor (1423) may control at least a portion of functions or states associated with at least one component (e.g., the display module (1460), the sensor module (1476), or the communication module (1490)) of the electronic device (1401), for example, on behalf of the main processor (1421) while the main processor (1421) is in an inactive (e.g., sleep) state, or together with the main processor (1421) while the main processor (1421) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1423) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1480) or a communication module (1490)). In one embodiment, the auxiliary processor (1423) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1401) where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1408)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0253] The memory (1430) can store various data used by at least one component (e.g., the processor (1420) or the sensor module (1476)) of the electronic device (1401). The data can include, for example, software (e.g., the program (1440)) and input data or output data for commands related thereto. The memory (1430) can include volatile memory (1432) or non-volatile memory (1434).

[0254] The program (1440) may be stored as software in memory (1430) and may include, for example, an operating system (1442), middleware (1444), or an application (1446).

[0255] The input module (1450) can receive commands or data to be used in a component of the electronic device (1401) (e.g., a processor (1420)) from an external source (e.g., a user) of the electronic device (1401). The input module (1450) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0256] The audio output module (1455) can output audio signals to the outside of the electronic device (1401). The audio output module (1455) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0257] The display module (1460) can visually provide information to an external party (e.g., a user) of the electronic device (1401). The display module (1460) may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling the device. In one embodiment, the display module (1460) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0258] The audio module (1470) can convert sound into an electrical signal, or vice versa. According to one embodiment, the audio module (1470) can acquire sound through the input module (1450), output sound through the sound output module (1455), or an external electronic device (e.g., electronic device (1402)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1401).

[0259] The sensor module (1476) can detect the operating status (e.g., power or temperature) of the electronic device (1401) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1476) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0260] The interface (1477) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1401) with an external electronic device (e.g., the electronic device (1402)). In one embodiment, the interface (1477) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0261] The connection terminal (1478) may include a connector through which the electronic device (1401) may be physically connected to an external electronic device (e.g., the electronic device (1402)). In one embodiment, the connection terminal (1478) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0262] The haptic module (1479) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (1479) may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0263] The camera module (1480) can capture still images and videos. In one embodiment, the camera module (1480) may include one or more lenses, image sensors, image signal processors, or flashes.

[0264] The power management module (1488) can manage the power supplied to the electronic device (1401). According to one embodiment, the power management module (1488) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0265] A battery (1489) may power at least one component of the electronic device (1401). In one embodiment, the battery (1489) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0266] The communication module (1490) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1401) and an external electronic device (e.g., electronic device (1402), electronic device (1404), or server (1408)), and the performance of communication through the established communication channel. The communication module (1490) may operate independently from the processor (1420) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1490) may include a wireless communication module (1492) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1494) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (1404) via a first network (1498) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1499) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network or a wide area network)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1492) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1496) to identify or authenticate the electronic device (1401) within a communication network such as the first network (1498) or the second network (1499).

[0267] The wireless communication module (1492) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1492) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1492) may support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1492) may support various requirements specified in the electronic device (1401), an external electronic device (e.g., the electronic device (1404)), or a network system (e.g., the second network (1499)). According to one embodiment, the wireless communication module (1492) may support a peak data rate (e.g., 20 Gbps or more) for eMBB implementation, a loss coverage (e.g., 164 dB or less) for mMTC implementation, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC implementation.

[0268] The antenna module (1497) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1497) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1497) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1498) or the second network (1499), may be selected from the plurality of antennas by, for example, the communication module (1490). A signal or power may be transmitted or received between the communication module (1490) and the external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1497).

[0269] According to various embodiments, the antenna module (1497) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0270] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0271] According to one embodiment, commands or data may be transmitted or received between the electronic device (1401) and an external electronic device (1404) via a server (1408) connected to a second network (1499). Each of the external electronic devices (1402 or 1404) may be the same or a different type of device as the electronic device (1401). According to one embodiment, all or part of the operations executed in the electronic device (1401) may be executed in one or more of the external electronic devices (1402, 1404, or 1408). For example, when the electronic device (1401) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1401) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1401). The electronic device (1401) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1401) may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1404) may include an Internet of Things (IoT) device. The server (1408) may be an intelligent server utilizing machine learning and / or a neural network.According to one embodiment, an external electronic device (1404) or server (1408) may be included within the second network (1499). The electronic device (1401) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0272] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0273] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0274] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0275] Various embodiments of the present document may be implemented as software (e.g., a program (1440)) including one or more instructions stored in a storage medium (e.g., an internal memory (1436) or an external memory (1438)) readable by a machine (e.g., an electronic device (1401)). For example, a processor (e.g., a processor (1420)) of the machine (e.g., an electronic device (1401)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0276] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0277] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0278] FIG. 15a illustrates an example of a perspective view of a wearable device.

[0279] FIG. 15b illustrates an example of one or more hardware devices arranged within a wearable device.

[0280] For example, the wearable device (1500) may have the form of glasses that are wearable on a part of the user's body (e.g., the head). The wearable device (1500) may include a head-mounted display (HMD). For example, the housing of the wearable device (1500) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (1500) may include one or more straps that are capable of being twined around the user's head, and / or one or more temples that are detachably attachable to the ears of the head.

[0281] Referring to FIG. 15A, a wearable device (1500) may include at least one display (1550) and a frame supporting at least one display (1550).

[0282] For example, the wearable device (1500) can be worn on a part of the user's body. The wearable device (1500) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to the user wearing the wearable device (1500). For example, the wearable device (1500) can display a virtual reality image provided from at least one optical device (1582, 1584) of FIG. 15B on at least one display (1550) in response to a user's designated gesture acquired through the motion recognition cameras (1560-2, 1560-3) of FIG. 15B.

[0283] For example, at least one display (1550) may provide visual information to a user. For example, at least one display (1550) may include a transparent or translucent lens. At least one display (1550) may include a first display (1550-1) and / or a second display (1550-2) spaced apart from the first display (1550-1). For example, the first display (1550-1) and the second display (1550-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0284] Referring to FIG. 15B, at least one display (1550) can provide visual information transmitted from external light to a user through a lens included in the at least one display (1550), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, the at least one display (1550) can include a first surface (1531) and a second surface (1532) opposite to the first surface (1531). A display area can be formed on the second surface (1532) of the at least one display (1550). When a user wears the wearable device (1500), external light can be transmitted to the user by being incident on the first surface (1531) and transmitted through the second surface (1532). As another example, at least one display (1550) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (1582, 1584) on a real screen transmitted through external light, in a display area formed on the second surface (1532).

[0285] For example, at least one display (1550) may include at least one waveguide (1533, 1534) that diffracts light emitted from at least one optical device (1582, 1584) and transmits the diffracted light to a user. The at least one waveguide (1533, 1534) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (1533, 1534). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (1533, 1534) may be propagated to the other end of the at least one waveguide (1533, 1534) by the nano-pattern. At least one waveguide (1533, 1534) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (1533, 1534) may be arranged within the wearable device (1500) to guide a screen displayed by at least one display (1550) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (1533, 1534).

[0286] The wearable device (1500) can analyze an object included in a real image collected through a shooting camera (1560-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (1550). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (1500) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (1500) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using the multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (1500) can view an image displayed on at least one display (1550).

[0287] For example, the frame may be configured as a physical structure that allows the wearable device (1500) to be worn on the user's body. For example, the frame may be configured so that, when the user wears the wearable device (1500), the first display (1550-1) and the second display (1550-2) can be positioned corresponding to the user's left and right eyes. The frame may support at least one display (1550). For example, the frame may support the first display (1550-1) and the second display (1550-2) to be positioned corresponding to the user's left and right eyes.

[0288] Referring to FIG. 15A, the frame may include a region (1520) that at least partially contacts a portion of the user's body when the user wears the wearable device (1500). For example, the region (1520) of the frame that contacts a portion of the user's body may include a region that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (1500) makes contact with. For example, the frame may include a nose pad (1510) that contacts a portion of the user's body. When the wearable device (1500) is worn by the user, the nose pad (1510) may contact a portion of the user's nose. The frame may include a first temple (1504) and a second temple (1505) that contact another portion of the user's body that is distinct from the portion of the user's body.

[0289] For example, the frame may include a first rim (1501) that surrounds at least a portion of a first display (1550-1), a second rim (1502) that surrounds at least a portion of a second display (1550-2), a bridge (1503) that is disposed between the first rim (1501) and the second rim (1502), a first pad (1511) that is disposed along a portion of an edge of the first rim (1501) from one end of the bridge (1503), a second pad (1512) that is disposed along a portion of an edge of the second rim (1502) from the other end of the bridge (1503), a first temple (1504) that extends from the first rim (1501) and is secured to a portion of an ear of the wearer, and a second temple (1505) that extends from the second rim (1502) and is secured to a portion of an ear opposite the ear. The first pad (1511) and the second pad (1512) may be in contact with a portion of the user's nose, and the first temple (1504) and the second temple (1505) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (1504, 1505) may be rotatably connected to the rim through the hinge units (1506, 1507) of FIG. 15B. The first temple (1504) may be rotatably connected to the first rim (1501) through the first hinge unit (1506) disposed between the first rim (1501) and the first temple (1504). The second temple (1505) may be rotatably connected to the second rim (1502) via a second hinge unit (1507) disposed between the second rim (1502) and the second temple (1505). For example, the wearable device (1500) may use a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame to identify an external object (e.g., a user's fingertip) touching the frame and / or a gesture performed by the external object.

[0290] For example, the wearable device (1500) may include hardwares that perform various functions (e.g., the hardwares described above based on the block diagram of FIG. 2). For example, the hardwares may include a battery module (1570), an antenna module (1575), at least one optical device (1582, 1584), speakers (e.g., speakers 1555-1, 1555-2), a microphone (e.g., microphones 1565-1, 1565-2, 1565-3), a light-emitting module, and / or a printed circuit board (PCB) (1590) (e.g., a printed circuit board). The various hardware components may be arranged within the frame.

[0291] For example, microphones (e.g., microphones 1565-1, 1565-2, 1565-3) of the wearable device (1500) may be arranged on at least a portion of the frame to acquire sound signals. A first microphone (1565-1) arranged on the bridge (1503), a second microphone (1565-2) arranged on the second rim (1502), and a third microphone (1565-3) arranged on the first rim (1501) are illustrated in FIG. 15B , but the number and arrangement of the microphones (1565) are not limited to the embodiment of FIG. 15B . When the number of microphones (1565) included in the wearable device (1500) is two or more, the wearable device (1500) may identify the direction of the sound signal by using a plurality of microphones arranged on different portions of the frame.

[0292] For example, at least one optical device (1582, 1584) can project a virtual object onto at least one display (1550) to provide various image information to a user. For example, at least one optical device (1582, 1584) can be a projector. At least one optical device (1582, 1584) can be positioned adjacent to at least one display (1550) or can be included within at least one display (1550) as a part of at least one display (1550). For example, a wearable device (1500) can include a first optical device (1582) corresponding to a first display (1550-1) and a second optical device (1584) corresponding to a second display (1550-2). For example, at least one optical device (1582, 1584) may include a first optical device (1582) disposed at an edge of a first display (1550-1) and a second optical device (1584) disposed at an edge of a second display (1550-2). The first optical device (1582) may transmit light to a first waveguide (1533) disposed on the first display (1550-1), and the second optical device (1584) may transmit light to a second waveguide (1534) disposed on the second display (1550-2).

[0293] For example, the camera (1560) may include a recording camera (1560-4), an eye tracking camera (ET CAM) (1560-1), and / or a motion recognition camera (1560-2, 1560-3). The recording camera (1560-4), the eye tracking camera (1560-1), and the motion recognition cameras (1560-2, 1560-3) may be positioned at different positions on the frame and may perform different functions. The eye tracking camera (1560-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (1500). For example, the wearable device (1500) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (1560-1).

[0294] The wearable device (1500) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (1560-1). The wearable device (1500) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (1500) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (1560-1). The wearable device (1500) can render an image (or screen) displayed on at least one display (1550) based on the position of the user's eyes.

[0295] For example, the visual quality of a first region related to the gaze within an image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI (pixels per inch)) of a second region distinct from the first region may be different from each other. The wearable device (1500) may obtain an image having the visual quality of the first region matching the user's gaze and the visual quality of the second region using foveated rendering. For example, if the wearable device (1500) supports an iris recognition function, user authentication may be performed based on iris information obtained using a gaze tracking camera (1560-1). Although an example in which the gaze tracking camera (1560-1) is positioned toward the user's right eye is illustrated in FIG. 15B, the embodiment is not limited thereto, and the gaze tracking camera (1560-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0296] For example, the capturing camera (1560-4) can capture an actual image or background to be aligned with a virtual image in order to implement augmented reality or mixed reality content. The capturing camera (1560-4) can be used to obtain a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (1560-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (1550). The at least one display (1550) can display a single image in which information about an actual image or background including an image of the specific object obtained using the capturing camera (1560-4) and a virtual image provided through at least one optical device (1582, 1584) are superimposed. The wearable device (1500) can compensate for depth information (e.g., the distance between the wearable device (1500) and an external object acquired through a depth sensor) using an image acquired through the capture camera (1560-4). The wearable device (1500) can perform object recognition using an image acquired using the capture camera (1560-4). The wearable device (1500) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capture camera (1560-4). The wearable device (1500) can perform a pass-through function to display an image acquired through the capture camera (1560-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (1550). For example, the shooting camera (1560-4) may be placed on a bridge (1503) that is placed between the first rim (1501) and the second rim (1502).

[0297] The gaze tracking camera (1560-1) can implement more realistic augmented reality by tracking the gaze of a user wearing a wearable device (1500) and matching the user's gaze with visual information provided to at least one display (1550). For example, when the wearable device (1500) looks straight ahead, the wearable device (1500) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (1550). The gaze tracking camera (1560-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (1560-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. For example, the gaze tracking camera (1560-1) can be placed at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (1560-1) may be positioned within the first rim (1501) and / or the second rim (1502) to face the direction in which the user wearing the wearable device (1500) is positioned.

[0298] The gesture recognition camera (1560-2, 1560-3) can provide a specific event on a screen provided on at least one display (1550) by recognizing the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the user's body. The gesture recognition camera (1560-2, 1560-3) can recognize the user's gesture (gesture recognition), obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (1550). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition camera (1560-2, 1560-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor may perform gesture recognition and / or object tracking functions using the motion recognition cameras (1560-2, 1560-3). For example, the motion recognition cameras (1560-2, 1560-3) may be positioned on the first rim (1501) and / or the second rim (1502).

[0299] The camera (1560) included in the wearable device (1500) is not limited to the above-described gaze tracking camera (1560-1) and motion recognition cameras (1560-2, 1560-3). For example, the wearable device (1500) can identify an external object included in the FoV of the camera (1560) by using a camera positioned toward the user's FoV. The wearable device (1500) can identify an external object based on a sensor for identifying the distance between the wearable device (1500) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (1560) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (1500) may include a camera (1560) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (1500).

[0300] Although not shown, for example, the wearable device (1500) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being captured using the camera (1560). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame and hinge units (1506, 1507).

[0301] For example, the battery module (1570) may supply power to electronic components of the wearable device (1500). For example, the battery module (1570) may be disposed within the first temple (1504) and / or the second temple (1505). For example, the battery module (1570) may be a plurality of battery modules (1570). The plurality of battery modules (1570) may be disposed within each of the first temple (1504) and the second temple (1505). For example, the battery module (1570) may be disposed at an end of the first temple (1504) and / or the second temple (1505).

[0302] The antenna module (1575) can transmit signals or power to the outside of the wearable device (1500), or receive signals or power from the outside. For example, the antenna module (1575) can be positioned within the first temple (1504) and / or the second temple (1505). For example, the antenna module (1575) can be positioned close to one surface of the first temple (1504) and / or the second temple (1505).

[0303] The speaker (1555) can output an audio signal to the outside of the wearable device (1500). The audio output module may be referred to as a speaker. For example, the speaker (1555) may be positioned within the first temple (1504) and / or the second temple (1505) so as to be positioned adjacent to the ear of a user wearing the wearable device (1500). For example, the speaker (1555) may include a second speaker (1555-2) positioned within the first temple (1504) and thus positioned adjacent to the user's left ear, and a first speaker (1555-1) positioned within the second temple (1505) and thus positioned adjacent to the user's right ear.

[0304] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state, in order to visually provide information regarding a specific state of the wearable device (1500) to the user. For example, when the wearable device (1500) requires charging, it may emit red light at a regular cycle. For example, the light-emitting module may be disposed on the first rim (1501) and / or the second rim (1502).

[0305] Referring to FIG. 15B, the wearable device (1500) may include a printed circuit board (PCB) (1590). The PCB (1590) may be included in at least one of the first temple (1504) or the second temple (1505). The PCB (1590) may include an interposer positioned between at least two sub-PCBs. One or more hardwares included in the wearable device (1500) (e.g., hardwares illustrated by different blocks in FIG. 2) may be positioned on the PCB (1590). The wearable device (1500) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0306] For example, the wearable device (1500) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (1500) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (1500). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). For example, the wearable device (1500) may identify a user's motion and / or gesture performed to execute or stop a specific function of the wearable device (1500) based on the IMU.

[0307] Figures 16a and 16b illustrate examples of the appearance of a wearable device.

[0308] The wearable device (1600) of FIGS. 16A and 16B may include at least a portion of the hardware of the wearable device (1500) described with reference to FIGS. 15A and / or 15B. For example, an example of the appearance of a first side (1610) of a housing of the wearable device (1600) may be illustrated in FIG. 16A, and an example of the appearance of a second side (1620) opposite to the first side (1610) may be illustrated in FIG. 16B.

[0309] Referring to FIG. 16A, for example, a first surface (1610) of a wearable device (1600) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (1600) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (1504) and / or the second temple (1505) of FIGS. 15A and 15B). A first display (1550-1) for outputting an image to a left eye among the user's two eyes, and a second display (1550-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (1610). The wearable device (1600) may be formed on the first surface (1610) and may further include a rubber or silicone packing to prevent interference from light (e.g., ambient light) different from the light emitted from the first display (1550-1) and the second display (1550-2).

[0310] For example, the wearable device (1600) may include cameras (1560-1) for photographing and / or tracking the user's two eyes adjacent to each of the first display (1550-1) and the second display (1550-2). The cameras (1560-1) may be referred to as the eye tracking camera (1560-1) of FIG. 15B. For example, the wearable device (1600) may include cameras (1560-5, 1560-6) for photographing and / or recognizing the user's face. The cameras (1560-5, 1560-6) may be referred to as FT cameras. The wearable device (1600) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (1560-5, 1560-6). For example, the wearable device (1600) may change the texture and / or shape of a portion of an avatar (e.g., a portion of the avatar representing a human face) using information obtained by cameras (1560-5, 1560-6) (e.g., FT cameras) and representing facial expressions of a user wearing the wearable device (1600).

[0311] Referring to FIG. 16B, a camera (e.g., cameras (1560-7, 1560-8, 1560-9, 1560-10, 1560-11, 1560-12)) and / or a sensor (e.g., a depth sensor (1630)) for obtaining information related to the external environment of the wearable device (1600) may be disposed on a second surface (1620) opposite to the first surface (1610) of FIG. For example, the cameras (1560-7, 1560-8, 1560-9, 1560-10) may be disposed on the second surface (1620) to recognize external objects. The cameras (1560-7, 1560-8, 1560-9, 1560-10) of FIG. 16b can correspond to the motion recognition cameras (1560-2, 1560-3) of FIG. 15b.

[0312] For example, using cameras (1560-11, 1560-12), the wearable device (1600) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (1560-11) can be placed on the second face (1620) of the wearable device (1600) to obtain an image to be displayed through the second display (1550-2) corresponding to the right eye among the two eyes. The camera (1560-12) can be placed on the second face (1620) of the wearable device (1600) to obtain an image to be displayed through the first display (1550-1) corresponding to the left eye among the two eyes. The cameras (1560-11, 1560-12) can correspond to the shooting camera (1560-4) of FIG. 15B.

[0313] For example, the wearable device (1600) may include a depth sensor (1630) disposed on the second face (1620) to identify a distance between the wearable device (1600) and an external object. Using the depth sensor (1630), the wearable device (1600) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (1600). Although not shown, a microphone may be disposed on the second face (1620) of the wearable device (1600) to obtain sound output from an external object. The number of microphones may be one or more depending on the embodiment.

[0314] For example, a wearable device may be required to display a visual object according to first depth data within a 3D (three dimensional) space provided through the display, and, while the visual object is displayed within the 3D space, identify that an input means is in contact with an area within the 3D space corresponding to second depth data offset from the first depth data using images acquired through the plurality of cameras, and recognize the input means in contact with the area as a touch input for the visual object.

[0315] As described above, the wearable device may include a plurality of cameras positioned toward the eyes of a user wearing the wearable device. The wearable device may include a display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide a three-dimensional (3D) space through the display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a visual object according to first depth data while providing at least a portion of the 3D space through the display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify second depth data by modifying the first depth data using reference data. The reference data may be used to apply a display location of the visual object recognized by a user of the wearable device to touch input recognition. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to set a touch recognition area for the visual object associated with the 3D space using the second depth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, using images acquired through the plurality of cameras while the visual object is displayed within the 3D space, that a location of an input means corresponds to the touch recognition area. The instructions may cause the wearable device to recognize the correspondence as a touch input to the visual object.

[0316] For example, the position of the input means may correspond to the touch recognition area before corresponding to the display area of ​​the visual object within the 3D space according to the first depth data.

[0317] For example, the reference data may be determined according to the direction of movement of the input means that changes with respect to the depth data in order to apply the display position of the visual object recognized by the user of the wearable device to touch input recognition.

[0318] For example, the visual object may be a first visual object, the 3D space may be a first 3D space, and the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a second visual object according to third depth data within a second 3D space provided through the display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify an input means moved relative to the second visual object using images acquired through the plurality of cameras while the second visual object is displayed within the second 3D space. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine a position at which a direction of the input means moved relative to the second visual object changes relative to the depth data based on the identification. The above instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain the reference data using the determined location.

[0319] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a different visual object within the 3D space provided through the display according to third depth data that is different from the first depth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, using images acquired through the plurality of cameras while the different visual object is displayed within the 3D space, that a location of the input means corresponds to a different touch recognition area within the 3D space corresponding to fourth depth data that is changed from the third depth data by applying the reference data to the third depth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to recognize a location of the input means corresponding to the different touch recognition area as a touch input for the different visual object. The difference between the first depth data and the second depth data may be the same as the difference between the third depth data and the fourth depth data.

[0320] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a different visual object within the 3D space provided through the display according to third depth data that is different from the first depth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, using images acquired through the plurality of cameras while the different visual object is displayed within the 3D space, that a location of the input means corresponds to a different touch recognition area within the 3D space corresponding to fourth depth data that is changed from the third depth data by applying different reference data to the third depth data, the different reference data being different from the reference data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to recognize a location of the input means corresponding to the different touch recognition area as a touch input for the different visual object. The above reference data may be determined based on a direction of movement of the input means that is changed based on depth data within a first reference depth range that includes the first depth data. The other reference data may be determined based on a direction of movement of the input means that is changed based on depth data within a second reference depth range that includes the third depth data and is distinct from the first reference depth range.

[0321] For example, the difference between the first depth data and the second depth data may be different from the difference between the third depth data and the fourth depth data.

[0322] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to receive an input for resetting the reference data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to, in response to the input, identify, using images acquired through the plurality of cameras, a movement of the input means after the position of the input means corresponds to the touch recognition area. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine, in response to the identification, a position at which a direction in which the input means is moved changes with respect to depth data after the position of the input means corresponds to the touch recognition area. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to at least partially reset the reference data applied to depth data of a visual object in 3D space to recognize a touch input using the determined location.

[0323] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to provide a function mapped to the visual object based on the recognition.

[0324] For example, the input means may correspond to the hand of the user of the wearable device, which is detected based on the acquired images.

[0325] As described above, the method may be performed in a wearable device including a plurality of cameras positioned toward the eyes of a user wearing the wearable device and a display. The method may include an operation of providing a three-dimensional (3D) space through the display. The method may include an operation of displaying a visual object according to first depth data while providing at least a portion of the 3D space through the display. The method may include an operation of identifying second depth data by modifying the first depth data using reference data. The reference data may be used to apply a display position of the visual object recognized by a user of the wearable device to touch input recognition. The method may include an operation of setting a touch recognition area for the visual object related to the 3D space using the second depth data. The method may include an operation of identifying that a position of an input means corresponds to the touch recognition area using images acquired through the plurality of cameras while the visual object is displayed in the 3D space. The method may include an action of recognizing the correspondence as a touch input to the visual object.

[0326] For example, the position of the input means may correspond to the touch recognition area before corresponding to the display area of ​​the visual object within the 3D space according to the first depth data.

[0327] For example, the reference data may be determined according to the direction of movement of the input means that changes with respect to the depth data in order to apply the display position of the visual object recognized by the user of the wearable device to touch input recognition.

[0328] For example, the visual object may be a first visual object, the 3D space may be a first 3D space, and the method may include an operation of displaying a second visual object according to third depth data within a second 3D space provided through the display. The method may include an operation of identifying an input means moved with respect to the second visual object using images acquired through the plurality of cameras while the second visual object is displayed within the second 3D space. The method may include an operation of determining a position at which a direction of the input means moved with respect to the second visual object changes with respect to the depth data based on the identification. The method may include an operation of acquiring the reference data using the determined position.

[0329] For example, the method may include an operation of displaying a different visual object according to third depth data different from the first depth data within the 3D space provided through the display. The method may include an operation of identifying, using images acquired through the plurality of cameras while the different visual object is displayed within the 3D space, that the position of the input means corresponds to another touch recognition area within the 3D space corresponding to fourth depth data changed from the third depth data by applying the reference data to the third depth data. The method may include an operation of recognizing the position of the input means corresponding to the different touch recognition area as a touch input for the different visual object. A difference between the first depth data and the second depth data may be the same as a difference between the third depth data and the fourth depth data.

[0330] For example, the method may include an operation of displaying a different visual object according to third depth data different from the first depth data within the 3D space provided through the display. The method may include an operation of identifying, using images acquired through the plurality of cameras while the different visual object is displayed within the 3D space, that the position of the input means corresponds to a different touch recognition area within the 3D space corresponding to fourth depth data changed from the third depth data by applying different reference data different from the reference data to the third depth data. The method may include an operation of recognizing the position of the input means corresponding to the different touch recognition area as a touch input for the different visual object. The reference data may be determined according to a direction of movement of the input means that changes based on depth data within a first reference depth range that includes the first depth data. The different reference data may be determined according to a direction of movement of the input means that changes based on depth data within a second reference depth range that includes the third depth data and is distinct from the first reference depth range.

[0331] For example, the difference between the first depth data and the second depth data may be different from the difference between the third depth data and the fourth depth data.

[0332] For example, the method may include an operation of receiving an input for resetting the reference data. The method may include an operation of identifying, based on the input, a movement of the input means after the position of the input means corresponds to the touch recognition area, using images acquired through the plurality of cameras. The method may include an operation of determining, based on the identification, a position at which a direction in which the input means moves changes with respect to depth data after the position of the input means corresponds to the touch recognition area. The method may include an operation of at least partially resetting, using the determined position, the reference data applied to depth data of a visual object in 3D space to recognize a touch input.

[0333] For example, the method may include an action of providing a function mapped to the visual object based on the recognition.

[0334] For example, the input means may correspond to the hand of the user of the wearable device, which is detected based on the acquired images.

[0335] In a computer-readable storage medium having one or more programs stored thereon, as described above, the one or more programs may include instructions that, when executed by a wearable device including a display and a plurality of cameras positioned toward the eyes of a user wearing the wearable device, cause the electronic device to provide a three-dimensional (3D) space through the display. The one or more programs may include instructions that, when executed by the electronic device, cause the wearable device to display a visual object according to first depth data while providing at least a portion of the 3D space through the display. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify second depth data by modifying the first depth data using reference data. The reference data may be used to apply a display position of the visual object recognized by a user of the wearable device to touch input recognition. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to set a touch recognition area for the visual object associated with the 3D space using the second depth data. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify, using images acquired through the plurality of cameras while the visual object is displayed within the 3D space, that a location of an input means corresponds to the touch recognition area.The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to recognize the correspondence as a touch input to the visual object.

[0336] For example, the position of the input means may correspond to the touch recognition area before corresponding to the display area of ​​the visual object within the 3D space according to the first depth data.

[0337] For example, the reference data may be determined according to the direction of movement of the input means that changes with respect to the depth data in order to apply the display position of the visual object recognized by the user of the wearable device to touch input recognition.

[0338] For example, the visual object may be a first visual object, the 3D space may be a first 3D space, and the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display a second visual object according to third depth data within a second 3D space provided through the display. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify an input means moved with respect to the second visual object using images acquired through the plurality of cameras while the second visual object is displayed within the second 3D space. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to determine a position at which a direction of the input means moved with respect to the second visual object changes with respect to depth data based on the identification. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to obtain the reference data using the determined location.

[0339] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display a different visual object according to third depth data that is different from the first depth data within the 3D space provided through the display. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify, using images acquired through the plurality of cameras while the different visual object is displayed within the 3D space, that the position of the input means corresponds to a different touch recognition area within the 3D space corresponding to fourth depth data that is changed from the third depth data by applying the reference data to the third depth data. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to recognize a position of the input means corresponding to the different touch recognition area as a touch input for the different visual object. The difference between the first depth data and the second depth data may be the same as the difference between the third depth data and the fourth depth data.

[0340] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display a different visual object according to third depth data that is different from the first depth data within the 3D space provided through the display. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify, using images acquired through the plurality of cameras while the different visual object is displayed within the 3D space, that the position of the input means corresponds to a different touch recognition area within the 3D space corresponding to fourth depth data that is changed from the third depth data by applying different reference data that is different from the reference data to the third depth data. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to recognize a position of the input means corresponding to the different touch recognition area as a touch input for the different visual object. The above reference data may be determined based on a direction of movement of the input means that is changed based on depth data within a first reference depth range that includes the first depth data. The other reference data may be determined based on a direction of movement of the input means that is changed based on depth data within a second reference depth range that includes the third depth data and is distinct from the first reference depth range.

[0341] For example, the difference between the first depth data and the second depth data may be different from the difference between the third depth data and the fourth depth data.

[0342] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to receive an input for resetting the reference data. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify, using images acquired through the plurality of cameras, a movement of the input means after the position of the input means corresponds to the touch recognition area, in response to the input. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to determine, based on the identification, a position at which a direction in which the input means is moved changes with respect to depth data after the position of the input means corresponds to the touch recognition area. The one or more programs, when executed by the wearable device, may include instructions that cause the wearable device to at least partially reset the reference data applied to depth data of a visual object in 3D space to recognize a touch input using the determined location.

[0343] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to provide a function mapped to the visual object based on the recognition.

[0344] For example, the input means may correspond to the hand of the user of the wearable device, which is detected based on the acquired images.

[0345] As described above, the wearable device may include a plurality of cameras positioned toward the eyes of a user wearing the wearable device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to provide a three-dimensional (3D) space through the display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a visual object based on depth data while providing at least a portion of the 3D space through the display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify an input means moved with respect to the visual object using images acquired through the plurality of cameras while the visual object is displayed in the 3D space. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine a position at which a direction of movement of the input means changes with respect to depth data based on the identification. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain reference data for applying a display position of the visual object recognized by a user of the wearable device to touch input recognition using the determined position.

[0346] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to obtain the reference data using the determined location based on depth data of the determined location that is different from the depth data used to display the visual object within the 3D space. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to refrain from obtaining the reference data using the determined location based on depth data of the determined location corresponding to the depth data used to display the visual object within the 3D space.

[0347] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, based on displaying the visual object within the 3D space, using images acquired through the plurality of cameras, whether the input means is located within a reference space range within the 3D space according to the depth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to bypass identifying whether the direction of movement of the input means changes relative to the depth data until the input means is located within the reference space range. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, based on identifying the input means as being located within the reference space range, whether the direction of movement of the input means changes relative to the depth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine the location based on identifying that the direction of movement of the input means has changed relative to the depth data.

[0348] For example, the depth data used to display the visual object within the 3D space may be first depth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display another visual object within the 3D space provided through the display according to second depth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, using images acquired through the plurality of cameras while the other visual object is displayed within the 3D space, that the position of the input means corresponds to another touch recognition area within the 3D space corresponding to third depth data changed from the second depth data by applying the reference data to the second depth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to recognize the position of the input means corresponding to the other touch recognition area as a touch input for the other visual object. The difference between the first depth data and the depth data indicating the determined position may be the same as the difference between the second depth data and the third depth data.

[0349] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to provide a function mapped to the other visual object based on the recognition.

[0350] For example, the depth data used to display the visual object within the 3D space may be first depth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a different visual object within the 3D space provided through the display according to second depth data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to identify, using images acquired through the plurality of cameras while the different visual object is displayed within the 3D space, that the position of the input means corresponds to a different touch recognition area within the 3D space corresponding to third depth data changed from the second depth data by applying different reference data to the second depth data, which is different from the reference data. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to recognize the input means in contact with the other touch recognition area as a touch input for the other visual object. The reference data may be determined according to a direction of movement of the input means that is changed based on depth data within a first reference depth range including the first depth data. The other reference data may be determined according to a direction of movement of the input means that is changed based on depth data within a second reference depth range that is distinct from the first reference depth range and includes the second depth data.

[0351] For example, the difference between the first depth data and the depth data indicating the determined position may be different from the difference between the second depth data and the third depth data.

[0352] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to provide a function mapped to the other visual object based on the recognition.

[0353] As described above, the method may be performed by a wearable device including a plurality of cameras positioned toward the eyes of a user wearing the wearable device and a display. The method may include an operation of providing a three-dimensional (3D) space through the display. The method may include an operation of displaying a visual object according to depth data while providing at least a portion of the 3D space through the display. The method may include an operation of identifying an input means moved with respect to the visual object using images acquired through the plurality of cameras while the visual object is displayed in the 3D space. The method may include an operation of determining a position at which a direction of movement of the input means changes with respect to the depth data based on the identification. The method may include an operation of acquiring reference data for recognizing a touch input occurring within the 3D space using the determined position.

[0354] For example, the method may include an operation of obtaining the reference data using the determined position based on depth data of the determined position that is different from the depth data used to display the visual object within the 3D space. The method may include an operation of refraining from obtaining the reference data using the determined position based on the depth data of the determined position corresponding to the depth data used to display the visual object within the 3D space.

[0355] For example, the method may include an operation of identifying, based on displaying the visual object within the 3D space, whether the input means is located within a reference space range within the 3D space according to the depth data, using images acquired through the plurality of cameras. The method may include an operation of bypassing identifying, until the input means is located within the reference space range, whether the direction of the movement of the input means changes with respect to the depth data. The method may include an operation of identifying, based on identifying the input means located within the reference space range, whether the direction of the movement of the input means changes with respect to the depth data. The method may include an operation of determining the position based on identifying that the direction of the movement of the input means changes with respect to the depth data.

[0356] For example, the depth data used to display the visual object within the 3D space may be first depth data. The method may include an operation of displaying another visual object according to second depth data within the 3D space provided through the display. The method may include an operation of identifying, using images acquired through the plurality of cameras while the other visual object is displayed within the 3D space, that the position of the input means corresponds to another touch recognition area within the 3D space corresponding to third depth data changed from the second depth data by applying the reference data to the second depth data. The method may include an operation of recognizing the position of the input means corresponding to the other touch recognition area as a touch input for the other visual object. A difference between the first depth data and the depth data representing the determined position may be the same as a difference between the second depth data and the third depth data.

[0357] For example, the method may include an action of providing a function mapped to the other visual object based on the recognition.

[0358] For example, the depth data used to display the visual object within the 3D space may be first depth data. The method may include an operation of displaying another visual object according to second depth data within the 3D space provided through the display. The method may include an operation of identifying that the position of the input means corresponds to another touch recognition area within the 3D space corresponding to third depth data changed from the second depth data by applying other reference data different from the reference data to the second depth data using images acquired through the plurality of cameras while the other visual object is displayed within the 3D space. The method may include an operation of recognizing the input means in contact with the other touch recognition area as a touch input for the other visual object. The reference data may be determined according to a direction of movement of the input means that is changed based on depth data within a first reference depth range including the first depth data. The above other reference data may be determined according to a direction of movement of the input means that is changed based on depth data within a second reference depth range that includes the second depth data and is distinct from the first reference depth range.

[0359] For example, the difference between the first depth data and the depth data indicating the determined position may be different from the difference between the second depth data and the third depth data.

[0360] For example, the method may include an action of providing a function mapped to the other visual object based on the recognition.

[0361] As described above, the non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by a wearable device including a plurality of cameras positioned toward the eyes of a user wearing the wearable device and a display, cause the electronic device to provide a three-dimensional (3D) space through the display. The one or more programs may include instructions that, when executed by the electronic device, cause the wearable device to display a visual object based on depth data while providing at least a portion of the 3D space through the display. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify an input means moved with respect to a visual object using images acquired through the plurality of cameras while the visual object is displayed in the 3D space. The one or more programs may include instructions causing the wearable device to determine a location at which the direction of movement of the input means changes relative to the depth data based on the identification. The one or more programs may include instructions causing the wearable device to obtain reference data for recognizing a touch input occurring within the 3D space using the determined location.

[0362] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to obtain the reference data using the determined location based on depth data of the determined location that is different from the depth data used to display the visual object within the 3D space. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to refrain from obtaining the reference data using the determined location based on the depth data of the determined location corresponding to the depth data used to display the visual object within the 3D space.

[0363] For example, the one or more programs may include instructions that cause the wearable device, when executed by the wearable device, to identify, based on displaying the visual object within the 3D space, using images acquired through the plurality of cameras, whether the input means is positioned within a reference space range within the 3D space according to the depth data. The one or more programs may include instructions that cause the wearable device, when executed by the wearable device, to bypass identifying, until the input means is positioned within the reference space range, whether the direction of the movement of the input means changes with respect to the depth data. The one or more programs may include instructions that cause the wearable device, when executed by the wearable device, to identify, based on identifying the input means as being positioned within the reference space range, whether the direction of the movement of the input means changes with respect to the depth data. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to determine the location based on identifying that the direction of movement of the input means has changed with respect to the depth data.

[0364] For example, the depth data used to display the visual object within the 3D space may be first depth data. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display a different visual object according to second depth data within the 3D space provided through the display. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify, using images acquired through the plurality of cameras while the different visual object is displayed within the 3D space, that the position of the input means corresponds to a different touch recognition area within the 3D space corresponding to third depth data changed from the second depth data by applying the reference data to the second depth data. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to recognize a location of the input means corresponding to the other touch recognition area as a touch input to the other visual object.

[0365] For example, the difference between the first depth data and the depth data indicating the determined position may be the same as the difference between the second depth data and the third depth data.

[0366] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to provide a function mapped to the other visual object based on the recognition.

[0367] For example, the depth data used to display the visual object within the 3D space may be first depth data. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to display a different visual object according to second depth data within the 3D space provided through the display. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to identify, using images acquired through the plurality of cameras while the different visual object is displayed within the 3D space, that the position of the input means corresponds to a different touch recognition area within the 3D space corresponding to third depth data changed from the second depth data by applying different reference data, which is different from the reference data, to the second depth data. The one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to recognize the input means in contact with the other touch recognition area as a touch input for the other visual object. The reference data may be determined according to a direction of movement of the input means that is changed based on depth data within a first reference depth range that includes the first depth data. The other reference data may be determined according to a direction of movement of the input means that is changed based on depth data within a second reference depth range that includes the second depth data and is distinct from the first reference depth range.

[0368] For example, the difference between the first depth data and the depth data indicating the determined position may be different from the difference between the second depth data and the third depth data.

[0369] For example, the one or more programs may include instructions that, when executed by the wearable device, cause the wearable device to provide a function mapped to the other visual object based on the recognition.

[0370] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0371] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0372] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0373] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0374] Therefore, other implementations, other embodiments, and equivalents of the claims are also included in the scope of the claims described below. For example, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a commodity. The computer program product may be distributed in the form of a storage medium that can be read by a machine (e.g., compact disc read only memory (CD-ROM)) or may be available through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0375] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In a wearable device (101), A memory (210) storing instructions and including one or more storage media; A plurality of cameras (220) positioned toward the eyes of a user (102) wearing the wearable device (101); display (110); and At least one processor (200) comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor (200), Provides a 3D (three dimensional) space (401) through the above display (110), While providing at least a part of the 3D space (401) through the display (110), displaying a visual object according to the first depth data, The second depth data is identified by changing the first depth data using reference data, and the reference data is used to apply the display position of the visual object recognized by the user (102) of the wearable device (101) to touch input recognition. Using the second depth data, a touch recognition area for the visual object related to the 3D space (401) is set, Using images acquired through the plurality of cameras (220) while the visual object is displayed within the 3D space (401), it is identified that the position of the input means (405) corresponds to the touch recognition area, To recognize the above response as a touch input to the above visual object, causing the above wearable device (101), Wearable device (101).

2. In claim 1, the position of the input means (405) is: Before corresponding to the display area of ​​the visual object in the 3D space (401) according to the first depth data, corresponding to the touch recognition area, Wearable device (101).

3. In claim 1, the reference data is: In order to apply the display position of the visual object recognized by the user (102) of the wearable device (101) to the touch input recognition, the direction of movement of the input means (405) that is changed with respect to the depth data is determined. Wearable device (101).

4. In claim 1, the visual object is: The first visual object, The above 3D space (401) is The first 3D space, The above instructions, when individually or collectively executed by the at least one processor (200), In the second 3D space provided through the above display (110), a second visual object is displayed according to the third depth data, Using the images acquired through the plurality of cameras (220) while the second visual object is displayed in the second 3D space, an input means (405) moving with respect to the second visual object is identified, According to the above identification, the direction of the input means (405) moving with respect to the second visual object is determined at a position where it changes with respect to the depth data, To obtain the reference data using the above determined location, causing the above wearable device (101), Wearable device (101).

5. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (200), Within the 3D space (401) provided through the display (110), different visual objects are displayed according to third depth data different from the first depth data, Using images acquired through the plurality of cameras (220) while the other visual object is displayed within the 3D space (401), the position of the input means (405) is identified as corresponding to another touch recognition area within the 3D space (401) corresponding to fourth depth data changed from the third depth data by applying the reference data to the third depth data, To recognize the position of the input means (405) corresponding to the other touch recognition area as a touch input for the other visual object, causing the above wearable device (101), The difference between the first depth data and the second depth data is Same as the difference between the third depth data and the fourth depth data, Wearable device (101).

6. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (200), Within the 3D space (401) provided through the display (110), different visual objects are displayed according to third depth data different from the first depth data, By using images acquired through the plurality of cameras (220) while the other visual object is displayed in the 3D space (401), the position of the input means (405) is applied to the third depth data by applying different reference data different from the reference data to the third depth data, thereby identifying that the position corresponds to another touch recognition area in the 3D space (401) corresponding to the fourth depth data changed from the third depth data, Causing the wearable device (101) to recognize the position of the input means (405) corresponding to the other touch recognition area as a touch input for the other visual object, The above reference data is, It is determined according to the direction of movement of the input means (405) that is changed based on the depth data within the first reference depth range including the first depth data, The other reference data above are, Determined according to the direction of movement of the input means (405) that is changed based on depth data within the second reference depth range that includes the third depth data and is distinct from the first reference depth range. Wearable device (101).

7. In claim 6, the difference between the first depth data and the second depth data is: The difference between the third depth data and the fourth depth data is different, Wearable device (101).

8. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (200), Receive input to reset the above reference data, According to the above input, using the images acquired through the plurality of cameras (220), the movement of the input means (405) is identified after the position of the input means (405) corresponds to the touch recognition area, After the position of the input means (405) corresponds to the touch recognition area according to the above identification, the position in which the direction in which the input means (405) moves is determined to change with respect to the depth data, Using the above-determined position, at least partially reset the reference data applied to the depth data of the visual object in the 3D space (401) to recognize the touch input. causing the above wearable device (101), Wearable device (101).

9. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (200), Based on the above recognition, to provide a function mapped to the visual object, causing the above wearable device (101), Wearable device (101).

10. In claim 1, the input means (405) Based on the acquired images, the hand of the user (102) is detected, Wearable device (101).

11. A method executed in a wearable device (101) comprising at least one processor (200) including a plurality of cameras (220) positioned toward the eyes of a user (102) wearing the wearable device (101) and a display (110), An operation of providing a 3D (three dimensional) space (401) through the above display (110), An operation of displaying a visual object according to first depth data while providing at least a part of the 3D space (401) through the display (110); An operation in which the second depth data is identified by changing the first depth data using reference data, and the reference data is used to apply the display position of the visual object recognized by the user (102) of the wearable device (101) to touch input recognition, An operation of setting a touch recognition area for the visual object related to the 3D space (401) using the second depth data; An operation of identifying that the position of the input means (405) corresponds to the touch recognition area using images acquired through the plurality of cameras (220) while the visual object is displayed within the 3D space (401), and An action that recognizes the above response as a touch input to the visual object, method.

12. In claim 11, the position of the input means (405) is: Before corresponding to the display area of ​​the visual object in the 3D space (401) according to the first depth data, corresponding to the touch recognition area, method.

13. In claim 11, the reference data is: In order to apply the display position of the visual object recognized by the user (102) of the wearable device (101) to the touch input recognition, the direction of movement of the input means (405) that is changed with respect to the depth data is determined. method.

14. In claim 11, the visual object is: The first visual object, The above 3D space (401) is The first 3D space, The above method, An operation of displaying a second visual object according to third depth data within a second 3D space provided through the above display (110), An operation of identifying an input means (405) moving with respect to the second visual object using images acquired through the plurality of cameras (220) while the second visual object is displayed within the second 3D space; An operation for determining a position at which the direction of the input means (405) moving with respect to the second visual object according to the above identification is changed with respect to the depth data; An operation of obtaining the reference data using the determined location, method.

15. In a non-transitory computer-readable storage medium storing one or more programs, The above one or more programs, when executed by the wearable device (101) including a plurality of cameras (220) and a display (110) positioned toward the eyes of a user (102) wearing the wearable device (101), Provides a 3D (three dimensional) space (401) through the above display (110), While providing at least a portion of the 3D space through the display (110), displaying a visual object according to the first depth data, The second depth data is identified by changing the first depth data using reference data, and the reference data is used to apply the display position of the visual object recognized by the user (102) of the wearable device (101) to touch input recognition. Using the second depth data, a touch recognition area for the visual object related to the 3D space (401) is set, Using images acquired through the plurality of cameras (220) while the visual object is displayed within the 3D space (401), it is identified that the position of the input means (405) corresponds to the touch recognition area, To recognize the above response as a touch input to the above visual object, Including instructions that cause the above wearable device (101), Non-transitory computer-readable storage medium.

Images