Wearable device and method for moving virtual object to obtain information about gaze positions

Abstract

A wearable device according to an embodiment may display a virtual object on at least one display. The wearable device may determine a user's gaze position by using eye images obtained from at least one camera. The wearable device, during detection of a gaze position in a detection area including the virtual object, may determine information indicating the displacement of gaze positions detected respectively at two viewpoints. The wearable device may determine the distance for moving the virtual object within a range less than the magnitude of the displacement indicated by the determined information. The wearable device, on the basis of determining that the distance is longer than a threshold distance, may move the virtual object by the determined distance in the direction of the displacement indicated by the information. The wearable device, on the basis of determining that the distance is shorter than the threshold distance, may stop moving the virtual object and execute a function related to the virtual object.

Description

Wearable device and method for moving a virtual object to obtain information on gaze position

[0001] The present disclosure relates to a wearable device and method for moving a virtual object to obtain information on a gaze position.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these electronic devices may be AR glasses and / or head-mounted devices (HMDs).

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] According to an embodiment, a wearable device may include at least one display, at least one camera configured to be positioned toward an eye of a user wearing the wearable device, at least one processor including processing circuitry, and a memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a virtual object on the at least one display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine a gaze position of the user using an image of the eye acquired from the at least one camera. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine information representing a displacement of gaze positions detected at two time points while detecting the gaze position within a detection area including the virtual object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine a distance for movement of the virtual object within a range less than a magnitude of the displacement indicated by the information based on the determined information. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to move the virtual object by the determined distance in the direction of the displacement indicated by the information based on determining the distance to be greater than a threshold distance.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to stop moving the virtual object based on determining that the distance is less than a threshold distance. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to perform a function associated with the virtual object based on stopping moving the virtual object.

[0005] In one embodiment, a non-transitory computer-readable storage medium comprising instructions may be provided. The instructions may be executed by a wearable device. The wearable device may include at least one display and at least one camera configured to be positioned toward an eye of a user wearing the wearable device. The instructions, when executed by the wearable device, may cause the wearable device to display a virtual object on the at least one display. The instructions, when executed by the wearable device, may cause the wearable device to determine a gaze position of the user using an image of the eye acquired from the at least one camera. The instructions, when executed by the wearable device, may cause the wearable device to determine information representing a displacement of the gaze positions detected at each of two time points while detecting the gaze positions within a detection area including the virtual object. The instructions, when executed by the wearable device, may cause the wearable device to determine a distance for movement of the virtual object within a range less than a magnitude of the displacement indicated by the information based on the determined information. The instructions, when executed by the wearable device, may cause the wearable device to move the virtual object by the determined distance in the direction of the displacement indicated by the information based on determining the distance to be greater than a threshold distance. The instructions, when executed by the wearable device, may cause the wearable device to stop moving the virtual object based on determining the distance to be less than the threshold distance.The above instructions, when executed by the wearable device, may cause the wearable device to execute a function associated with the virtual object based on the wearable device ceasing to move the virtual object.

[0006] According to an embodiment, a wearable device may include at least one display, at least one camera configured to be positioned toward an eye of a user wearing the wearable device, at least one processor including processing circuitry, and a memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a virtual object on the at least one display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine a gaze position of the user using an image of the eye acquired from the at least one camera. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine information indicative of a displacement of the gaze position within a detection area, based on detecting the gaze position within the detection area, which includes the virtual object. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to move the virtual object in the direction of the displacement. The distance by which the virtual object is moved in the direction may be less than the magnitude of the displacement. The at least one processor may be individually or collectively configured to repeatedly move the virtual object in the direction while the distance exceeds a threshold distance.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to stop moving the virtual object using the gaze position based on identifying that the distance is less than a threshold distance. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to perform a function associated with the virtual object based on stopping moving the virtual object.

[0007] In one embodiment, a non-transitory computer-readable storage medium comprising instructions may be provided. The instructions may be executed by a wearable device. The wearable device may include at least one display and at least one camera configured to be positioned toward an eye of a user wearing the wearable device. The instructions, when executed by the wearable device, may cause the wearable device to display a virtual object on the at least one display. The instructions, when executed by the wearable device, may cause the wearable device to determine a gaze position of the user using an image of the eye acquired from the at least one camera. The instructions, when executed by the wearable device, may cause the wearable device to determine information indicative of a displacement of the gaze position within the detection area based on detecting the gaze position within the detection area, which includes the virtual object. The instructions, when executed by the wearable device, may cause the wearable device to move the virtual object in the direction of the displacement. A distance of the virtual object moved in the direction may be less than a magnitude of the displacement. The instructions, when executed by the wearable device, may cause the wearable device to repeatedly move the virtual object in the direction while the distance exceeds a threshold distance. The instructions, when executed by the wearable device, may cause the wearable device to stop moving the virtual object using the gaze position based on determining that the distance is less than the threshold distance.The above instructions, when executed by the wearable device, may cause a function associated with the virtual object to be executed based on stopping movement of the virtual object.

[0008] In one embodiment, a method for a wearable device may be provided. The wearable device may include at least one display and at least one camera configured to be positioned toward the eyes of a user wearing the wearable device. The method may include displaying a plurality of virtual objects on the at least one display. The method may include determining a gaze position of the user using an image of the eye acquired from the at least one camera. The method may include moving the plurality of virtual objects so that the plurality of virtual objects move away from the gaze position based on detecting the gaze position within an overlapping area of ​​detection areas of the plurality of virtual objects. The method may include determining a first virtual object focused on by the user among the plurality of virtual objects using a displacement of the gaze position detected after the plurality of virtual objects are moved. The method may include storing the displacement of the gaze position as information to be used for calibration related to the gaze position adjacent to the first virtual object based on determining the first virtual object.

[0009] In one embodiment, a wearable device may include at least one display, at least one camera configured to be positioned toward an eye of a user wearing the wearable device, at least one processor including a processing circuit, and a memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to display a plurality of virtual objects on the at least one display. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine a gaze position of the user using an image of the eye acquired from the at least one camera. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to move the plurality of virtual objects away from the gaze location based on detecting the gaze location within an overlapping area of ​​the detection areas of the plurality of virtual objects. The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to determine a first virtual object focused on by the user among the plurality of virtual objects using a displacement of the gaze location detected after the plurality of virtual objects have been moved.The instructions, when individually or collectively executed by the at least one processor, may cause the wearable device to store the displacement of the gaze position as information to be used for calibration related to the gaze position adjacent to the first virtual object based on determining the first virtual object.

[0010] FIG. 1 illustrates an embodiment of a wearable device that displays a screen including a virtual object.

[0011] FIG. 2 illustrates a block diagram of a wearable device according to one embodiment.

[0012] FIG. 3 illustrates a flowchart of a wearable device according to one embodiment.

[0013] FIG. 4 illustrates a screen displayed by a wearable device according to one embodiment.

[0014] Figures 5a, 5b, 5c and 5d illustrate states of a wearable device that detects gaze position.

[0015] Figures 6a and 6b illustrate the operation of a wearable device that executes a function related to a virtual object using a gaze position.

[0016] Figure 7 illustrates the operation of a wearable device that moves a virtual object according to the gaze position.

[0017] FIG. 8 illustrates the operation of a wearable device that displays a virtual object related to a gaze position on a lock screen.

[0018] Figures 9a, 9b, 9c and 9d illustrate the operation of a wearable device that detects gaze positions within overlapping areas of detection areas.

[0019] Figures 10a, 10b and 10c illustrate the operation of a wearable device that detects gaze positions within an overlapping area of ​​detection areas.

[0020] FIG. 11 illustrates the operation of a wearable device that changes the position and / or shape of a virtual object using the gaze position.

[0021] Figures 12a and 12b illustrate the operation of a wearable device that changes the depth of a virtual object using the gaze position.

[0022] FIG. 13A illustrates an example of a perspective view of a wearable device, according to one embodiment.

[0023] FIG. 13b illustrates an example of one or more hardware elements disposed within a wearable device, according to one embodiment.

[0024] FIGS. 14A and 14B illustrate an example of an appearance of a wearable device according to one embodiment.

[0025] Hereinafter, various embodiments of this document are described with reference to the attached drawings.

[0026] The various embodiments of this document and the terminology used therein are not intended to limit the technology described in this document to a specific embodiment, but should be understood to include various modifications, equivalents, and / or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expression may include plural expressions unless the context clearly indicates otherwise. In this document, expressions such as "A or B", "at least one of A and / or B", "A, B, or C", or "at least one of A, B, and / or C" may include all possible combinations of the items listed together. Expressions such as "first", "second", "first", or "second" may modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), said component may be directly connected to said other component, or may be connected via another component (e.g., a third component).

[0027] The term "module" as used in this document includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimal unit or portion thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0028] FIG. 1 illustrates an embodiment of a wearable device (101) that displays a screen (120) including a virtual object (o). The wearable device (101) may include a head-mounted display (HMD) that can be worn on a user's head. The wearable device (101) may be referred to as a head-mounted display (HMD) device, a headgear electronic device, a glasses-type (or goggle-type) electronic device, a video see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. Although the external appearance of the wearable device (101) in the form of glasses is illustrated, the embodiment is not limited thereto. An example of a hardware configuration included in the wearable device (101) is exemplarily described with reference to FIG. 2. An example of the structure of a wearable device (101) that can be worn on a user's head is described with reference to FIGS. 13a, 13b, 14a, and / or 14b. The wearable device (101) may be referred to as an electronic device. For example, the electronic device may include an accessory (e.g., a strap) for attaching to the user's head.

[0029] According to one embodiment, a wearable device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, while a user wears the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the user's eyes. The wearable device (101) may combine ambient light passing through the lens with light emitted from a display of the wearable device (101). A display area of ​​the display may be formed within the lens through which the ambient light passes. Because the wearable device (101) combines the ambient light and the light emitted from the display, the user may see a mixed image of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0030] According to one embodiment, a wearable device (101) may perform functions related to video see-through (VST) and / or virtual reality (VR). For example, in a state where a user wears the wearable device (101), the wearable device (101) may include a housing that covers the user's eyes. In the state, the wearable device (101) may include a display arranged on a first surface of the housing facing the eyes. Referring to FIG. 1, the wearable device (101) may include a first display (110-1) configured to face the left eye (140-1) of a user wearing the wearable device (101) and a second display (110-2) configured to face the right eye (140-2) of a user wearing the wearable device (101).

[0031] According to one embodiment, a wearable device (101) may include at least one camera configured to be arranged toward a user's eyes. Referring to FIG. 1, the wearable device (101) may include a first camera (130-1) configured to face a left eye (140-1) of a user wearing the wearable device (101) and a second camera (130-2) configured to face a right eye (140-2) of a user wearing the wearable device (101). The wearable device (101) may obtain images and / or videos of the user's two eyes (e.g., the left eye (130-1) and / or the right eye (130-2)) from the first camera (130-1) and / or the second camera (130-2), and may calculate or determine a location gazed by the user. The location detected by the wearable device (101) and gazed at by the user may be referred to as a gaze location, gaze point, and / or gaze point.

[0032] Referring to FIG. 1, an exemplary screen (120) displayed by a wearable device (101) is illustrated. The wearable device (101) can display a virtual object (o) having a three-dimensional position within the screen (120). The wearable device (101) can change a first position of the virtual object (ol) within a first display (110-1) and a second position of the virtual object (or) within a second display (110-2), thereby changing a distance of the virtual object (o) from the wearable device (101) recognized by the user. For example, by adjusting a difference between the first position and the second position (e.g., a difference in position on the horizontal axis of the screen (120)), the wearable device (101) can change the perspective of the virtual object (o). As the above difference increases, the binocular disparity (b) of the virtual object (o) may increase, and the wearable device (101) may provide a user viewing the virtual object (o) with a sensation that the virtual object (o) is approaching. As the above difference decreases, the binocular disparity (b) of the virtual object (o) may decrease, and the wearable device (101) may provide a user viewing the virtual object (o) with a sensation that the virtual object (o) is moving away.

[0033] According to one embodiment, the wearable device (101) may be configured to track a gaze position using at least one camera, such as a first camera (130-1) and a second camera (130-2). The accuracy of the gaze position calculated by the wearable device (101) may change due to the positional relationship between the user's two eyes and the wearable device (101) that changes each time the user wears the wearable device (101) (or while the user is wearing the wearable device (101). According to one embodiment, the wearable device (101) may perform calibration related to the gaze position to compensate for or reduce an error in the gaze position caused by the positional relationship.

[0034] In one embodiment, to perform calibration, the wearable device (101) may obtain calibration information related to a gaze position. The calibration information may include information related to calculation of the gaze position. For example, the calibration information may include at least one of an image and / or a video acquired using the first camera (130-1) and / or the second camera (130-2), and / or a result of calculating the gaze position from the image and / or the video. In one embodiment, the wearable device (101) may obtain calibration information using a screen for a function other than the function of obtaining calibration information, such as a lock screen, instead of a user interface (UI) dedicated to obtaining calibration information (e.g., a UI including one or more virtual objects and / or one or more visual objects).

[0035] For example, by moving the virtual object (o) displayed for the above-described other function, the wearable device (101) can cause the user's gaze to move, such as following the virtual object (o). Using the gaze movement, the wearable device (101) can store information related to the user's gaze looking at a specific location within the screen (120) as calibration information. An exemplary operation of the wearable device (101) to trigger the gaze movement by moving the virtual object (o) within the screen (120) is described with reference to FIG. 3. An exemplary operation of the wearable device (101) to obtain calibration information using the lock screen is described with reference to FIG. 4, FIG. 6A to FIG. 6B, FIG. 8, and FIG. 9A to FIG. 9D. An exemplary operation of a wearable device (101) to obtain calibration information using virtual objects displayed on various screens including a lock screen is described with reference to FIGS. 5A to 5D, FIG. 7, FIGS. 10A to 10C, FIG. 11, and FIGS. 12A to 12B.

[0036] FIG. 2 illustrates a block diagram of a wearable device (101) according to one embodiment. The wearable device (101) of FIG. 2 may include the wearable device (101) of FIG. 1.

[0037] Referring to FIG. 2, a wearable device (101) according to one embodiment may include at least one of a processor (210), a memory (215), a display (110) (e.g., the first display (110-1) and / or the second display (110-2) of FIG. 1), a camera (130) (e.g., the first camera (130-1) and / or the second camera (130-2) of FIG. 1), a sensor (230), or a communication circuit (235). The processor (210), the memory (215), the display (110), the camera (130), the sensor (230), and / or the communication circuit (235) may be electrically and / or operatively connected to each other by electronic components such as a communication bus (202). In the present disclosure, the operational connection of electronic components may include a direct connection established between the electronic components and / or an indirect connection established between the electronic components, such that a first electronic component among the electronic components is controlled by a second electronic component among the electronic components. The type and / or number of electronic components included in the wearable device (101) is not limited to those illustrated in FIG. 2. For example, the wearable device (101) may include only some of the electronic components illustrated in FIG. 2.

[0038] A processor (210) of a wearable device (101) according to one embodiment may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the wearable device (101) may include one or more processors. The processor (210) may have a multi-core processor structure such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor structure of the processor (210) may include a structure based on a plurality of core circuits (e.g., a big-little structure) that are distinguished by power consumption, clock, and / or calculation amount per unit time. In one embodiment comprising a processor (210) having a multi-core processor architecture, the operations and / or functions of the present disclosure may be performed individually or collectively by one or more cores included in the processor (210).

[0039] According to one embodiment, the memory (215) of the wearable device (101) may include electronic components for storing data and / or instructions input and / or output to the processor (210). The memory (215) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multi media card (eMMC). In one embodiment, the memory (215) may be referred to as storage.

[0040] In one embodiment, a display (110) of a wearable device (101) can output visualized information to a user of the wearable device (101). The display (110), which is arranged in front of the eyes of a user wearing the wearable device (101), can be arranged on at least a portion of a housing of the wearable device (101) (e.g., the first display (110-1) and / or the second display (110-2) of FIG. 1 ). For example, the display (110) can be controlled by a processor (210) including a circuit such as a graphic processing unit (GPU) to output visualized information to the user. The display (110) can include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (110) can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LED may include an OLED (organic LED). The embodiment is not limited thereto, and for example, if the wearable device (101) includes a lens for transmitting external light (or ambient light), the display (110) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (110) may be referred to as a display panel and / or a display module. The pixels included in the display (110) may be arranged to face one of the user's two eyes when the wearable device (101) is worn by the user. For example, the display (110) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0041] In one embodiment, the camera (130) of the wearable device (101) may include one or more optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The camera (130) may be referred to as an image sensor and may be included in the sensor (230) of FIG. 2. The plurality of optical sensors included in the camera (130) may be arranged in the form of a two-dimensional array. The camera (130) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously, and generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (130) may mean one (a) two-dimensional frame data acquired from the camera (130). For example, video data captured using a camera (130) may mean a sequence of a plurality of two-dimensional frame data acquired from the camera (130) according to a frame rate. The camera (130) may be positioned toward the direction in which the camera (130) receives light and may further include a flash light for outputting light toward the direction.

[0042] According to one embodiment, the wearable device (101) may include a plurality of cameras, for example, cameras (130), arranged in different directions. As described above with reference to FIG. 1, the plurality of cameras may include gaze tracking cameras (e.g., camera 130-1 and / or second camera 130-2 of FIG. 1) configured to be arranged toward the eyes of a user wearing the wearable device (101). The plurality of cameras may include outward cameras. The processor (210) may identify the direction of the user's gaze using images and / or videos acquired from the gaze tracking cameras. The gaze tracking cameras may include infrared (IR) sensors. The gaze tracking cameras may be referred to as eye sensors and / or eye trackers.

[0043] For example, the external camera may be positioned facing the front of a user wearing the wearable device (101) (e.g., in a direction that both eyes may face). The wearable device (101) may include multiple external cameras. The embodiment is not limited thereto, and the external camera may be positioned facing an external space. Using images and / or videos acquired from the external cameras, the processor (210) may identify external objects. For example, based on images and / or videos acquired from the external cameras, the processor (210) may identify the position, shape, and / or gesture (e.g., hand gesture) of a hand of a user wearing the wearable device (101). Using images and / or videos of the external environment acquired from the external cameras, the processor (210) may recognize or track one or more objects within the external environment.

[0044] According to one embodiment, a sensor (230) of a wearable device (101) may generate electrical information from non-electronic information related to the wearable device (101), which may be processed and / or stored by a processor (210) and / or a memory (215) of the wearable device (101). The information may be referred to as sensor data. The sensor (230) may include a global positioning system (GPS) sensor for detecting a geographic location of the wearable device (101), an image sensor, an audio sensor (e.g., a microphone and / or a microphone array including a plurality of microphones), an ambient light sensor, an inertial measurement unit (IMU) (e.g., an accelerometer, a gyro sensor, and / or a geomagnetic sensor), and / or a time-of-flight (ToF) sensor (or a ToF camera). As with the ToF sensor, the wearable device (101) may include a sensor configured to detect the distance between the wearable device (101) and an external object. A sensor that detects the distance between external objects may be referred to as a depth sensor.

[0045] In one embodiment, the depth sensor included in the wearable device (101) may include a ToF sensor and / or a structured light (SL) sensor. The SL sensor may be referred to as an SL camera. The ToF sensor may be referred to as a ToF camera. The SL sensor may emit or output a light pattern (e.g., a plurality of dots) of a specific wavelength (e.g., an infrared wavelength). When an external object reflects the light pattern, the light pattern may be distorted by embossing on the surface of the external object. By detecting the reflected light for the light pattern, the SL sensor and / or the processor (210) connected to the SL sensor may recognize the distortion. Using the distortion, the processor (210) may calculate the distance between the wearable device (101) and the external object. The ToF sensor may emit light of a specific wavelength (e.g., an infrared wavelength) in units of nanoseconds. The ToF sensor can measure the time it takes for light reflected by an external object to propagate to the ToF sensor. Using the measured time, the ToF sensor and / or processor (210) can calculate or determine the distance between the external object and the wearable device (101). Information calculated using the ToF sensor and / or the SL sensor can be referred to as a depth map.

[0046] In one embodiment, the communication circuit (235) of the wearable device (101) may include circuitry for supporting transmission and / or reception of electrical signals between the wearable device (101) and an external electronic device. The communication circuit (235) may include, for example, at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit (235) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G new radio (NR), 6G and / or above-6G. In one embodiment, the communication circuit (235) may be referred to as a communication processor and / or a communication module.

[0047] According to one embodiment, one or more instructions (or commands) representing data to be processed, calculations to be performed, and / or operations to be performed by the processor (210) of the wearable device (101) may be stored in the memory (215) of the wearable device (101). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or a software application (hereinafter, “application”). For example, the wearable device (101) and / or the processor (210) may perform at least one of the operations of FIG. 3 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, a program, and / or a software application is executed. Hereinafter, the fact that a software application is installed in a wearable device (101) may mean that one or more instructions provided in the form of a software application (or package) are stored in a memory (215), and that the one or more applications are stored in a format executable by the processor (210) (e.g., a file having an extension specified by the operating system of the wearable device (101)). As an example, the application may include a program and / or a library related to a service provided to a user.

[0048] Referring to FIG. 2, programs installed in the wearable device (101) may be included in any one of different layers, including the application layer (240), the framework layer (250), and / or the hardware abstraction layer (HAL) (280), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware (e.g., the display (110), the camera (130), the sensor (230), and / or the communication circuit (235)) of the wearable device (101) may be included in the hardware abstraction layer (280). The framework layer (250) may be referred to as an XR framework layer from the perspective of including one or more programs for providing an XR (extended reality) service. For example, the layers illustrated in FIG. 2 may be logically (or for convenience of explanation) separated, and may not mean that the address space of the memory (215) is separated by the layers.

[0049] For example, within the framework layer (250), programs designed to target at least one of the hardware abstraction layer (280) and / or the application layer (240) (e.g., a position tracker (271), a space recognizer (272), a gesture tracker (273), and / or an eye-gaze tracker (274)) may be included. The programs included in the framework layer (250) may provide an API (application programming interface) that is executable (or callable) based on other programs.

[0050] For example, the application layer (240) may include a program designed to target users of the wearable device (101). As an example of programs included in the application layer (240), an extended reality (XR) system user interface (UI) (241) and / or an XR application (242) are exemplified, but the embodiment is not limited thereto. For example, programs (e.g., software applications) included in the application layer (240) may call an API to cause execution of functions supported by programs included in the framework layer (250).

[0051] For example, the wearable device (101) may display one or more visual objects on the display (110) for performing interaction with the user based on the execution of the XR system UI (241). A visual object may refer to an object that can be placed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (101) may provide the user with functions available within a virtual space based on the execution of the XR system UI (241).

[0052] Referring to FIG. 2, a lightweight renderer (243) and / or an XR plug-in (244) are illustrated to be included within the XR system UI (241), but are not limited thereto. For example, based on the XR system UI (241), the processor (210) may execute a lightweight renderer (243) and / or an XR plug-in (244) within the framework layer (250).

[0053] For example, the wearable device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (243). The lightweight renderer (243) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (243) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (244). The XR plugin (244) can be referred to as an open XR native client from the perspective of defining (or setting up) the entire rendering pipeline.

[0054] For example, the wearable device (101) may display a screen representing at least a portion of a virtual space on the display (110) based on the execution of the XR application (242). The XR plug-in (244-1) included in the XR application (242) may include instructions that support functions similar to those of the XR plug-in (244) of the XR system UI (241). Descriptions of the XR plug-in (244-1) that overlap with those of the XR plug-in (244) may be omitted. The wearable device (101) may cause the execution of the virtual space manager (251) based on the execution of the XR application (242).

[0055] According to one embodiment, the wearable device (101) may provide a virtual space service based on the execution of the virtual space manager (251). For example, the virtual space manager (251) may include a platform for supporting the virtual space service. The wearable device (101) may identify a virtual space formed based on the user's location indicated by data acquired through the sensor (230) based on the execution of the virtual space manager (251), and may display at least a portion of the virtual space on the display (110). The virtual space manager (251) may be referred to as a composition presentation manager (CPM).

[0056] For example, the virtual space manager (251) may include a runtime service (252). As an example, the runtime service (252) may be referred to as an OpenXR runtime module (or an OpenXR runtime program). The wearable device (101) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (252). As an example, the wearable device (101) may perform rendering for a virtual space service to the user based on the execution of the runtime service (252). For example, a function related to a virtual space, executable by the application layer (240), may be supported based on the execution of the runtime service (252).

[0057] For example, the virtual space manager (251) may include a pass-through manager (253). Based on the execution of the pass-through manager (253), the wearable device (101) may display a screen representing a virtual space (e.g., screen (120) of FIG. 1) on the display (110), while displaying an image and / or video representing an actual space acquired through an external camera by overlaying it on at least a portion of the screen.

[0058] For example, the virtual space manager (251) may include an input manager (254). The wearable device (101) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (270) based on the execution of the input manager (254). The wearable device (101) may use the acquired data to identify a user input related to the wearable device (101). The user input may be related to a motion (e.g., a hand gesture), gaze, and / or speech of the user identified by the sensor (230) and / or the camera (110) (e.g., an external camera). The user input may be identified based on an external electronic device connected (or paired) via the communication circuit (235).

[0059] For example, the perception abstract layer (260) can be used for data exchange between the virtual space manager (251) and the perception service layer (270). From the perspective of being used for data exchange between the virtual space manager (251) and the perception service layer (270), the perception abstract layer (260) can be referred to as an interface. For example, the perception abstract layer (260) can be referenced as OpenPX. The perception abstract layer (260) can be used for a perception client and a perception service.

[0060] According to one embodiment, the recognition service layer (270) may include one or more programs for processing data acquired from the sensor (230) and / or the camera (130). The one or more programs may include at least one of a position tracker (271), a space recognizer (272), a gesture tracker (273), and / or an eye tracker (274). The type and / or number of the one or more programs included in the recognition service layer (270) are not limited to those illustrated in FIG. 2.

[0061] For example, the wearable device (101) can identify the pose of the wearable device (101) using the sensor (230) based on the execution of the position tracker (271). The wearable device (101) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) using data acquired using an external camera and / or an IMU (e.g., a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor) based on the execution of the position tracker (271). The position tracker (271) may be referred to as a head tracking (HeT) module (or head tracker, a head tracking program).

[0062] For example, the wearable device (101) may obtain information for providing a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (101) (or the user of the wearable device (101)) based on the execution of the space recognizer (272). The wearable device (101) may reproduce the surrounding environment of the wearable device (101) in three dimensions using data obtained using an external camera (e.g., camera (130)) based on the execution of the space recognizer (272). The wearable device (101) may identify at least one of a plane, a slope, and stairs based on the surrounding environment of the wearable device (101) reproduced in three dimensions based on the execution of the space recognizer (272). The space recognizer (272) may be referred to as a scene understanding (SU) module (or a scene recognition program).

[0063] For example, the wearable device (101) may identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (101) based on the execution of the gesture tracker (273). As an example, the wearable device (101) may identify a pose and / or gesture of a hand of a user using data acquired from an external camera (e.g., camera (130)) based on the execution of the gesture tracker (273). As an example, the wearable device (101) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using an external camera based on the execution of the gesture tracker (273). The gesture tracker (273) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0064] For example, the wearable device (101) may identify (or track) eye movements of a user of the wearable device (101) based on the execution of the gaze tracker (274). As an example, the wearable device (101) may identify eye movements of the user using data acquired from a gaze tracking camera (e.g., camera (130)) based on the execution of the gaze tracker (274). The gaze tracker (274) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0065] For example, the processor (210) of the wearable device (101) may execute a gaze calibrator (275) to calibrate the gaze position. For example, the processor (210) may estimate or compensate for an error included in the gaze position calculated by an image of the user's eye obtained through the camera (130) (e.g., the first camera (130-1) and / or the second camera (130-2) of FIG. 1). To reduce the error, the processor (210) may calibrate the gaze position by at least partially changing information related to the gaze position. The at least partially changed gaze position may be used to execute the gaze tracker (274) of the processor (210). The at least partially changed gaze position may be used to receive user input based on the gaze position. The processor (210) that executes the gaze calibrator (275) can perform calibration of the gaze position using calibration information stored in the memory (215).

[0066] For example, the processor (210) of the wearable device (101) may execute the virtual object controller (276) to change the properties of the virtual object displayed on the display (110) (e.g., the position (e.g., two-dimensional coordinates and / or three-dimensional coordinates of the virtual object), shape, color and / or size) of the virtual object while acquiring calibration information to be stored in the memory (215). The processor (210) executing the virtual object controller (276) may control the display (110) to change the display of the virtual object according to the properties. For example, the processor (210) may change the virtual object included in the screen displayed on the display (110) to display a screen for acquiring calibration information.

[0067] In one embodiment, the virtual object controller (276) may include a virtual object position adjuster (291) and a validity determiner (292). The processor (210) may execute the virtual object controller (276) and the gaze calibrator (275) simultaneously, and control the flow of information between the virtual object controller (276) and the gaze calibrator (275). For example, the result of adjusting the position of the virtual object using the virtual object position adjuster (291) may be processed by the processor (210) executing the gaze calibrator (275) and transmitted to another program and / or hardware (e.g., the display (110)). In one embodiment, the processor (210) may execute the virtual object controller (276) using information (e.g., gaze position) acquired using the gaze calibrator (275).

[0068] In one embodiment, while executing the virtual object controller (276) to change the position of a virtual object within the display (110), the processor (210) may execute the gaze tracker (274) to obtain information indicating a gaze position within the display (110) (or a display area of ​​the display (110). The information may include two-dimensional coordinates and / or three-dimensional coordinates indicating the gaze position. The information may be generated based on a calibration algorithm executed by the gaze calibrator (275).

[0069] In one embodiment, the processor (210) executing the virtual object controller (276) may use information from the gaze calibrator (275) (e.g., the result of calculating the gaze position) to initiate changing the position of the virtual object, or may stop changing the position of the virtual object. For example, the processor (210) may execute the gaze calibrator (275) to determine whether to stop changing properties such as the position, shape, color, and / or size of the virtual object. The processor (210) may execute the gaze calibrator (275) to determine whether to execute a function mapped to the virtual object. The processor (210) executing the gaze calibrator (275) may use the gaze position acquired using the gaze tracker (274) to determine the point in time to stop changing the virtual object. The point in time determined by the gaze calibrator (275) may be used to control the virtual object based on the execution of the virtual object controller (276). Up to this point, the processor (210) that has executed the gaze calibrator (275) can store the gaze position acquired using the gaze tracker (274), the position of the virtual object, and / or the image acquired from the camera (130) as calibration information.

[0070] In one embodiment, calibration information acquired using the gaze calibrator (275) may be used to calibrate the gaze position and / or determine the validity of the gaze position. The processor (210) executing the validity determination unit (292) may determine the validity of the gaze position. Based on the determined validity, the processor (210) may determine whether to stop acquiring the calibration information. After stopping the acquisition of the calibration information, the calibration information stored in the memory (215) may be used to calibrate the gaze position.

[0071] Below, with reference to FIG. 3, an exemplary operation of the processor (210) performed to obtain calibration information is described.

[0072] FIG. 3 illustrates a flowchart of a wearable device according to one embodiment. The wearable device (101) of FIG. 1 and / or FIG. 2 and / or the processor (210) of FIG. 2 may perform at least one of the operations of FIG. 3. The operations of FIG. 3 may be related to operations of the wearable device executing the virtual object controller (276) of FIG. 2. The order in which the operations of FIG. 3 are performed is not limited to the order illustrated in FIG. 3. For example, the processor of the wearable device may perform the operations of FIG. 3 in a different order than the order illustrated in FIG. 3. For example, the processor of the wearable device may perform at least two of the operations of FIG. 3 substantially simultaneously.

[0073] Referring to FIG. 3, in operation (310), a processor of a wearable device according to one embodiment may display a virtual object (e.g., the virtual object (o) of FIG. 1). The processor of the wearable device may display the virtual object on a display of the wearable device (e.g., the display (110) of FIG. 2). The virtual object of operation (310) may be configured to execute a function related to a user input. For example, the virtual object of operation (310) may be displayed to receive a pattern input for unlocking.

[0074] Referring to FIG. 3, within operation (320), a processor of a wearable device according to one embodiment may determine a gaze position using an image of an eye acquired from a camera (e.g., camera (130) of FIG. 2). The processor may determine a user's gaze position using an image of an eye acquired from at least one camera (e.g., first camera (130-1) and / or second camera (130-2) of FIG. 1). The gaze position may be calculated based on the execution of an eye tracker (274) and / or an eye calibrator (275) of FIG. 2.

[0075] Referring to FIG. 3, in operation (330), a processor of a wearable device according to one embodiment may detect a gaze position within a detection area associated with a virtual object. The detection area of ​​operation (330) may be set to initiate control of the virtual object. The detection area of ​​operation (330) may be referred to as a control initiation determination area. The processor may compare the coordinates (e.g., two-dimensional coordinates) of the gaze position with the coordinates of the detection area within the display. Based on the comparison, the processor may determine whether the gaze position is included within the detection area.

[0076] In one embodiment, the detection area of ​​the operation (330) may be formed to include a virtual object within the display (or a display area of ​​the display). For example, the detection area may have a circular shape surrounding the virtual object. The shape of the detection area is not limited to a circle. The detection area may be formed as a condition for determining whether the gaze position is included in the reference position. The detection area may have an irregular shape. For example, information for defining the detection area (e.g., dimensions such as the radius, width, and / or height of the detection area) may be stored in the electronic device or set by the electronic device. For example, the size of the detection area (e.g., the diameter and / or radius of a circle) may be determined using calibration information stored in the wearable device. For example, by utilizing the error in the gaze position indicated by the calibration information, the processor can determine the size of the detection area such that when the user gazes at the virtual object, the probability that the gaze position measured by the wearable device is included in the detection area exceeds a specified probability (e.g., 96%). For example, the size of the detection area may be related to the accuracy of estimating the user's intention of gazing at the virtual object.

[0077] Referring to FIG. 3, within operation (340), a processor of a wearable device according to one embodiment may move a virtual object based on a displacement of a gaze position within a detection area (e.g., a vector representing a change in the position of the gaze position in the time domain). For example, the processor may perform operation (340) based on whether the gaze position is included within the detection area. Referring to FIG. 3, when the processor performs operation (340) to move a virtual object, the processor may repeatedly perform operations (344, 346) of FIG. 3.

[0078] Referring to FIG. 3, in operation (342), a processor of a wearable device according to one embodiment may, in response to detecting a gaze position that has entered a detection area, move a virtual object using the gaze position and the position of the virtual object. After detecting the gaze position within the detection area, the processor may perform the movement of operation (342) if the gaze position is located within the detection area for a specified time (e.g., 0.5 seconds). When the processor starts the movement of operation (342), the processor may calculate a timing to stop the movement of operation (342). The movement of operation (342) may be performed only before the timing. The movement of operation (342) may be stopped if the gaze position moves out of the detection area before the completion of the timing.

[0079] Referring to FIG. 3, in operation (344), according to one embodiment, a processor of a wearable device may move a virtual object using a displacement of a gaze position according to a movement of the gaze position. For example, a processor that detects a gaze position within a detection area including a virtual object may determine information indicating a displacement of the gaze position within the detection area. The processor may move the virtual object according to a direction of the displacement. A distance of the virtual object moved according to the direction may be less than a magnitude of the displacement. For example, the processor may move the virtual object by a magnitude of the displacement to which a specified ratio is applied. The specified ratio may be set such that the virtual object is moved less than a magnitude of the displacement (e.g., a distance by which the gaze position is moved).

[0080] In one embodiment, a processor that moves a virtual object by performing at least one of the operations (342, 344) may store the results of tracking the gaze position. For example, the processor may store gaze positions detected at different points in time. For example, the processor may store the gaze position at a first point in time and the gaze position at a second point in time, and may determine or calculate a displacement of the gaze position using the gaze positions at the first and second points in time.

[0081] Referring to FIG. 3, in operation (346), according to one embodiment, a processor of a wearable device may determine whether to stop the movement of a virtual object by using a condition for stopping the movement of the virtual object. The condition may be related to the timing determined when moving the virtual object based on operation (342). For example, in response to the expiration of the timing, the processor may stop the movement of the virtual object. The condition may be related to whether the distance by which the virtual object is moved based on operation (344) is less than a threshold distance. The condition may be related to whether the distance calculated based on operation (344) is less than the threshold distance. For example, if the distance by which the virtual object is moved is less than the threshold distance, the processor may stop the movement of the virtual object. The condition may be related to whether the gaze position is located within the detection area of ​​operation (330). For example, if the gaze position moves outside the detection area, the processor may stop the movement of the virtual object.

[0082] In one embodiment, within operation (346), if it is determined that the movement of the virtual object is not to be stopped, the processor may perform operation (344) of FIG. 3 again. For example, if the condition for stopping the movement of the virtual object is not satisfied, the processor may perform operation (344) of FIG. 3 again. For example, if the timing has not expired or the distance the virtual object has moved is greater than or equal to a threshold distance, the processor may perform operation (344) of FIG. 3 again. The processor performing operation (344) of FIG. 3 again may move the virtual object by using the displacement between the gaze position before performing operation (346) and the gaze position obtained by performing operation (344) again.

[0083] Referring to FIG. 3, within operation (350), according to one embodiment, a processor of a wearable device may store information acquired while moving a virtual object as information for calibrating a gaze position. The processor may store at least one of a displacement of a position of a virtual object or a gaze position acquired while the virtual object is being moved, based on stopping the movement of the virtual object.

[0084] In one embodiment, based on operation (346), the processor that has stopped moving the virtual object may execute a function associated with the virtual object. For example, the processor may determine that a user input associated with the virtual object has been received. For example, the processor may execute a function associated with the virtual object associated with the user input. For example, the user input may include an input indicating selection of the virtual object.

[0085] In one embodiment, after storing information of operation (350) or after executing the function, the processor may perform operation (350) based on detecting a gaze position adjacent to a location where a virtual object is displayed, and may perform calibration related to the gaze position using the stored information (e.g., at least one of the position or displacement of the virtual object).

[0086] As described above, according to one embodiment, the processor of the wearable device can display a virtual object that moves according to the displacement of the gaze position calculated at one or more points in time. For example, the processor can move the virtual object in proportion to the displacement. The processor can track the gaze following the moving virtual object to obtain calibration information. The calibration information obtained by the processor can be used to improve the accuracy of the gaze position. The processor can continuously check the gaze following the virtual object while the virtual object is moving. For example, the processor can obtain calibration information based on the moving virtual object more accurately.

[0087] Below, with reference to FIG. 4, a detection area formed to include a virtual object is exemplarily described.

[0088] FIG. 4 illustrates a screen (420) displayed by a wearable device (101) according to one embodiment. The wearable device (101) of FIG. 1 and / or FIG. 2 and / or the processor (210) of FIG. 2 may perform at least one of the operations of FIG. 4. The operations of FIG. 4 may be related to the operations of FIG. 3.

[0089] Referring to FIG. 4, an exemplary state of a wearable device (101) worn by a user (410) is illustrated. According to one embodiment, the wearable device (101) may display an AR or VR-based screen (420) on a display (e.g., the display (110) of FIG. 2). When displaying an AR-based screen (420) according to one embodiment, when the wearable device (101) acquires images and / or videos of an external environment through an external camera (e.g., the camera (130) of FIG. 2), the screen (420) displayed by the wearable device (101) may include an image (432) corresponding to an external object (430) included in the external environment.

[0090] According to one embodiment, the states of the wearable device (101) may include a locked state and an unlocked state. The locked state may include a state requiring authentication of a user (410) wearing the wearable device (101) in order to switch to an unlocked state, as shown in the exemplary screen (420) of FIG. 4. Within the locked state, access to information stored in the wearable device (101) and / or execution of software applications of the wearable device (101) may be restricted or blocked. A screen displayed in the locked state (e.g., screen (420) of FIG. 4) may be referred to as a lock screen. A screen displayed in the locked state may be referred to as an unlock screen from the perspective of being displayed for unlocking.

[0091] According to one embodiment, within a state in which a lock screen is displayed, the wearable device (101) may perform an operation for authentication. For example, within a locked state of the wearable device (101), the wearable device (101) may display one or more virtual objects configured to receive a pattern for switching (or changing) the locked state. Referring to an exemplary screen (420) of FIG. 4, the wearable device (101) may display virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) selectable by the user. Although virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) having the shape of circles and / or dots are illustrated, the embodiment is not limited thereto. The wearable device (101) may display text (e.g., “Enter pattern to unlock”) guiding the input of information for authentication within the screen (420).

[0092] According to one embodiment, the wearable device (101) may receive an input for connecting at least two virtual objects among virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449). Using a pattern in which at least two virtual objects among the virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) are connected, indicated by the input, the wearable device (101) may authenticate a user. For example, the wearable device (101) may authenticate a user of the wearable device (101) by comparing the order in which the at least two virtual objects are connected and / or the shapes of the virtual objects connected to each other, indicated by the input, with a pre-registered order and / or shape. Based on the above authentication, the wearable device (101) can switch from a locked state to an unlocked state.

[0093] According to one embodiment, the operation performed by the wearable device (101) to authenticate a user may not be limited to pattern input based on virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) included in the screen (420). The wearable device (101) may authenticate a user (410) wearing the wearable device (101) using biometric information such as a fingerprint, face, and / or iris.

[0094] According to one embodiment, the wearable device (101) may obtain calibration information by using virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) for unlocking the wearable device (101), as shown in the screen (420) of FIG. 4. For example, the wearable device (101) may receive an input for connecting at least two virtual objects among the virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) based on movement of the gaze position. Along with the input, the wearable device (101) may store information related to the gaze position tracked for the input as calibration information.

[0095] Referring to FIG. 4, by using the detection areas formed in each of the virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449), the wearable device (101) can detect or determine a gaze position directed toward any one of the virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449). Referring to FIG. 4, a detection area (451) corresponding to the virtual object (441) is exemplarily illustrated. The detection area (451) and the virtual object (441) may be concentric circles. The position of the detection area (451) (or the position of the center of the detection area (451)) may be fixed to the position of the center of the virtual object (441). For example, the detection area (451) may not be visible through the screen (420). Detection areas similar to the detection area (451) may be formed in portions of the screen (420) where each of the other virtual objects (442, 443, 444, 445, 446, 447, 448, 449) different from the virtual object (441) are displayed.

[0096] In one embodiment, the size (e.g., radius, diameter, and / or area) of the detection area (451) having a shape surrounding the virtual object (441) may be determined using calibration information (e.g., calibration information acquired by the wearable device (101). For example, the wearable device (101) may determine the size of the detection area (451) using information indicating an error in the gaze position at the location of the screen (420) on which the virtual object (441) is displayed. The information indicating the error in the gaze position may be included in the calibration information. For example, the size of the detection area (451) may be set such that, when the user gazes at the virtual object (441), the probability that the gaze position is included in the detection area (451) exceeds a specified probability, as described above with reference to operation (330) of FIG. 3 .

[0097] In one embodiment, within a state in which a plurality of virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) are displayed, the wearable device (101) may track or identify a gaze position. If the gaze position is away from all of the detection areas (e.g., detection area (451)) corresponding to each of the plurality of virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449), the wearable device may not respond to the gaze position or may not perform any function.

[0098] Within an exemplary state displaying the screen (420) of FIG. 4, the wearable device (101) can move the virtual object (441) based on detecting the gaze position included in the detection area (451). The wearable device (101) can determine or change the position of the virtual object (441) within the movable area (461) by using the distance by which the gaze position has moved within the detection area (451) and / or the direction in which the gaze position has moved within the detection area (451). For example, the wearable device (101) can translate or shift the virtual object within the movable area (461).

[0099] Referring to FIG. 4, a movable area (461) is exemplarily illustrated, which has a different size from the detection area (451) and a center that is at least partially coincident with the center of the detection area (451) and / or the virtual object (441). For example, although the movable area (461) is illustrated as having a circular shape concentric with the virtual object (441) and / or the detection area (451), the shape, position and / or size of the movable area (461) is not limited to the embodiment of FIG. 4. For example, the movable area (461) may coincide with the detection area (451). Moving the virtual object (441) based on a gaze position included in the detection area (451) may be interrupted or terminated when the virtual object (441) contacts a boundary and / or edge of the movable area (461). The shape of the virtual object (441), detection area (451) and / or movable area (461) is not limited to the shapes of concentric circles illustrated in FIG. 4.

[0100] As described above, according to one embodiment, the wearable device (101) can simultaneously receive input for unlocking and calibration information using virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) included in the lock screen. For example, the wearable device (101) can receive an input for unlocking based on a movement of a gaze position. While receiving the input, the wearable device (101) can move any virtual object corresponding to the input, thereby causing a motion of the user's (410) eyes following the virtual object (or a virtual object that appears to be focused by the gaze position). The wearable device (101) can store an image and / or video of the eyes while the motion occurs as calibration information for the gaze position.

[0101] In one embodiment, by combining a function for collecting calibration information with virtual objects displayed for essential functions, such as virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) on the lock screen, the wearable device (101) can acquire calibration information more naturally. For example, the wearable device (101) can acquire the calibration information without displaying a UI dedicated to providing the calibration information.

[0102] Hereinafter, with reference to FIGS. 5a to 5d, FIG. 6a and / or FIG. 6b, an exemplary operation of a wearable device (101) for moving a virtual object using a gaze position is described.

[0103] Figures 5a, 5b, 5c, and 5d illustrate states of a wearable device that detects a gaze position. The wearable device (101) of Figures 1 and / or 2 and / or the processor (210) of Figure 2 may perform operations of the wearable device described with reference to Figures 5a to 5d. The operations of Figures 5a to 5d may be related to at least one of the operations of Figure 3 (e.g., operation (340) and / or operations (342, 344, 346)).

[0104] Referring to FIGS. 5A to 5D , exemplary states (501, 502, 503, 504) of a wearable device displaying a virtual object (510) are illustrated. In the state (501) of displaying a virtual object (510) on the display (110) of FIG. 2 , the wearable device can identify or confirm a gaze position using a camera (e.g., the camera (130) of FIG. 2 ). Identifying the gaze position can be performed based on the execution of the gaze calibrator (275) of FIG. 2 . The virtual object (510) of FIG. 5A is a virtual object displayed on the display to receive user input, and may include virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) of FIG. 4 .

[0105] According to one embodiment, a wearable device that detects a gaze position (531) within a state in which a virtual object (510) is displayed may determine whether to move the virtual object (510) by using the positional relationship between the gaze position (531) and the detection area (520). Determining whether to move the virtual object (510) may be performed based on the execution of the virtual object controller (276) of FIG. 2. In one embodiment, the detection area (520) may not be displayed within the display. In one embodiment, the detection area (520) may have a specified transparency (e.g., a transparency greater than 100% or 0%), a specified alpha value, and / or a specified color.

[0106] In the state (501) of FIG. 5A, the wearable device (101) that detects a gaze position (531) located within a detection area (520) formed around a virtual object (510) can determine whether the gaze position (531) is located within the detection area (520) for a specified period of time (e.g., about 0.5 seconds). Even when the user gazes at the virtual object (510), the gaze position (531) calculated by the wearable device may deviate from the virtual object (510) due to an error associated with the gaze position (531). For example, since the gaze position (531) may be incorrectly determined to be located within the detection area (520) due to eye blinking and / or rapid eye movement, the wearable device (101) can measure the period of time during which the gaze position (531) is maintained within the detection area (520). If the measured period exceeds the specified period, the wearable device may decide to move the virtual object (510) using the gaze position (531).

[0107] For example, if it is decided to move the virtual object (510) using the gaze position (531), the wearable device can calculate or identify the position difference between the gaze position (531) and the virtual object (510). In the exemplary state (501) of FIG. 5A, the wearable device can calculate the position difference (x1) between the gaze position (531) and the virtual object (510) on the x-axis and the position difference (y1) between the gaze position (531) and the virtual object (510) on the y-axis. Using the position differences (x1, y1) on the x-axis and the y-axis, respectively, the wearable device can move the virtual object (510). Independently of the movement of the virtual object (510), the position, shape and / or size of the detection area (520) within the display can be maintained or fixed. Moving the virtual object (510) can be performed based on the execution of the virtual object position controller (291) of FIG. 2. The wearable device can determine the distance and / or direction to move the virtual object (510) using the gaze position (531) calculated using the gaze calibrator (275) of FIG. 2.

[0108] Referring to FIG. 5B, a state (502) is illustrated after the wearable device moves the virtual object (510) using the position differences (x1, y1) in the x-axis and y-axis, respectively. The point (v1) in FIG. 5B may represent the position of the virtual object (510) within the state (501) prior to the state (502). For example, while switching from the state (501) to the state (502), the wearable device may move the virtual object (510) along the y-axis by y1 / a and may move the virtual object (510) along the x-axis by x1 / b. While switching from the state (501) to the state (502), the direction in which the virtual object (510) moves may correspond to the direction from the gaze position (531) within the state (501) toward the virtual object (510). In the above example, a and b may be integers (or real numbers) greater than 1. In the above example, a and b may be equal to each other. Since a and b exceed 1, the distance that the virtual object (510) moves may be less than the distance between the gaze position (531) and the virtual object (510). a and / or b may be determined based on the reliability and / or accuracy associated with the calculation of the gaze position (531) indicated by the calibration information stored in the wearable device.

[0109] In one embodiment, since the virtual object (510) is moved while switching from state (501) to state (502), the gaze of the user gazing at the virtual object (510) may also be moved. Within the exemplary state (503) of FIG. 5C , the wearable device according to one embodiment may calculate or determine the gaze position (532) of the user that has moved to gaze at the virtual object (510). The wearable device may determine whether to move the virtual object (510) by using the position difference and / or displacement between the gaze position (531) calculated prior to the state (503) (e.g., states (501, 502)) and the gaze position (532) within the state (503). The determination of whether to move the virtual object (510) may be performed based on the execution of the validity determiner (292) of FIG. 2 .

[0110] Referring to FIG. 5C, the gaze position (532) has been moved along the x-axis by x2 with respect to the gaze position (531) before the state (503). The gaze position (532) has been moved along the y-axis by y2 with respect to the gaze position (531) before the state (503). In one embodiment, the wearable device can calculate the position difference (x2) of the gaze positions (531, 532) on the x-axis and the position difference (y2) of the gaze positions (531, 532) on the y-axis. After the movement of the virtual object (510) begins, the wearable device can change the position, shape, color and / or size of the virtual object (510) by using the displacement (e.g., movement direction and / or movement distance) of the gaze position (532) in the time domain. For example, within a state (503) in which a gaze position (532) is detected that is moved by x2 on the x-axis and by y2 on the y-axis, the wearable device can move a virtual object (510) using a specified ratio (e.g., 1 / a) on the x-axis and a specified ratio (e.g., 1 / b) on the y-axis.

[0111] Referring to FIG. 5d, a state (504) is illustrated in which the wearable device moves the virtual object (510) using the displacement (e.g., (x2, y2)) of the gaze position (532) detected in the state (503). While switching from the state (503) to the state (504), the wearable device can move the virtual object (510) along the y-axis by y2 / c and can move the virtual object (510) along the x-axis by x2 / d. For example, a, b, c, and d of FIGS. 5b and 5d can be integers (or real numbers) greater than 1. For example, a, b, c, and d of FIGS. 5b and 5d can be equal to each other. For example, a and c of FIGS. 5a and 5d can be equal to each other. For example, b and d of FIGS. 5a and 5d can be equal to each other. For example, a, b, c, and d in FIGS. 5A and 5D may be predetermined values. For example, a, b, c, and d in FIGS. 5A and 5D may be determined based on the reliability of the gaze position determined by a gaze calibrator (e.g., the gaze calibrator (275) of FIG. 2). During the transition from state (503) to state (504), the direction in which the virtual object (510) moves may correspond to the direction of the displacement of the gaze position (532) detected within the state (503). As described above with reference to FIG. 5B, since a and b are integers (or real numbers) exceeding 1, the distance in which the virtual object (510) moves may be less than the magnitude of the displacement of the gaze position (532).

[0112] As described above with reference to FIGS. 5A to 5D , the wearable device can move the virtual object (510) by using the displacement of the gaze position (532) and / or the position of the virtual object (510). When the user gazes at the virtual object (510), since the distance by which the virtual object (510) moves is less than the magnitude of the displacement of the gaze position (532), the movement distance of the moved virtual object (510) based on the distance by which the gaze position (532) moves can be gradually reduced. For example, the position of the virtual object (510) can converge.

[0113] In one embodiment, the wearable device may stop moving the virtual object (510) if the distance by which the virtual object (510) has moved is reduced below a specified threshold distance. In one embodiment, the wearable device may stop moving the virtual object (510) based on a displacement of the gaze position (532) if the virtual object (510) contacts a boundary line of a movable area corresponding to the virtual object (510) (e.g., the movable area (461) of FIG. 4) and / or is moved out of the movable area. In one embodiment, the wearable device may move the virtual object (510) for a specified period of time calculated within a state (501) in which the virtual object (510) is initially detected within a detection area (520), and in response to expiration of the specified period of time, stop moving the virtual object (510). In one embodiment, the wearable device may stop moving the virtual object (510) based on whether the calculated distance for moving the virtual object (510) is less than a threshold distance. For example, if the calculated distance is less than the threshold distance, the processor may stop moving the virtual object (510).

[0114] In one embodiment, a wearable device that decides to stop moving a virtual object (510) may move the virtual object (510) away from the center of the detection area (520) back toward the center. For example, based on stopping moving the virtual object (510), the wearable device may display the virtual object (510) at the location where it was displayed before detecting the gaze position (531) within the detection area (520).

[0115] In one embodiment, if the distance that the virtual object (510) has moved exceeds a specified threshold distance, if the virtual object (510) is positioned within a movable area, or if the specified period of time has expired, the wearable device may continue to move the virtual object (510) within the detection area (520). For example, based on the update of the gaze position after state (504), the wearable device may move the virtual object (510) using the updated gaze position and the displacement of the gaze position (532). For example, by applying the specified ratios (e.g., 1 / b and 1 / a) described above to the x-axis size and the y-axis size of the displacement, respectively, the wearable device may calculate the movement distance of the virtual object (510) in each of the x-axis and the y-axis. The wearable device may move the virtual object (510) according to the calculated movement distance.

[0116] In the state (504) of FIG. 5d, when the movement of the virtual object (510) is stopped, the wearable device can store the position of the virtual object (510) in the state (504) (e.g., the position of the virtual object (510) at the time when the movement of the virtual object (510) is stopped). The position of the virtual object (510) in the detection area (520) in the state (504) and / or the position difference (or displacement) of the virtual object (510) with respect to the center of the detection area (520) can be stored as calibration information.

[0117] Referring to FIGS. 5A to 5D , the positions of the virtual object (510) within states (501, 502, 503, 504) in which gaze positions (e.g., gaze positions (531, 532)) are detected within the detection area (520) can be stored as calibration information within the memory of the wearable device (e.g., the memory (215) of FIG. 2 ). In one embodiment, the result of moving the virtual object (510) using the positional relationship between the virtual object (510) and the gaze position (531) (e.g., the result of moving the virtual object (510) within the state (502) of FIG. 5B ) can be compared with another result of moving the virtual object (510) according to the displacement of the gaze position (e.g., the result of moving the virtual object (510) within the state (503) of FIG. 5C ). Based on the above comparison, the wearable device can determine whether to store (or use) the result of moving the virtual object (510) using the positional relationship between the virtual object (510) and the gaze position (531). For example, if the result is similar to the other result, the wearable device can store the result as calibration information.

[0118] As described above, the wearable device can start moving the virtual object (510) within a state (501) in which the gaze position (531) is detected and maintained within the detection area (520). When starting to move the virtual object (510), the wearable device can move the virtual object (510) using the positional relationship between the virtual object (510) and the gaze position (531). After moving the virtual object (510) using the positional relationship, the wearable device can move the virtual object (510) using the displacement of the gaze position (e.g., the displacement of the gaze positions (531, 532) in each of the states (503, 502). For example, the virtual object (510) can be moved based on the difference between the current gaze position and another gaze position that was detected prior to the current gaze position. To prevent the virtual object (510) from being emitted, the wearable device can limit the movement distance of the virtual object (510) to less than the size of the displacement of the gaze position.

[0119] In one embodiment, the wearable device may check a condition for stopping the movement of the virtual object (510) while moving the virtual object (510). Stopping the movement of the virtual object (510) may indicate that acquisition of calibration information based on the movement of the virtual object (510) is stopped. Stopping the movement of the virtual object (510) may indicate that a function related to (or mapped to) the virtual object (510) is executed. Hereinafter, with reference to FIG. 6A and / or FIG. 6B, an exemplary operation of a wearable device that has started moving a virtual object (510) and stops moving the virtual object (510) is described.

[0120] FIGS. 6A and 6B illustrate operations of a wearable device that executes a function related to a virtual object (441) using a gaze position. The wearable device (101) of FIGS. 1 and / or 2 and / or the processor (210) of FIG. 2 may perform operations of the wearable device described with reference to FIGS. 6A and / or 6B. The operations of FIGS. 6A and 6B may be related to at least one of the operations of FIG. 3 (e.g., operation (346)).

[0121] Referring to FIG. 6a and / or FIG. 6b, different states (601, 602) of a lock screen including virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) are illustrated. The wearable device can display virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) together with virtual objects (611, 612, 613, 614, 615, 616, 617, 618, 619) having a shape that surrounds each of the virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449). The virtual objects (611, 612, 613, 614, 615, 616, 617, 618, 619) may have a radius of r1 and may be aligned to the center points of the virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449), respectively.

[0122] According to one embodiment, the wearable device (101) may gradually change an attribute (e.g., at least one of size, shape, and / or color) of one of the virtual objects (611, 612, 613, 614, 615, 616, 617, 618, 619) associated with a specific virtual object based on detecting a gaze position associated with the specific virtual object among the virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449). The wearable device may gradually change the attribute to provide visual feedback regarding the duration of gazing at the specific virtual object.

[0123] In the exemplary state (601) of FIG. 6A, a wearable device that detects a gaze position included in a detection area corresponding to a virtual object (441) may reduce the size of a virtual object (611) corresponding to the virtual object (441). For example, the wearable device may gradually reduce the radius of the virtual object (611) from r1 to a radius less than r1 (e.g., r2). For example, the radius of the virtual object (611) may be gradually reduced from r1 to 0 (or the radius of the virtual object (441)) to indicate the expiration of the timing described with reference to operation (346) of FIG. 3. For example, the end point of the time interval in which the radius of the virtual object (611) is reduced may be the same as the timing described with reference to operation (346) of FIG. 3. At the above termination point, since the radius of the virtual object (611) is 0 or less than or equal to the radius of the virtual object (441), the virtual object (611) may disappear from the screen or may be hidden.

[0124] In one embodiment, a wearable device that detects a gaze position included in a detection area corresponding to a virtual object (441) may gradually change the color of the virtual object (441). For example, in a state (601) in which virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) of a first designated color (e.g., black) are displayed, the wearable device that detects a gaze position included in a detection area corresponding to the virtual object (441) may gradually change the color of the virtual object (441) to a second designated color (e.g., a primary color such as red) different from the first designated color. The end point of the time period in which the color of the virtual object (441) changes from the first designated color to the second designated color may be the same as the timing described with reference to operation (346) of FIG. 3. Based on the expiration of the above timing, the wearable device may display a virtual object (441) of a second designated color. The embodiment is not limited thereto, and the wearable device may play a visual effect (e.g., an animation) indicating that the movement of the virtual object (441) has stopped based on the expiration of the designated time interval during which the virtual object (441) moves.

[0125] Referring to FIG. 6B, a state (602) is illustrated after movement of the virtual object (441) is stopped. When movement of the virtual object (441) is stopped, the wearable device may determine that an input for selecting the virtual object (441) has been received. Based on the determination, the wearable device may set the size, shape, position, and / or color of the virtual object (441) to a designated size, a designated shape, a designated position, and / or a designated color indicating that the input has been received.

[0126] Within state (602), the wearable device may display a virtual object in the form of a line extending from the virtual object (441) to the gaze position (621) detected within state (602). Within state (602), based on detecting the gaze position (621) included in the detection area of ​​the virtual object (445), the wearable device may start moving the virtual object (445) as described above with reference to FIG. 3, FIGS. 5A to 5D, and / or FIG. 6A.

[0127] As described above, when a user moves his / her gaze along the virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) of the lock screen, some of the virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) may be sequentially selected. The wearable device may display a virtual object (or visual object) indicating the selection of the some, such as a line extending from the virtual object (441) of FIG. 6B to the gaze position (621). If the pattern indicated by the line matches a pattern pre-registered in the wearable device, the wearable device may complete authentication based on the pattern. Based on the completion of the authentication, the wearable device may switch from a locked state to a different state (e.g., an unlocked state).

[0128] In one embodiment, the timing for controlling the display (e.g., the frame rate based on units of frames per second (fps)) and the timing for calculating the gaze position (e.g., the period for analyzing images acquired from a camera) may be different from each other or may be changed independently. Below, with reference to FIG. 7, exemplary operations of a wearable device related to the difference between the timing for updating the gaze position and the timing for controlling the display are described.

[0129] Fig. 7 illustrates an operation of a wearable device that moves a virtual object (710) according to a gaze position. The wearable device (101) of Fig. 1 and / or Fig. 2 and / or the processor (210) of Fig. 2 may perform the operation of the wearable device described with reference to Fig. 7. The operation of the wearable device described with reference to Fig. 7 may be related to at least one of the operations of Fig. 3 (e.g., operations (340, 342, 344)).

[0130] In one embodiment of FIG. 7, the wearable device may update or calculate the gaze position based on the period of Ts (e.g., a period of 1 / 10 seconds). While an embodiment of periodically tracking the gaze position is described, the embodiment is not limited thereto. In one embodiment, the wearable device may control the display (e.g., the display (110) of FIG. 2) using a frame rate less than the period of Ts (e.g., 60 Hz and / or 120 Hz). For example, the wearable device may be configured to update the screen displayed on the display based on the frame rate.

[0131] Referring to FIG. 7, an exemplary state of a wearable device displaying a virtual object (710) is illustrated. By comparing a detection area (720) including the virtual object (710) and a gaze position, the wearable device can determine whether to move the virtual object (710). The wearable device that has decided to move the virtual object (710) can determine a speed and / or velocity of moving the virtual object (710) using the period of Ts in which the gaze position is updated in order to continuously move the virtual object (710). For example, the wearable device can move the virtual object (710) such that the virtual object (710) moves a distance less than the magnitude of the displacement during the time interval (in one embodiment of FIG. 7, the period of Ts) in which the gaze position is updated.

[0132] Referring to the exemplary case of FIG. 7, the wearable device that detects the gaze position (g0) at time t0 can detect the gaze position (g1) at time t1 after a period of Ts after time t0. As described above with reference to FIGS. 5A to 5D , the wearable device can calculate or determine a position (o2) to which the virtual object (710) will move by using the displacement of the gaze positions (g0, g1). The position (o2) may be a position that is moved by an amount of the displacement to which a specified ratio is applied, depending on the direction of the displacement, from the virtual object (710) displayed at time t1. If the virtual object (710) is immediately moved to the position (o2) at time t1 when the position (o2) is calculated, the virtual object (710) may be stationary from time t1 to time t2 after a period of Ts. That is, the virtual object (710) can be discretely moved along the cycle of Ts. When gazing at a discretely moved virtual object (710), rapid eye movements following the virtual object (710) can cause user fatigue.

[0133] According to one embodiment, a wearable device can reduce user fatigue by smoothly moving a virtual object (710). For example, a wearable device that has decided to move a virtual object (710) to a location (o2) can determine the speed and / or location of the virtual object (710) to be moved to the location (o2) according to a frame rate. For example, if the current coordinates of the virtual object (710) at time t1 are (r1, s1) and the coordinates of the location (o2) are (r2, s2), the coordinates (r, s) of the virtual object (710) at time t between t1 and t2 can be calculated or determined as in mathematical expression 1.

[0134]

[0135] For example, if the current coordinates (or current position) of the virtual object (710) are (r1, s1) and the position to which the virtual object (710) is to be moved is determined as (r2, s2), the wearable device can determine the position of the virtual object (710) for each frame of the display according to the frequency of the display. In the above example, if the frequency of the display is 60 Hz, t2 may be t1 + 1 / 60. In the above example, in order to smoothly express the movement of the virtual object (710), t2 may be defined as t1 = t1 + ta, and ta may be set to a value greater than 1 / 60. In the above example, if ta = 1 / 10, the coordinates (r, s) (or position) of the virtual object (710) at time point t within t1 ≤ t ≤ t2 may be determined based on the above mathematical expression 1.

[0136] In one embodiment, a wearable device that sequentially displays frames distinguished by a frame rate, during a time interval between t1 and t2, the coordinates (r) of a virtual object (710) in the previous frame t-1 , s t-1 ), the current coordinates (r) of the virtual object (710) t , s t ) can be calculated or determined as in mathematical expression 2.

[0137]

[0138] According to one embodiment, in Equation 2 It can be a real number greater than 0 and less than 1. The wearable device that determines the position of the virtual object (710) using Equation 1 and / or Equation 2 can gradually move the virtual object (710) during a period from time t1 to Ts. Referring to FIG. 7, at time ta after time t1, the wearable device can move the virtual object (710) from the position (o1) of time t2 to the position (oa). At time tb after time ta, the wearable device can move the virtual object (710) from the position (oa) of time ta to the position (ob). The positions (oa, ob) of the virtual object (710) at time points ta and tb can be located on a line connecting the position (o1) of the virtual object (710) at time t1 and the calculated position (o2) of the virtual object (710). At time t2, which is a period of Ts after time t1, the wearable device can display a virtual object (710) at a location (o2). Using the updated gaze location at time t2, the wearable device can recalculate the location of the virtual object (710). Towards the newly calculated location, the wearable device can gradually move the virtual object (710) according to the frame rate.

[0139] According to one embodiment, in one embodiment of gradually moving the virtual object (710), the first position of the virtual object (710) at the first timing when an image is acquired using a gaze tracking camera (e.g., camera (130) of FIG. 2) and the second position of the virtual object (710) at the second timing when the gaze position is acquired using the image may be different. In this case, since the gaze position calculated at the second timing corresponds to the gaze position of the user gazing at the virtual object (710) displayed at the first position, the wearable device may store a pair of the gaze position calculated at the second timing and the first position of the virtual object (710) as calibration information.

[0140] In one embodiment, in an embodiment in which the virtual object (710) is gradually moved, since the gaze gazing at the virtual object (710) gradually moves along the virtual object (710), the wearable device can eliminate or reduce noise in the gaze position calculated from the gaze tracking camera. For example, while gradually moving the virtual object (710), the frequency of acquiring images from the gaze tracking camera to obtain the gaze position can be reduced by increasing the period for updating the gaze position (e.g., the period of Ts). For example, the wearable device can increase the period and, among the images received within the period, select an image containing relatively little noise to calculate the gaze position. For example, the wearable device can exclude or remove an image containing relatively large noise from among the images received within the increased period.

[0141] FIG. 8 illustrates an operation of a wearable device that displays a virtual object (812) related to a gaze position on a lock screen (810). The wearable device (101) of FIG. 1 and / or FIG. 2 and / or the processor (210) of FIG. 2 may perform the operation of the wearable device described with reference to FIG. 8. The operation of the wearable device described with reference to FIG. 8 may be related to at least one of the operations of FIG. 3.

[0142] Referring to FIG. 8, an exemplary state of a lock screen (810) displayed on a display (e.g., display (110) of FIG. 2) is illustrated. Referring to FIG. 8, among virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) for receiving a pattern, virtual objects (441, 445, 443, 446, 449) may be sequentially selected based on the operations described with reference to FIG. 3, FIG. 5A to FIG. 5D, FIG. 6A and / or FIG. 6B. For example, by using gaze positions continuously measured within the screen (810), the wearable device may detect or receive an input for sequentially selecting virtual objects (441, 445, 443, 446, 449).

[0143] In one embodiment, the wearable device may further display a virtual object (812) for obtaining calibration information related to a specific location of the display. For example, it is assumed that a pattern of sequentially connecting virtual objects (441, 445, 443, 446, 449) illustrated in FIG. 8 matches a pre-registered pattern that authenticates a user of the wearable device. Under the assumption, the wearable device that identifies a pattern that matches the pre-registered pattern may determine whether to add a virtual object (812) for obtaining calibration information using calibration information stored (or accumulated) in the wearable device. For example, if an error in the gaze position indicated by the calibration information is relatively large in a specific portion of the display, the wearable device may display a virtual object (812) in the specific portion.

[0144] Referring to FIG. 8, an exemplary state in which a virtual object (812) is displayed in a portion having a relatively large error is illustrated. The wearable device may display the virtual object (812) in a portion where the error indicated by the calibration information exceeds a specified threshold, or in a portion having the largest error. If the error indicated by the calibration information is below the threshold at the location of the display, the wearable device may not display the virtual object (812).

[0145] Referring to FIG. 8, while the virtual object (812) is further displayed, the wearable device can switch from a locked state to an unlocked state based on a user input related to the virtual object (812). The user input can be identified based on a gaze position included in a detection area related to the virtual object (812). For example, the wearable device that detects the gaze position included in the detection area can perform the operations described above with reference to FIGS. 5A to 5D and / or FIGS. 6A to 6B. For example, the wearable device can move the virtual object (812) based on the displacement of the gaze position, and can acquire or store calibration information while the virtual object (812) is moving.

[0146] As described above, according to one embodiment, the wearable device may add a virtual object (812) different from the virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) for receiving a pattern to the lock screen (810) in order to obtain calibration information. Using the added virtual object (812), the wearable device may obtain calibration information for a specific portion of the display where the virtual object (812) is displayed, along with authentication.

[0147] Below, with reference to FIGS. 9A to 9D, the operation of a wearable device in one embodiment where detection areas of virtual objects overlap each other is described.

[0148] FIGS. 9A, 9B, 9C, and 9D illustrate the operation of a wearable device that detects a gaze position within an overlapping area (920) of detection areas (911, 912, 913, 914, 915, 916, 917, 918, 919). The wearable device (101) of FIGS. 1 and / or 2 and / or the processor (210) of FIG. 2 may perform the operation of the wearable device described with reference to FIGS. 9A to 9D. The operation of the wearable device described with reference to FIGS. 9A to 9D may be related to at least one of the operations of FIG. 3.

[0149] Referring to FIGS. 9A to 9D , exemplary states (901, 902, 903, 904) of a wearable device displaying exemplary virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) are illustrated. The virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) may be virtual objects arranged to receive a pattern within a lock screen. The wearable device may detect or identify a gaze position within the lock screen using a camera (e.g., camera (130) of FIG. 2 ).

[0150] Referring to FIG. 9A, detection areas (911, 912, 913, 914, 915, 916, 917, 918, 919) aligned with the center points of virtual objects (441, 442, 443, 444, 445, 446, 447, 448, 449) are illustrated. The detection areas (911, 912, 913, 914, 915, 916, 917, 918, 919) are illustrated for convenience of explanation and may not be recognized by a user wearing a wearable device. Since each of the detection areas (911, 912, 913, 914, 915, 916, 917, 918, 919) is set to distinguish the intention of the user gazing at the corresponding virtual object, the size of each of the detection areas (911, 912, 913, 914, 915, 916, 917, 918, 919) may increase according to the error of the gaze position. Referring to Fig. 9a, the detection areas (911, 912, 913, 914, 915, 916, 917, 918, 919) may at least partially overlap.

[0151] In one embodiment, when the wearable device detects a gaze position located in an overlapping area (e.g., overlapping area (920)) of detection areas (911, 912, 913, 914, 915, 916, 917, 918, 919), the wearable device may move all of a plurality of virtual objects associated with the overlapping area. By moving the plurality of virtual objects in opposite directions, the wearable device may trigger a movement of the gaze position along one of the plurality of virtual objects. By comparing the direction in which the gaze position is moved and the directions in which each of the plurality of virtual objects is moved, the wearable device may determine or identify a virtual object gazed by the user among the plurality of virtual objects. When a gaze position located in only one of the detection areas (911, 912, 913, 914, 915, 916, 917, 918, 919) is detected, the wearable device can move only the virtual object corresponding to the detection area based on the operation described above with reference to FIGS. 5A to 5D.

[0152] Referring to FIG. 9b, an exemplary state (902) of a wearable device that identifies a gaze position (g2) located in an overlapping area (920) of detection areas (911, 912) corresponding to virtual objects (441, 442) is illustrated. The wearable device can move the virtual objects (441, 442) corresponding to each of the detection areas (911, 912) that include the gaze position (g2). Referring to FIG. 9b, the wearable device can move the virtual objects (441, 442) such that the virtual objects (441, 442) approach the gaze position (g2). For example, the virtual object (441) can move from the position (o1) of FIG. 9a to a position adjacent to the gaze position (g2), and the virtual object (442) can move from the position (o1) of FIG. 9a to a position adjacent to the gaze position (g2).

[0153] Referring to FIG. 9b, since a virtual object (441) located to the left of the gaze position (g2) moves toward the gaze position (g2) and a virtual object (442) located to the right of the gaze position (g2) moves toward the gaze position (g2), the virtual objects (441, 442) can move in opposite directions. When a user gazes at a specific virtual object among the virtual objects (441, 442), since the wearable device moves the virtual objects (441, 442) in opposite directions, the gaze position can move along the direction in which the specific virtual object focused on by the user moves. The wearable device can accurately detect a virtual object focused on by the user among the virtual objects (441, 442) by using the direction in which the gaze position moves.

[0154] Referring to FIG. 9C, a state (903) following the state (902) of FIG. 9B is illustrated. Within the state (903), the wearable device can detect a gaze position (g3) different from the gaze position (g2) detected within the state (902). Within the state (903) in which the gaze position (g3) located to the right of the gaze position (g2) is detected, the wearable device can detect a movement direction (e.g., a direction toward the right) of the gaze position (g3). As described above with reference to FIGS. 9A and 9B , since the virtual object (441) among the virtual objects (441, 442) has moved in a direction toward the right, the wearable device can determine that the virtual object (441) moved in a direction similar to the movement direction of the gaze position (g3) is focused by the user. After determining that the virtual object (441) is focused by the user, the wearable device can maintain moving the virtual object (441) according to the gaze position. After determining that the virtual object (441) is focused by the user, the wearable device can fix the position of another virtual object (e.g., virtual object (442)) different from the virtual object (441) or move it to its initial position.

[0155] Referring to FIG. 9d, within a state (904) subsequent to the state (903) of FIG. 9c, the wearable device can move the virtual object (441) according to the gaze position (g3). For example, the virtual object (441) can be further moved from the position (o3) within the state (903). Within the exemplary state (904) of FIG. 9d, the wearable device can move another virtual object (442) different from the virtual object (441) focused on by the user to the initial position (the position within the state (901) of FIG. 9a). For example, the virtual object (442) can be moved from the position (g3) within the state (903) to the initial position. After state (904) of FIG. 9d, the wearable device can move the virtual object (441) according to the gaze position based on the operation described with reference to FIGS. 5a to 5d and / or FIGS. 6a to 6b.

[0156] In one embodiment, the directions in which virtual objects (441, 442) move based on the gaze position (g2) included in the overlapping area (920) are not limited to the embodiments of FIGS. 9A to 9D . Hereinafter, exemplary operations of a wearable device that detects the gaze position (g2) included in the overlapping area (920) are described with reference to FIGS. 10A to 10C .

[0157] FIGS. 10A, 10B, and 10C illustrate the operation of a wearable device that detects a gaze position (g1) within an overlapping area of ​​detection areas (1021, 1022, 1023, and 1024). The wearable device (101) of FIGS. 1 and / or 2 and / or the processor (210) of FIG. 2 may perform the operation of the wearable device described with reference to FIGS. 10A to 10C. The operation of the wearable device described with reference to FIGS. 10A to 10C may be related to at least one of the operations of FIG. 3.

[0158] Referring to FIGS. 10A to 10C , exemplary states of a wearable device displaying exemplary virtual objects (1011, 1012, 1013, 1014) are illustrated. The positions, shapes, and / or layouts of the virtual objects (1011, 1012, 1013, 1014) illustrated in FIGS. 10A to 10C are exemplary, and the positions, shapes, and / or layouts of the virtual objects displayed on a display (e.g., the display (110) of FIG. 2 ) are not limited. For example, the virtual objects (1011, 1012, 1013, 1014) may be included in a lock screen. The wearable device can identify or detect the intention of a user gazing at a specific virtual object by using detection areas (1021, 1022, 1023, 1024) including virtual objects (1011, 1012, 1013, 1014).

[0159] Referring to FIG. 10A, according to one embodiment, a wearable device can display a plurality of virtual objects (1011, 1012, 1013, 1014) on a display. The wearable device can determine a gaze position (g1) of a user wearing the wearable device using an image of the eye acquired from a camera (e.g., camera (130) of FIG. 2). Referring to FIG. 10A, an exemplary state is illustrated in which the gaze position (g1) is included in an overlapping area of ​​detection areas (1021, 1022, 1023, 1024). A wearable device that detects a gaze position (g1) within an overlapping area of ​​detection areas (1021, 1022, 1023, 1024) of virtual objects (1011, 1012, 1013, 1014) can move the plurality of virtual objects (1011, 1012, 1013, 1014) so ​​that the plurality of virtual objects (1011, 1012, 1013, 1014) move away from the gaze position (g1). Referring to FIG. 10b, an exemplary state in which the wearable device that detected the gaze position (g1) of FIG. 10a moves the plurality of virtual objects (1011, 1012, 1013, 1014) is illustrated.

[0160] Within the state of FIG. 10b, the wearable device can move the plurality of virtual objects (1011, 1012, 1013, 1014) so ​​that the plurality of virtual objects (1011, 1012, 1013, 1014) move away from the gaze position (g1). For example, the virtual object (1011) can move from the initial position (o1) toward the upper left. For example, the virtual object (1012) can move from the initial position (o2) toward the upper right. For example, the virtual object (1013) can move from the initial position (o3) toward the lower left. For example, the virtual object (1014) can move from the initial position (o4) toward the lower right. For example, multiple virtual objects (1011, 1012, 1013, 1014) can be moved in different directions.

[0161] Referring to FIG. 10C, an exemplary state after a plurality of virtual objects (1011, 1012, 1013, 1014) have been moved is illustrated. According to one embodiment, the wearable device can determine or detect a first virtual object focused on by the user among the plurality of virtual objects (1011, 1012, 1013, 1014) by using the displacement (vg) of the gaze position (g2) detected after the plurality of virtual objects (1011, 1012, 1013, 1014) have been moved. Referring to FIG. 10C, in the exemplary state in which the gaze position (g2) is detected, the wearable device can calculate or identify the displacement (vg) of the gaze position (g1) and the gaze position (g2) before the state of FIG. 10C.

[0162] According to one embodiment, a wearable device can compare the directions (v1, v2, v3, v4) in which each of a plurality of virtual objects (1011, 1012, 1013, 1014) is moved and the direction of the displacement (vg). For example, the wearable device can calculate the direction of the displacement (vg) and the angular differences between each of the directions (v1, v2, v3, v4). A virtual object corresponding to a direction having the smallest angular difference among the directions (v1, v2, v3, v4) can be determined as a virtual object focused on by the user. In the exemplary case of FIG. 10c, since the angular difference between the direction in which the virtual object (1011) is moved (v1) and the direction of displacement (vg) is smaller than the angular differences for other directions (v2, v3, v4), the wearable device can determine the virtual object (1011) among the virtual objects (1011, 1012, 1013, 1014) as the virtual object focused on by the user.

[0163] In one embodiment, the wearable device can calculate a similarity using the magnitude, position, and / or direction of the displacement (vg). Using the similarity, the wearable device can determine whether the gaze follows a particular virtual object (or the validity of data including the displacement (vg)). For example, data determined to be invalid (e.g., a pair of the result of the position of the virtual object and / or the gaze position (g1)) can be removed. For example, if multiple virtual objects are controlled based on gaze positions, such as multiple virtual objects (1011, 1012, 1013, 1014), and at least one has valid data, the wearable device can stop or terminate control of the virtual object containing the invalid data. The virtual object for which control has been stopped can be moved to its original position. The wearable device can maintain control of the valid virtual object.

[0164] In one embodiment, when a virtual object (1011) is determined to be focused by a user, the wearable device may store a displacement (vg) of the gaze position (g2) as information to be used for calibration related to a gaze position (g2) adjacent to the virtual object (1011). When a virtual object (1011) is determined to be focused by a user, the wearable device may move the virtual object (1011) using the displacement (vg) of the gaze position (g2). For example, the distance that the virtual object (1011) moves along the direction of the displacement (vg) may be less than the magnitude of the displacement (vg). For example, the wearable device may move the virtual object (1011) by the magnitude of the displacement (vg) to which a specified ratio is applied. For example, the wearable device can move the virtual object (1011) and other virtual objects (1012, 1013, 1014) to their initial locations (o2, o3, o4), respectively.

[0165] In one embodiment, when the virtual object (1011) is determined to be focused by the user, the wearable device may move the virtual object (1011) or stop moving the virtual object (1011) based on the operations described with reference to FIGS. 5A to 5D and / or FIGS. 6A to 6B. When stopping moving the virtual object (1011), the wearable device may execute a function associated with the virtual object (1011).

[0166] As described above, according to one embodiment, the wearable device can cause a movement of a gaze position along any one of the virtual objects (1011, 1012, 1013, 1014) by moving all of the virtual objects (1011, 1012, 1013, 1014) corresponding to each of the detection areas (1021, 1022, 1023, 1024) when the detection areas (1021, 1022, 1023, 1024) overlap each other. By comparing the directions in which the virtual objects (1011, 1012, 1013, 1014) are moved and the direction of the displacement (vg) of the gaze position (g2), the wearable device can identify any one virtual object focused on by the user.

[0167] Fig. 11 illustrates an operation of a wearable device that changes the position and / or shape of a virtual object using a gaze position (g1). The wearable device (101) of Fig. 1 and / or Fig. 2 and / or the processor (210) of Fig. 2 may perform the operation of the wearable device described with reference to Fig. 11. The operation of the wearable device described with reference to Fig. 11 may be related to at least one of the operations of Fig. 3.

[0168] Referring to FIG. 11, an exemplary state of a wearable device executing multiple software applications is illustrated. The wearable device may display panels (1111, 1112) provided by the multiple software applications on a display (e.g., display (110) of FIG. 2). Panels (1111, 1112) having a two-dimensional plane shape (e.g., a square or a square with rounded corners) are illustrated as an exemplary state, but the embodiment is not limited thereto. In one embodiment, the panels (1111, 1112) may be referred to as windows and / or activities. An exemplary state of a wearable device displaying a screen (1101) including a panel (1111) based on the execution of a web browser and a panel (1112) based on a messenger is illustrated as an exemplary state, but the embodiment is not limited thereto.

[0169] Referring to FIG. 11, the wearable device may display visual objects (1113, 1114, 1115) corresponding to respective options related to the size of the panel (1112) on the panel (1112). For example, the visual object (1113) may correspond to an option to minimize the size of the panel (1112) or add an icon for restoring the panel (1112) to a designated area (e.g., an area referred to as an app tray). The visual object (1113) may be referred to as a minimize button. For example, the visual object (1114) may correspond to an option to maximize the size of the panel (1112) (e.g., to a size that covers the entire screen (1101). The visual object (1114) may be referred to as a maximize button. For example, a visual object (1115) may correspond to an option to stop displaying a panel (1112) and terminate execution of a software application (e.g., a messenger application) corresponding to the panel (1112). The visual object (1115) may be referred to as an exit button.

[0170] In one embodiment, using adjacently positioned visual objects (1113, 1114, 1115), the wearable device can perform an operation to obtain calibration information. Referring to FIG. 11, enlarged views (1101, 1102) of a portion (1115) of a screen (1101) including visual objects (1113, 1114, 1115) are shown. Referring to the enlarged view (1101), detection areas (1116, 1117, 1118) corresponding to each of the visual objects (1113, 1114, 1115) can be formed. Referring to the enlarged view (1101), according to one embodiment, the wearable device can detect or identify a gaze position (g1) included in a detection area (1117) corresponding to the visual object (1114).

[0171] In one embodiment, a wearable device that detects a gaze position (g1) included in a detection area (1117) can change the position and / or shape of a visual object (1114), as illustrated in the enlarged view (1102). For example, a wearable device that detects a gaze position (g2) can change the position and / or shape of the visual object (1114) by using the displacement of the gaze positions (g1, g2). For example, by changing the position and / or shape of a shape (e.g., a square shape) included in the visual object (1114), the wearable device can cause an eye movement to follow the shape. The wearable device can store the gaze position moved by the eye movement as calibration information. While the gaze position (g2) is maintained within the detection area (1117), the wearable device can complete or stop storing the calibration information based on the expiration of a specified period set to measure the calibration information. Based on the expiration of the above specified period, the wearable device may execute a function associated with the visual object (1114), such as a function of maximizing the size of the panel (1112).

[0172] As described above, the wearable device can perform operations to obtain calibration information using virtual objects included in the lock screen and other virtual objects. The wearable device can obtain calibration information using a virtual object designed to obtain calibration information or having functions for obtaining calibration information mapped to other functions, without a virtual object dedicated to obtaining calibration information. For example, the wearable device can obtain calibration information without a separate session for obtaining calibration information.

[0173] In one embodiment, the wearable device can move a virtual object not only horizontally and vertically within the screen, but also in a depth direction (e.g., moving away from or toward the user). Below, an exemplary operation of a wearable device that acquires calibration information using a virtual object moving along the depth direction is described.

[0174] FIGS. 12A and 12B illustrate operations of a wearable device (101) that change the depth of a virtual object (e.g., virtual objects (v1, v2, v3)) using a gaze position. The wearable device (101) of FIGS. 1 and / or 2 and / or the processor (210) of FIG. 2 may perform operations of the wearable device (101) described with reference to FIGS. 12A and / or 12B. Operations of the wearable device (101) described with reference to FIGS. 12A and / or 12B may be related to at least one of the operations of FIG. 3.

[0175] Referring to FIGS. 12A and / or 12B , exemplary states (1201, 1202) of a wearable device (101) displaying virtual objects (v1, v2, v3) are illustrated. The virtual objects (vl1, vl2, vl3) displayed on the first display (110-1) positioned toward the user's left eye may correspond to the virtual objects (v1, v2, v3), respectively. The virtual objects (vr1, vr2, vr3) displayed on the second display (110-2) positioned toward the user's right eye may correspond to the virtual objects (v1, v2, v3), respectively. For example, a user viewing the virtual objects (vl1, vr1) with the left eye (140-1) and the right eye (140-2), respectively, may recognize the virtual object (v1). For example, a user who views virtual objects (vl2, vr2) with the left eye (140-1) and the right eye (140-2) respectively can recognize virtual object (v2). For example, a user who views virtual objects (vl3, vr3) with the left eye (140-1) and the right eye (140-2) respectively can recognize virtual object (v3).

[0176] In one embodiment, the wearable device (101) can change the depth perception of virtual objects (v1, v2, v3) by using binocular disparity. The positions of each of the virtual objects (vl1, vl2, vl3) within the first screen (1210-1) displayed on the first display (110-1) can be spaced apart from the positions of the virtual objects (vr1, vr2, vr3) within the second screen (1210-2) displayed on the second display (110-2) by the binocular disparities set for each of the virtual objects (v1, v2, v3). For example, the binocular disparity of the virtual object (v1) can be greater than that of the other virtual objects (v2, v3). The binocular disparity of the virtual object (v3) can be less than that of the other virtual objects (v1, v2). The binocular disparity of the virtual object (v2) may be smaller than the binocular disparity of the virtual object (v1) and larger than the binocular disparity of the virtual object (v3). A user viewing the virtual objects (v1, v2, v3) may perceive the virtual object (v1) as being closer than the other virtual objects (v2, v3), and may perceive the virtual object (v3) as being further than the other virtual objects (v1, v2).

[0177] Within the state (1201) of FIG. 12A, the wearable device (101) can adjust the sizes of virtual objects (v1, v2, v3) so that the virtual objects (v1, v2, v3) are displayed in the same size. Referring to the first screen (1210-1), the virtual objects (vl1, vl2, vl3) within the first screen (1210-1) can have the same size. Referring to the second screen (1210-2), the virtual objects (vr1, vr2, vr3) within the second screen (1210-2) can have the same size.

[0178] According to one embodiment, the wearable device (101) can calculate or determine the gaze position (g1) using images of two eyes (e.g., the left eye (140-1) and the right eye (140-2)) acquired from a camera (e.g., the camera (130) of FIG. 2). For example, the wearable device (101) can calculate or determine three-dimensional coordinates representing the gaze position (g1). By comparing the gaze position (g1) with detection areas (1211, 1212, 1213) corresponding to virtual objects (v1, v2, v3), respectively, the wearable device (101) can determine whether to move each of the virtual objects (v1, v2, v3).

[0179] In an exemplary state (1201) of FIG. 12A, the wearable device (101) can detect a gaze position (g1) included in a detection area (1212) corresponding to a virtual object (v2). The wearable device (101) that has detected the gaze position (g1) within the detection area (1212) can move the virtual object (v2) corresponding to the detection area (1212). For example, the wearable device (101) can change the position of the virtual object (v2) by using the positional relationship between the gaze position (g1) and the virtual object (v2).

[0180] Referring to FIG. 12B, an exemplary state (1202) is illustrated after the state (1201) of FIG. 12A, in which the position of the virtual object (v2) has changed. Within the state (1202), the wearable device (101) may increase the depth value of the virtual object (v2) or decrease the binocular disparity of the virtual object (v2). Due to the increased depth value and / or the decreased binocular disparity, the virtual object (v2) may be perceived as being farther away from the user. In one embodiment, the wearable device (101) may change the size of the virtual object (v2) so that the virtual object (v2) within the state (1202) matches the size of the virtual object (v2) that was perceived by the user within the state (1201) prior to the state (1202). For example, the size of the virtual object (vl2) within the screen (1210-1) of the state (1202) and the size of the virtual object (vl2) within the screen (1210-1) of the state (1201) may be the same. For example, in the state (1202) where the virtual object (v2) is moved according to the gaze position (g1), the wearable device (101) may change the size of the virtual object (v2).

[0181] Referring to FIG. 12B, the wearable device (101) can detect a gaze position (g2). In a state (1202) in which the gaze position (g2) is detected, the wearable device (101) can move a virtual object (v2) using the displacement of the gaze positions (g1, g2). Moving the virtual object (v2) using the gaze position can be stopped based on at least one of a period set to move the virtual object (v2), whether the gaze position is maintained within the detection area (1212), or whether the position of the virtual object (v2) converges. Until moving the virtual object (v2) is stopped, the wearable device (101) can store information related to the gaze position as calibration information.

[0182] As described above, according to one embodiment, the wearable device (101) may utilize a virtual object other than a virtual object dedicated to obtaining calibration information, such as a virtual object within the lock screen, to obtain calibration information. For example, the wearable device (101) may move the virtual object according to the displacement of the gaze position. Since the wearable device (101) moves the virtual object less than the magnitude of the displacement, the position of the virtual object may converge. The wearable device (101) may store information related to the gaze position (e.g., images and / or videos of the user's two eyes as a result of calculating the gaze position) as calibration information until the position of the virtual object converges.

[0183] Hereinafter, with reference to FIGS. 13A, 13B, 14A and / or 14B, exemplary appearances of wearable devices (101) described with reference to FIGS. 1 to 11, 12A and / or 12B are illustrated. The wearable devices (1300) of FIGS. 13A and / or 13B and / or the wearable devices (1400) of FIGS. 14A and / or 14B may be examples of the wearable devices (101) of FIG. 1.

[0184] FIG. 13A illustrates an example of a perspective view of a wearable device, according to one embodiment. According to one embodiment, the wearable device (1300) may have the form of glasses that are wearable on a body part of a user (e.g., the head). The wearable device (1300) may include a head-mounted display (HMD). For example, the housing of the wearable device (1300) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (1300) may include one or more straps that are capable of being twined around the user's head, and / or one or more temples that are detachably attachable to the ears of the head.

[0185] Referring to FIG. 13A, according to one embodiment, a wearable device (1300) may include at least one display (1350) and a frame supporting at least one display (1350).

[0186] According to one embodiment, a wearable device (1300) may be worn on a part of a user's body. The wearable device (1300) may provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (1300). For example, the wearable device (1300) may display a virtual reality image provided from at least one optical device (1382, 1384) of FIG. 13B on at least one display (1350) in response to a user's designated gesture acquired through the motion recognition cameras (1360-2, 1360-3) of FIG. 13B.

[0187] According to one embodiment, at least one display (1350) may provide visual information to a user. For example, at least one display (1350) may include a transparent or translucent lens. At least one display (1350) may include a first display (1350-1) and / or a second display (1350-2) spaced apart from the first display (1350-1). For example, the first display (1350-1) and the second display (1350-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0188] Referring to FIG. 13B, at least one display (1350) can provide visual information transmitted from external light to a user through a lens included in the at least one display (1350), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (1350) can include a first surface (1331) and a second surface (1332) opposite to the first surface (1331). A display area can be formed on the second surface (1332) of the at least one display (1350). When a user wears the wearable device (1300), external light can be transmitted to the user by being incident on the first surface (1331) and transmitted through the second surface (1332). As another example, at least one display (1350) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (1382, 1384) on a real screen transmitted through external light, in a display area formed on the second surface (1332).

[0189] In one embodiment, at least one display (1350) may include at least one waveguide (1333, 1334) that diffracts light emitted from at least one optical device (1382, 1384) and transmits the diffracted light to a user. The at least one waveguide (1333, 1334) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (1333, 1334). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (1333, 1334) may be propagated to the other end of the at least one waveguide (1333, 1334) by the nano-pattern. At least one waveguide (1333, 1334) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (1333, 1334) may be arranged within the wearable device (1300) to guide a screen displayed by at least one display (1350) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (1333, 1334).

[0190] The wearable device (1300) can analyze an object included in a real image collected through a shooting camera (1360-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (1350). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (1300) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (1300) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using the multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (1300) can view an image displayed on at least one display (1350).

[0191] According to one embodiment, the frame may be formed as a physical structure that allows the wearable device (1300) to be worn on the user's body. According to one embodiment, the frame may be configured so that, when the user wears the wearable device (1300), the first display (1350-1) and the second display (1350-2) can be positioned corresponding to the user's left and right eyes. The frame may support at least one display (1350). For example, the frame may support the first display (1350-1) and the second display (1350-2) to be positioned corresponding to the user's left and right eyes.

[0192] Referring to FIG. 13A, the frame may include a region (1320) that at least partially contacts a portion of the user's body when the user wears the wearable device (1300). For example, the region (1320) of the frame that contacts a portion of the user's body may include a region that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (1300) makes contact with. According to one embodiment, the frame may include a nose pad (1310) that contacts a portion of the user's body. When the wearable device (1300) is worn by the user, the nose pad (1310) may contact a portion of the user's nose. The frame may include a first temple (1304) and a second temple (1305) that contact another portion of the user's body that is distinct from the portion of the user's body.

[0193] For example, the frame may include a first rim (1301) that surrounds at least a portion of a first display (1350-1), a second rim (1302) that surrounds at least a portion of a second display (1350-2), a bridge (1303) that is disposed between the first rim (1301) and the second rim (1302), a first pad (1311) that is disposed along a portion of an edge of the first rim (1301) from one end of the bridge (1303), a second pad (1312) that is disposed along a portion of an edge of the second rim (1302) from the other end of the bridge (1303), a first temple (1304) that extends from the first rim (1301) and is secured to a portion of an ear of the wearer, and a second temple (1305) that extends from the second rim (1302) and is secured to a portion of an ear opposite the ear. The first pad (1311) and the second pad (1312) may be in contact with a portion of the user's nose, and the first temple (1304) and the second temple (1305) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (1304, 1305) may be rotatably connected to the rim through the hinge units (1306, 1307) of FIG. 13B. The first temple (1304) may be rotatably connected to the first rim (1301) through the first hinge unit (1306) disposed between the first rim (1301) and the first temple (1304). The second temple (1305) may be rotatably connected to the second rim (1302) via a second hinge unit (1307) disposed between the second rim (1302) and the second temple (1305). In one embodiment, the wearable device (1300) may use a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame to identify an external object (e.g., a user's fingertip) touching the frame and / or a gesture performed by the external object.

[0194] According to one embodiment, the wearable device (1300) may include hardwares (e.g., hardwares described above based on the block diagram of FIG. 2) that perform various functions. For example, the hardwares may include a battery module (1370), an antenna module (1375), at least one optical device (1382, 1384), speakers (e.g., speakers 1355-1, 1355-2), microphones (e.g., microphones 1365-1, 1365-2, 1365-3), a light-emitting module, and / or a printed circuit board (PCB) (1390) (e.g., a printed circuit board). The various hardware components may be arranged within a frame.

[0195] According to one embodiment, the microphones (e.g., microphones 1365-1, 1365-2, 1365-3) of the wearable device (1300) may be disposed on at least a portion of the frame to acquire sound signals. A first microphone (1365-1) disposed on the bridge (1303), a second microphone (1365-2) disposed on the second rim (1302), and a third microphone (1365-3) disposed on the first rim (1301) are illustrated in FIG. 13B , but the number and arrangement of the microphones (1365) are not limited to the embodiment of FIG. 13B . When the number of microphones (1365) included in the wearable device (1300) is two or more, the wearable device (1300) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame.

[0196] According to one embodiment, at least one optical device (1382, 1384) may project a virtual object onto at least one display (1350) to provide various image information to a user. For example, at least one optical device (1382, 1384) may be a projector. At least one optical device (1382, 1384) may be disposed adjacent to at least one display (1350) or may be included within at least one display (1350) as a part of at least one display (1350). According to one embodiment, the wearable device (1300) may include a first optical device (1382) corresponding to a first display (1350-1) and a second optical device (1384) corresponding to a second display (1350-2). For example, at least one optical device (1382, 1384) may include a first optical device (1382) disposed at an edge of a first display (1350-1) and a second optical device (1384) disposed at an edge of a second display (1350-2). The first optical device (1382) may transmit light to a first waveguide (1333) disposed on the first display (1350-1), and the second optical device (1384) may transmit light to a second waveguide (1334) disposed on the second display (1350-2).

[0197] In one embodiment, the camera (1360) may include a recording camera (1360-4), an eye tracking camera (ET CAM) (1360-1), and / or motion recognition cameras (1360-2, 1360-3). The recording camera (1360-4), the eye tracking camera (1360-1), and the motion recognition cameras (1360-2, 1360-3) may be positioned at different locations on the frame and may perform different functions. The eye tracking camera (1360-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (1300). For example, the wearable device (1300) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (1360-1).

[0198] The wearable device (1300) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (1360-1). The wearable device (1300) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (1300) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (1360-1). The wearable device (1300) can render an image (or screen) displayed on at least one display (1350) based on the position of the user's eyes.

[0199] For example, the visual quality of a first region related to the gaze within an image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI (pixels per inch)) of a second region distinct from the first region may be different from each other. The wearable device (1300) may obtain an image having the visual quality of the first region matching the user's gaze and the visual quality of the second region using foveated rendering. For example, if the wearable device (1300) supports an iris recognition function, user authentication may be performed based on iris information obtained using the gaze tracking camera (1360-1). Although an example in which the gaze tracking camera (1360-1) is positioned toward the user's right eye is illustrated in FIG. 13B, the embodiment is not limited thereto, and the gaze tracking camera (1360-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0200] In one embodiment, the capturing camera (1360-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (1360-4) can be used to obtain a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (1360-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (1350). The at least one display (1350) can display a single image in which information about an actual image or background including an image of the specific object obtained using the capturing camera (1360-4) and a virtual image provided through at least one optical device (1382, 1384) are superimposed. The wearable device (1300) can compensate for depth information (e.g., the distance between the wearable device (1300) and an external object acquired through a depth sensor) using an image acquired through the capture camera (1360-4). The wearable device (1300) can perform object recognition using an image acquired using the capture camera (1360-4). The wearable device (1300) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capture camera (1360-4). The wearable device (1300) can perform a pass-through function to display an image acquired through the capture camera (1360-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (1350) while displaying a screen. In one embodiment, the shooting camera (1360-4) may be positioned on a bridge (1303) positioned between the first rim (1301) and the second rim (1302).

[0201] The gaze tracking camera (1360-1) can implement more realistic augmented reality by tracking the gaze of a user wearing a wearable device (1300) and matching the user's gaze with visual information provided to at least one display (1350). For example, when the wearable device (1300) looks straight ahead, the wearable device (1300) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (1350). The gaze tracking camera (1360-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (1360-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (1360-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (1360-1) may be positioned within the first rim (1301) and / or the second rim (1302) to face the direction in which the user wearing the wearable device (1300) is positioned.

[0202] The gesture recognition cameras (1360-2, 1360-3) can recognize the movement of the user's entire body, such as the user's torso, hands, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (1350). The gesture recognition cameras (1360-2, 1360-3) can recognize the user's gesture (gesture recognition), obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (1350). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition cameras (1360-2, 1360-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor may perform gesture recognition and / or object tracking functions using the motion recognition cameras (1360-2, 1360-3). In one embodiment, the motion recognition cameras (1360-2, 1360-3) may be positioned on the first rim (1301) and / or the second rim (1302).

[0203] The camera (1360) included in the wearable device (1300) is not limited to the above-described gaze tracking camera (1360-1) and motion recognition cameras (1360-2, 1360-3). For example, the wearable device (1300) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (1300) can identify an external object based on a sensor for identifying the distance between the wearable device (1300) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (1360) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (1300) may include a camera (1360) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (1300).

[0204] Although not shown, in one embodiment, the wearable device (1300) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being captured using the camera (1360). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame and hinge units (1306, 1307).

[0205] According to one embodiment, the battery module (1370) may supply power to the electronic components of the wearable device (1300). In one embodiment, the battery module (1370) may be disposed within the first temple (1304) and / or the second temple (1305). For example, the battery module (1370) may be a plurality of battery modules (1370). The plurality of battery modules (1370) may be disposed within each of the first temple (1304) and the second temple (1305). In one embodiment, the battery module (1370) may be disposed at an end of the first temple (1304) and / or the second temple (1305).

[0206] The antenna module (1375) can transmit signals or power to the outside of the wearable device (1300), or receive signals or power from the outside. In one embodiment, the antenna module (1375) can be positioned within the first temple (1304) and / or the second temple (1305). For example, the antenna module (1375) can be positioned close to one surface of the first temple (1304) and / or the second temple (1305).

[0207] The speaker (1355) can output an audio signal to the outside of the wearable device (1300). The audio output module may be referred to as a speaker. In one embodiment, the speaker (1355) may be positioned within the first temple (1304) and / or the second temple (1305) so as to be positioned adjacent to the ear of a user wearing the wearable device (1300). For example, the speaker (1355) may include a second speaker (1355-2) positioned within the first temple (1304) and thus adjacent to the user's left ear, and a first speaker (1355-1) positioned within the second temple (1305) and thus adjacent to the user's right ear.

[0208] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (1300) to the user. For example, when the wearable device (1300) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (1301) and / or the second rim (1302).

[0209] Referring to FIG. 13B, according to one embodiment, a wearable device (1300) may include a printed circuit board (PCB) (1390). The PCB (1390) may be included in at least one of the first temple (1304) or the second temple (1305). The PCB (1390) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (1300) (e.g., hardwares illustrated by different blocks in FIG. 2) may be disposed on the PCB (1390). The wearable device (1300) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0210] According to one embodiment, a wearable device (1300) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (1300) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (1300). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (1300) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (1300) based on the IMU.

[0211] Figures 14a and 14b illustrate an example of an exterior appearance of a wearable device according to one embodiment. The wearable device (1400) of Figures 14a and 14b may include at least a portion of the hardware of the wearable device (1300) described with reference to Figures 13a and / or 13b. An example of an exterior appearance of a first side (1410) of a housing of the wearable device (1400) according to one embodiment is illustrated in Figure 14a, and an example of an exterior appearance of a second side (1420) opposite to the first side (1410) may be illustrated in Figure 14b.

[0212] Referring to FIG. 14A, a first surface (1410) of a wearable device (1400) according to one embodiment may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (1400) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (1304) and / or the second temple (1305) of FIGS. 13A and 13B). A first display (1350-1) for outputting an image to a left eye among the user's two eyes, and a second display (1350-2) for outputting an image to a right eye among the two eyes, may be disposed on the first surface (1410). The wearable device (1400) may be formed on the first surface (1410) and may further include a rubber or silicone packing to prevent interference from light (e.g., ambient light) different from the light emitted from the first display (1350-1) and the second display (1350-2).

[0213] According to one embodiment, a wearable device (1400) may include cameras (1360-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (1350-1) and the second display (1350-2). The cameras (1360-1) may be referred to as the gaze tracking camera (1360-1) of FIG. 13B. According to one embodiment, a wearable device (1400) may include cameras (1360-5, 1360-6) for photographing and / or recognizing a face of a user. The cameras (1360-5, 1360-6) may be referred to as FT cameras. The wearable device (1400) can control an avatar representing the user in a virtual space based on the facial motion of the user identified using cameras (1360-5, 1360-6). For example, the wearable device (1400) can change the texture and / or shape of a portion of the avatar (e.g., a portion of the avatar representing a human face) using information obtained by cameras (1360-5, 1360-6) (e.g., FT cameras) and representing the facial expression of the user wearing the wearable device (1400).

[0214] Referring to FIG. 14B, a camera (e.g., cameras (1360-7, 1360-8, 1360-9, 1360-10, 1360-11, 1360-12)) and / or a sensor (e.g., a depth sensor (1430)) for obtaining information related to the external environment of the wearable device (1400) may be disposed on a second surface (1420) opposite to the first surface (1410) of FIG. 14A. For example, the cameras (1360-7, 1360-8, 1360-9, 1360-10) may be disposed on the second surface (1420) to recognize external objects. The cameras (1360-7, 1360-8, 1360-9, 1360-10) of FIG. 14b can correspond to the motion recognition cameras (1360-2, 1360-3) of FIG. 13b.

[0215] For example, using cameras (1360-11, 1360-12), the wearable device (1400) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (1360-11) can be placed on the second face (1420) of the wearable device (1400) to obtain an image to be displayed through the second display (1350-2) corresponding to the right eye among the two eyes. The camera (1360-12) can be placed on the second face (1420) of the wearable device (1400) to obtain an image to be displayed through the first display (1350-1) corresponding to the left eye among the two eyes. The cameras (1360-11, 1360-12) can correspond to the shooting camera (1360-4) of FIG. 13B.

[0216] According to one embodiment, the wearable device (1400) may include a depth sensor (1430) disposed on a second face (1420) to identify a distance between the wearable device (1400) and an external object. Using the depth sensor (1430), the wearable device (1400) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (1400). Although not illustrated, a microphone may be disposed on the second face (1420) of the wearable device (1400) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0217] In one embodiment, a method may be required to obtain calibration information related to gaze position without a virtual object dedicated to calibration. In one embodiment, a method may be required to obtain calibration information related to gaze position using a virtual object for unlocking. According to one embodiment, a wearable device (e.g., the wearable device (101) of FIG. 1 , the wearable device (1300) of FIGS. 13A-13B , and / or the wearable device (1400) of FIGS. 14A-14B ) as described above may include at least one display (e.g., the display (110) of FIG. 2 ), at least one camera (e.g., the camera (130) of FIG. 2 ) configured to be positioned toward an eye of a user wearing the wearable device, at least one processor including a processing circuit (e.g., the processor (210) of FIG. 2 ), and a memory (e.g., the memory (215) of FIG. 2 ) including one or more storage media for storing instructions. The at least one processor may be individually or collectively configured to display a virtual object (e.g., the virtual object (o) of FIG. 1 ) on the at least one display. The at least one processor may be individually or collectively configured to determine a gaze position of the user (e.g., gaze position (g1) in FIG. 1) using an image of the eye acquired from the at least one camera.The at least one processor is configured to individually or collectively determine information representing a displacement of the gaze positions detected at each of two points in time while detecting the gaze positions within a detection area including the virtual object (e.g., the detection area (451) of FIG. 4, the detection area (520) of FIGS. 5A to 5D, the detection area (720) of FIG. 7, the detection areas (911, 912, 913, 914, 915, 916, 917, 918, 919) of FIGS. 9A to 9D, the detection areas (1021, 1022, 1023, 1024) of FIGS. 10A to 10C, the detection areas (1116, 1117, 1118) of FIG. 11, the detection areas (1211, 1212, 1213) of FIGS. 12A to 12B). The at least one processor may be individually or collectively configured to determine a distance for movement of the virtual object within a range less than the magnitude of the displacement indicated by the information based on the determined information. The at least one processor may be individually or collectively configured to move the virtual object by the determined distance in the direction of the displacement indicated by the information based on determining the distance to be greater than a threshold distance. The at least one processor may be individually or collectively configured to stop moving the virtual object based on determining the distance to be less than a threshold distance. The at least one processor may be individually or collectively configured to execute a function related to the virtual object based on stopping moving the virtual object.

[0218] According to one embodiment, a wearable device (e.g., wearable device (101) of FIG. 1, wearable device (1300) of FIGS. 13A-13B, and / or wearable device (1400) of FIGS. 14A-14B) may include at least one display (e.g., display (110) of FIG. 2), at least one camera (e.g., camera (130) of FIG. 2) configured to be positioned toward an eye of a user wearing the wearable device, at least one processor including processing circuitry (e.g., processor (210) of FIG. 2), and a memory (e.g., memory (215) of FIG. 2) including one or more storage media for storing instructions. The at least one processor may be individually or collectively configured to display a virtual object (e.g., virtual object (o) of FIG. 1) on the at least one display. The at least one processor may be individually or collectively configured to determine a gaze position of the user (e.g., gaze position (g1) in FIG. 1) using an image of the eye acquired from the at least one camera. The at least one processor individually or collectively detects the gaze position within a detection area including the virtual object (e.g., the detection area (451) of FIG. 4, the detection area (520) of FIGS. 5A to 5D, the detection area (720) of FIG. 7, the detection areas (911, 912, 913, 914, 915, 916, 917, 918, 919) of FIGS. 9A to 9D, the detection areas (1021, 1022, 1023, 1024) of FIGS. 10A to 10C, the detection areas (1116, 1117, 1118) of FIG. 11, the detection areas (1211, 1212, 1213) of FIGS. 12A to 12B), and determines information representing a displacement of the gaze position within the detection area. Can be composed.The at least one processor may be individually or collectively configured to move the virtual object in the direction of the displacement. The distance of the virtual object moved in the direction may be less than the magnitude of the displacement. The at least one processor may be individually or collectively configured to repeatedly move the virtual object in the direction while the distance exceeds a threshold distance. The at least one processor may be individually or collectively configured to stop moving the virtual object using the gaze position based on identifying that the distance is less than the threshold distance. The at least one processor may be individually or collectively configured to execute a function related to the virtual object based on stopping moving the virtual object. In one embodiment, a wearable device may obtain calibration information related to a gaze position without a virtual object dedicated for calibration. In one embodiment, a wearable device may obtain calibration information related to a gaze position using a virtual object for unlocking.

[0219] For example, the at least one processor may be individually or collectively configured to move the virtual object by an amount of the displacement to which a specified ratio is applied.

[0220] For example, the at least one processor may be configured, individually or collectively, to store at least one of the gaze position or the displacement of the gaze position obtained while the virtual object is being moved, based on stopping the movement of the virtual object.

[0221] For example, the at least one processor may be configured, individually or collectively, to perform calibration related to the gaze position using at least one of the stored gaze position or the displacement, based on detecting the gaze position adjacent to the location where the virtual object is displayed, after executing the function.

[0222] For example, the at least one processor may be configured, individually or collectively, to display the virtual object at a location where it was displayed prior to detecting the gaze position within the detection area, based on the virtual object ceasing to move.

[0223] For example, the at least one processor may be configured, individually or collectively, to determine the size of the detection area using information representing an error in the gaze position at the location where the virtual object is displayed.

[0224] For example, the at least one processor may be individually or collectively configured to move the virtual object by the distance less than the magnitude of the displacement during the time interval during which the gaze position is updated.

[0225] For example, the at least one processor may be individually or collectively configured to receive a pattern for switching the locked state, within a locked state of the wearable device, and to display the virtual object and other virtual objects.

[0226] As described above, in one embodiment, a non-transitory computer-readable storage medium comprising instructions may be provided. The instructions may be executed by a wearable device. The wearable device may include at least one display and at least one camera configured to be positioned toward an eye of a user wearing the wearable device. The instructions, when executed by the wearable device, may cause the wearable device to display a virtual object on the at least one display. The instructions, when executed by the wearable device, may cause the wearable device to determine a gaze position of the user using an image of the eye acquired from the at least one camera. The instructions, when executed by the wearable device, may cause the wearable device to determine information representing a displacement of the gaze positions detected at each of two time points while detecting the gaze positions within a detection area including the virtual object. The instructions, when executed by the wearable device, may cause the wearable device to determine a distance for movement of the virtual object within a range less than a magnitude of the displacement indicated by the information based on the determined information. The instructions, when executed by the wearable device, may cause the wearable device to move the virtual object by the determined distance in the direction of the displacement indicated by the information based on determining the distance to be greater than a threshold distance. The instructions, when executed by the wearable device, may cause the wearable device to stop moving the virtual object based on determining the distance to be less than the threshold distance.The above instructions, when executed by the wearable device, may cause the wearable device to execute a function associated with the virtual object based on the wearable device ceasing to move the virtual object.

[0227] As described above, in one embodiment, a non-transitory computer-readable storage medium comprising instructions may be provided. The instructions may be executed by a wearable device. The wearable device may include at least one display and at least one camera configured to be positioned toward an eye of a user wearing the wearable device. The instructions, when executed by the wearable device, may cause the wearable device to display a virtual object on the at least one display. The instructions, when executed by the wearable device, may cause the wearable device to determine a gaze position of the user using an image of the eye acquired from the at least one camera. The instructions, when executed by the wearable device, may cause the wearable device to determine information indicative of a displacement of the gaze position within the detection area based on detecting the gaze position within the detection area, which includes the virtual object. The instructions, when executed by the wearable device, may cause the wearable device to move the virtual object in the direction of the displacement. A distance of the virtual object moved in the direction may be less than a magnitude of the displacement. The instructions, when executed by the wearable device, may cause the wearable device to repeatedly move the virtual object in the direction while the distance exceeds a threshold distance. The instructions, when executed by the wearable device, may cause the wearable device to stop moving the virtual object using the gaze position based on determining that the distance is less than the threshold distance.The above instructions, when executed by the wearable device, may cause the wearable device to execute a function associated with the virtual object based on the wearable device ceasing to move the virtual object.

[0228] For example, the instructions, when executed by the wearable device, may cause the wearable device to move the virtual object by an amount equal to the displacement to which a specified ratio has been applied.

[0229] For example, the instructions, when executed by the wearable device, may cause the wearable device to store at least one of the gaze position or the displacement of the gaze position obtained while the virtual object was being moved, based on the wearable device ceasing to move the virtual object.

[0230] For example, the instructions, when executed by the wearable device, may cause the wearable device to perform calibration related to the gaze position using at least one of the stored gaze position or the displacement, based on detecting the gaze position adjacent to the location where the virtual object is displayed after executing the function.

[0231] For example, the instructions, when executed by the wearable device, may cause the wearable device to display the virtual object at a location where it was displayed prior to detecting the gaze location within the detection area based on the wearable device ceasing to move the virtual object.

[0232] For example, the instructions, when executed by the wearable device, may cause the wearable device to determine the size of the detection area using information representing an error in the gaze position at the location where the virtual object is displayed.

[0233] For example, the instructions, when executed by the wearable device, may cause the wearable device to move the virtual object by the distance less than the magnitude of the displacement during the time interval in which the gaze position is updated.

[0234] For example, the instructions, when executed by the wearable device, may cause the wearable device to display the virtual object and other virtual objects, within a locked state of the wearable device, configured to receive a pattern for switching the locked state.

[0235] As described above, in one embodiment, a method of a wearable device may be provided. The wearable device may include at least one display and at least one camera configured to be positioned toward the eyes of a user wearing the wearable device. The method may include an operation of displaying a plurality of virtual objects on the at least one display. The method may include an operation of determining a gaze position of the user using an image of the eye acquired from the at least one camera. The method may include an operation of moving the plurality of virtual objects so that the plurality of virtual objects move away from the gaze position based on detecting the gaze position within an overlapping area of ​​detection areas of the plurality of virtual objects. The method may include an operation of determining a first virtual object focused on by the user among the plurality of virtual objects using a displacement of the gaze position detected after the plurality of virtual objects are moved. The method may include an operation of storing the displacement of the gaze position as information to be used for calibration related to the gaze position adjacent to the first virtual object based on determining the first virtual object.

[0236] For example, the storing operation may include, after determining the first virtual object, moving the first virtual object according to the direction of the displacement of the gaze position. The distance by which the first virtual object is moved according to the direction may be less than the magnitude of the displacement of the gaze position. The storing operation may include, while the distance by which the first virtual object is moved exceeds a threshold distance, repeatedly moving the first virtual object according to the direction. The storing operation may include, based on identifying that the distance by which the first virtual object is moved is less than the threshold distance, stopping moving the first virtual object using the gaze position. The storing operation may include, based on stopping moving the first virtual object, executing a function related to the first virtual object.

[0237] For example, after determining the first virtual object, the action of moving the first virtual object may include an action of moving the first virtual object by the amount of the displacement to which the specified ratio is applied.

[0238] For example, the operation of determining the first virtual object may include an operation of comparing the directions in which each of the plurality of virtual objects has moved and the direction of the displacement.

[0239] As described above, according to one embodiment, a wearable device (e.g., wearable device (101) of FIG. 1, wearable device (1300) of FIGS. 13A-13B and / or wearable device (1400) of FIGS. 14A-14B) may include at least one display (e.g., display (110) of FIG. 2), at least one camera (e.g., camera (130) of FIG. 2) configured to be positioned toward the eyes of a user wearing the wearable device), at least one processor including processing circuitry (e.g., processor (210) of FIG. 2), and a memory (e.g., memory (215) of FIG. 2) including one or more storage media for storing instructions. The at least one processor may be individually or collectively configured to display a plurality of virtual objects (e.g., virtual objects 441, 442, 443, 444, 445, 446, 447, 448, 449 of FIGS. 9A to 9D and / or virtual objects 1011, 1012, 1013, 1014 of FIGS. 10A to 10B) on the at least one display. The at least one processor may be individually or collectively configured to determine a gaze position of the user (e.g., gaze position g2 of FIG. 9B and / or gaze position g1 of FIG. 10A) using an image of the eye acquired from the at least one camera. The at least one processor may be configured, individually or collectively, to move the plurality of virtual objects away from the gaze position based on detecting the gaze position within an overlapping area of ​​the detection areas of the plurality of virtual objects (e.g., the overlapping area (920) of FIG. 9A).The at least one processor may be individually or collectively configured to determine a first virtual object focused on by the user among the plurality of virtual objects by using a displacement of the gaze position (e.g., displacement (vg) of FIG. 10c) detected after the plurality of virtual objects have moved. The at least one processor may be individually or collectively configured to store the displacement of the gaze position as information to be used for calibration related to the gaze position adjacent to the first virtual object, based on determining the first virtual object.

[0240] For example, the at least one processor may be individually or collectively configured to, after determining the first virtual object, move the first virtual object in a direction of displacement of the gaze position. The distance by which the first virtual object is moved in the direction may be less than the magnitude of the displacement of the gaze position. The at least one processor may be individually or collectively configured to repeatedly move the first virtual object in the direction while the distance by which the first virtual object is moved exceeds a threshold distance. The at least one processor may be individually or collectively configured to stop moving the first virtual object using the gaze position based on identifying that the distance by which the first virtual object is moved is less than the threshold distance. The at least one processor may be individually or collectively configured to execute a function related to the first virtual object based on stopping moving the first virtual object.

[0241] For example, the at least one processor may be individually or collectively configured to move the first virtual object by an amount of the displacement to which a specified ratio is applied.

[0242] For example, the at least one processor may be configured, individually or collectively, to compare the directions in which each of the plurality of virtual objects has moved and the direction of the displacement.

[0243] As used herein, the term "if" will be understood to mean "when, upon," "in response to determining," or "in response to detecting," depending on the context. Similarly, "if it is determined to," or "if [the stated condition or event] is detected," will optionally be understood to mean "upon determining," or "in response to determining," "upon detecting [the stated condition or event]," or "in response to detecting [the stated condition or event]."

[0244] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0245] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0246] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program commands, including ROM, RAM, and flash memory. In addition, examples of other media may include recording media or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0247] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0248] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In wearable devices, At least one display; At least one camera configured to be positioned toward the eyes of a user wearing the wearable device; At least one processor comprising a processing circuit; and A memory comprising one or more storage media for storing instructions, The above instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Displaying a virtual object (o) on at least one display; Using the image of the eye acquired from at least one camera, the user's gaze position (g1) is determined; While detecting the gaze position within the detection area including the virtual object, information indicating the displacement of the gaze positions detected at each of the two points in time is determined; Based on the information determined above: Determine the distance for movement of the virtual object within a range less than the size of the displacement indicated by the above information; Based on determining the distance longer than the threshold distance, moving the virtual object by the determined distance in the direction of the displacement indicated by the information; and Based on determining that the distance is shorter than the threshold distance, stop moving the virtual object, Causing a function associated with said virtual object to be executed based on stopping the movement of said virtual object; Wearable devices.

2. In claim 1, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: causing the virtual object to move by the amount of the displacement to which the specified ratio is applied; Wearable devices.

3. In claims 1 and 2, when the instructions are individually or collectively executed by the at least one processor, the wearable device, Causing at least one of the gaze position or the displacement of the gaze position, acquired while the virtual object is being moved, to be stored based on stopping the movement of the virtual object; Wearable devices.

4. In claim 3, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: After executing the above function, based on detecting the gaze position adjacent to the position where the virtual object is displayed, causing calibration related to the gaze position to be performed using at least one of the stored gaze position or the displacement. Wearable devices.

5. In claims 1 to 4, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Based on the stopping of the movement of the virtual object, causing the virtual object to be displayed at the location where it was displayed before detecting the gaze location within the detection area. Wearable devices.

6. In claims 1 to 5, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: By using information indicating the error of the gaze position at the location where the virtual object is displayed, the size of the detection area is determined, Wearable devices.

7. In claims 1 to 6, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: causing the virtual object to move by the distance less than the magnitude of the displacement during the time interval in which the gaze position is updated; Wearable devices.

8. In claims 1 to 7, the instructions, when individually or collectively executed by the at least one processor, cause the wearable device to: Causing the virtual object and other virtual objects to be displayed, within the locked state of the wearable device, configured to receive a pattern for switching the locked state; Wearable devices.

9. A non-transitory computer-readable storage medium comprising instructions, wherein the instructions, when executed by a wearable device, cause the wearable device to include at least one display and at least one camera configured to be positioned toward the eyes of a user wearing the wearable device, wherein the wearable device: Displaying a virtual object on at least one display; Using the image of the eye acquired from at least one camera, determining the gaze position of the user; While detecting the gaze positions within the detection area including the virtual object, information indicating the displacement of the gaze positions detected at each of the two time points is determined; Based on the information determined above: Determine the distance for movement of the virtual object within a range less than the size of the displacement indicated by the above information; Based on determining the distance longer than the threshold distance, moving the virtual object by the determined distance in the direction of the displacement indicated by the information; and Based on determining that the distance is shorter than the threshold distance, stop moving the virtual object, Causing a function associated with said virtual object to be executed based on stopping the movement of said virtual object; Non-transitory computer-readable storage medium.

10. In claim 9, the instructions, when executed by the wearable device, cause the wearable device to: causing the virtual object to move by the amount of the displacement to which the specified ratio is applied; Non-transitory computer-readable storage medium.

11. In claims 9 to 10, the instructions, when executed by the wearable device, cause the wearable device to: Causing at least one of the gaze position or the displacement of the gaze position, acquired while the virtual object is being moved, to be stored based on stopping the movement of the virtual object; Non-transitory computer-readable storage medium.

12. In claim 11, the instructions, when executed by the wearable device, cause the wearable device to: After executing the above function, based on detecting the gaze position adjacent to the position where the virtual object is displayed, causing calibration related to the gaze position to be performed using at least one of the stored gaze position or the displacement. Non-transitory computer-readable storage medium.

13. In claims 9 to 12, the instructions, when executed by the wearable device, cause the wearable device to: Based on the stopping of the movement of the virtual object, causing the virtual object to be displayed at the location where it was displayed before detecting the gaze location within the detection area. Non-transitory computer-readable storage medium.

14. In claims 9 to 13, the instructions, when executed by the wearable device, cause the wearable device to: By using information indicating the error of the gaze position at the location where the virtual object is displayed, the size of the detection area is determined, Non-transitory computer-readable storage medium.

15. In claims 9 to 14, the instructions, when executed by the wearable device, cause the wearable device to: causing the virtual object to move by the distance less than the magnitude of the displacement during the time interval in which the gaze position is updated; Non-transitory computer-readable storage medium.

Images