Wearable device for providing virtual object for guiding capturing of image or video, and method thereof

Abstract

A wearable device according to one embodiment can acquire images by using one or more cameras. The wearable device can: display, on a first display, a screen representing the surrounding environment of the wearable device by using at least some of the images; in response to a first input for executing a camera application, visually emphasize a portion of the screen, corresponding to an area to be captured using the camera application, with respect to the remaining portion of the screen; and, while displaying the visually emphasized portion, capture an area in response to a second input for capturing the area, and display, on a second display, an indicator indicating that photographing is performed using the one or more cameras.

Description

Wearable device and method for providing a virtual object that guides the shooting of an image or video

[0001] The disclosure relates to a wearable device and method for providing a virtual object that guides shooting.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these electronic devices may be AR glasses and / or head-mounted devices (HMDs).

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] In one embodiment, a wearable device may include a housing, a first display positioned toward an eye of a user based on being worn by a user, a second display facing a second direction opposite a first direction in which the first display faces, one or more cameras, a memory including one or more storage media storing instructions, and at least one processor including a processing circuit. The at least one processor may be individually or collectively configured to execute the instructions and may be configured to cause the wearable device to acquire images using the one or more cameras. The at least one processor may be individually or collectively configured to cause the wearable device to control the first display to display a screen representing an environment surrounding the wearable device using at least a portion of the images. The at least one processor may be individually or collectively configured to cause the wearable device to receive a first input for executing a camera application while displaying the screen. At least one processor may be individually or collectively configured to cause the wearable device, in response to the first input, to control the first display to visually highlight a portion of the screen corresponding to an area to be captured using the camera application relative to the remaining portion of the screen. The at least one processor may be individually or collectively configured to cause the wearable device, while displaying the portion of the screen visually highlighted relative to the remaining portion of the screen, to receive a second input to capture the area displayed as the portion of the screen.At least one processor may be individually or collectively configured to cause the wearable device, in response to the second input, to capture the area. At least one processor may be individually or collectively configured to cause the wearable device to control the second display to display an indicator that provides notification of the performance of a photographing operation using the one or more cameras.

[0005] In one embodiment, a method of operating a wearable device may be provided. The wearable device may include a housing, a first display positioned toward the user's eyes based on the wearer being worn by the wearer, a second display facing a second direction opposite to a first direction in which the first display faces, and one or more cameras. The method may include an operation of acquiring images using the one or more cameras. The method may include an operation of controlling the first display to display a screen representing an environment surrounding the wearable device using at least a portion of the images. The method may include an operation of receiving a first input for executing a camera application while displaying the screen. The method may include an operation of controlling the first display to visually highlight a portion of the screen corresponding to an area to be captured using the camera application relative to the remaining portion of the screen, in response to the first input. The method may include receiving a second input for capturing the area indicated as the portion of the screen while displaying the portion of the screen visually highlighted with respect to the remaining portion of the screen. The method may include capturing the area in response to the second input. The method may include controlling the second display to display an indicator indicating that filming is being performed using the one or more cameras.

[0006] In one embodiment, a wearable device may include a housing, a first display disposed on a first side of the housing facing a face of a user based on the wearable device being worn by the user, a second display disposed on a second side of the housing facing an external environment of the wearable device based on the wearable device being worn by the user, a plurality of cameras configured to acquire a plurality of images of at least a portion of an external environment of the wearable device, a memory storing instructions, the memory including one or more storage media, and at least one processor including a processing circuit. The at least one processor may be individually or collectively configured to execute the instructions, and may be configured to cause the wearable device to display, through the first display, a composite image of at least a portion of the external environment generated based on the plurality of images and a view finder object at least partially overlapping the composite image. At least one processor may be individually or collectively configured to cause the wearable device, in an image capture mode, to display a first visual notification corresponding to the image capture mode through the second display while the composite image and the viewfinder object are displayed through the first display. At least one processor may be individually or collectively configured to cause the wearable device, in a video capture mode, to display a second visual notification corresponding to the video capture mode through the second display while the composite image and the viewfinder object are displayed through the second display.At least one processor may be individually or collectively configured to cause the wearable device, in the image capture mode or the video capture mode, to, in response to a user input, store at least a portion of the composite image corresponding to the viewfinder object in the memory.

[0007] According to one embodiment, a wearable device may include a housing, a display disposed in at least a portion of the housing and arranged in front of a user's eyes based on wearing the wearable device, a plurality of cameras for acquiring images of at least a portion of an external environment of the wearable device, a memory for storing instructions, and at least one processor including processing circuitry. The at least one processor may be individually or collectively configured to execute the instructions, and may be configured to cause the wearable device, in response to a first input, to display a view finder object on a composite image of the images, the composite image being displayed to represent a portion of the external environment beyond the display. At least one processor may be configured, individually or collectively, to cause the wearable device, while displaying the viewfinder object on the composite image, to change at least one of a position or a size of the viewfinder object in response to a second input for moving or resizing the viewfinder object. At least one processor may be configured, individually or collectively, to cause the wearable device, while displaying the viewfinder object on the composite image, to store a portion of the composite image corresponding to the viewfinder object in the memory in response to a third input for capturing.

[0008] In one embodiment, a method of operating a wearable device may be provided. The wearable device may include a housing, a display disposed within at least a portion of the housing and arranged in front of the eyes of a user wearing the wearable device, and a plurality of cameras for acquiring images of at least a portion of an external environment of the wearable device. The method may include, in response to a first input, displaying a viewfinder object on a composite image of the images, the composite image being displayed to represent a portion of the external environment beyond the display. The method may include, in response to a second input for moving or resizing the viewfinder object while displaying the viewfinder object on the composite image, changing at least one of a position or a size of the viewfinder object. The method may include, while displaying the viewfinder object on the composite image, an operation of storing a portion of the composite image corresponding to the viewfinder object in the memory in response to a third input for taking a picture.

[0009] In one embodiment, a non-transitory computer-readable storage medium comprising instructions may be provided. The instructions, when individually and / or collectively executed by at least one processor of a wearable device, the wearable device comprising a housing, a display disposed within at least a portion of the housing and arranged in front of the eyes of a user wearing the wearable device, and a plurality of cameras configured to acquire images of at least a portion of an external environment of the wearable device, may cause the wearable device to display a viewfinder object on the display, superimposed on a composite image of the images. The instructions, when executed by the at least one processor, may cause the wearable device to store a first portion of the composite image corresponding to the first location in the memory, in response to receiving an input for capturing while displaying the viewfinder object at a first location on the display. The instructions, when executed by at least one processor, may cause the wearable device to, in response to receiving an input for capturing while displaying the viewfinder object at a second location on the display, store a second portion of the composite image corresponding to the second location in the memory.

[0010] In one embodiment, a wearable device may include a housing, a display disposed in at least a portion of the housing and arranged in front of a user's eyes based on wearing the wearable device, a plurality of cameras configured to acquire images of at least a portion of an external environment of the wearable device, a memory storing instructions, and at least one processor including processing circuitry. The at least one processor may be individually and / or collectively configured to execute the instructions and cause the wearable device to display a viewfinder object on the display, superimposed on a composite image of the images. The at least one processor may be individually or collectively configured to cause the wearable device to store a first portion of the composite image corresponding to the first location in the memory, in response to receiving an input for capturing while displaying the viewfinder object on the display at a first location. At least one processor may be individually or collectively configured to cause the wearable device, in response to receiving an input for capturing while displaying the viewfinder object at a second location on the display, to store a second portion of the composite image corresponding to the second location in the memory.

[0011] The above-described and other aspects, features, and advantages of some embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0012] FIG. 1 is a diagram illustrating an example of a wearable device displaying a viewfinder object associated with a camera, according to various embodiments;

[0013] FIG. 2 is a block diagram illustrating an exemplary configuration of a wearable device according to various embodiments;

[0014] FIGS. 3A, 3B, and 3C are diagrams illustrating exemplary operations of a wearable device displaying a viewfinder object according to various embodiments;

[0015] FIG. 4 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments;

[0016] FIG. 5 is a diagram illustrating the operation of a wearable device that displays a composite image of images from multiple cameras according to various embodiments;

[0017] FIG. 6 is a diagram illustrating an exemplary operation of a wearable device for adjusting a focal length of at least one of a plurality of cameras according to various embodiments;

[0018] FIG. 7 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments;

[0019] FIGS. 8A, 8B, 8C, and 8D are diagrams illustrating exemplary operations of a wearable device for adjusting the position and / or size of a viewfinder object, according to various embodiments;

[0020] FIG. 9 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments;

[0021] FIGS. 10A, 10B, and 10C are diagrams illustrating exemplary operations of a wearable device for acquiring images and / or videos, according to various embodiments;

[0022] FIG. 11 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments;

[0023] FIG. 12 is a diagram illustrating exemplary operation of a wearable device with respect to a control handle displayed together with a viewfinder object, according to various embodiments;

[0024] FIG. 13 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments;

[0025] FIGS. 14A and 14B are diagrams illustrating exemplary operations of a wearable device in connection with a depth sensor, according to various embodiments;

[0026] FIG. 15 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments;

[0027] FIG. 16 is a diagram illustrating exemplary operations of a wearable device for tracking an external object according to various embodiments;

[0028] FIG. 17 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments;

[0029] FIGS. 18A and 18B are diagrams illustrating exemplary operations of a wearable device in relation to a virtual object, according to various embodiments;

[0030] FIG. 19 is a diagram illustrating an exemplary operation of a wearable device for acquiring video according to various embodiments;

[0031] FIG. 20 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments;

[0032] FIG. 21A is a perspective view of an exemplary wearable device according to various embodiments;

[0033] FIG. 21B is a perspective view illustrating an exemplary configuration of a wearable device according to various embodiments;

[0034] FIGS. 22A and 22B are perspective views illustrating exemplary exterior views of an exemplary wearable device according to various embodiments.

[0035] Hereinafter, various embodiments of the disclosure are described with reference to the attached drawings.

[0036] The various embodiments of the disclosure and the terminology used therein are not intended to limit the technology described in the disclosure to the various exemplary embodiments, but should be understood to encompass various modifications, equivalents, and / or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar components. The singular expressions may include plural expressions unless the context clearly indicates otherwise. In the disclosure, expressions such as “A or B,” “at least one of A and / or B,” “A, B, or C,” or “at least one of A, B, and / or C” can include all possible combinations of the listed items. Expressions such as “first,” “second,” “first,” or “second” can modify the corresponding components without regard to order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), said component may be directly connected to said other component, or may be connected via another component (e.g., a third component).

[0037] The term "module" as used in the disclosure includes a unit composed of hardware, software, or firmware, or any combination thereof, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integrally formed component or a minimum unit or portion thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0038] FIG. 1 is a diagram illustrating an example of a wearable device (101) displaying a viewfinder object (150) associated with a camera, according to various embodiments. The wearable device (101) may include a head-mounted display (HMD) wearable on the head of a user (110). The wearable device (101) may be referred to as a head-mounted display (HMD) device, a headgear electronic device, a glasses-type (or goggle-type) electronic device, a video see-through (VST) device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device, or the like. Although the external appearance of the wearable device (101) having the form of glasses is shown, the disclosure is not limited thereto. An example of a hardware configuration included in a wearable device (101) is described with reference to FIG. 2. An example of a structure of a wearable device (101) that can be worn on the head of a user (110) is described in more detail below with reference to FIGS. 21A, 21B, 22A, and / or 22B. The wearable device (101) may be referred to as an electronic device. For example, the electronic device may include an accessory (e.g., a strap) for attaching to the head of the user (110).

[0039] According to one embodiment, a wearable device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, when a user (110) wears the wearable device (101), the wearable device (101) may include at least one lens positioned adjacent to the user's (110) eyes. The wearable device (101) may combine ambient light passing through the lens with light emitted from a display of the wearable device (101). A display area of ​​the display may be formed within the lens through which the ambient light passes. Because the wearable device (101) combines the ambient light and the light emitted from the display, the user (110) may see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0040] According to one embodiment, the wearable device (101) can perform functions related to video see-through (VST) and / or virtual reality (VR). For example, in a state where a user (110) wears the wearable device (101), the wearable device (101) can include a housing that covers the eyes of the user (110). In the state, the wearable device (101) can include a display disposed on a first surface of the housing facing the eyes. The wearable device (101) can include at least one display that forms at least a portion of the housing of the wearable device (101) so as to be arranged in front of the eyes of the user (110) wearing the wearable device (101).

[0041] The wearable device (101) may include a camera disposed on a second side opposite the first side. The wearable device (101) may include one or more cameras for capturing images of at least a portion of an external environment. The wearable device (101) may include a plurality of cameras exposed through at least a portion of a housing of the wearable device (101) to capture images (or videos) of at least a portion of the external environment. Using the cameras, the wearable device (101) may capture images and / or videos representing ambient light. The wearable device (101) may output the images and / or videos within a display disposed on the first side, thereby allowing a user (110) to recognize the ambient light through the display. The displaying area (or displaying region) (or active area or active region) of the display arranged on the first surface may be formed by one or more pixels included in the display. The wearable device (101) may synthesize a virtual object into an image and / or video output through the display, thereby allowing the user (110) to recognize the virtual object together with a real object recognized by ambient light.

[0042] Referring to FIG. 1, an exemplary state of a wearable device (101) providing a user experience related to AR and / or VST is illustrated. A user (110) wearing the wearable device (101) can view an external environment (e.g., an external environment including external objects such as a chair (121) and / or a lamp (122)) through the display of the wearable device (101). Referring to FIG. 1, an exemplary screen (130) is illustrated including a visual object (141) corresponding to a chair (121) included in the external environment and a visual object (142) corresponding to a lamp (122). While a user (110) wearing a wearable device (101) stands facing an external environment including a chair (121) and a lamp (122), the wearable device (101) can display a screen (130) including visual objects (141, 142) corresponding to the chair (121) and the lamp (122), respectively. Within the screen (130), an image of the external environment may be referred to as a background image and / or an outer image. The operation of the wearable device (101) for displaying a background image related to a VST using one or more cameras is described in more detail below with reference to FIG. 5.

[0043] According to one embodiment, the wearable device (101) may display a user interface (UI) for controlling a camera that captures at least a portion of an external environment. For example, the wearable device (101) may display, within a screen (130) including an image of at least a portion of the external environment, a viewfinder object (150) superimposed on the image. An exemplary operation of the wearable device (101) displaying the viewfinder object (150) is described in more detail below with reference to FIGS. 3A, 3B, and / or 3C. The viewfinder object (150) having the shape of a rectangle (or a round-corner rectangle) may have a position and / or size linked to the external environment (or external space) displayed through the display of the wearable device (101). Using the viewfinder object (150), the wearable device (101) may guide the capture (or recording) of the external environment specified by the viewfinder object (150). For example, the wearable device (101) may obtain an image (160) and / or a video using the viewfinder object (150) superimposed on a background image related to the VST. In order to adjust the focus of the image (160) and / or the video obtained using the viewfinder object (150), the wearable device (101) may display a visual object (152) inside the viewfinder object (150). For example, an image (160) acquired (or stored) by a wearable device (101) may include a portion corresponding to a viewfinder object (150) within a background image.

[0044] For example, by using a viewfinder object (150) superimposed on a background image provided for VST, the wearable device (101) can guide a portion of the external environment corresponding to an image file and / or a video file without additional panels, windows (or, for example, activities of an Android operating system), and / or virtual objects. The wearable device (101) can obtain an image (160) and / or a video of the external environment corresponding to the viewfinder object (150) by using motions of the user (110) relative to the viewfinder object (150) (e.g., head, gaze, hand gestures, and / or speech of the user (110). The motion of the user (110) may include a hand gesture of the user (110) located away from the wearable device (101), a motion of the user (110) detected by an external electronic device (e.g., a remote controller) connected to the wearable device (101), and / or a motion of the user (110) associated with a dial (109) and / or button exposed externally through the housing of the wearable device (101).

[0045] Using the motion of the user (110) associated with the viewfinder object (150), the wearable device (101) can execute a function related to the camera. For example, the operation of the wearable device (101) for adjusting the focal length of the camera using the gaze of the user (110) toward the inside of the viewfinder object (150) is described with reference to FIG. 6. The operation of the wearable device (101) for adjusting the position and / or size of the viewfinder object (150) using the motion of the user (110) is described in more detail below with reference to FIGS. 8A to 8D. The operation of the wearable device (101) for acquiring an image (160) and / or a video in response to the motion of the user (110) is described in more detail below with reference to FIGS. 10A to 10C.

[0046] Hereinafter, with reference to FIG. 2, an exemplary hardware configuration of a wearable device (101) displaying a viewfinder object (150) is described in more detail.

[0047] FIG. 2 is a block diagram illustrating an exemplary configuration of a wearable device (101) according to various embodiments. The wearable device (101) of FIG. 2 may include the wearable device (101) of FIG. 1.

[0048] Referring to FIG. 2, a wearable device (101) according to one embodiment may include at least one of a processor (210) (e.g., including a processing circuit), a memory (215), a display (220), a camera (225), a sensor (230), and / or a communication circuit (235). The processor (210), the memory (215), the display (220), the camera (225), the sensor (230), and / or the communication circuit (235) may be electrically and / or operably coupled with each other by an electronic component, such as a communication bus (202). The type and / or number of hardware components included in the wearable device (101) is not limited to those illustrated in FIG. 2. For example, the wearable device (101) may include only some of the hardware components illustrated in FIG. 2.

[0049] A processor (210) of a wearable device (101) according to one embodiment may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the wearable device (101) may include one or more processors. The processor (210) may have a multi-core processor structure such as a dual core, a quad core, a hexa core, and / or an octa core. The multi-core processor structure of the processor (210) may include a structure based on a plurality of core circuits (e.g., a big-little structure) that are distinguished by power consumption, clock, and / or calculation amount per unit time. In one embodiment comprising a processor (210) having a multi-core processor architecture, the operations and / or functions of the present disclosure may be collectively performed by one or more cores included in the processor (210). The processor (210) may include various processing circuits and / or multiple processors. For example, as used herein, including in the claims, the term “processor” may include various processing circuits, including at least one processor, wherein one or more of the at least one processor may be configured to individually and / or collectively perform the various functions described herein in a distributed manner.As used herein, when "a processor," "at least one processor," and "one or more processors" are described as being configured to perform several functions, these expressions also cover, for example, but not limited to, situations where a processor performs some of the described functions and other processor(s) perform the remainder of the described functions, and situations where a single processor performs all of the described functions. Additionally, the at least one processor may comprise a combination of processors that perform the various described / described functions, e.g., in a distributed manner. The at least one processor may execute program instructions to achieve or perform the various functions.

[0050] The memory (215) of the wearable device (101) according to one embodiment may include a hardware component for storing data and / or instructions input to and / or output from the processor (210). The memory (215) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multi media card (eMMC). In one embodiment, the memory (215) may be referred to as storage.

[0051] In one embodiment, a display (220) of a wearable device (101) can output visualized information to a user of the wearable device (101) (e.g., the user (110) of FIG. 1). The display (220), which is arranged in front of the eyes of a user wearing the wearable device (101), can be arranged on at least a portion of a housing of the wearable device (101). For example, the display (220) can be configured to visualize information provided by a processor (210) including a circuit such as a graphic processing unit (GPU). The display (220) can include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (220) can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs). The disclosure is not limited thereto, and for example, if the wearable device (101) includes a lens for transmitting external light (or ambient light), the display (220) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (220) may be referred to as a display panel and / or a display module. The pixels included in the display (220) may be arranged to face either of the user's two eyes when the wearable device (101) is worn by the user. For example, the display (220) may include display areas (or active areas) corresponding to each of the user's two eyes.

[0052] In one embodiment, the camera (225) of the wearable device (101) may include optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The camera (225) may be referred to as an image sensor and may be included in the sensor (230) of FIG. 2. The plurality of optical sensors included in the camera (225) may be arranged in the form of a two-dimensional array. The camera (225) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously, and generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (225) may refer to, for example, one (a) two-dimensional frame data acquired from the camera (225). For example, video data captured using a camera (225) may refer to a sequence of a plurality of two-dimensional frame data obtained from the camera (225) according to, for example, a frame rate. The camera (225) may further include a flash light positioned toward the direction in which the camera (225) receives light and outputs light toward the direction.

[0053] According to one embodiment, the wearable device (101) may include a plurality of cameras, for example, cameras (225), arranged in different directions. Referring to FIG. 2, the camera (225) included in the wearable device (101) may include a gaze tracking camera and / or an outward camera. The gaze tracking camera may be arranged toward at least one of the two eyes of a user wearing the wearable device (101). The processor (210) may identify the direction of the user's gaze using images and / or videos acquired from the gaze tracking camera. The gaze tracking camera may include an infrared (IR) sensor. The gaze tracking camera may be referred to as an eye sensor, a gaze tracker, and / or an eye tracker.

[0054] For example, the external camera may be positioned facing the front of a user wearing the wearable device (101) (e.g., in a direction in which both eyes may face). The wearable device (101) may include multiple external cameras. The disclosure is not limited thereto, and the external camera may be positioned facing an external space. Using images and / or videos acquired from the external cameras, the processor (210) may identify external objects. For example, the processor (210) may identify the position, shape, and / or gesture (e.g., hand gesture) of a hand of a user wearing the wearable device (101) (e.g., the user (110) of FIG. 1) based on images and / or videos acquired from the external cameras.

[0055] According to one embodiment, a sensor (230) of a wearable device (101) may generate electrical information from non-electronic information related to the wearable device (101), which may be processed and / or stored by a processor (210) and / or a memory (215) of the wearable device (101). The information may be referred to as sensor data. The sensor (230) may include a global positioning system (GPS) sensor for detecting a geographic location of the wearable device (101), an image sensor, an audio sensor (e.g., a microphone and / or a microphone array including a plurality of microphones), an ambient light sensor, an inertial measurement unit (IMU) (e.g., an accelerometer, a gyro sensor, and / or a geomagnetic sensor), and / or a time-of-flight (ToF) sensor (or a ToF camera). As with the ToF sensor, the wearable device (101) may include a sensor configured to detect the distance between the wearable device (101) and an external object. A sensor that detects the distance between the wearable device (101) and an external object may be referred to as a depth sensor.

[0056] In one embodiment, the depth sensor included in the wearable device (101) may include a ToF sensor and / or a structured light (SL) sensor. The SL sensor may be referred to as an SL camera. The ToF sensor may be referred to as a ToF camera. The SL sensor may emit or output a light pattern (e.g., a plurality of dots) of a specific wavelength (e.g., an infrared wavelength). When an external object reflects the light pattern, the light pattern may be distorted by embossing on the surface of the external object. By detecting the reflected light for the light pattern, the SL sensor and / or the processor (210) connected to the SL sensor may recognize the distortion.

[0057] The ToF sensor can emit light of a specific wavelength (e.g., an infrared wavelength) in units of nanoseconds. The ToF sensor can measure the time it takes for light reflected by an external object to propagate to the ToF sensor. Using the measured time, the ToF sensor and / or the processor (210) can calculate or determine the distance between the external object and the wearable device (101). Using the ToF sensor, the processor (210) can output light in different directions. Using the time it takes for the reflected light for each of the outputted lights to propagate to the ToF sensor, the processor (210) can detect the distances between the external object and the wearable device (101) located in each of the directions. The two-dimensional distribution of the distances can be referred to as a depth map.

[0058] In one embodiment where the wearable device (101) includes a ToF sensor and an SL sensor, the processor (210) may use the SL sensor to detect an external object that is less than a specified distance (e.g., 10 m) away from the wearable device (101), and may use the ToF sensor to detect an external object that is more than the specified distance away from the wearable device (101). The disclosure is not limited thereto.

[0059] In one embodiment, the communication circuit (235) of the wearable device (101) may include circuitry for supporting transmission and / or reception of electrical signals between the wearable device (101) and an external electronic device. The communication circuit (235) may include, for example, at least one of a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit (235) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G new radio (NR), 6G and / or above-6G. In one embodiment, the communication circuit (235) may be referred to as a communication processor and / or a communication module.

[0060] According to one embodiment, one or more instructions (or commands) representing data to be processed, calculations to be performed, and / or operations to be performed by the processor (210) of the wearable device (101) may be stored in the memory (215) of the wearable device (101). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or a software application (hereinafter, “application”). For example, the wearable device (101) and / or the processor (210) may perform at least one of the operations described in more detail below in FIG. 4, FIG. 7, FIG. 9, FIG. 11, FIG. 13, FIG. 15, FIG. 17, and / or FIG. 20 when a set of a plurality of instructions distributed in the form of an operating system, firmware, driver, program, and / or application is executed. Hereinafter, the fact that an application is installed in a wearable device (101) may refer to one or more instructions provided in the form of an application being stored in a memory (215), for example, one or more applications being stored in a format executable by a processor (210) (e.g., a file having an extension specified by an operating system of the wearable device (101)). As an example, an application may include a program and / or a library related to a service provided to a user.

[0061] Referring to FIG. 2, programs installed in the wearable device (101) may be included in any one of different layers, including an application layer (240), a framework layer (250), and / or a hardware abstraction layer (HAL) (280), based on the target. For example, programs (e.g., modules or drivers) designed to target the hardware (e.g., display (220), camera (225), sensor (230), and / or communication circuit (235)) of the wearable device (101) may be included in the hardware abstraction layer (280). The framework layer (250) may be referred to as an XR framework layer in terms of including one or more programs for providing an XR (extended reality) service. For example, the layers illustrated in FIG. 2 may be logically (or for convenience of explanation) separated, and may not refer to, for example, the address space of the memory (215) being separated by the layers.

[0062] For example, within the framework layer (250), programs designed to target at least one of the hardware abstraction layer (280) and / or the application layer (240) (e.g., a position tracker (271), a space recognizer (272), a gesture tracker (273), and / or an eye-gaze tracker (274)) may be included. The programs included in the framework layer (250) may provide an API (application programming interface) that is executable (or callable) based on other programs.

[0063] For example, a program designed to target users of a wearable device (101) may be included within the application layer (240). As an example of programs classified into the application layer (240), an extended reality (XR) system user interface (UI) (241) and / or an XR application (242) are exemplified, but the disclosure is not limited thereto. For example, programs (e.g., software applications) included in the application layer (240) may call an application programming interface (API) to cause execution of functions supported by programs classified into the framework layer (250).

[0064] For example, the wearable device (101) may display one or more visual objects on the display (220) for performing interaction with the user based on the execution of the XR system UI (241). A visual object may refer to an object that can be deployed within a screen for transmitting and / or interacting with information, such as, for example, text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The wearable device (101) may provide the user with functions available within a virtual space based on the execution of the XR system UI (241).

[0065] Referring to FIG. 2, a lightweight renderer (243) and / or an XR plug-in (244) are illustrated to be included within the XR system UI (241), but are not limited thereto. For example, based on the XR system UI (241), the processor (210) may execute a lightweight renderer (243) and / or an XR plug-in (244) within the framework layer (250).

[0066] For example, the wearable device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (243). The lightweight renderer (243) may be referred to as a lightweight render pipeline in terms of defining a rendering pipeline that allows partial changes. The lightweight renderer (243) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the wearable device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (244). The XR plugin (244) can be referred to as an open XR native client from the perspective of defining (or setting up) the entire rendering pipeline.

[0067] For example, the wearable device (101) may display a screen representing at least a portion of a virtual space on the display (220) based on the execution of the XR application (242). The XR plug-in (244-1) included in the XR application (242) may include instructions that support functions similar to those of the XR plug-in (244) of the XR system UI (241). Descriptions of the XR plug-in (244-1) that overlap with those of the XR plug-in (244) may be omitted. The wearable device (101) may cause the execution of the virtual space manager (251) based on the execution of the XR application (242).

[0068] According to one embodiment, the wearable device (101) may provide a virtual space service based on the execution of the virtual space manager (251). For example, the virtual space manager (251) may include a platform for supporting the virtual space service. The wearable device (101) may identify a virtual space formed based on the user's location indicated by data acquired through the sensor (230) based on the execution of the virtual space manager (251), and may display at least a portion of the virtual space on the display (220). The virtual space manager (251) may be referred to as a composition presentation manager (CPM).

[0069] For example, the virtual space manager (251) may include a runtime service (252). As an example, the runtime service (252) may be referred to as an OpenXR runtime module (or an OpenXR runtime program). The wearable device (101) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (252). As an example, the wearable device (101) may perform rendering for a virtual space service to the user based on the execution of the runtime service (252). For example, a function related to a virtual space, executable by the application layer (240), may be supported based on the execution of the runtime service (252).

[0070] For example, the virtual space manager (251) may include a pass-through manager (253). Based on the execution of the pass-through manager (253), the wearable device (101) may display a screen representing a virtual space (e.g., screen (130) of FIG. 1) on the display (220), while displaying an image and / or video representing an actual space acquired through an external camera by overlaying it on at least a portion of the screen.

[0071] For example, the virtual space manager (251) may include an input manager (254). The wearable device (101) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (270) based on the execution of the input manager (254). The wearable device (101) may use the acquired data to identify a user input related to the wearable device (101). The user input may be related to a motion (e.g., a hand gesture), gaze, and / or speech of the user identified by the sensor (230) and / or the camera (225) (e.g., an external camera). The user input may be identified based on an external electronic device connected (or paired) via the communication circuit (235).

[0072] For example, the perception abstract layer (260) can be used for data exchange between the virtual space manager (251) and the perception service layer (270). From the perspective of being used for data exchange between the virtual space manager (251) and the perception service layer (270), the perception abstract layer (260) can be referred to as an interface. For example, the perception abstract layer (260) can be referenced as OpenPX. The perception abstract layer (260) can be used for a perception client and a perception service.

[0073] According to one embodiment, the recognition service layer (270) may include one or more programs for processing data acquired from a sensor (230) and / or a camera (225). The one or more programs may include at least one of a position tracker (271), a space recognizer (272), a gesture tracker (273), and / or an eye tracker (274). The type and / or number of the one or more programs included in the recognition service layer (270) are not limited to those illustrated in FIG. 2.

[0074] For example, the wearable device (101) can identify the pose of the wearable device (101) using the sensor (230) based on the execution of the position tracker (271). The wearable device (101) can identify the 6 degrees of freedom pose (6 dof pose) of the wearable device (101) using data acquired using the camera (225) and / or IMU (e.g., a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor) based on the execution of the position tracker (271). The position tracker (271) may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0075] For example, the wearable device (101) may obtain information for providing a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the wearable device (101) (or the user of the wearable device (101)) based on the execution of the space recognizer (272). The wearable device (101) may reconstruct the surrounding environment of the wearable device (101) in three dimensions using data obtained using an external camera based on the execution of the space recognizer (272). The wearable device (101) may identify at least one of a plane, a slope, and stairs based on the surrounding environment of the wearable device (101) reconstructed in three dimensions based on the execution of the space recognizer (272). The space recognizer (272) may be referred to as a scene understanding (SU) module (or a scene recognition program).

[0076] For example, the wearable device (101) may identify (or recognize) a pose and / or gesture of a hand of a user of the wearable device (101) based on the execution of the gesture tracker (273). As an example, the wearable device (101) may identify a pose and / or gesture of a hand of a user using data acquired from an external camera based on the execution of the gesture tracker (273). As an example, the wearable device (101) may identify a pose and / or gesture of a hand of a user based on data (or images) acquired using an external camera based on the execution of the gesture tracker (273). The gesture tracker (273) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0077] For example, the wearable device (101) may identify (or track) eye movements of a user of the wearable device (101) based on the execution of the gaze tracker (274). As an example, the wearable device (101) may identify eye movements of the user using data acquired from a gaze tracking camera based on the execution of the gaze tracker (274). The gaze tracker (274) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0078] In one embodiment, the processor (210) of the wearable device (101) may execute the pass-through manager (253) to display an image of the external environment (e.g., a background image of the screen (130) of FIG. 1) within the display (220). To represent a portion of the external environment beyond the display (220), the processor (210) may display a composite image of images from external cameras. On the composite image displayed within the display (220), the processor (210) may display a viewfinder object (e.g., the viewfinder object (150) of FIG. 1). While displaying the viewfinder object on the composite image, the processor (210) may receive or detect an input for moving or resizing the viewfinder object. In response to the input, the processor (210) may change at least one of the position or the size of the viewfinder object. While displaying a viewfinder object on a composite image, the processor (210) may receive or detect an input for shooting (hereinafter, referred to as a shooting input). In response to the shooting input, the processor (210) may store, in the memory (215), a portion of the composite image corresponding to the viewfinder object. In response to the shooting input, the processor (210) may store, in the memory (215), a portion of the composite image associated with or designated by the viewfinder object. The portion of the composite image may be stored in the format of an image and / or a video.

[0079] A software application (e.g., a camera application) for acquiring images and / or videos using a viewfinder object may be installed in the memory (215) of the wearable device (101). Hereinafter, exemplary operations of the wearable device (101) related to user input for executing the software application will be described in more detail with reference to FIGS. 3A, 3B, and / or 3C.

[0080] FIGS. 3A, 3B, and 3C are diagrams illustrating exemplary operations of a wearable device (101) displaying a viewfinder object (150) according to various embodiments. The wearable device (101) of FIGS. 1-2 and / or the processor (210) of FIG. 2 may perform the operations of the wearable device (101) described with reference to FIGS. 3A, 3B, and 3C (which may be referred to as FIGS. 3A-3C).

[0081] Referring to FIGS. 3A to 3C , exemplary screens (301, 302, 303, 304, 305, and 306) displayed by the wearable device (101) are illustrated. Referring to FIG. 3A , the wearable device (101) may display icons (e.g., a B icon, a C icon, a D icon, an E icon, and / or an F icon) corresponding to each of a plurality of software applications installed in the wearable device (101) within the screen (301). The screen (301) is a screen for providing a list of a plurality of software applications installed in the wearable device (101) and may be referred to as a home screen and / or a launcher screen. Within the screen (301), the wearable device (101) may display a panel (322) for providing a list of frequently executed software applications. The panel (322) may be referred to as a dock. Within the panel (322), the wearable device (101) may display icons representing frequently executed software applications (e.g., a G icon and / or an H icon). Within the panel (322), the wearable device (101) may display the current time and / or information related to the battery of the wearable device (101).

[0082] Within the exemplary screen (301) of FIG. 3A, along with the icons, the wearable device (101) may display an icon (320) corresponding to a software application (e.g., a camera application) for capturing images and / or videos using a camera (225) (e.g., an external camera) included in the wearable device (101). While an embodiment is described in which the wearable device (101) displays an icon (320) representing the software application, the disclosure is not limited thereto, and the wearable device (101) may display text, images, and / or videos representing the software application.

[0083] In one embodiment, the wearable device (101) may receive an input for executing a camera application. The input may include an input for selecting an icon (320) representing the camera application within the screen (301). The input for selecting the icon (320) may include a hand gesture detected by the hand (112) of the user (110). For example, the wearable device (101) may use a camera (e.g., camera (225) of FIG. 2 and / or an external camera) to obtain an image and / or video of a body part including the hand (112) of the user (110). Upon detecting the hand (112), the wearable device (101) may display a virtual object (340) corresponding to the hand (112) within the screen (301). The virtual object (340) may include a three-dimensional graphical object representing the posture of the hand (112). For example, the posture and / or orientation of the virtual object (340) may correspond to the posture and / or orientation of the hand (112) detected by the wearable device (101).

[0084] Referring to FIG. 3A, while displaying a virtual object (340), the wearable device (101) may display a virtual object (342) having a shape of a line extending from the virtual object (340). The virtual object (342) may be referred to as a ray, a ray object, a cursor, a pointer, and / or a pointer object. The virtual object (342) may have a shape of a line extending from a portion of the hand (112) (e.g., a palm and / or a designated finger such as the index finger). Although a virtual object (342) having a curved shape is illustrated, the disclosure is not limited thereto. A user (110) wearing the wearable device (101) may change the position and / or direction of the virtual object (340) and / or the virtual object (342) within the screen (301) by moving the hand (112).

[0085] Referring to FIG. 3A, while a virtual object (342) in the form of a line is displayed on an exemplary screen (301) extending toward an icon (320), the wearable device (101) can detect or identify a pinch gesture of a hand (112). For example, after acquiring an image (311) of a hand (112) in which the fingertips of all fingers included in the hand (112) are spaced apart from each other, the wearable device (101) can acquire an image (312) of a hand (112) including at least two fingers in the form of a ring, in which at least two fingertips (e.g., the fingertips of the thumb and the index finger) are in contact with each other. The wearable device (101) that acquired the image (312) can detect a pinch gesture expressed by at least two fingers in the form of a ring. The duration of a pinch gesture may refer to the period of time during which the fingertips of at least two fingers of a hand (112) are in contact with each other, for example, as in image (312). The pinch gesture may correspond to, or be mapped to, a click and / or tap gesture.

[0086] For example, in response to a pinch gesture detected while displaying a screen (301) including a virtual object (342) extended toward an icon (320), the wearable device (101) may launch a camera application corresponding to the icon (320). While an exemplary operation of launching the camera application using a hand gesture such as a pinch gesture is described, the disclosure is not limited thereto. For example, the wearable device (101) may launch the camera application in response to a user's utterance (e.g., "launch the camera application") and / or rotation and / or pressing of a dial (e.g., dial (109) of FIG. 1 ).

[0087] Referring to FIG. 3A, an exemplary screen (302) displayed by a wearable device (101) executing a camera application is illustrated. Referring to the screen (302), in response to an input for executing the camera application, the wearable device (101) may display a viewfinder object (150) on a composite image of external cameras. For example, by displaying a composite image including visual objects corresponding to external objects included in the external environment (e.g., a visual object (391) corresponding to a picture of a dog, and a visual object (392) corresponding to a flower pot), the wearable device (101) may provide a user experience similar to that of a user (110) viewing the external environment. Referring to FIG. 3A, a viewfinder object (150) having a rectangular shape surrounding a visual object (391) in the composite image is illustrated.

[0088] In one embodiment, the wearable device (101) may display a visual object (152) related to a focal length on the inside of the viewfinder object (150). The wearable device (101) may display a control handle (330) at a location adjacent to the viewfinder object (150). While the control handle (330) is illustrated adjacent to the right side of the viewfinder object (150), the location, size, and / or shape of the control handle (330) is not limited thereto. Functions executable using the control handle (330) are described in more detail below with reference to FIGS. 12 to 18B. The control handle (330) may be referred to as a handle object.

[0089] In one embodiment, the wearable device (101) can execute multiple software applications substantially simultaneously (e.g., multitasking). Referring to FIG. 3B, the wearable device (101) can display a screen (303) including virtual objects (351, 352, 353) provided from one or more software applications. Each of the virtual objects (351, 352, 353) can be a two-dimensional panel provided by the execution of a software application. While displaying a composite image based on VST, the wearable device (101) can display the virtual objects (351, 352, 353) as an overlay on the composite image. While displaying the screen (303), the wearable device (101) can receive or detect an input for executing a camera application. In response to the above input, the wearable device (101) may display a viewfinder object (150) superimposed on the composite image, such as the screen (304).

[0090] To display a viewfinder object (150) superimposed on the composite image, the wearable device (101) may stop displaying or hide at least one of the virtual objects (351, 352, 353) that were displayed prior to executing the camera application (or prior to displaying the viewfinder object (150). For example, in response to an input for executing the camera application, a virtual object (352) positioned at a location where the viewfinder object (150) provided by the camera application is to be displayed may be removed from the screen (304) or displayed with a specified opacity (or transparency) (e.g., 100% or less than 100%). For example, a portion of a virtual object (353) overlapping a viewfinder object (150) may be removed from the screen (304) or displayed with a specified transparency (e.g., 100% or less than 100%). When the viewfinder object (150) is moved within the screen (304), the wearable device (101) may remove another virtual object overlapping the viewfinder object (150), or adjust (or set) the transparency of the other virtual object to the specified transparency.

[0091] Referring to FIG. 3C, the wearable device (101) can display a screen (305) including a virtual object (354). While displaying the screen (305), the wearable device (101) that has received an input for executing a camera application can display a viewfinder object (150). Referring to an exemplary screen (306) of FIG. 3C, the wearable device (101) that has detected a virtual object (354) positioned on a location where the viewfinder object (150) is to be displayed can move the virtual object (354) within the screen (306) to a location that does not overlap with the viewfinder object (150). For example, within the screen (306), the virtual object (354) can be positioned in an area different from an area inside the viewfinder object (150). Since the virtual object (354) is placed on a portion of the screen (306) outside the viewfinder object (150), a user (110) wearing the wearable device (101) can accurately recognize the external environment to be captured by the viewfinder object (150).

[0092] Hereinafter, with reference to FIG. 4, an exemplary operation of a wearable device (101) displaying a viewfinder object (150) is described in more detail.

[0093] FIG. 4 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments. The wearable device (101) of FIGS. 1 to 2 and / or the processor (210) of FIG. 2 may perform operations of the wearable device described with reference to FIG. 4. The operations of the wearable device of FIG. 4 may be related to the operations of the wearable device (101) described with reference to FIGS. 3A to 3C. Instructions stored in a memory of the wearable device (e.g., the memory (215) of FIG. 2) may, when executed by the processor of the wearable device, cause the wearable device to perform the operations of FIG. 4 (or the processor itself may be configured to cause).

[0094] Referring to FIG. 4, in operation (410), according to one embodiment, a processor of a wearable device may, in response to an input for executing a camera application, display a viewfinder object superimposed on an image associated with a pass-through. The image associated with a pass-through may include a composite image of images from cameras of the wearable device (e.g., camera (225) of FIG. 2 ). The image associated with a pass-through may be displayed to maintain visibility of an external environment beyond a display positioned in front of the eyes of a user wearing the wearable device. By superimposing one or more virtual objects on the image of operation (410), the processor may display one or more virtual objects combined with a real environment. The input of operation (410) may include an input for executing a camera application, as described with reference to FIGS. 3A to 3C .

[0095] In one embodiment, the processor may guide at least a portion of the image to be captured by user input within the image of the real environment using an image overlaid with a viewfinder object. The input of operation (410) may be identified or detected by a hand gesture (e.g., a pinch gesture) associated with a virtual object representing a camera application (e.g., icon (320) of FIG. 3A ). In response to the input of operation (410), the wearable device may display a composite image of images from external cameras and a viewfinder object (e.g., viewfinder object (150) of FIG. 1 ) superimposed on the composite image. If the wearable device includes multiple external cameras, the processor may obtain or generate a composite image of the images by compositing (e.g., stitching) the images from the multiple external cameras. The operation of the processor displaying the composite image is described in more detail below with reference to FIG. 5 .

[0096] Referring to FIG. 4, in operation (420), a processor of a wearable device according to one embodiment may receive an input for taking a picture while displaying a viewfinder object having a position and / or size adjustable by a user's motion. The input of operation (420) may include a picture input. The position and / or size of the viewfinder object in the display of operation (420) may be changed by an exemplary input described with reference to FIGS. 8A to 8D. The position of the viewfinder object may be moved three-dimensionally within the virtual space. For example, the processor may move or adjust the position and / or size of the viewfinder object along the x-axis, the y-axis, and / or the z-axis.

[0097] Prior to receiving the input of the action (420), the processor may detect other inputs for executing functions related to the viewfinder object. The other inputs may include inputs for obtaining depth information (e.g., a depth map) using a depth sensor. The other inputs may include inputs for combining or synthesizing one or more virtual objects displayed on a display with images and / or videos of the external environment obtained through a camera. The other inputs may include inputs for tracking external objects and obtaining images and / or videos related to the external objects.

[0098] Referring to FIG. 4, in operation (430), according to one embodiment, a processor of a wearable device may store an image and / or video corresponding to a viewfinder object. In response to an input for capturing in operation (420), the processor may perform operation (430). The processor, which displays a viewfinder object on the image of operation (410), may perform operation (430) to obtain or store an image and / or video including at least a portion of the image of operation (410). When obtaining an image in response to an input of operation (420), the processor may store the image corresponding to the viewfinder object at the time when the input of operation (420) is received. When obtaining a video in response to an input of operation (420), the processor may initiate obtaining a video corresponding to the viewfinder object from the time when the input of operation (420) is received. The video acquired from the above point can be completed by additional input to stop recording of the video.

[0099] Hereinafter, with reference to FIG. 5, an exemplary operation of a wearable device that displays an image related to the pass-through of operation (410) and a viewfinder object superimposed on the image is described in more detail.

[0100] FIG. 5 is a diagram illustrating exemplary operations of a wearable device (101) that displays a composite image (520) of images from multiple cameras according to various embodiments. The wearable device (101) of FIGS. 1 to 2 and / or the processor (210) of FIG. 2 may perform operations of the wearable device (101) described with reference to FIG. 5. Operations of the wearable device (101) described with reference to FIG. 5 may be related to at least one of the operations of FIG. 4 (e.g., operation (410)).

[0101] Referring to FIG. 5, an exemplary state of a wearable device (101) positioned toward an external space including an external object such as a flower pot (510) is illustrated. Within the exemplary state of FIG. 5, the wearable device (101) may acquire images and / or videos of the external space using one or more cameras (e.g., camera (225) of FIG. 2). In one embodiment including multiple external cameras, a composite image (520) of the external space may be acquired or generated by synthesizing images of the multiple external cameras.

[0102] Referring to an exemplary composite image (520) of FIG. 5, the composite image (520) may represent at least a portion of an external space including the front of the wearable device (101). The wearable device (101) may generate the composite image (520) by combining images of a plurality of external cameras having different fields-of-views (FoVs) and / or different directions on a virtual two-dimensional plane and / or three-dimensional curved surface. The plane and / or the curved surface on which the composite image (520) is generated may be referred to as a skybox. The composite image (520) may be used to provide a background image and / or an external image.

[0103] A composite image (520) generated by the wearable device (101) may be displayed on a display (e.g., display (220) of FIG. 2) to provide a user experience related to VST. Referring to FIG. 5, an exemplary screen (530) including at least a portion of the composite image (520) is illustrated. In one embodiment where the display of the wearable device (101) includes display areas directed toward each of the user's two eyes, the screen (530) may be displayed in at least one of the display areas. The location within the display where the composite image (520) is displayed and / or the size of the composite image (520) may be related to the positional relationship between the wearable device (101) and external space. Within a screen (530) including at least a portion of a composite image (520), a position, size, and / or distance (e.g., binocular disparity) of a visual object (522) corresponding to a flower pot (510) may correspond to the position, size, and / or distance of the flower pot (510) as seen from the position of the wearable device (101), respectively.

[0104] The disclosure is not limited thereto, and in one embodiment including one external camera, the wearable device (101) may display a screen (530) having an image of the one external camera as a background image. In one embodiment including one external camera, the external camera may be positioned to face the front of a user (110) wearing the wearable device (101), or to face at least a portion of an external space including the front.

[0105] Within the screen (530) of FIG. 5, the wearable device (101) may receive or detect an input for executing a camera application, as described above with reference to FIGS. 3A to 3C. In response to the input, the wearable device (101) may display a viewfinder object (150) within the screen (530). The viewfinder object (150) may be superimposed on a portion of a composite image (520) included in the screen (530) to guide an image and / or video to be acquired by the photographing input. In response to the photographing input, the wearable device (101) may store a portion of the composite image (520) corresponding to the viewfinder object (150) as an image (540). Referring to the exemplary image (540) of FIG. 5, a portion of the external space represented by the image (540) may be matched to a portion of the composite image (520) designated by the viewfinder object (150). The disclosure is not limited thereto, and the wearable device (101) may, in response to the photographing input, perform video recording of a portion of the composite image (520) corresponding to the viewfinder object (150), thereby acquiring or storing the video.

[0106] As described above, according to one embodiment, the wearable device (101) can customize at least a portion of an image (540) (or video) to be stored within the composite image (520) using the viewfinder object (150). Hereinafter, with reference to FIG. 6, an exemplary operation of the wearable device (101) for adjusting the focal length of at least one camera included in the wearable device (101) prior to receiving a photographing input is described in more detail.

[0107] FIG. 6 is a diagram illustrating an exemplary operation of a wearable device (101) for adjusting a focal length of at least one of a plurality of cameras according to various embodiments. The wearable device (101) of FIGS. 1 and 2 and / or the processor (210) of FIG. 2 may perform the operation of the wearable device (101) described with reference to FIG. 6. The operation of the wearable device (101) described with reference to FIG. 6 may be related to at least one of the operations of FIG. 4.

[0108] Referring to FIG. 6, screens (302, 603) displayed by a wearable device (101) executing a camera application are illustrated. The screen (302) of FIG. 6 may correspond to the screen (302) of FIG. 3A. According to one embodiment, the wearable device (101) may acquire an image (610) of at least one eye of a user (110) wearing the wearable device (101) using a camera (e.g., the camera (225) of FIG. 2 and / or an eye tracking camera). For example, the camera may be positioned to face the eye of the user (110). The wearable device (101) can determine or identify the direction of at least one eye (e.g., direction of gaze) from the image (610) by using feature points (or key points) included in the image (610) or by using a neural network trained to obtain information related to the eyes. While wearing the wearable device (101), the user's (110) two eyes can be directed toward a specific point of a display covering the two eyes. The wearable device (101) can calculate or identify the direction of gaze toward the specific point using the image (610).

[0109] Referring to FIG. 6, while displaying the screen (302), the wearable device (101) can determine the direction of the eyes toward the point (p1) within the screen (302) using the image (610). Referring to FIG. 6, the wearable device (101) that has detected the direction toward a portion of the composite image designated by the viewfinder object (150) can change the focal length of at least one of the plurality of cameras that acquire the composite image using a portion of the external environment corresponding to the direction. For example, the wearable device (101) that has detected the direction of the gaze toward the point (p1) within the viewfinder object (150) can change the focal length of at least one of the external cameras using an external object arranged in the direction of the wearable device (101) corresponding to the point (p1).

[0110] Referring to FIG. 6, an exemplary screen (603) displayed by the wearable device (101) is illustrated after changing the focal length using the direction of the gaze toward the point (p1). The wearable device (101) can move a visual object (152) in the form of a crosshair to the point (p1). Within the screen (603), the center of the visual object (152) in the form of a crosshair can correspond to the point (p1) within the display toward which both eyes of the user (110) wearing the wearable device (101) are directed. The wearable device (101) can control at least one of the external cameras (e.g., a camera associated with a portion of the composite image corresponding to the viewfinder object (150)) to have a focal length corresponding to an external object corresponding to the point (p1) (e.g., an external object visible at the point (p1) of the composite image displayed through the display). Since the focal length of at least one of the cameras driven to display the composite image is changed based on the point (p1), an external object (e.g., a wall surface) related to the point (p1) can be clearly seen within the screen (603), and other external objects (e.g., a flower pot and / or a picture of an animal) other than the external object can be blurred.

[0111] As described above, the wearable device (101) can change the focal length of at least one camera driven to provide external space by using the direction of the gaze toward the point (p1) within the viewfinder object (150). Hereinafter, with reference to FIG. 7, an exemplary operation of the wearable device (101) described with reference to FIG. 6 will be described in more detail.

[0112] FIG. 7 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments. The wearable device (101) of FIGS. 1 to 2 and / or the processor (210) of FIG. 2 may perform the operations of the wearable device described with reference to FIG. 7. The operations of the wearable device of FIG. 7 may be related to the operations of the wearable device (101) described with reference to FIG. 6. Instructions stored in a memory of the wearable device (e.g., memory (215) of FIG. 2) may cause the wearable device to perform the operations of FIG. 7 when executed by the processor of the wearable device.

[0113] Referring to FIG. 7, within operation (710), a processor of a wearable device according to one embodiment may display a viewfinder object (e.g., the viewfinder object (150) of FIG. 1 and / or FIG. 6) superimposed on a composite image (e.g., the composite image (520) of FIG. 5). The processor may perform operation (710) of FIG. 7 similarly to operation (410) of FIG. 4 to display the viewfinder object.

[0114] Referring to FIG. 7, in operation (720), a processor of a wearable device according to one embodiment can detect a direction of a gaze toward a point within a viewfinder object. The processor can detect the direction of operation (720) using a camera for gaze tracking (e.g., a gaze tracking camera). For example, using image (610) of FIG. 6, the processor can detect the direction of the gaze. If the direction of the gaze is toward a point within the viewfinder object (e.g., point (p1) of FIG. 6), the processor can perform an operation of controlling at least one camera using the gaze.

[0115] Referring to FIG. 7, in operation (730), a processor of a wearable device according to one embodiment may adjust a focal length of a camera using a direction of a detected gaze. The processor may change the focal length of operation (730) using a distance between an external object positioned in the direction of the gaze and the wearable device. During, before, or after performing operation (730), the processor may move a location of a visual object (e.g., a visual object (152) of FIG. 1 and / or FIG. 6) associated with the focal length to a location within the display (or a location within a viewfinder object) that overlaps the direction of operation (720).

[0116] In response to a received photographing input after adjusting the focal length of operation (730), the processor of the wearable device may store a portion of the composite image corresponding to the viewfinder object in the format of an image and / or video. In response to the photographing input, the processor may store an image and / or video having the adjusted focal length of operation (730).

[0117] Hereinafter, exemplary operations of a wearable device for adjusting the position and / or size of a viewfinder object within a display are described in more detail with reference to FIGS. 8A, 8B, 8C, and 8D (which may be referred to as FIGS. 8A to 8D), and / or FIG. 9.

[0118] FIGS. 8A, 8B, 8C, and 8D are diagrams illustrating exemplary operations of a wearable device (101) for adjusting a position and / or size of a viewfinder object (150) according to various embodiments. The wearable device (101) and / or the processor (210) of FIGS. 1 to 2 may perform the operations of the wearable device (101) described with reference to FIGS. 8A to 8D.

[0119] Referring to FIGS. 8A to 8D , screens (801, 802, 803, 804, 805, 806) displayed by a wearable device (101) executing a camera application are illustrated. The wearable device (101) executing the camera application can display a viewfinder object (150) as an overlay on a composite image (e.g., the composite image (520) of FIG. 2 ). Referring to the exemplary screen (801) of FIG. 8A , while a user (110) wearing the wearable device (101) looks in a direction (d1), the wearable device (101) can display a viewfinder object (150) superimposed on a composite image representing a portion of external space corresponding to the direction (d1).

[0120] The viewfinder object (150) may be anchored to a specific point (e.g., the center point of the display) of the screen (801) (or display). For example, when the direction (d1) of the user's (110) head changes, a portion of the composite image overlapping the viewfinder object (150) anchored to the specific point of the display may change, and the position and / or size of the viewfinder object (150) within the display may be maintained. Referring to FIG. 8A, when the user (110) turns his / her head toward the direction (d2) while displaying the screen (801), the wearable device (101) may display a composite image associated with the direction (d2).

[0121] For example, within the screen (802) displayed while the user (110) is looking in the direction (d2), the wearable device (101) may display a viewfinder object (150) centered on the screen (802). Referring to the screens (801, 802), the wearable device (101) may maintain the position and / or size of the viewfinder object (150) on the display while the head of the user (110) rotates. For example, the viewfinder object (150) may have 0 DoF (degree of freedom) and may follow the FoV of the user (110).

[0122] The reference position and / or size of the viewfinder object (150) on the display can be changed by a user input related to the viewfinder object (150). Referring to FIG. 8B, the wearable device (101) that detects a body part including a hand (112) of a user (110) can display a virtual object (340) corresponding to the hand (112) within the screen (801). Together with the virtual object (340), the wearable device (101) can display a virtual object (342) (e.g., a pointer object) extending from a point of the composite image related to the body part including the hand (112) (e.g., a point where the virtual object (340) is displayed). Using the direction and / or position of the hand (112), the wearable device (101) can change the position and / or direction of the virtual objects (340, 342) within the screen (801).

[0123] Referring to FIG. 8B, while displaying a virtual object (342) toward a boundary line (150-1) of a viewfinder object (150), the wearable device (101) can detect or receive an input for moving the viewfinder object (150) using the shape of the hand (112). For example, while the direction of the virtual object (342) corresponding to the direction of the hand (112) is toward the boundary line (150-1), the wearable device (101) can continuously acquire images (311, 312). Using the continuously acquired images (311, 312), the wearable device (101) can detect a pinch gesture of the hand (112). For example, an input for moving a viewfinder object (150) can be detected by the direction of the hand (112) toward the boundary line (150-1) of the viewfinder object (150) and a pinch gesture of the hand (112).

[0124] Referring to FIG. 8B, the wearable device (101) that receives an input for moving the viewfinder object (150) can change the position of the viewfinder object (150) using the position and / or direction of the hand (112). For example, while the hand (112) of the right-handed user (110) moves from the right side of the user (110) to the left, the wearable device (101) can display the viewfinder object (150) moved to the left, as shown in the screen (802). While the shape of the hand (112) maintains the shape of a pinch gesture, the wearable device (101) can move the viewfinder object (150) according to the position and / or direction of the hand (112).

[0125] The wearable device (101) can move the viewfinder object (150) three-dimensionally according to the path of the hand (112) that maintains the pinch gesture. For example, the wearable device (101) can adjust the horizontal direction and / or vertical direction of the display, as well as the binocular parallax and / or depth value, to cause the viewfinder object (150) to move away from or toward the wearable device (101). When the viewfinder object (150) moves away from the wearable device (101) by the hand (112) that maintains the pinch gesture, the size of the viewfinder object (150) displayed through the display can be reduced. When the viewfinder object (150) approaches the wearable device (101) by a hand (112) maintaining a pinch gesture, the size of the viewfinder object (150) displayed through the display can be enlarged.

[0126] While moving the viewfinder object (150) using the direction of the virtual object (342) corresponding to the hand (112) that maintains the pinch gesture, the wearable device (101) can detect or determine whether the shape of the hand (112) changes to a shape different from the pinch gesture. For example, if two fingers that were in contact with each other for the pinch gesture separate from each other, the wearable device (101) can stop moving the viewfinder object (150) along the path of the hand (112). The viewfinder object (150) can be fixed at the position within the display at the time when the cessation of the pinch gesture is identified. For example, if the head of the user (110) is rotated after the viewfinder object (150) is re-fixed at a specific position at the above point in time, the wearable device (101) can maintain the position of the viewfinder object (150) within the display at the specific position.

[0127] In one embodiment, the wearable device (101) can detect or receive an input for resizing the viewfinder object (150). Referring to an exemplary screen (801) of FIG. 8C, while displaying a virtual object (342) toward a vertex (150-2) of the viewfinder object (150) from a virtual object (340) according to the direction of the hand (112), the wearable device (101) can detect a pinch gesture of the hand (112). In response to the pinch gesture detected while displaying the virtual object (342) toward the vertex (150-2) of the viewfinder object (150) having a rectangular shape, the wearable device (101) can detect an input for resizing the viewfinder object (150). In response to the above input, the wearable device (101) can change the size of the viewfinder object (150) using the position and / or direction of the virtual object (342), which is a pointer object.

[0128] Referring to FIG. 8C, the wearable device (101) can display a screen (804) including a viewfinder object (150) having an increased size by using the direction and / or position of the hand (112) detected using an image (e.g., images (311, 312)). While the pinch gesture of the hand (112) is maintained, the wearable device (101) can use the position and / or direction of the hand (112) to increase or decrease the size of the viewfinder object (150) on the display. For example, the wearable device (101) can provide a user experience in which a vertex (150-2) of the viewfinder object (150) toward which the virtual object (342) is directed follows the virtual object (342) moved by the hand (112) making the pinch gesture. When the pinch gesture of the hand (112) ends, the wearable device (101) may stop adjusting the size of the viewfinder object (150).

[0129] Adjusting the size of the viewfinder object (150) using the position and / or direction of the hand (112) having the pinch gesture can be performed below a maximum size. For example, the maximum size can be set to maintain visibility of the boundary line of the viewfinder object (150) within the display. The wearable device (101) can change the size of the viewfinder object (150) below the maximum size. For example, the maximum size can correspond to a specified ratio (e.g., 90%) of the FoV of the display.

[0130] In one embodiment, the wearable device (101) may provide a user experience that is disconnected from the external space, such as VR. For example, the wearable device (101) may not display a composite image of the external space. The wearable device (101) may execute a VR-related software application and display at least a portion of a virtual space provided by the software application. For example, the entire display area of ​​the display may be occupied by the virtual space.

[0131] Referring to FIG. 8D , an exemplary screen (805) displayed by a wearable device (101) that receives an input for executing a camera application while providing a virtual space for VR is illustrated. Within the screen (805), the wearable device (101) may display a viewfinder object (150) as an overlay on a virtual space (840) associated with VR. Since the viewfinder object (150) is overlaid on the virtual space (840), a portion of the virtual space (840) overlapping with the viewfinder object (150) may not be displayed. The disclosure is not limited thereto, and the wearable device (101) may display the viewfinder object (150) using a specified transparency to at least partially maintain the visibility of a portion of the virtual space (840) overlapping with the viewfinder object (150).

[0132] With the viewfinder object (150), the wearable device (101) can display an external image (e.g., a composite image (520) of FIG. 5) acquired using a camera of the wearable device (101) (e.g., a camera (225) of FIG. 2). Within the screen (805), the image can be bounded by the viewfinder object (150). By displaying the viewfinder object (150) including at least a portion of the external image, the wearable device (101) can transition from a VR experience that provides a user experience that is disconnected from the external space to a partial pass-through experience that at least partially visualizes the external space.

[0133] Referring to FIG. 8D, while displaying a viewfinder object (150), the wearable device (101) can receive an input for moving the viewfinder object (150). For example, while displaying a virtual object (342) toward a boundary line (150-1) of the viewfinder object (150), in response to a pinch gesture of a hand (112) detected, the wearable device (101) can move the viewfinder object (150) within the display using the position and / or direction of the hand (112). For example, while displaying a screen (805), the wearable device (101) that detects a pinch gesture of a hand (112) can move the viewfinder object (150) according to the path of the hand (112). For example, within a screen (806) displayed after a screen (805), a viewfinder object (150) may be moved toward the left edge of the display along the path of the hand (112). As the viewfinder object (150) is moved, a portion of the composite image included in the viewfinder object (150) may change.

[0134] As described above, the wearable device (101) can adjust the position and / or size of the viewfinder object (150) displayed on the display. The wearable device (101) displaying the viewfinder object (150) at a first position on the display can, in response to receiving an input for capturing, store a first portion of the composite image corresponding to the first position in the memory. Similarly, while displaying the viewfinder object (150) at a second position on the display different from the first position, the wearable device (101) can, in response to receiving an input for capturing, store a second portion of the composite image corresponding to the second position. For example, the portion of the external space corresponding to the image and / or video stored in the memory can be related to the position and / or size of the viewfinder object (150) within the display at the time the image and / or video was acquired. For example, the wearable device (101) may acquire or store images and / or videos of a portion of external space bounded by the position and / or size of the viewfinder object (150) within the display.

[0135] Hereinafter, with reference to FIG. 9, exemplary operations of the wearable device (101) described with reference to FIGS. 8a to 8d are described in more detail.

[0136] FIG. 9 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments. The wearable device (101) of FIGS. 1 to 2 and / or the processor (210) of FIG. 2 may perform operations of the wearable device described with reference to FIG. 9. The operations of the wearable device of FIG. 9 may be related to the operations of the wearable device (101) described with reference to FIGS. 8A to 8D. Instructions stored in a memory of the wearable device (e.g., memory (215) of FIG. 2), when executed by the processor of the wearable device, may cause the wearable device to perform operations of FIG. 9.

[0137] Referring to FIG. 9 , within operation (910), a processor of a wearable device according to one embodiment may display a viewfinder object (e.g., the viewfinder object (150) of FIGS. 1 and / or 8A-8D) superimposed on a composite image (e.g., the composite image (520) of FIG. 5 ). The processor may perform operation (910) of FIG. 9 to display the viewfinder object, similar to operation (410) of FIG. 4 and / or operation (710) of FIG. 7 . For example, the processor may perform operation (910) in response to an input for executing a software application for controlling a camera of the wearable device (e.g., camera (225) of FIG. 2 and / or an external camera), such as a camera application.

[0138] Referring to FIG. 9, in operation (920), according to one embodiment, a processor of a wearable device may receive an input related to a first point on a viewfinder object. For example, the wearable device may detect a hand of a user wearing the wearable device (e.g., the user (110) of FIG. 1 and / or FIGS. 8A-8D ) from an image of an external camera. Using the direction and / or shape of the hand, the processor may receive or detect an input related to the first point of operation (920). While displaying a viewfinder object having a shape of a line, the first point on the viewfinder object may be located on the line (e.g., a side and / or a vertex). For example, if a hand extended along a direction corresponding to the first point has a shape of a pinch gesture, the processor may determine that the input of operation (920) has been detected.

[0139] Referring to FIG. 9 , in operation (930), a processor of a wearable device according to an embodiment may determine whether an input received based on operation (920) corresponds to an input for moving a viewfinder object. For example, while displaying a viewfinder object having a rectangular shape, the processor may detect an input for moving the viewfinder object of operation (930) based on whether a first point of operation (920) corresponds to an edge. In response to the input for moving the viewfinder object (930 - Yes), the processor may perform operation (940). If the input of operation (920) is different from the input for moving the viewfinder object (930 - No), the processor may perform operation (950). For example, while displaying a viewfinder object having a rectangular shape, if the first point of operation (920) does not correspond to an edge, the processor may perform operation (950).

[0140] Referring to FIG. 9, in operation (940), a processor of a wearable device according to an embodiment may move a viewfinder object using a received input. Screen (803) of FIG. 8B may be an exemplary screen displayed while moving a viewfinder object (150) using operation (940). The processor may move the viewfinder object according to operation (940) until it detects that the input is discontinued due to a change in the shape of the hand (e.g., the shape of a pinch gesture).

[0141] Referring to FIG. 9, in operation (950), a processor of a wearable device according to an embodiment may determine whether an input received based on operation (920) corresponds to an input for changing the size of a viewfinder object. For example, while displaying a viewfinder object having a rectangular shape, the processor may detect an input for changing the size of the viewfinder object of operation (950) based on whether a first point of operation (920) corresponds to a vertex. If the first point corresponds to a vertex, the processor may determine that the input of operation (920) corresponds to an input for changing the size of the viewfinder object. If an input for changing the size of the viewfinder object is received (950 - Yes), the processor may perform operation (960). If the input of action (920) is different from all of the inputs of actions (930, 950) (950-No), the processor may continue to display the viewfinder object based on action (910).

[0142] Referring to FIG. 9, an embodiment of sequentially performing operations (930, 950) in response to an input related to a first point on a viewfinder object is described, but the disclosure is not limited thereto. For example, the processor may perform operations (930, 950) substantially simultaneously, or in an order opposite to the order illustrated in FIG. 9.

[0143] Referring to FIG. 9, in operation (960), a processor of a wearable device according to an embodiment may change the size of a viewfinder object using the received input. Screen (804) of FIG. 8C may be an exemplary screen displayed by the processor performing operation (960). While detecting a hand of a pinch gesture, the processor may perform operation (960) to continuously change the size of the viewfinder object. If the shape of the hand changes to a shape different from that of the pinch gesture, the processor may stop changing the size of the viewfinder object of operation (960).

[0144] For example, a user may change the size and / or position of a viewfinder object displayed within a display by performing the input described with reference to FIGS. 8A to 8D , and may change a focal length by performing the input described with reference to FIG. 6 . Having changed the size and / or position of the viewfinder object and changed the focal length, the user may perform a photographing input to acquire an image and / or video. Hereinafter, exemplary operations of a wearable device receiving a photographing input will be described in more detail with reference to FIGS. 10A , 10B , and 10C , and / or 11 (which may be referred to as FIGS. 10A to 10C ).

[0145] FIGS. 10A, 10B, and 10C are diagrams illustrating exemplary operations of a wearable device (101) for acquiring images and / or videos according to various embodiments. The wearable device (101) and / or the processor (210) of FIGS. 1 and 2 may perform the operations of the wearable device (101) described with reference to FIGS. 10A to 10C.

[0146] Referring to FIGS. 10A to 10C, screens (302, 1002) displayed by a wearable device (101) executing instructions included in a camera application are illustrated. While displaying a viewfinder object (150) superimposed on a composite image related to a VST, the wearable device (101) can track a hand (112). For example, a virtual object (340) representing the position and / or direction of the hand (112) can be displayed within the screen (302).

[0147] Referring to FIG. 10A, within a screen (302) including a viewfinder object (150), the wearable device (101) can display a virtual object (340) that moves according to the position and / or direction of the hand (112) and a virtual object (342) having the form of a line extending from the virtual object (340). Referring to the exemplary screen (302) of FIG. 10A, while displaying a virtual object (342) that is directed toward a point (p1) within a portion different from the portion of the composite image designated by the viewfinder object (150), the wearable device (101) can detect or receive a photographing input based on the posture of the hand (112). For example, while displaying a virtual object (342) pointing to a point (p1), a wearable device (101) that detects a pinch gesture using continuously acquired images (311, 312) may determine that a photographing input has been received. For example, the photographing input may be received by a pinch gesture performed by a hand (112) having a direction corresponding to a point (p1) outside a viewfinder object (150) within a screen (302).

[0148] Within an exemplary state displaying the screen (302) of FIG. 10A, the wearable device (101) may acquire an image (1010) in response to a pinch gesture performed by a hand (112) toward a point (p1) outside the viewfinder object (150) and performed for less than a specified period of time (e.g., several seconds). For example, a pinch gesture performed for less than a specified period of time by a hand (112) having a direction corresponding to the point (p1) outside the viewfinder object (150) within the screen (302) may be a photographing input for generating a photograph of a specific point in time. Referring to FIG. 10A, in response to the photographing input, the wearable device (1011) may store an image (1010) including a portion of the composite image corresponding to the viewfinder object (150) in a memory (e.g., the memory (215) of FIG. 2).

[0149] In one embodiment, the wearable device (101) can record video based on the viewfinder object (150). Referring to the screen (302) of FIG. 10B, in response to a pinch gesture performed by a hand (112) toward a point (p1) outside the viewfinder object (150) and performed for a specified period of time, the wearable device (101) can initiate video recording. From the time of detecting the pinch gesture performed for a specified period of time, the wearable device (101) can acquire or store video corresponding to the viewfinder object (150). For example, a pinch gesture performed for a specified period of time by a hand (112) having a direction corresponding to a point (p1) outside the viewfinder object (150) within the screen (302) can be a capture input that initiates video recording.

[0150] Referring to FIG. 10b, while recording video in response to a capture input for recording video, the wearable device (101) may receive an input for executing a software application different from the camera application. In response to said input, detected while acquiring video for a portion of the composite image, the wearable device (101) may stop displaying the viewfinder object (150), as shown in the screen (1002) of FIG. 10b.

[0151] Referring to the screen (1002) of FIG. 10B, the wearable device (101) may display a virtual object (1021) provided by the software application in response to the input. The wearable device (101) that has stopped displaying the viewfinder object (150) may display an indicator (1022) indicating video recording. The indicator (1022) may be displayed after a specified time (e.g., several seconds) from the time at which the viewfinder object (150) stopped displaying. The indicator (1022) may include a conventional icon (e.g., a red circle) and / or text (e.g., “REC”) indicating video recording. The size and / or shape of the indicator (1022) is not limited to the embodiment of FIG. 10B. A wearable device (101) that has stopped displaying a viewfinder object (150) may display a virtual object (1023) to provide a thumbnail (or preview) of a video being captured using a camera. The virtual object (1023) may have a smaller size than the viewfinder object (150).

[0152] In response to an input selecting an indicator (1022) and / or a virtual object (1023) (e.g., a pinch gesture of a hand (112) having a direction corresponding to the indicator (1022) and / or the virtual object (1023), the wearable device (101) may redisplay the viewfinder object (150). For example, the wearable device (101) may redisplay the viewfinder object (150) at a location where the viewfinder object (150) was displayed prior to receiving an input to execute a software application corresponding to the virtual object (1021).

[0153] After receiving an input for selecting an indicator (1022) and / or a virtual object (1023), the wearable device (101) may determine whether to stop displaying the virtual object (1021) before displaying the viewfinder object (150) again. For example, the wearable device (101) may display a virtual object for determining whether to reduce or minimize the size of the virtual object (1021). In response to an input indicating selection of the virtual object, the wearable device (101) may stop displaying the virtual object (1021) and display the viewfinder object (150) again.

[0154] A wearable device (101) that acquires a video corresponding to a viewfinder object (150) in response to an input for recording a video may further receive an input for stopping recording of the video. The input may also be detected by a pinch gesture of a hand (112) having a direction corresponding to a point outside the viewfinder object (150), similar to the input for recording the video. The disclosure is not limited thereto, and during video recording, a visual object (e.g., a shutter) may further be displayed for receiving an input for stopping recording of the video. The wearable device (101) may stop recording the video in response to an input related to the visual object (e.g., an input indicating selection of the visual object using a pinch gesture).

[0155] Referring to FIG. 10C, an embodiment of a wearable device (101) further comprising a display (1030) facing the external space while being worn by a user (110) is illustrated. Using the display (1030), the wearable device (101) can inform other users and / or the external environment of the status of the user (110) wearing the wearable device (101) different from the user (110). Referring to FIG. 10C, while displaying a viewfinder object (150) provided from a camera application, the wearable device (101) can display a visual object (1032) related to the viewfinder object (150) on the display (1030). The visual object (1032) can include a portion of a composite image corresponding to the viewfinder object (150). The location of the visual object (1032) within the display (1030) may correspond to the location of the viewfinder object (150) within the screen (302) visible to the user (110) wearing the wearable device (101).

[0156] For example, a user (110) wearing a wearable device (101) can view a screen (302) using a display (e.g., display (220) of FIG. 2) facing both eyes of the user (110). The wearable device (101) can display a screen including a visual object (1032) on a display (1030) disposed on a side opposite to one side of the housing on which the display is disposed. The visual object (1032) can be displayed by the wearable device (101) that has received a photographing input (e.g., a photographing input for an image and / or video). The disclosure is not limited thereto, and the wearable device (101) can display a visual object (1032) on the display (1030) while displaying a viewfinder object (150) within the screen (302).

[0157] An example of displaying a visual object (1032) comprising a portion of a composite image corresponding to a viewfinder object (150) is described, but the disclosure is not limited thereto. For example, the wearable device (101) may, in response to a photographing input, display an image, text, and / or icon representing the photographing input on the display (1030). For example, while recording a video, the wearable device (101) may display an indicator (1022) of FIG. 10B on the display (1030).

[0158] Hereinafter, with reference to FIG. 11, exemplary operations of the wearable device (101) described with reference to FIGS. 10a to 10c are described in more detail.

[0159] FIG. 11 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments. The wearable device (101) of FIGS. 1 and 2 and / or the processor (210) of FIG. 2 may perform operations of the wearable device described with reference to FIG. 11. The operations of the wearable device of FIG. 11 may be related to the operations of the wearable device (101) described with reference to FIGS. 10A to 10C. Instructions stored in a memory of the wearable device (e.g., memory (215) of FIG. 2), when executed by the processor of the wearable device, may cause the wearable device to perform operations of FIG. 11.

[0160] Referring to FIG. 11, within operation (1110), a processor of a wearable device according to one embodiment may display a viewfinder object (e.g., the viewfinder object (150) of FIGS. 1 and / or 8A-8D) superimposed on a composite image (e.g., the composite image (520) of FIG. 5). The processor may perform operation (1110) of FIG. 11 to display the viewfinder object similarly to operation (410) of FIG. 4, operation (710) of FIG. 7, and / or operation (910) of FIG. 9.

[0161] Referring to FIG. 11, in operation (1120), a processor of a wearable device according to one embodiment may receive a photographing input including a pinch gesture. The processor may receive the photographing input of operation (1120) in response to a pinch gesture of a hand having a direction corresponding to a point of the display outside the viewfinder object, as described above with reference to FIGS. 10A and / or 10B. Using an image and / or video corresponding to the hand, the processor may calculate or identify a duration for which the pinch gesture is maintained.

[0162] Referring to FIG. 11, in operation (1130), a processor of a wearable device according to one embodiment may determine whether a period of time during which a pinch gesture is performed exceeds a specified period of time. The specified period of time may be set to determine whether to acquire either an image or a video in response to a photographing input indicated by the pinch gesture. If a pinch gesture exceeding the specified period of time is detected (1130 - Yes), the processor may perform operation (1140). If a pinch gesture less than the specified period of time is detected (1130 - No), the processor may perform operation (1150).

[0163] Referring to FIG. 11, in operation (1150), a processor of a wearable device according to one embodiment may obtain an image (e.g., image (1010) of FIG. 10A) of a portion included in a viewfinder object. The processor may store the image of operation (1150) in memory. After obtaining the image using operation (1150), the processor may maintain display of the viewfinder object. For example, in order to receive further photographing input, the processor may continue to display the viewfinder object.

[0164] Referring to FIG. 11, in operation (1140), according to one embodiment, a processor of a wearable device may acquire a video of a portion included in a viewfinder object. While acquiring a video of a portion corresponding to the viewfinder object, the processor may further display a visual object to guide acquisition of the video. While acquiring the video, the processor may maintain displaying the viewfinder object. While acquiring the video, the processor may change the position and / or size of the viewfinder object in response to an input for adjusting the position and / or size of the viewfinder object, as described with reference to FIGS. 8A to 8D. The video acquired by the wearable device may include a portion corresponding to the viewfinder object having the changed position and / or the changed size.

[0165] Referring to FIG. 11, in operation (1160), according to one embodiment, the processor of the wearable device may determine whether a virtual object is displayed as an overlay on the viewfinder object. For example, when a user runs a software application, a virtual object provided by the software application may be displayed on the display. If the virtual object of operation (1160) is not overlaid on the viewfinder object or is spaced apart from the viewfinder object (1160-No), the processor may perform operation (1140) to maintain video acquisition and continue displaying the viewfinder object. If the virtual object overlaid on the viewfinder object is identified (1160-Yes), the processor may perform operation (1170).

[0166] Referring to FIG. 11, within operation (1170), a processor of a wearable device according to one embodiment may display an indicator indicating video recording (e.g., indicator (1022) of FIG. 10b) and stop displaying the viewfinder object. The disclosure is not limited thereto, and the processor may display a virtual object (e.g., virtual object (1023) of FIG. 10b) for providing a preview of the video. Upon receiving input related to the indicator of operation (1170), the processor may display the viewfinder object again.

[0167] Hereinafter, the operation of the wearable device in relation to the control handle displayed together with the viewfinder object (e.g., the control handle (330) of FIGS. 3A to 3C) is described in more detail with reference to FIGS. 12 to 20.

[0168] FIG. 12 is a diagram illustrating exemplary operations of a wearable device (101) with respect to a control handle (330) displayed together with a viewfinder object (150), according to various embodiments. The wearable device (101) and / or the processor (210) of FIGS. 1 and 2 may perform the operations of the wearable device (101) described with reference to FIG. 12.

[0169] Referring to FIG. 12, an exemplary screen (302) of a wearable device (101) is illustrated, which includes a viewfinder object (150) and a control handle (330) provided from a camera application. While an embodiment is illustrated in which the control handle (330) is displayed along the left edge of the viewfinder object (150), the position, shape, and / or size of the control handle (330) is not limited thereto.

[0170] While displaying the screen (302), the wearable device (101) that detects a body part including a hand (112) from an image and / or video acquired from an external camera can display a virtual object (340) corresponding to the hand (112) within the screen (302). Together with the virtual object (340), the wearable device (101) can display a virtual object (342) representing the direction of the hand (112). In response to a pinch gesture of the hand (112) detected while the direction of the hand (112) represented by the virtual object (342) is directed toward a point (p1) on the control handle (330), the wearable device (101) can detect or receive an input indicating selection of the control handle (330).

[0171] In response to an input indicating a selection of a control handle (330), the wearable device (101) can display visual objects (1221, 1222, 1223, 1224, 1225, 1226, 1227, 1228, 1229, 1230, 1231) corresponding to each of the photographing-related functions along the direction of the edge of the viewfinder object (150) having a rectangular shape. Referring to FIG. 12, the wearable device (101) can display a control menu (1210) including the visual objects (1221, 1222, 1223, 1224, 1225, 1226, 1227, 1228, 1229, 1230, 1231). The control menu (1210) may be referred to as a context menu and / or a pop-up menu. The control menu (1210) may be at least partially overlapped on a control handle (330) within the screen (302).

[0172] Referring to FIG. 12, a control menu (1210) including different icons is illustrated. A visual object (1221) may correspond to a function for displaying a virtual object and / or window containing different options related to shooting. A visual object (1222) may correspond to a function for activating a flashlight included in the wearable device (101). A visual object (1223) may correspond to a function for adjusting a delay between the time of receiving a shooting input and the time of capturing an image and / or video (e.g., a timer). A visual object (1224) may correspond to a function for adjusting the aspect ratio of an image and / or video to be captured in response to a viewfinder object (150) and / or a shooting input. A visual object (1225) may correspond to a function for capturing an image in which motion is recorded for a specified period of time (e.g., several seconds), such as a motion photo. The visual object (1226) may correspond to a function for adjusting the color, brightness and / or saturation of an image and / or video to be acquired by a shooting input, or for applying a visual effect.

[0173] A visual object (1227) may correspond to a function for obtaining a depth map using a depth sensor, together with an image and / or video using a synthetic image. An exemplary operation of a wearable device (101) executing the function in response to an input indicating selection of a visual object (1227) is described in more detail below with reference to FIGS. 14A, 14B, and 15.

[0174] A visual object (1228) may correspond to a function for combining or synthesizing a virtual object displayed on a display within an image and / or video using a composite image. An exemplary operation of a wearable device (101) receiving an input indicating selection of a visual object (1228) is described in more detail below with reference to FIGS. 18A and 18B .

[0175] A visual object (1229) may correspond to a function for acquiring images and / or videos related to a specific external object. An exemplary operation of a wearable device (101) that executes the function in response to the selection of a visual object (1229) is described in more detail below with reference to FIGS. 16 and 17 .

[0176] The visual object (1230) may be referred to as a shutter. In response to an input related to the visual object (1230) (e.g., a pinch gesture of a hand (112) having a direction corresponding to the visual object (1230), the wearable device (101) may acquire an image and / or video. For example, a capture input may include an input indicating selection of the visual object (1230).

[0177] A visual object (1231) may correspond to a function for exploring images and / or videos stored in a memory of the wearable device (101) (e.g., memory (215) of FIG. 2). In response to an input indicating selection of a visual object (1231), the wearable device (101) may execute a designated software application, such as a gallery application, to display a panel and / or virtual object containing a list of images and / or videos stored in the wearable device (101).

[0178] Hereinafter, with reference to FIG. 13, exemplary operations of the wearable device (101) related to the control handle (330) are described in more detail.

[0179] FIG. 13 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments. The wearable device (101) of FIGS. 1 and 2 and / or the processor (210) of FIG. 2 may perform operations of the wearable device described with reference to FIG. 13. The operations of the wearable device of FIG. 13 may be related to the operations of the wearable device (101) described with reference to FIG. 12. Instructions stored in a memory of the wearable device (e.g., memory (215) of FIG. 2) may cause the wearable device to perform operations of FIG. 13 when executed by the processor of the wearable device.

[0180] Referring to FIG. 13, within operation (1310), a processor of a wearable device according to one embodiment may display a viewfinder object (e.g., viewfinder object (150) of FIGS. 1 and / or FIGS. 8A-8D) and a control handle (e.g., control handle (330) of FIGS. 1 and / or FIG. 12) superimposed on a composite image (e.g., composite image (520) of FIG. 5). Operation (1310) may be performed similarly to operation (410) of FIG. 4, operation (710) of FIG. 7, operation (910) of FIG. 9, and / or operation (1110) of FIG. 11.

[0181] Referring to FIG. 13, in operation (1320), a processor of a wearable device according to one embodiment may receive an input related to a control handle. For example, as described above with reference to FIG. 12, the processor may receive an input including a pinch gesture of a hand having a direction related to the control handle. The processor, having received the input of operation (1320), may perform operation (1330).

[0182] Referring to FIG. 13, in operation (1330), according to one embodiment, a processor of a wearable device may display a control menu including executable objects. The control menu of operation (1330) may include the control menu (1210) of FIG. 12. The control menu may be displayed at a location on a display associated with a viewfinder object and / or a control handle of operation (1310). The control menu may be displayed as an overlay on the control handle. The processor may replace the control handle with the control menu.

[0183] Referring to FIG. 13, in operation (1340), a processor of a wearable device according to one embodiment may execute a function mapped to any one of the objects in response to an input related to objects included in a control menu. The objects of operation (1340) may include visual objects (1221, 1222, 1223, 1224, 1225, 1226, 1227, 1228, 1229, 1230, 1231) of FIG. 12. In response to an input indicating a selection of at least one of the objects of operation (1340), the processor may execute a function associated with the object corresponding to the input.

[0184] FIGS. 14A and 14B are diagrams illustrating exemplary operations of a wearable device (101) in connection with a depth sensor according to various embodiments. The wearable device (101) of FIGS. 1 and 2 and / or the processor (210) of FIG. 2 may perform the operations of the wearable device described with reference to FIGS. 14A and 14B.

[0185] Referring to FIGS. 14A and 14B , screens (1401, 1402) displayed by a wearable device (101) executing a camera application are illustrated. While displaying a viewfinder object (150) superimposed on a composite image, the wearable device (101) may display a control handle (330). Upon receiving an input related to the control handle (330), the wearable device (101) may display a control menu (1210) of FIG. 12 .

[0186] Referring to FIG. 14A, a screen (1401) displayed by a wearable device (101) facing external objects (1411, 1412, 1413) is illustrated. Within the screen (1401), the wearable device (101) can display a composite image including visual objects (1417, 1418, 1419) corresponding to each of the external objects (1411, 1412, 1413) using a camera (e.g., camera (225) of FIG. 2 and / or an external camera). Referring to FIG. 14A, an external object (1411) may be spaced apart from the wearable device (101) by a distance (d1), an external object (1412) may be spaced apart from the wearable device (101) by a distance (d2) that exceeds the distance (d1), and an external object (1413) may be spaced apart from the wearable device (101) by a distance (d3) that exceeds the distance (d2).

[0187] Referring to FIG. 14A, a screen (1402) displayed by a wearable device (101) is illustrated that receives an input for obtaining a depth map (e.g., an input indicating selection of a visual object (1227) of FIG. 12) using a control handle (330). Within the screen (1402), the wearable device (101) can display a virtual object representing a depth map obtained from a depth sensor as an overlay on a portion of the composite image designated by the viewfinder object (150). For example, the virtual object can include shapes having different colors, brightnesses, and / or transparencies. The wearable device (101) can visualize a depth map corresponding to the portion of the composite image designated by the viewfinder object (150).

[0188] Within the screen (1402) of FIG. 14A, the wearable device (101) can visualize a depth map by adjusting the color of a portion of the composite image corresponding to the viewfinder object (150). For example, using a designated color, such as blue, the wearable device (101) can visualize the depth map. Within the portion of the composite image designated by the viewfinder object (150), an area relatively close to the wearable device (101) can be filled with a relatively bright color, and an area relatively far from the wearable device (101) can be filled with a relatively dark color. An area corresponding to a maximum and / or minimum value of the depth map can be displayed without adjusting the color.

[0189] For example, among the external objects (1411, 1412, 1413), a visual object (1417) corresponding to the external object (1411) located closest to the wearable device (101) may be displayed in light blue. For example, a visual object (1418) corresponding to the external object (1412) located farther from the wearable device (101) than the external object (1411) may be displayed in blue. For example, a visual object (1419) corresponding to the external object (1413) located farther from the wearable device (101) than the external object (1412) may be displayed in dark blue.

[0190] Referring to FIG. 14A, in response to a photographing input received while displaying a screen (1402) visualizing a depth map, the wearable device (101) may store an image (1421) including a portion of the composite image corresponding to the viewfinder object (150) and a depth map (1422) indicating depth values ​​(e.g., distances from the wearable device (101)) of each pixel included in the image (1421). The depth map (1422) may be stored in the metadata of the image (1421). The disclosure is not limited thereto, and the depth map (1422) may be stored in a file associated with the image (1421). The wearable device (101) receiving the photographing input may store the depth map (1422) acquired using a depth sensor and corresponding to the portion of the composite image designated by the viewfinder object (150).

[0191] In one embodiment, a depth map (1422) stored with an image (1421) can be used to display the image (1421) stereoscopically. For example, the wearable device (101) can, in response to an input for displaying the image (1421), set the binocular disparity of each pixel of the image (1421) using depth values ​​individually assigned to the pixels by the depth map (1422). Using the exemplary depth map (1422) of FIG. 14A, the wearable device (101) can stereoscopically display the image (1421) including visual objects (1417, 1418, 1419) corresponding to each of the external objects (1411, 1412, 1413). For example, since the distance (d1) between the external object (1411) and the wearable device (101) is smaller than the distance (d3) between the external object (1413) and the wearable device (101), the wearable device (101) displaying the image (1421) can set the binocular disparity of the visual object (1417) corresponding to the external object (1411) to be larger than the binocular disparity of the visual object (1419) corresponding to the external object (1413).

[0192] In one embodiment, the wearable device (101) may receive input for adjusting the maximum and / or minimum distance to be measured using the depth sensor. Referring to the exemplary screen (1403) of FIG. 14B, the wearable device (101) may display a virtual object (1430) representing a range measurable by the depth sensor inside the viewfinder object (150). To maintain visibility of a portion of the depth map and / or composite image visualized through the inside of the viewfinder object (150), the wearable device (101) may display the virtual object (1430) with a specified transparency (e.g., a transparency greater than 0%).

[0193] Referring to an exemplary screen (1403) of FIG. 14B, the wearable device (101) may display a virtual object (1430) including a plane (1431) representing a maximum value (e.g., maximum distance) of a depth map to be acquired by a photographing input, and a plane (1432) representing a minimum value (e.g., minimum distance) of the depth map. In response to an input for moving any one of the planes (1431, 1432), the wearable device (101) may move at least one of the planes (1431, 1432) displayed on the display. In response to the input, the wearable device (101) may change the maximum value and / or minimum value of the depth map.

[0194] Referring to FIG. 14B, a wearable device (101) that detects a pinch gesture (e.g., a pinch gesture of a hand facing a direction related to the point (p1)) on a plane (1431) can move the plane (1431) while detecting the pinch gesture, according to a path of the hand that performed the pinch gesture. Similarly, a wearable device (101) that detects a pinch gesture for movement of a plane (1432) can move the plane (1432) while the pinch gesture is maintained, according to a position and / or path of the hand that performed the pinch gesture. The disclosure is not limited thereto, and a wearable device (101) that receives a pinch gesture associated with a plane (1432) may move the plane (1432) along the path of the hand that performed the pinch gesture while detecting the pinch gesture.

[0195] Referring to FIG. 14B, a screen (1404) displayed by a wearable device (101) that receives an exemplary input for moving a plane (1431) from a point (p2) to a point (p3) is illustrated. By the input for moving the plane (1431) corresponding to the maximum distance, the maximum distance may be reduced. For example, the maximum distance may be reduced to a distance greater than the distance (d2) between an external object (1412) and the wearable device (101) and less than the distance (d3) between the external object (1413) and the wearable device (101). Since the reduced maximum distance is less than the distance (d3), the wearable device (101) may determine the distance between the external object (1413) and the wearable device (101) as the reduced maximum distance, instead of the distance (d3). Within the viewfinder object (150) of the exemplary screen (1404) of FIG. 14b, the wearable device (101) can display a visual object (1419) corresponding to an external object (1413) without changing the color associated with the depth map.

[0196] Referring to FIG. 14B, an image (1441) and a depth map (1442) stored by a wearable device (101) that received a photographing input while displaying a screen (1404) are illustrated. The image (1441) may include a portion of a composite image corresponding to a viewfinder object (150), similar to the image (1421) of FIG. 14A. Within the depth map (1442), unlike the depth map (1422) of FIG. 14A, a depth value corresponding to an external object (1413) that is farthest from the wearable device (101) among the external objects (1411, 1412, 1413) may be set to a maximum distance corresponding to a plane (1431).

[0197] As described above, according to one embodiment, the wearable device (101) may acquire or store information (e.g., the depth map (1422) of FIG. 14A and / or the depth map (1442) of FIG. 14B) for stereoscopically displaying an image (e.g., the image (1421) of FIG. 14A and / or the image (1441) of FIG. 14B) to be acquired by a photographing input. The wearable device (101) may select or determine at least one external object to be stereoscopically displayed using a depth range adjustable by the user.

[0198] Hereinafter, the operation of the wearable device (101) described with reference to FIG. 14 will be described in more detail with reference to FIG. 15.

[0199] FIG. 15 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments. The wearable device (101) of FIGS. 1 to 2 and / or the processor (210) of FIG. 2 may perform operations of the wearable device described with reference to FIG. 15. The operations of the wearable device of FIG. 15 may be related to the operations of the wearable device (101) described with reference to FIG. 14. Instructions stored in a memory of the wearable device (e.g., memory (215) of FIG. 2) may cause the wearable device to perform operations of FIG. 15 when executed by the processor of the wearable device.

[0200] Referring to FIG. 15 , within operation (1510), a processor of a wearable device according to one embodiment may display a viewfinder object (e.g., the viewfinder object (150) of FIGS. 1 and / or 8A-8D) superimposed on a composite image (e.g., the composite image (520) of FIG. 5 ). Operation (1510) of FIG. 15 may be performed similarly to operation (410) of FIG. 4 , operation (710) of FIG. 7 , operation (910) of FIG. 9 , operation (1110) of FIG. 11 , and / or operation (1310) of FIG. 13 .

[0201] Referring to FIG. 15, within operation (1520), a processor of a wearable device according to one embodiment may identify that an option associated with a depth sensor has been activated. For example, in response to an input selecting a visual object (e.g., visual object (1227) of FIG. 12) corresponding to an option of operation (1520) included in a control menu (e.g., control menu (1210) of FIG. 12) provided by a control handle (e.g., control handle (330) of FIG. 1 and / or FIG. 14) displayed together with a viewfinder object, the option may be activated or deactivated. For example, in response to the input, activation of the option may be toggled. If the option of operation (1520) is activated, the processor may perform operation (1530).

[0202] Referring to FIG. 15, in operation (1530), according to one embodiment, a processor of a wearable device may display a visual object for guiding a depth map of a depth sensor inside a viewfinder object. The visual object of operation (1530) may include one or more virtual objects overlapping inside the viewfinder object to visualize the depth map, as described above with reference to FIG. 14A and / or FIG. 14B. The visual object of operation (1530) may include one or more shapes having a color representing a depth distribution (or distance distribution) represented by the depth map, as described above with reference to FIG. 14A and / or FIG. 14B. The visual object of operation (1530) may include a virtual object (1430) displayed inside the viewfinder object.

[0203] Referring to FIG. 15, in operation (1540), according to one embodiment, a processor of a wearable device may change the range of a depth map to be detected by a depth sensor based on an input related to a visual object. The visual object of operation (1540) may be displayed inside a viewfinder object to change the range of the depth map, such as a virtual object (1430) including planes (1431, 1432) of FIG. 14. The screen (1404) of FIG. 14B may be displayed by the wearable device (101) that has changed the range of the depth map based on the input of operation (1540).

[0204] Referring to FIG. 15, within operation (1550), a processor of a wearable device according to one embodiment may store a depth map (e.g., depth map 1422 of FIG. 14A and / or depth map 1442 of FIG. 14B) based on a photographing input. The depth map of operation (1550) may be associated with an image and / or video to be acquired based on the photographing input. For example, the depth map of operation (1550) may be stored in association with an image and / or video acquired by the photographing input.

[0205] Hereinafter, exemplary operations of a wearable device for acquiring images and / or videos to track external objects are described in more detail with reference to FIGS. 16 and / or 17.

[0206] FIG. 16 is a diagram illustrating exemplary operations of a wearable device (101) for tracking an external object according to various embodiments. The wearable device (101) of FIGS. 1 and 2 and / or the processor (210) of FIG. 2 may perform the operations of the wearable device described with reference to FIG. 16.

[0207] Referring to FIG. 16, an exemplary screen (1601) displayed by a wearable device (101) is illustrated while a user (110) looks in a direction (d1). The wearable device (101) executing a camera application can display a composite image and a viewfinder object (150) superimposed on the composite image on the screen (1601). The wearable device (101) displaying the viewfinder object (150) and the control handle (330) can display the control menu (1210) of FIG. 12 in response to an input related to the control handle (330).

[0208] Referring to FIG. 16, a screen (1601) displayed by a wearable device (101) is illustrated that has received an input (e.g., an input indicating selection of a visual object (1229) of FIG. 12) for obtaining an image and / or video associated with a particular external object using a control handle (330). The wearable device (101) that has received the input may detect or receive an input indicating tracking of the external object within a portion of the composite image designated by the viewfinder object (150). The disclosure is not limited thereto, and the wearable device (101) may detect or receive an input for selecting any one of the external objects associated with the composite image.

[0209] Referring to FIG. 16, a wearable device (101) that receives an input (e.g., an input detected by a pinch gesture of a hand having a direction corresponding to the visual object (1611)) for selecting a visual object (1611) corresponding to a specific external object (e.g., a flower pot (1610)) may determine to track the external object. In response to the input, the wearable device (101) may display a virtual object (1612) having a position linked to the visual object (1611) included in the composite image. While a pin-shaped virtual object (1612) is illustrated, the disclosure is not limited thereto. The wearable device (101) may adjust the position of the viewfinder object (150) associated with the composite image so that the visual object (1611) is included within the viewfinder object (150). Referring to the exemplary screen (1601) of FIG. 16, the wearable device (101) can display a viewfinder object (150) as an overlay on a portion of a composite image that includes a visual object (1611).

[0210] Prior to receiving an input to acquire an image and / or video including a particular external object, the wearable device (101) may display a viewfinder object (150) fixed to a particular point on the display. Referring to FIG. 16 , within an exemplary state in which an input to acquire an image and / or video related to a flower pot (1610) is received, the wearable device (101) may change the position of the viewfinder object (150) depending on the position of a visual object (1611) corresponding to the flower pot (1610) within the composite image.

[0211] For example, when a user (110) looking at a direction (d1) looks at a direction (d2), the wearable device (101) can display a screen (1602) including a composite image corresponding to the direction (d2). Within the screen (1602), the wearable device (101) can display a viewfinder object (150) such that a visual object (1611) corresponding to a flower pot (1610) is positioned inside the viewfinder object (150). Together with the viewfinder object (150), the wearable device (101) can display a virtual object (1612) adjacent to the visual object (1611).

[0212] A virtual object (1612) may be displayed while the wearable device (101) moves the position of the viewfinder object (150) using the position of the visual object (1611) within the composite image. In response to additional input related to the virtual object (1612) (e.g., a pinch gesture of a hand having a direction related to the virtual object (1612), the wearable device (101) may stop displaying the virtual object (1612) or change the external object associated with the virtual object (1612) to another external object. For example, an input indicating selection of the virtual object (1612) may be mapped to a function of stopping moving the viewfinder object (150) using the position of the visual object (1611) corresponding to the virtual object (1612) within the composite image. The wearable device (101) receiving the input may stop displaying the virtual object (1612).

[0213] As described above, in response to an input for tracking an external object (e.g., a flower pot (1610)), the wearable device (101) may change the position of the viewfinder object (150) on the composite image using the position of the external object relative to the composite image. When receiving a capture input, the wearable device (101) may acquire or store an image and / or video associated with a portion of the composite image corresponding to the viewfinder object (150) that includes a visual object (e.g., a visual object (1611)) corresponding to the external object. While recording a video in response to the capture input, the wearable device (101) may acquire or store an image and / or video including a visual object (1611) corresponding to the external object within the composite image, independently of the rotation and / or movement of the head of the user (110) wearing the wearable device (101).

[0214] As described above with reference to FIG. 16, the wearable device (101) can track a visual object in a composite image corresponding to the external object by adjusting the position of the viewfinder object (150), independently of whether the user (110) wearing the wearable device (101) is looking in the same direction as the external object corresponding to the input.

[0215] Hereinafter, with reference to FIG. 17, an exemplary operation of the wearable device (101) described with reference to FIG. 16 is described in more detail.

[0216] FIG. 17 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments. The wearable device (101) of FIGS. 1 to 2 and / or the processor (210) of FIG. 2 may perform operations of the wearable device described with reference to FIG. 17. The operations of the wearable device of FIG. 17 may be related to the operations of the wearable device (101) described with reference to FIG. 16. Instructions stored in a memory of the wearable device (e.g., memory (215) of FIG. 2) may cause the wearable device to perform operations of FIG. 17 when executed by the processor of the wearable device.

[0217] Referring to FIG. 17, within operation (1710), a processor of a wearable device according to one embodiment may display a viewfinder object (e.g., viewfinder object (150) of FIGS. 1 and / or 8A-8D) superimposed on a composite image (e.g., composite image (520) of FIG. 5) and linked to a position and / or orientation of a head wearing the wearable device. As described above with reference to FIG. 8A, the position of the viewfinder object may be fixed to a specific point on a display (e.g., display (220) of FIG. 2). By displaying a viewfinder object having a position fixed to a point within the display, the processor may provide a user experience such that the viewfinder object follows the user's head and / or FoV.

[0218] Referring to FIG. 17, within operation (1720), a processor of a wearable device according to one embodiment may receive an input for tracking an external object. The input of operation (1720) may be received via a control handle (e.g., control handle (330) of FIGS. 1 and / or 16) and / or a control menu (e.g., control menu (1210) of FIG. 12) displayed together with a viewfinder object. For example, the input of operation (1720) may include an input indicating a selection of a visual object (1229) within the control menu (1210) of FIG. 12. The input of operation (1720) may include an input indicating a selection of a visual object (e.g., visual object (1611) of FIG. 16) corresponding to a particular external object within at least a portion of a composite image displayed via the display.

[0219] Referring to FIG. 17, in operation (1730), a processor of a wearable device according to an embodiment may display a viewfinder object within a display using a location of an external object identified by a plurality of cameras (e.g., camera (225) of FIG. 12 and / or an external camera). During operation (1730), the processor may display the viewfinder object using a location of a visual object corresponding to the external object within the display. During operation (1730), the processor may display the viewfinder object such that the viewfinder object includes a visual object within the display. For example, the location of the viewfinder object within the display may be related to a location of the external object tracked by the processor.

[0220] Referring to FIG. 17, in operation (1740), a processor of a wearable device according to one embodiment may, in response to a photographing input, obtain an image and / or video associated with an external object corresponding to the input. The photographing input of operation (1740) may be received by a pinch gesture as described with reference to FIGS. 10A to 10C. While performing operation (1740) to obtain an image and / or video, the processor may repeatedly (or periodically) change the position of the viewfinder object using the position of the external object. For example, the image and / or video obtained by performing operation (1740) may include a visual object (e.g., visual object 1611 of FIG. 16) associated with the external object.

[0221] Hereinafter, exemplary operations of a wearable device in relation to a virtual object are described in more detail with reference to FIGS. 18a, 18b and / or 19.

[0222] FIGS. 18A and 18B are diagrams illustrating exemplary operations of a wearable device (101) in relation to a virtual object (1810), according to various embodiments. The wearable device (101) of FIGS. 1 and 2 and / or the processor (210) of FIG. 2 may perform the operations of the wearable device (101) described with reference to FIGS. 18A and 18B.

[0223] Referring to FIGS. 18A and 18B , screens (1801, 1802, 1803, 1804) displayed by a wearable device (101) executing a camera application are illustrated. The wearable device (101) displaying a virtual object (1810) superimposed on a composite image may display a viewfinder object (150) and a control handle (330) in response to an input for executing the camera application. Referring to the exemplary screen (1801) of FIG. 18A , the wearable device (101) may display a virtual object (1810) with at least a portion (1811) overlapping the viewfinder object (150) masked together with the viewfinder object (150). For example, a portion (1811) of a virtual object (1810) may have a specified transparency (e.g., a transparency greater than 0%). In one embodiment, the wearable device (101) may completely remove a portion (1811) of a virtual object (1810) that overlaps a viewfinder object (150) from within the screen (1801).

[0224] While displaying the screen (1801) of FIG. 18A, the wearable device (101) may receive an input related to the control handle (330). Based on a control menu displayed by the input (e.g., a control menu (1210) of FIG. 12), the wearable device (101) may receive an input (e.g., an input indicating selection of a visual object (1228) of FIG. 12) for executing a function of combining at least one virtual object with a composite image. After receiving the input, the wearable device (101) may display a portion (1811) of the virtual object (1810) overlapping the viewfinder object (150) using a specified transparency (e.g., a transparency greater than 0% and less than 100%).

[0225] Based on a portion (1811) displayed using a specified transparency, the wearable device (101) can receive an input for selecting an area (1820) of a virtual object (1810) to be combined with a portion of a composite image corresponding to the viewfinder object (150). For example, the wearable device (101) can receive an input for selecting an area (1820) in response to a drag gesture (e.g., a path moved by a hand (112) maintaining a pinch gesture) within the portion (1811). The wearable device (101) receiving the input can display an area (1820) of the virtual object (1810) corresponding to the input within the viewfinder object (150), such as a screen (1802). For example, the wearable device (101) can normally display the area (1820) using a transparency of 0%.

[0226] While displaying an area (1820) as shown in the screen (1802) of FIG. 18A, the wearable device (101) may receive a photographing input. In response to the photographing input, the wearable device (101) may store an image (1822) including a visual object (1824) representing the area (1820). The visual object (1824) included in the image (1822) may correspond to a portion of a virtual object (1810) displayed through the area (1820) at the time the photographing input was received. Referring to FIG. 18A, the wearable device (101) may combine an area (1824) of a virtual object (1810) within an image (1822) representing an external environment. For example, the image (1822) may be a screenshot of a portion of the screen (1802) corresponding to the viewfinder object (150) at the time the capture input was received.

[0227] Although exemplary operations of a wearable device (101) in connection with a virtual object (1810) having the form of a two-dimensional panel have been described, the disclosure is not limited thereto. Referring to an exemplary screen (1803) of FIG. 18B , the wearable device (101) may display a viewfinder object (150) while displaying a virtual object (1830) that is a three-dimensional graphical object. At least a portion of the virtual object (1830) overlapping the viewfinder object (150) may be hidden or removed within the screen (1803).

[0228] While displaying the screen (1803) of FIG. 18b, the wearable device (101) may receive an input (e.g., an input indicating selection of a visual object (1228) of FIG. 12) for executing a function of combining at least a portion of a virtual object (1830) with a portion of a composite image corresponding to the viewfinder object (150). After receiving the input, the wearable device (101) may at least partially display a portion of the virtual object (1830) hidden by the viewfinder object (150). After receiving the input, the wearable device (101) that further receives an input indicating selection of the virtual object (1830) may completely display the virtual object (1830) overlapping the viewfinder object (150), as in the screen (1804) of FIG. 18b (e.g., with a transparency of 0%).

[0229] In response to a photographing input received while displaying the screen (1804) of FIG. 18B, the wearable device (101) may store an image (1842) including a portion of the composite image corresponding to the viewfinder object (150). The wearable device (101) may store an image (1842) including a visual object (1844) corresponding to a portion of the virtual object (1830) overlapping the viewfinder object (150) along with the portion of the composite image. The image (1842) may be a screenshot of a portion of the screen (1804) corresponding to the viewfinder object (150) at the time the photographing input was received.

[0230] As described above, according to one embodiment, the wearable device (101) can combine a virtual object displayed on the display with an image and / or video for a composite image representing an external space.

[0231] Although the operation of a wearable device (101) that displays a viewfinder object (150) in response to an input executing a camera application and stores images and / or videos associated with the viewfinder object (150) in response to a photographing input has been described, the disclosure is not limited thereto. For example, the wearable device (101) may support a function of automatically acquiring images and / or videos. Hereinafter, with reference to FIGS. 19 and 20, exemplary operations of a wearable device (101) that automatically stores images and / or videos in response to a specific time and / or specific conditions will be described in more detail.

[0232] FIG. 19 is a diagram illustrating an exemplary operation of a wearable device (101) for acquiring video according to various embodiments. The wearable device (101) of FIGS. 1 and 2 and / or the processor (210) of FIG. 2 may perform the operation of the wearable device (101) described with reference to FIG. 19.

[0233] In one embodiment, the wearable device (101) may determine whether to acquire images and / or videos by checking preset conditions. The conditions may be preset by user input related to the camera application. For example, the preset conditions may be related to times and / or intervals reserved by the user (110). For example, if the user (110) wears the wearable device (101) at a time associated with the preset conditions, the wearable device (101) may automatically acquire images and / or videos. In the above example, the wearable device (101) may repeatedly acquire images and / or videos using intervals associated with the preset conditions.

[0234] For example, the preset condition may be related to a geographical location set by the user (110). Using a geofence and / or GPS coordinates, the wearable device (101) may compare the current location of the wearable device (101) with the geographical location included in the preset condition. When the wearable device (101) detects a current location corresponding to a location included in the preset condition while being worn by the user (110), the wearable device (101) may acquire an image and / or video.

[0235] For example, the preset condition may relate to whether or not an external object of a specific type (or a specific category) set by the user (110) has been detected. If an external object of a specific type set by the user (110) has been detected, the wearable device (101) may acquire or store an image and / or video containing the external object.

[0236] For example, the preset condition may relate to whether a specific user registered by the user (110) has been detected. If the specific user has been detected, the wearable device (101) may acquire or store images and / or videos including the specific user.

[0237] For example, the preset condition may be related to a sound detected by the wearable device (101). For example, the wearable device (101) may initiate acquisition of an image and / or video upon receiving a sound having a volume exceeding a specified volume (e.g., a volume having a unit of decibels).

[0238] For example, the preset conditions may relate to motion detected by an external camera. For example, if an external object is detected using the external camera moving beyond a specified distance and / or speed, the wearable device (101) may initiate acquisition of images and / or video.

[0239] Referring to FIG. 19, exemplary screens (1901, 1902) displayed by a wearable device (101) for acquiring images and / or videos using preset conditions are illustrated. While displaying a plurality of virtual objects (351, 352, 353), the wearable device (101) that has decided to acquire images and / or videos using preset conditions may further display a virtual object (1910) within the screen (1901) to provide a preview of the video acquired using the camera. The virtual object (1910) may be displayed at a location within the screen (1901) that does not overlap with other virtual objects (351, 352, 353). In response to an input for selecting the virtual object (1910), the wearable device (101) may display a viewfinder object corresponding to the video, such as the screen (304) of FIG. 3B.

[0240] As shown in screen (1901), after displaying a virtual object (1910), based on the expiration of a specified period of time (e.g., several seconds), the wearable device (101) may display an indicator (1920) to indicate acquisition of video, instead of displaying the virtual object (1910) as shown in screen (1902). The indicator (1920) of FIG. 19 may correspond to the indicator (1022) of FIG. 10B. The virtual object (1910) of FIG. 19 may correspond to the virtual object (1023) of FIG. 10B. The indicator (1920) may blink periodically. When acquisition of video is complete, the wearable device (101) may stop displaying the indicator (1920) and / or the virtual object (1910).

[0241] Hereinafter, with reference to FIG. 20, an exemplary operation of the wearable device (101) described with reference to FIG. 19 is described in more detail.

[0242] FIG. 20 is a flowchart illustrating exemplary operations of a wearable device according to various embodiments. The wearable device (101) of FIGS. 1 to 2 and / or the processor (210) of FIG. 2 may perform operations of the wearable device described with reference to FIG. 20. The operations of the wearable device of FIG. 20 may be related to the operations of the wearable device (101) described with reference to FIG. 19. Instructions stored in a memory of the wearable device (e.g., memory (215) of FIG. 2) may cause the wearable device to perform operations of FIG. 20 when executed by the processor of the wearable device.

[0243] Referring to FIG. 20, in operation (2010), a processor of a wearable device according to one embodiment may determine conditions for automatically acquiring images and / or videos. The conditions of operation (2010) may include preset conditions described with reference to FIG. 19. The conditions may be determined based on the execution of a process different from a foreground process set to occupy a display and / or display area, such as a background process. The conditions of operation (2010) may be preset by user input or may be stored.

[0244] Referring to FIG. 20, within operation (2020), a processor of a wearable device according to one embodiment may determine whether a specified condition for acquiring an image is satisfied. If the specified condition for acquiring an image is satisfied (2020—Yes), the processor may perform operation (2030). If the specified condition for acquiring an image is not satisfied, or before the specified condition is satisfied (2020—No), the processor may perform operation (2040).

[0245] Referring to FIG. 20, in operation (2030), according to one embodiment, a processor of a wearable device may acquire an image and output a notification regarding the acquisition of the image. The processor, having confirmed a specified condition for acquiring an image, may perform operation (2030) to acquire or store the image without a viewfinder object (e.g., the viewfinder object (150) of FIG. 1). The processor may display a notification message (e.g., a pop-up window and / or a notification message provided by an operating system, such as a toast) to notify the acquisition of the image. The disclosure is not limited thereto, and the processor, having confirmed a specified condition for acquiring an image, may display a visual object (e.g., a dialog window) for confirming whether to acquire the image before acquiring the image.

[0246] Referring to FIG. 20, in operation (2040), a processor of a wearable device according to an embodiment may determine whether a specified condition for acquiring a video is satisfied. If the specified condition for acquiring a video is satisfied (2040 - Yes), the processor may perform operation (2050). If the specified condition for acquiring a video and the specified condition for acquiring an image are not satisfied (2040 - No), the processor may not perform operations (2030, 2050) and may continue to check the condition of operation (2010).

[0247] Referring to FIG. 20, within operation (2050), a processor of a wearable device according to one embodiment may initiate acquisition of a video and display a virtual object (e.g., virtual object (1910) of FIG. 19) and / or an indicator (e.g., indicator (1920) of FIG. 19) associated with the video. The processor may activate one or more external cameras (e.g., camera (225) of FIG. 2) and initiate recording of the video using a composite image of images of the one or more external cameras. Screens (1901, 1902) of FIG. 19 may be exemplary screens displayed by the processor performing operation (2050).

[0248] As described above, according to one embodiment, a wearable device can capture a portion of an image of an external environment displayed through at least one display covering both eyes of a user using a viewfinder object. The viewfinder object can be moved on the image displayed through the display. The size of the viewfinder object can be adjusted by a user input. The wearable device can change the position and / or size of the viewfinder object based on a gesture (e.g., a hand gesture) of a user wearing the wearable device. The wearable device can obtain an image and / or a video based on the user's gesture. The wearable device can obtain a depth map together with the image and / or the video based on the user's gesture. The wearable device can obtain an image and / or a video of a specific external object included in the external environment based on the user's gesture, independently of the motion of the user wearing the wearable device. The wearable device can acquire images and / or videos including at least one virtual object displayed through an external environment and a display based on a user's gesture.

[0249] Hereinafter, with reference to FIGS. 21A, 21B, 22A and / or 22B, exemplary appearances of wearable devices (101) described with reference to FIGS. 1 to 20 are illustrated. The wearable devices (2100) of FIGS. 21A and / or 21B and / or the wearable devices (2200) of FIGS. 22A and / or 22B may be examples of the wearable devices (101) of FIG. 1.

[0250] FIG. 21A is a perspective view illustrating an exemplary wearable device according to various embodiments. In one embodiment, the wearable device (2100) may have the form of glasses that are wearable on a body part of a user (e.g., head). The wearable device (2100) may include a head-mounted display (HMD). For example, the housing of the wearable device (2100) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (2100) may include one or more straps that are capable of being twined around the user's head, and / or one or more temples that are detachably attachable to the ears of the head.

[0251] Referring to FIG. 21A, according to one embodiment, a wearable device (2100) may include at least one display (2150) and a frame supporting at least one display (2150).

[0252] According to one embodiment, a wearable device (2100) can be worn on a part of a user's body. The wearable device (2100) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (2100). For example, the wearable device (2100) can display a virtual reality image provided from at least one optical device (2182, 2184) of FIG. 21B on at least one display (2150) in response to a user's designated gesture acquired through the motion recognition cameras (2160-2, 2160-3) of FIG. 21B.

[0253] According to one embodiment, at least one display (2150) may provide visual information to a user. For example, at least one display (2150) may include a transparent or translucent lens. At least one display (2150) may include a first display (2150-1) and / or a second display (2150-2) spaced apart from the first display (2150-1). For example, the first display (2150-1) and the second display (2150-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0254] Referring to FIG. 21B, at least one display (2150) can provide visual information transmitted from external light to a user through a lens included in the at least one display (2150), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (2150) can include a first surface (2131) and a second surface (2132) opposite to the first surface (2131). A display area can be formed on the second surface (2132) of the at least one display (2150). When a user wears the wearable device (2100), external light can be transmitted to the user by being incident on the first surface (2131) and transmitted through the second surface (2132). As another example, at least one display (2150) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (2182, 2184) on a real screen transmitted through external light, in a display area formed on the second surface (2132).

[0255] In one embodiment, at least one display (2150) may include at least one waveguide (2133, 2134) that diffracts light emitted from at least one optical device (2182, 2184) and transmits the diffracted light to a user. The at least one waveguide (2133, 2134) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (2133, 2134). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (2133, 2134) may be propagated to the other end of the at least one waveguide (2133, 2134) by the nano-pattern. At least one waveguide (2133, 2134) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (2133, 2134) may be arranged within the wearable device (2100) to guide a screen displayed by at least one display (2150) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (2133, 2134).

[0256] The wearable device (2100) can analyze an object included in a real image collected through a shooting camera (2160-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (2150). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (2100) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (2100) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using the multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (2100) can view an image displayed on at least one display (2150).

[0257] According to one embodiment, the frame may be formed as a physical structure that allows the wearable device (2100) to be worn on the user's body. According to one embodiment, the frame may be configured so that, when the user wears the wearable device (2100), the first display (2150-1) and the second display (2150-2) can be positioned corresponding to the user's left and right eyes. The frame may support at least one display (2150). For example, the frame may support the first display (2150-1) and the second display (2150-2) to be positioned corresponding to the user's left and right eyes.

[0258] Referring to FIG. 21A, the frame may include a region (2120) that at least partially contacts a portion of the user's body when the user wears the wearable device (2100). For example, the region (2120) of the frame that contacts a portion of the user's body may include a region that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (2100) makes contact with. According to one embodiment, the frame may include a nose pad (2110) that contacts a portion of the user's body. When the wearable device (2100) is worn by the user, the nose pad (2110) may contact a portion of the user's nose. The frame may include a first temple (2104) and a second temple (2105) that contact another portion of the user's body that is distinct from the portion of the user's body.

[0259] For example, the frame may include a first rim (2101) that surrounds at least a portion of the first display (2150-1), a second rim (2102) that surrounds at least a portion of the second display (2150-2), a bridge (2103) that is disposed between the first rim (2101) and the second rim (2102), a first pad (2111) that is disposed along a portion of an edge of the first rim (2101) from one end of the bridge (2103), a second pad (2112) that is disposed along a portion of an edge of the second rim (2102) from the other end of the bridge (2103), a first temple (2104) that extends from the first rim (2101) and is secured to a portion of an ear of the wearer, and a second temple (2105) that extends from the second rim (2102) and is secured to a portion of an ear opposite the ear. The first pad (2111) and the second pad (2112) may be brought into contact with a portion of the user's nose, and the first temple (2104) and the second temple (2105) may be brought into contact with a portion of the user's face and a portion of the user's ear. The temples (2104, 2105) may be rotatably connected to the rim through the hinge units (2106, 2107) of FIG. 21B. The first temple (2104) may be rotatably connected to the first rim (2101) through the first hinge unit (2106) disposed between the first rim (2101) and the first temple (2104). The second temple (2105) may be rotatably connected to the second rim (2102) via a second hinge unit (2107) disposed between the second rim (2102) and the second temple (2105). In one embodiment, the wearable device (2100) may use a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame to identify an external object (e.g., a user's fingertip) touching the frame and / or a gesture performed by the external object.

[0260] According to one embodiment, the wearable device (2100) may include hardwares (e.g., hardwares described above based on the block diagram of FIG. 2) that perform various functions. For example, the hardwares may include a battery module (2170), an antenna module (2175), at least one optical device (2182, 2184), speakers (e.g., speakers 2155-1, 2155-2), a microphone (e.g., microphones 2165-1, 2165-2, 2165-3), a light-emitting module, and / or a printed circuit board (PCB) (2190) (e.g., a printed circuit board). The various hardware components may be arranged within a frame.

[0261] According to one embodiment, the microphones (e.g., microphones 2165-1, 2165-2, 2165-3) of the wearable device (2100) may be disposed on at least a portion of the frame to acquire sound signals. A first microphone (2165-1) disposed on the bridge (2103), a second microphone (2165-2) disposed on the second rim (2102), and a third microphone (2165-3) disposed on the first rim (2101) are illustrated in FIG. 21B , but the number and arrangement of the microphones (2165) are not limited to the embodiment of FIG. 21B . When the number of microphones (2165) included in the wearable device (2100) is two or more, the wearable device (2100) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame.

[0262] According to one embodiment, at least one optical device (2182, 2184) can project a virtual object onto at least one display (2150) to provide various image information to a user. For example, at least one optical device (2182, 2184) can be a projector. At least one optical device (2182, 2184) can be positioned adjacent to at least one display (2150) or can be included within at least one display (2150) as a part of at least one display (2150). According to one embodiment, the wearable device (2100) can include a first optical device (2182) corresponding to a first display (2150-1) and a second optical device (2184) corresponding to a second display (2150-2). For example, at least one optical device (2182, 2184) may include a first optical device (2182) disposed at an edge of a first display (2150-1) and a second optical device (2184) disposed at an edge of a second display (2150-2). The first optical device (2182) may transmit light to a first waveguide (2133) disposed on the first display (2150-1), and the second optical device (2184) may transmit light to a second waveguide (2134) disposed on the second display (2150-2).

[0263] In one embodiment, the camera (2160) may include a recording camera (2160-4), an eye tracking camera (ET CAM) (2160-1), and / or a motion recognition camera (2160-2, 2160-3). The recording camera (2160-4), the eye tracking camera (2160-1), and the motion recognition cameras (2160-2, 2160-3) may be positioned at different locations on the frame and may perform different functions. The eye tracking camera (2160-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (2100). For example, the wearable device (2100) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (2160-1).

[0264] The wearable device (2100) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (2160-1). The wearable device (2100) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (2100) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (2160-1). The wearable device (2100) can render an image (or screen) displayed on at least one display (2150) based on the position of the user's eyes.

[0265] For example, the visual quality of a first region related to the gaze within an image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI (pixels per inch)) of a second region distinct from the first region may be different from each other. The wearable device (2100) may obtain an image having the visual quality of the first region matching the user's gaze and the visual quality of the second region using foveated rendering. For example, if the wearable device (2100) supports an iris recognition function, user authentication may be performed based on iris information obtained using the gaze tracking camera (2160-1). Although an example in which the gaze tracking camera (2160-1) is positioned toward the user's right eye is illustrated in FIG. 21B , the disclosure is not limited thereto, and the gaze tracking camera (2160-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0266] In one embodiment, the capturing camera (2160-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (2160-4) can be used to obtain a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (2160-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (2150). The at least one display (2150) can display a single image in which information about an actual image or background including an image of the specific object obtained using the capturing camera (2160-4) and a virtual image provided through at least one optical device (2182, 2184) are superimposed. The wearable device (2100) can compensate for depth information (e.g., the distance between the wearable device (2100) and an external object acquired through a depth sensor) using an image acquired through the capture camera (2160-4). The wearable device (2100) can perform object recognition using an image acquired using the capture camera (2160-4). The wearable device (2100) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capture camera (2160-4). The wearable device (2100) can perform a pass-through function to display an image acquired through the capture camera (2160-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (2150) while displaying a screen. In one embodiment, the camera (2160-4) may be positioned on a bridge (2103) between the first rim (2101) and the second rim (2102).

[0267] The gaze tracking camera (2160-1) can implement more realistic augmented reality by tracking the gaze of a user wearing a wearable device (2100) and matching the user's gaze with visual information provided to at least one display (2150). For example, when the wearable device (2100) looks straight ahead, the wearable device (2100) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (2150). The gaze tracking camera (2160-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (2160-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (2160-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (2160-1) may be positioned within the first rim (2101) and / or the second rim (2102) to face the direction in which the user wearing the wearable device (2100) is positioned.

[0268] The gesture recognition camera (2160-2, 2160-3) can recognize the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (2150). The gesture recognition camera (2160-2, 2160-3) can recognize the user's gesture (gesture recognition), obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (2150). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition camera (2160-2, 2160-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor may perform gesture recognition and / or object tracking functions using the motion recognition cameras (2160-2, 2160-3). In one embodiment, the motion recognition cameras (2160-2, 2160-3) may be positioned on the first rim (2101) and / or the second rim (2102).

[0269] The camera (2160) included in the wearable device (2100) is not limited to the above-described gaze tracking camera (2160-1) and motion recognition cameras (2160-2, 2160-3). For example, the wearable device (2100) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (2100) can identify an external object based on a sensor for identifying the distance between the wearable device (2100) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (2160) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (2100) may include a camera (2160) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (2100).

[0270] Although not shown, in one embodiment, the wearable device (2100) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being captured using the camera (2160). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame and hinge units (2106, 2107).

[0271] In one embodiment, the battery module (2170) may supply power to the electronic components of the wearable device (2100). In one embodiment, the battery module (2170) may be disposed within the first temple (2104) and / or the second temple (2105). For example, the battery module (2170) may be a plurality of battery modules (2170). The plurality of battery modules (2170) may be disposed within each of the first temple (2104) and the second temple (2105). In one embodiment, the battery module (2170) may be disposed at an end of the first temple (2104) and / or the second temple (2105).

[0272] The antenna module (2175) can transmit signals or power to the outside of the wearable device (2100), or receive signals or power from the outside. In one embodiment, the antenna module (2175) can be positioned within the first temple (2104) and / or the second temple (2105). For example, the antenna module (2175) can be positioned close to one surface of the first temple (2104) and / or the second temple (2105).

[0273] The speaker (2155) can output an audio signal to the outside of the wearable device (2100). The audio output module may be referred to as a speaker. In one embodiment, the speaker (2155) can be positioned within the first temple (2104) and / or the second temple (2105) so as to be positioned adjacent to the ear of a user wearing the wearable device (2100). For example, the speaker (2155) can include a second speaker (2155-2) positioned within the first temple (2104) and thus positioned adjacent to the user's left ear, and a first speaker (2155-1) positioned within the second temple (2105) and thus positioned adjacent to the user's right ear.

[0274] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (2100) to the user. For example, when the wearable device (2100) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (2101) and / or the second rim (2102).

[0275] Referring to FIG. 21B, according to one embodiment, a wearable device (2100) may include a printed circuit board (PCB) (2190). The PCB (2190) may be included in at least one of the first temple (2104) or the second temple (2105). The PCB (2190) may include an interposer disposed between at least two sub-PCBs. One or more hardwares included in the wearable device (2100) (e.g., hardwares illustrated by different blocks in FIG. 2) may be disposed on the PCB (2190). The wearable device (2100) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0276] According to one embodiment, a wearable device (2100) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (2100) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (2100). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., an x-axis, a y-axis, and a z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (2100) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (2100) based on the IMU.

[0277] Figures 22a and 22b are perspective views illustrating exemplary exterior appearances of a wearable device according to various embodiments. The wearable device (2200) of Figures 22a and 22b may include at least a portion of the hardware of the wearable device (2100) described with reference to Figures 21a and / or 21b. An example of the exterior appearance of a first side (2210) of a housing of the wearable device (2200) according to one embodiment is illustrated in Figure 22a, and an example of the exterior appearance of a second side (2220) opposite to the first side (2210) may be illustrated in Figure 22b.

[0278] Referring to FIG. 22A, a first surface (2210) of a wearable device (2200) according to one embodiment may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (2200) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (2104) and / or the second temple (2105) of FIGS. 21A and 21B). A first display (2150-1) for outputting an image to a left eye among the user's two eyes, and a second display (2150-2) for outputting an image to a right eye among the two eyes, may be disposed on the first surface (2210). The wearable device (2200) may further include a rubber or silicone packing formed on the first surface (2210) to prevent and / or reduce interference by light (e.g., ambient light) different from the light radiated from the first display (2150-1) and the second display (2150-2).

[0279] According to one embodiment, a wearable device (2200) may include cameras (2160-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (2150-1) and the second display (2150-2). The cameras (2160-1) may be referred to as the gaze tracking camera (2160-1) of FIG. 21B. According to one embodiment, a wearable device (2200) may include cameras (2160-5, 2160-6) for photographing and / or recognizing a face of a user. The cameras (2160-5, 2160-6) may be referred to as FT cameras. The wearable device (2200) can control an avatar representing the user in a virtual space based on the facial motion of the user identified using cameras (2160-5, 2160-6). For example, the wearable device (2200) can change the texture and / or shape of a portion of the avatar (e.g., a portion of the avatar representing a human face) using information obtained by cameras (2160-5, 2160-6) (e.g., FT cameras) and representing the facial expression of the user wearing the wearable device (2200).

[0280] Referring to FIG. 22b, a camera (e.g., cameras (2160-7, 2160-8, 2160-9, 2160-10, 2160-11, 2160-12)) and / or a sensor (e.g., a depth sensor (2230)) for obtaining information related to the external environment of the wearable device (2200) may be disposed on a second surface (2220) opposite to the first surface (2210) of FIG. 22a. For example, the cameras (2160-7, 2160-8, 2160-9, 2160-10) may be disposed on the second surface (2220) to recognize external objects. The cameras (2160-7, 2160-8, 2160-9, 2160-10) of FIG. 22b can correspond to the motion recognition cameras (2160-2, 2160-3) of FIG. 21b.

[0281] For example, using cameras (2160-11, 2160-12), the wearable device (2200) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (2160-11) can be placed on the second face (2220) of the wearable device (2200) to obtain an image to be displayed through the second display (2150-2) corresponding to the right eye among the two eyes. The camera (2160-12) can be placed on the second face (2220) of the wearable device (2200) to obtain an image to be displayed through the first display (2150-1) corresponding to the left eye among the two eyes. The cameras (2160-11, 2160-12) can correspond to the shooting camera (2160-4) of FIG. 21B.

[0282] According to one embodiment, the wearable device (2200) may include a depth sensor (2230) disposed on a second face (2220) to identify a distance between the wearable device (2200) and an external object. Using the depth sensor (2230), the wearable device (2200) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (2200). Although not shown, a microphone may be disposed on the second face (2220) of the wearable device (2200) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0283] In one embodiment, a method for acquiring images and / or videos using a user interface (UI) that is adjustable by the user may be required. As described above, according to one embodiment, a wearable device (e.g., wearable device (101) of FIG. 1, wearable device (2100) of FIG. 21A and / or FIG. 21B, wearable device (2200) of FIG. 22A and / or FIG. 22B) may include a housing, a display (e.g., display (220) of FIG. 2) disposed in at least a portion of the housing and arranged in front of the eyes of a user (e.g., user (110) of FIG. 1) based on wearing the wearable device, a plurality of cameras (e.g., cameras (225) of FIG. 2) configured to acquire images of at least a portion of an external environment of the wearable device, a memory (e.g., memory (215) of FIG. 2) for storing instructions, and at least one processor (e.g., processor (210) of FIG. 2) including processing circuitry. At least one processor may be configured, individually and / or collectively, to execute the instructions, and may be configured to cause the wearable device, in response to a first input, to display a view finder object (e.g., the view finder object (150) of FIG. 1) on a composite image of the images (e.g., the composite image (520) of FIG. 5), wherein the composite image may be displayed to represent a portion of the external environment beyond the display. The at least one processor may be configured, individually and / or collectively, to cause the wearable device, while displaying the view finder object on the composite image, to change at least one of a position or a size of the view finder object, in response to a second input for moving or resizing the view finder object.At least one processor may be individually and / or collectively configured to cause the wearable device, while displaying the viewfinder object on the composite image, to, in response to a third input for capturing, store a portion of the composite image corresponding to the viewfinder object in the memory.

[0284] For example, at least one processor may be individually and / or collectively configured to cause the wearable device to display a pointer object (e.g., a virtual object (342) of FIG. 3A) extending from a point in the composite image associated with a body part that includes a hand (e.g., a hand (112) of FIG. 3A). The at least one processor may be individually and / or collectively configured to cause the wearable device to detect the second input using the shape of the hand while displaying the pointer object pointing toward a boundary line of the viewfinder object (e.g., boundary line (150-1) of FIG. 8B).

[0285] For example, at least one processor may be individually and / or collectively configured to cause the wearable device to change the position of the viewfinder object using the direction of the pointer object in response to the second input detected while the wearable device is displaying the pointer object pointing toward the edge of the boundary line having a rectangular shape.

[0286] For example, at least one processor may be individually and / or collectively configured to cause the wearable device to change the size of the viewfinder object using the direction of the pointer object in response to the second input detected while the wearable device is displaying the pointer object pointing toward a vertex of the boundary line having a rectangular shape (e.g., vertex (150-2) of FIG. 8B).

[0287] For example, at least one processor may be individually and / or collectively configured to cause the wearable device, in response to the second input, to change the size of the border within the display to a size less than or equal to a specified maximum size to maintain visibility of the border.

[0288] For example, at least one processor may be individually and / or collectively configured to cause the wearable device to detect the third input using the form of a hand while displaying a pointer object pointing to a different portion of the composite image than the portion designated by the viewfinder object.

[0289] For example, the wearable device may include another camera positioned to face the user's eye. At least one processor may be individually and / or collectively configured to cause the wearable device to determine the direction of the eye using the other camera. At least one processor may be individually and / or collectively configured to cause the wearable device to change the focal length of at least one of the plurality of cameras based on the direction toward the portion of the composite image designated by the viewfinder object, using a portion of the external environment corresponding to the direction.

[0290] For example, at least one processor may be individually and / or collectively configured to cause the wearable device, in response to a fourth input indicating selection of a handle object (e.g., control handle (330) of FIGS. 3A-3C) displayed within the display together with the viewfinder object, to display visual objects corresponding to each of the photographing-related functions along a direction of one edge of the viewfinder object having a rectangular shape.

[0291] For example, the wearable device may include a depth sensor. At least one processor may be individually and / or collectively configured to cause the wearable device, in response to the third input, to store, in the memory, a depth map acquired using the depth sensor and corresponding to the portion of the composite image.

[0292] For example, at least one processor may be individually and / or collectively configured to cause the wearable device to display a virtual object representing a range measurable by the depth sensor inside the viewfinder object.

[0293] For example, at least one processor may be individually and / or collectively configured to cause the wearable device to display a virtual object representing the depth map obtained from the depth sensor, overlaid on the portion of the composite image designated by the viewfinder object, prior to detecting the third input.

[0294] For example, at least one processor may be individually and / or collectively configured to cause the wearable device to display the viewfinder object in response to the first input to execute a first software application. The at least one processor may be individually and / or collectively configured to cause the wearable device to stop displaying the viewfinder object and display an indicator indicating recording of the video (e.g., indicator (1022) of FIG. 10b and / or indicator (1920) of FIG. 19) in response to a fourth input to execute a second software application, different from the first software application, detected while acquiring video of the portion of the composite image in response to the third input.

[0295] For example, at least one processor may be individually and / or collectively configured to cause the wearable device, in response to a fifth input indicating selection of the indicator, to display the viewfinder object at a location where the viewfinder object was displayed prior to receiving the fourth input.

[0296] For example, at least one processor may be individually and / or collectively configured to cause the wearable device, in response to the first input detected while displaying the virtual object overlaid on the composite image, to display, together with the viewfinder object, the virtual object, at least a portion of which overlaps the viewfinder object, masked. The at least one processor may be individually and / or collectively configured to cause the wearable device, in response to a fourth input for capturing the portion of the composite image together with the virtual object, to display, within the viewfinder object, the at least a portion of the masked virtual object, using a specified transparency.

[0297] For example, at least one processor may be individually and / or collectively configured to cause the wearable device, while displaying the viewfinder object on the composite image, to detect a fourth input indicative of tracking of an external object within the portion of the composite image designated by the viewfinder object. The at least one processor may be individually and / or collectively configured to cause the wearable device, in response to the fourth input, to change the position of the viewfinder object on the composite image using a position of the external object relative to the composite image.

[0298] In one embodiment, as described above, a method of a wearable device may be provided. The wearable device may include a housing, a display disposed within at least a portion of the housing and arranged in front of the eyes of a user wearing the wearable device, and a plurality of cameras for acquiring images of at least a portion of an external environment of the wearable device. The method may include, in response to a first input, displaying a viewfinder object on a composite image of the images, the composite image being displayed to represent a portion of the external environment beyond the display. The method may include, in response to a second input for moving or resizing the viewfinder object while displaying the viewfinder object on the composite image, changing at least one of a position or a size of the viewfinder object. The method may include, while displaying the viewfinder object on the composite image, an operation of storing a portion of the composite image corresponding to the viewfinder object in the memory in response to a third input for taking a picture.

[0299] For example, the changing action may include displaying a pointer object extending from a point of the composite image associated with a body part including a hand. The method may include detecting the second input using the shape of the hand while displaying the pointer object toward the boundary line of the viewfinder object.

[0300] For example, the changing action may include an action of changing the position of the viewfinder object using the direction of the pointer object in response to the second input detected while displaying the pointer object pointing toward the edge of the boundary line having a rectangular shape.

[0301] For example, the changing action may include an action of changing the size of the viewfinder object using the direction of the pointer object in response to the second input detected while displaying the pointer object pointing toward a vertex of the boundary line having a rectangular shape.

[0302] For example, the changing action may include, in response to the second input, changing the size to a maximum size or less specified to maintain visibility of the border line within the display.

[0303] For example, the storing operation may include detecting the third input using the shape of a hand while displaying a pointer object pointing to a different portion of the composite image than the portion designated by the viewfinder object.

[0304] For example, the method may include an operation of determining the direction of the eye using another camera positioned to face the user's eye. The method may include an operation of changing the focal length of at least one of the plurality of cameras using a portion of the external environment corresponding to the direction, based on the direction toward the portion of the composite image designated by the viewfinder object.

[0305] For example, the method may include, in response to a fourth input indicating selection of a handle object displayed within the display together with the viewfinder object, displaying visual objects corresponding to each of the photographing-related functions along a direction of one edge of the viewfinder object having a rectangular shape.

[0306] For example, the storing operation may include, in response to the third input, storing, in the memory, a depth map obtained using a depth sensor of the wearable device, together with the portion of the composite image, and corresponding to the portion.

[0307] For example, the displaying action may include an action of displaying a virtual object representing a range measurable by the depth sensor inside the viewfinder object.

[0308] For example, the method may include, prior to detecting the third input, displaying a virtual object representing the depth map obtained from the depth sensor as an overlay on the portion of the composite image designated by the viewfinder object.

[0309] For example, the displaying action may include displaying the viewfinder object in response to the first input for executing the first software application. The method may include displaying an indicator indicating recording of the video in response to a fourth input for executing a second software application, different from the first software application, detected while acquiring video of the portion of the composite image in response to the third input.

[0310] In one embodiment, the method may include, in response to a fifth input indicating selection of the indicator, displaying the viewfinder object at a location where the viewfinder object was displayed prior to receiving the fourth input.

[0311] For example, the displaying operation may include, in response to the first input detected while displaying the virtual object overlaid on the composite image, displaying the virtual object, with at least a portion overlapping the viewfinder object masked, together with the viewfinder object. The method may include, in response to a fourth input for capturing the portion of the composite image together with the virtual object, displaying, within the viewfinder object, the at least a portion of the masked virtual object using a specified transparency.

[0312] In one embodiment, the method may include an operation of detecting, while displaying the viewfinder object on the composite image, a fourth input indicating tracking of an external object within the portion of the composite image designated by the viewfinder object. The method may include an operation of changing, in response to the fourth input, the position of the viewfinder object on the composite image using a position of the external object relative to the composite image.

[0313] In one embodiment, as described above, a non-transitory computer-readable storage medium comprising instructions may be provided. The instructions, when individually and / or collectively executed by at least one processor of a wearable device, the wearable device comprising a housing, a display disposed within at least a portion of the housing and arranged in front of a user's eyes based on wearing the wearable device, and a plurality of cameras configured to acquire images of at least a portion of an external environment of the wearable device, may cause the wearable device to display a viewfinder object on the display, superimposed on a composite image of the images. The instructions, when executed by the processor, may cause the wearable device to store a first portion of the composite image corresponding to the first location in the memory, in response to receiving an input for capturing while displaying the viewfinder object at a first location on the display. The instructions, when executed by the processor, may cause the wearable device to, in response to receiving an input for capturing while displaying the viewfinder object at a second location on the display, store a second portion of the composite image corresponding to the second location in the memory.

[0314] According to one embodiment, a wearable device (e.g., wearable device (101) of FIG. 1 , wearable device (2100) of FIG. 21A and / or FIG. 21B , wearable device (2200) of FIG. 22A and / or FIG. 22B ) as described above may include a housing, a display (e.g., display (220) of FIG. 2 ) disposed in at least a portion of the housing and arranged in front of the eyes of a user (e.g., user (110) of FIG. 1 ) based on wearing the wearable device), a plurality of cameras (e.g., cameras (225) of FIG. 2 ) configured to acquire images of at least a portion of an external environment of the wearable device, a memory (e.g., memory (215) of FIG. 2 ) for storing instructions, and at least one processor (e.g., processor (210) of FIG. 2 ) including processing circuitry. At least one processor may be individually and / or collectively configured to execute the instructions, and may be configured to cause the wearable device to display a viewfinder object (e.g., the viewfinder object (150) of FIG. 1) on the display, overlaid on a composite image of the images (e.g., the composite image (520) of FIG. 5). The at least one processor may be individually and / or collectively configured to cause the wearable device, in response to receiving an input for capturing while displaying the viewfinder object at a first location on the display, to store a first portion of the composite image corresponding to the first location in the memory. The at least one processor may be individually and / or collectively configured to cause the wearable device, in response to receiving an input for capturing while displaying the viewfinder object at a second location on the display, to store a second portion of the composite image corresponding to the second location in the memory.

[0315] According to one embodiment, a wearable device as described above may include a housing, a first display configured to be positioned toward an eye of a user when worn by the user, a second display oriented in a second direction opposite a first direction in which the first display is oriented, one or more cameras, a memory including one or more storage media storing instructions, and at least one processor including a processing circuit. The at least one processor may be individually and / or collectively configured to execute the instructions and cause the wearable device to acquire images using the one or more cameras. The at least one processor may be individually and / or collectively configured to cause the wearable device to control the first display to display a screen representing an environment surrounding the wearable device using at least a portion of the images. The at least one processor may be individually and / or collectively configured to cause the wearable device, while displaying the screen, to receive a first input for executing a camera application. At least one processor may be individually and / or collectively configured to cause the wearable device, in response to the first input, to control the first display to visually highlight a portion of the screen corresponding to an area to be captured using the camera application relative to the remaining portion of the screen. The at least one processor may be individually and / or collectively configured to cause the wearable device, while displaying the portion of the screen visually highlighted relative to the remaining portion of the screen, to receive a second input to capture the area indicated as the portion of the screen.At least one processor may be individually and / or collectively configured to cause the wearable device, in response to the second input, to capture the area. At least one processor may be individually and / or collectively configured to cause the wearable device to control the second display to display an indicator indicating that the one or more cameras are performing a photographing operation.

[0316] For example, at least one processor may be individually or collectively configured to cause the wearable device to control the first display to display executable objects related to the capture at a location adjacent to the portion of the screen that is visually highlighted relative to the remaining portion of the screen.

[0317] For example, the executable objects may be overlapped on the screen.

[0318] For example, at least one processor may be configured, individually or collectively, to cause the wearable device to control the first display to further display a visual object associated with a focus on the portion of the screen when the portion of the screen is visually highlighted relative to the remaining portion of the screen.

[0319] For example, at least one processor may be individually or collectively configured to cause the wearable device, in response to the second input, to store the area and a depth map for the area.

[0320] For example, the wearable device may include a button that is at least partially visible through the housing. At least one processor may be individually or collectively configured to cause the wearable device to receive the second input through the button.

[0321] According to one embodiment, a wearable device as described above may include a housing, a first display disposed on a first side of the housing facing a face of a user based on the wearable device being worn by the user, a second display disposed on a second side of the housing facing an external environment of the wearable device based on the wearable device being worn by the user, a plurality of cameras configured to acquire a plurality of images of at least a portion of an external environment of the wearable device, a memory storing instructions, the memory including one or more storage media, and at least one processor including a processing circuit. The at least one processor may be individually or collectively configured to execute the instructions, and may be configured to cause the wearable device to display, through the first display, a composite image generated based on the plurality of images and of at least a portion of the external environment, and a view finder object at least partially overlapping the composite image. The instructions, when individually or collectively executed by the at least one processor, may be configured to cause the wearable device, in an image capture mode, to display a first visual notification corresponding to the image capture mode through the second display while the composite image and the viewfinder object are displayed through the first display. The instructions, when individually or collectively executed by the at least one processor, may be configured to cause the wearable device, in a video capture mode, to display a second visual notification corresponding to the video capture mode through the second display while the composite image and the viewfinder object are displayed through the second display.The instructions, when individually or collectively executed by the at least one processor, may be configured to cause the wearable device, in the image capture mode or the video capture mode, to store at least a portion of the composite image corresponding to the viewfinder object in the memory, in response to a user input.

[0322] For example, at least one processor may be individually or collectively configured to cause the wearable device to display a control handle through the first display, at a location adjacent to the viewfinder object.

[0323] For example, at least one processor may be individually or collectively configured to cause the wearable device to display, through the display, a control handle, such that the control handle includes at least one visual object corresponding to a function for navigating an image or video stored in the memory. The instructions, when individually or collectively executed by the at least one processor, may be configured to cause the wearable device to display, through the first display, a list including the image or video, based on an input selecting the visual object.

[0324] For example, at least one processor may be individually or collectively configured to cause the wearable device to control the first display to display a visual object associated with a focus positioned within the viewfinder object.

[0325] For example, the wearable device may include a button that is at least partially visible through the housing. At least one processor may be individually or collectively configured to cause the wearable device to receive the input through the button.

[0326] For example, at least one processor may be individually or collectively configured to cause the wearable device, in response to the input, to store at least a portion of the composite image and a depth map for at least a portion of the composite image.

[0327] For example, at least one processor may be individually or collectively configured to cause the wearable device, in response to the input, to display at least one of the first visual notification or the second visual notification via the second display.

[0328] For example, at least one processor may be individually or collectively configured to cause the wearable device to display executable objects associated with the image capture mode and the video capture mode via the first display.

[0329] For example, at least one processor may be individually or collectively configured to cause the wearable device to display the viewfinder object by visually highlighting a portion of the screen displayed on the first display, relative to other portions, that corresponds to at least a portion of the composite image to be stored by the input.

[0330] In one embodiment, as described above, a method of operating a wearable device may be provided. The wearable device may include a housing, a first display configured to be positioned toward a user's eyes based on the wearer being worn by the wearable device, a second display facing a second direction opposite a first direction in which the first display faces, and one or more cameras. The method may include an operation of acquiring images using the one or more cameras. The method may include an operation of controlling the first display to display a screen representing an environment surrounding the wearable device using at least a portion of the images. The method may include an operation of receiving a first input for executing a camera application while displaying the screen. The method may include an operation of controlling the first display to visually highlight a portion of the screen corresponding to an area to be captured using the camera application relative to the remaining portion of the screen, in response to the first input. The method may include receiving a second input for capturing the area indicated as the portion of the screen while displaying the portion of the screen visually highlighted with respect to the remaining portion of the screen. The method may include capturing the area in response to the second input. The method may include controlling the second display to display an indicator indicating that filming is being performed using the one or more cameras.

[0331] For example, the act of controlling the first display may include the act of controlling the first display to display executable objects related to shooting at a location adjacent to the portion of the screen that is visually highlighted relative to the remaining portion of the screen.

[0332] For example, the executable objects may be overlapped on the screen.

[0333] For example, the act of controlling the first display may include the act of controlling the first display to further display a visual object related to focus on the portion of the screen based on displaying the portion of the screen visually highlighted relative to the remaining portion of the screen.

[0334] For example, the capturing operation may include, in response to the second input, storing the area and a depth map for the area.

[0335] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in various embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0336] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0337] The method according to the exemplary embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may store a computer-executable program or temporarily store it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and those configured to store program instructions, including ROM, RAM, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0338] While the disclosure has been illustrated and described with reference to various embodiments, it is to be understood that the various exemplary embodiments are intended to be illustrative and not limiting. Those skilled in the art will further recognize that various changes in form and detail may be made without departing from the full scope of the present disclosure, including the appended claims and their equivalents. It will also be understood that any of the embodiment(s) described herein may be used in combination with any other embodiment(s) described herein.

Claims

1. In wearable devices, housing; A first display configured to be positioned toward the eyes of a user based on being worn by the user; A second display facing a second direction opposite to the first direction in which the first display faces; One or more cameras; A memory comprising one or more storage media storing instructions; and comprising at least one processor, comprising a processing circuit; At least one processor is configured, individually or collectively, to execute said instructions, wherein said wearable device, Acquiring images using one or more of the above cameras; Controlling the first display to display a screen representing an environment surrounding the wearable device using at least a portion of the images; While displaying the above screen, a first input for executing a camera application is received; In response to said first input, controlling said first display to visually highlight a portion of said screen corresponding to an area to be captured using said camera application over the remaining portion of said screen; While displaying said part of said screen visually highlighted over said remaining part of said screen, receiving a second input for capturing said area displayed as said part of said screen; and In response to the second input, Capturing the above area; and configured to cause said second display to be controlled to display an indicator that provides notification of the performance of shooting using said one or more cameras; Wearable devices.

2. In claim 1, at least one processor, individually or collectively, the wearable device, configured to cause said first display to be controlled to display executable objects related to shooting at a location adjacent to said portion of said screen that is visually highlighted with respect to said remaining portion of said screen; Wearable devices.

3. In claim 2, the executable objects are: Overlapping on the above screen, Wearable devices.

4. In claim 1, At least one processor, individually or collectively, the wearable device, When displaying said portion of said screen visually highlighted over said remaining portion of said screen, controlling said first display to further display a visual object related to focus on said portion of said screen; Wearable devices.

5. In claim 1, At least one processor, individually or collectively, the wearable device, In response to said second input, configured to cause storing of said region and a depth map for said region, Wearable devices.

6. In claim 1, further comprising a button at least partially visible through said housing, At least one processor, individually or collectively, the wearable device, configured to cause the second input to be received through the operation of the above button, Wearable devices.

7. In wearable devices, housing; A first display disposed on a first surface of the housing, directed toward the face of the user, based on the wearable device being worn by the user; A second display disposed on a second surface of the housing, facing the external environment of the wearable device, based on the wearable device being worn by the user; A plurality of cameras configured to acquire a plurality of images of at least a portion of an external environment of the wearable device; A memory storing instructions, comprising one or more storage media; and comprising at least one processor including a processing circuit; At least one processor is configured, individually or collectively, to execute said instructions, wherein said wearable device, Displaying a composite image of at least a part of the external environment and a view finder object at least partially superimposed on the composite image through the first display based on the plurality of images; In the image capturing mode, while the composite image and the viewfinder object are displayed through the first display, a first visual notification corresponding to the image capturing mode is displayed through the second display; In the video shooting mode, while the composite image and the viewfinder object are displayed through the second display, a second visual notification corresponding to the video shooting mode is displayed through the second display; and In the image capturing mode or the video capturing mode, in response to an input, configured to cause storing at least a portion of the composite image corresponding to the viewfinder object in the memory, Wearable devices.

8. In claim 7, at least one processor, individually or collectively, the wearable device, configured to cause a control handle to be displayed through the first display at a location adjacent to the viewfinder object; Wearable devices.

9. In claim 8, at least one processor is configured to individually or collectively execute the instructions, and the wearable device, displaying the control handle through the first display, so as to include at least one visual object corresponding to a function for navigating an image or video stored in the memory within the control handle; and Based on an input selecting said visual object, configured to cause a list including said image or said video to be displayed through said first display, Wearable devices.

10. In claim 7, at least one processor is configured to individually or collectively execute the instructions, and the wearable device, configured to cause said first display to be controlled so as to display a visual object associated with a focus positioned within said viewfinder object; Wearable devices.

11. In claim 7, further comprising a button at least partially visible through said housing, At least one processor is configured, individually or collectively, to execute said instructions, wherein said wearable device, configured to cause receiving said input via said button, Wearable devices.

12. In claim 7, At least one processor is configured, individually or collectively, to execute said instructions, wherein said wearable device, In response to said input, configured to cause storing of at least a portion of said composite image and a depth map for at least a portion of said composite image; Wearable devices.

13. In claim 7, At least one processor is configured, individually or collectively, to execute said instructions, wherein said wearable device, In response to said input, configured to cause at least one of said first visual notification or said second visual notification to be displayed through said second display; Wearable devices.

14. In claim 7, At least one processor is configured, individually or collectively, to execute said instructions, wherein said wearable device, configured to cause executable objects related to the image capturing mode and the video capturing mode to be displayed through the first display; Wearable devices.

15. In claim 7, At least one processor is configured, individually or collectively, to execute said instructions, wherein said wearable device, configured to cause the viewfinder object to be displayed by visually highlighting a portion of the screen displayed on the first display corresponding to at least a portion of the composite image to be stored by the input, relative to other portions; Wearable devices.

Images