Electronic device for synthesizing image obtained from camera and method thereof

Abstract

According to one embodiment, a processor of a head-mounted display (HMD) device may control a camera to obtain a first image of the outside of the HMD device. The processor may detect a portion of the first image corresponding to an external HMD device. The processor may receive information about a body part covered by the external HMD device from the external HMD device via the communication circuit. The processor may obtain a second image representing the body part by performing rendering on the body part on the basis of the information. The processor may display, by means of at least one display, the second image located in the portion of the first image, and a synthetic image including the first image.

Description

Electronic device and method for synthesizing images acquired from a camera

[0001] The present disclosure relates to an electronic device (e.g., a head-mounted display (HMD) device) and a method thereof for synthesizing images acquired from a camera.

[0002] To provide an enhanced user experience, electronic devices are being developed that provide augmented reality (AR) services, which display computer-generated information in conjunction with external objects in the real world. These electronic devices may be wearable devices worn by the user. For example, these electronic devices may be AR glasses and / or head-mounted displays (HMDs).

[0003] The information is disclosed solely as supporting background information for the present disclosure. No determination or assertion is made as to whether any of the above is applicable as prior art to the present disclosure.

[0004] One aspect of the present disclosure is to address at least the problems and / or disadvantages described above and to provide at least the advantages described below. Accordingly, one aspect of the present disclosure is to provide an electronic device (e.g., a head-mounted display (HMD) device) and a method thereof for synthesizing images acquired from a camera.

[0005] Additional aspects will be set forth in part in the description which follows, and in part will be apparent from the description, or may be learned by practice of the disclosed embodiments.

[0006] According to one aspect of the present disclosure, a head-mounted display (HMD) device is provided. The HMD device may include a camera, communication circuitry, at least one display, a memory storing one or more computer programs, and one or more processors communicatively coupled to the camera, the communication circuitry, the at least one display, and the memory. The one or more computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the HMD device to control the camera to acquire a first image of an exterior surface in front of the HMD device. The one or more computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the HMD device to detect a portion of the first image corresponding to an external HMD device. The one or more computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the HMD device to receive information about a body part covered by the external HMD device from the external HMD device through the communication circuit. The one or more computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the HMD device to perform rendering of the body part based on the information to obtain a second image representing the body part.The one or more computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the HMD device to display, through the at least one display, a composite image including the first image and the second image, wherein the second image may be positioned at the portion of the first image.

[0007] According to another aspect of the present disclosure, a method is provided, which is performed by an HMD device including a camera, a communication circuit, at least one display, and one or more processors. The method may include an operation of controlling the camera to obtain a first image external to the HMD device by the HMD device. The method may include an operation of detecting, by the HMD device, a portion of the first image corresponding to an external HMD device. The method may include an operation of receiving, by the HMD device, information about a body part covered by the external HMD device from the external HMD device through the communication circuit. The method may include an operation of performing rendering on the body part based on the information by the HMD device to obtain a second image representing the body part. The method may include an operation of generating, by the HMD device, a composite image of the first image and the second image through the at least one display, wherein the second image may be positioned on the portion of the first image.

[0008] According to another embodiment of the present disclosure, an electronic device is provided. The electronic device may include a camera, a communication circuit, at least one display, a memory storing one or more computer programs, and one or more processors communicatively coupled to the camera, the communication circuit, the at least one display, and the memory. The one or more computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the electronic device to detect, within a video acquired through the camera, a portion corresponding to a head-mounted device (HMD) device. The one or more computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the electronic device to obtain, based on information received from the HMD device through the communication circuit, a virtual object representing a body part covered by the HMD device. The one or more computer programs may include computer-executable instructions that, when executed individually or collectively by the one or more processors, cause the electronic device to display at least a portion of the virtual object on at least a portion of the at least one display on which the detected portion is displayed while displaying the video on the at least one display.

[0009] According to another aspect of the present disclosure, a method is provided, comprising a camera, a communication circuit, at least one display, and one or more processors. The method may include an operation of detecting, by the electronic device, a portion corresponding to an HMD device within a video acquired through the camera. The method may include an operation of obtaining, by the electronic device, a virtual object representing a body part covered by the HMD device based on information received from the HMD device through the communication circuit. The method may include an operation of displaying, by the electronic device, at least a portion of the virtual object on at least a portion of the at least one display on which the detected portion is displayed while displaying the video on the at least one display.

[0010] According to another aspect of the present disclosure, one or more non-transitory computer-readable storage media are provided storing one or more computer programs including computer-readable instructions, wherein the computer-readable instructions, when individually or collectively executed by one or more processors of an HMD device, cause the HMD device to perform operations. The above operations may include: controlling a camera of the HMD device to obtain a first image external to the HMD device; detecting, by the HMD device, a portion of the first image corresponding to an external HMD device; receiving, by the HMD device, information about a body part covered by the external HMD device from the external HMD device through a communication circuit of the HMD device; performing, by the HMD device, rendering of the body part based on the information to obtain a second image representing the body part; and displaying, by the HMD device, a composite image of the first image and the second image through at least one display of the HMD device, wherein the second image is positioned on the portion of the first image.

[0011] Other aspects, advantages, and salient features of the present disclosure will become apparent to those skilled in the art from the following detailed description of various embodiments of the present disclosure, taken in conjunction with the accompanying drawings.

[0012] The above-described and other aspects, features, and advantages of some embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings:

[0013] FIG. 1 illustrates an example of a screen displayed by a head-mounted display (HMD) device according to one embodiment of the present disclosure;

[0014] FIGS. 2A and 2B illustrate an example of a block diagram of an HMD device according to various embodiments of the present disclosure;

[0015] FIG. 3 illustrates an example of a flowchart of an HMD device according to one embodiment of the present disclosure;

[0016] FIG. 4 illustrates an example of operation of an HMD device based on images and / or videos acquired from a camera, according to one embodiment of the present disclosure;

[0017] FIG. 5 illustrates an example of an operation of an HMD device for processing information received from an external HMD device, according to one embodiment of the present disclosure;

[0018] FIGS. 6A and 6B illustrate an example of an operation of an HMD device for processing information received from an external HMD device, according to one embodiment of the present disclosure;

[0019] FIG. 7A and FIG. 7B illustrate an example of operation of an HMD device according to various embodiments of the present disclosure;

[0020] FIG. 8 illustrates a signal flow diagram between HMD devices according to one embodiment of the present disclosure;

[0021] FIG. 9A illustrates an example of a perspective view of a wearable device according to one embodiment of the present disclosure;

[0022] FIG. 9b illustrates an example of one or more hardware elements disposed within a wearable device according to one embodiment of the present disclosure;

[0023] FIG. 10A and FIG. 10B illustrate an example of an appearance of a wearable device according to various embodiments of the present disclosure; and

[0024] FIG. 11 is a diagram of a network environment related to a metaverse service according to one embodiment of the present disclosure.

[0025] It should be noted that in the drawings, like reference numerals are used to describe the same or similar elements, features, and structures.

[0026] The following description, with reference to the accompanying drawings, is provided to assist in a comprehensive understanding of various embodiments of the present disclosure as defined by the claims and their equivalents. While it includes various specific details that aid in this understanding, these are to be considered merely exemplary. Accordingly, those skilled in the art will recognize that various modifications and variations of the various embodiments described herein can be made without departing from the scope of the present disclosure. Additionally, descriptions of well-known functions and aspects may be omitted for clarity and brevity.

[0027] The terms and words used in the following description and claims are not limited to their bibliographic meanings, but are used solely by the inventors to facilitate a clear and consistent understanding of the present disclosure. Accordingly, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only and is not intended to limit the present disclosure to the appended claims and their equivalents.

[0028] The singular forms "a," "an," and "the" are to be understood to include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a part plane" includes one or more such planes.

[0029] The various embodiments of this document and the terminology used therein are not intended to limit the technology described in this document to a specific embodiment, but should be understood to include various modifications, equivalents, and / or substitutes of the embodiment. In connection with the description of the drawings, similar reference numerals may be used for similar components. In this document, expressions such as “A or B,” “at least one of A and / or B,” “A, B, or C,” or “at least one of A, B, and / or C” can include all possible combinations of the items listed together. Expressions such as “first,” “second,” “first,” or “second” can modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another, but do not limit the corresponding components. When it is said that a component (e.g., a first component) is “(functionally or communicatively) connected” or “connected” to another component (e.g., a second component), said component may be directly connected to said other component, or may be connected via another component (e.g., a third component).

[0030] The term "module" as used in this document includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimal unit or portion thereof that performs one or more functions. For example, a module may be composed of an application-specific integrated circuit (ASIC).

[0031] It is apparent that the blocks of each drawing and combination of drawings are executed by one or more computer programs containing instructions. The one or more computer programs may be entirely stored in a single memory device, or the one or more computer programs may be divided into different portions of different memory devices.

[0032] The functions or operations of the present disclosure may be processed by a single processor or a combination of processors. A single processor or a combination of processors is a circuit that performs processing and includes circuits such as an application processor (AP, e.g., a central processing unit (CPU)), a communication processor (CP, e.g., a modem), a graphics processing unit (GPU), a neural processing unit (NPU) (e.g., an artificial intelligence (AI) chip), a Wi-Fi chip, a Bluetooth® chip, a global positioning system (GPS) chip, a near field communication (NFC) chip, connectivity chips, a sensor controller, a touch controller, a fingerprint sensor controller, a display driver integrated circuit (IC), an audio CODEC chip, a universal serial bus (USB) controller, a camera controller, an image processing IC, a microprocessor unit (MPU), a system on chip (SoC), an IC, or the like.

[0033] FIG. 1 illustrates an example of a screen (130) displayed by an electronic device (e.g., a head-mounted display (HMD) device) (101) according to one embodiment of the present disclosure.

[0034] The electronic device (101) may include a head-mounted display (HMD) that is wearable on the head of a user (110). The electronic device (101) may be referred to as a wearable device, a head-mounted device (HMD) (or HMD device), a headgear electronic device, glasses-type (or goggle-type) electronic device, a video see-through (VST) or visible see-through device, an extended reality (XR) device, a virtual reality (VR) device, and / or an augmented reality (AR) device. Although the external appearance of the electronic device (101) in the form of glasses is illustrated, the embodiment is not limited thereto. An example of a hardware configuration included in the electronic device (101) is exemplarily described with reference to FIGS. 2A and 2B . An example of the structure of an electronic device (101) wearable on the head of a user (110) is described with reference to FIGS. 8A, 8B, 9A, and / or 9B. The electronic device (101) may be referred to as an electronic device. For example, the electronic device may include an accessory (e.g., a strap) for attaching to the head of the user (110).

[0035] According to one embodiment, the electronic device (101) may perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, when a user (110) wears the electronic device (101), the electronic device (101) may include at least one lens positioned adjacent to the eyes of the user (110). The electronic device (101) may combine ambient light passing through the lens with light emitted from a display of the electronic device (101). A display area of ​​the display may be formed within the lens through which the ambient light passes. Since the electronic device (101) combines the ambient light and the light emitted from the display, the user (110) may see an image that is a mixture of a real object recognized by the ambient light and a virtual object formed by the light emitted from the display. The augmented reality, mixed reality, and / or virtual reality described above may be referred to as extended reality (XR).

[0036] According to one embodiment, the electronic device (101) can perform functions related to video see-through (VST) and / or virtual reality (VR). For example, when a user (110) wears the electronic device (101), the electronic device (101) can include a housing that covers the eyes of the user (110). The electronic device (101), in this state, can include a display disposed on a first side of the housing facing the eyes. The electronic device (101) can include a camera disposed on a second side opposite the first side. Using the camera, the electronic device (101) can obtain images and / or videos representing ambient light. The electronic device (101) can output the images and / or videos within the display disposed on the first side, thereby allowing the user (110) to recognize the ambient light through the display. The displaying area (or displaying region) (or active area or active region) of the display arranged on the first surface may be formed by one or more pixels included in the display. The electronic device (101) may synthesize a virtual object into an image and / or video output through the display, thereby allowing the user (110) to recognize the virtual object together with an actual object recognized by ambient light.

[0037] Referring to FIG. 1, an example of a screen (130) displayed by an electronic device (101) is illustrated. The electronic device (101) may display images and / or videos of an external environment including a front direction of a user (110) wearing the electronic device (101) based on VST (or pass-through). By using the screen (130) including the images and / or videos, the electronic device (101) may provide a user experience similar to that of a user (110) wearing the electronic device (101) viewing the external environment (or similar to that of a user not wearing the electronic device (101). Hereinafter, VST and / or pass-through may refer to a function supported by the electronic device (101) and a function of displaying a screen (130) including images and / or videos of an external environment to a user (110) wearing the electronic device (101).

[0038] Referring to FIG. 1, an environment is illustrated including a user (110) wearing an electronic device (101) and another user (125) wearing an external electronic device (120). For convenience of explanation, the user (110) may be referred to as a first user (110), and the other user (125) may be referred to as a second user (125). Similarly, the electronic device (101) may be referred to as a first electronic device (101), and the external electronic device (120) may be referred to as a second electronic device (120). In the environment of FIG. 1, since the electronic device (101) covers both eyes of the first user (110) and the external electronic device (120) covers both eyes of the second user (125), the first user (110) cannot see a part of the face including both eyes of the second user (125), and the second user (125) cannot see a part of the face including both eyes of the first user (110). When the first user (110) and the second user (125) are conversing with each other, nonverbal expressions including facial expressions may be blocked by the electronic device (101) and / or the external electronic device (120).

[0039] In one embodiment, the electronic device (101) may change at least a portion of the face of the second user (125) that is covered by the second electronic device (120) within a screen (130) displayed to the first user (110) to support exchanging nonverbal expressions between the first user (110) and the second user (125). Referring to FIG. 1, within an image and / or video within the screen (130) displayed based on pass-through, the electronic device (101) may display a visual object (140) corresponding to the second user (125).

[0040] In one embodiment, the electronic device (101) may combine a virtual object (e.g., a three-dimensional graphical object), an image and / or a video representing a body part of the second user (125) covered by the second electronic device (120) with a portion (142) covered (or obscured) by the second electronic device (120) within the visual object (140). The virtual object, image and / or video combined with the portion (142) may be determined based on information (150) received from the second electronic device (120). For example, the information (150) may relate to two eyes of the second user (125) detected by the second electronic device (120). The information (150) may include an image and / or a video of two eyes of the second user (125). Information (150) may include parameters indicating the directions of the two eyes of the second user (125).

[0041] Referring to FIG. 1, since a portion (142) corresponding to a second electronic device (120) within a screen (130) displayed by an electronic device (101) is replaced with a virtual object, image and / or video representing the face of a second user (125), a first user (110) wearing the electronic device (101) can see the entire face of the second user (125). The first user (110) seeing the face of the second user (125) can recognize nonverbal expressions (e.g., facial expressions) of the second user (125). The electronic device (101) can support interaction between the first user (110) and the second user (125) by using the screen (130) including the entire face of the second user (125). Similarly, the electronic device (101) may transmit information related to a facial expression of a first user (110) wearing the electronic device (101) to an external electronic device (120). The information may include images, videos, and / or parameters related to both eyes of the first user (110), similar to the information (150) transmitted from the external electronic device (120) to the electronic device (101).

[0042] As described above, according to one embodiment, the electronic device (101) may provide a screen (130) related to an external environment to a first user (110) wearing the electronic device (101) based on MR, AR and / or XR. In order to support interaction between users (e.g., the first user (110) and / or the second user (125)) wearing electronic devices (e.g., the electronic device (101) and / or the external electronic device (120)), the electronic device (101) may display a screen (130) based on pass-through. While displaying a visual object (140) for a counterpart (e.g., the second user (125)) within the screen (130), the electronic device (101) may perform partial rendering for a portion (142) covered by the external electronic device (120) within the visual object (140). The hardware configuration of the electronic device (101) for the above partial rendering is described with reference to FIG. 2a and / or FIG. 2b.

[0043] Based on the above partial rendering, the electronic device (101) can at least partially display the face of the other party covered (or hidden) by the external electronic device (120) instead of the visual object representing the external electronic device (120) within the portion (142). An operation of the electronic device (101) to display the face of the other party covered by the external electronic device (120) is described with reference to FIGS. 3, 5, 6A, 6B, 7A, and / or 7B. An operation of the electronic device (101) receiving information (150) from the external electronic device (120) and similarly transmitting other information to the external electronic device (120) is described with reference to FIG. 8.

[0044] Although the operation of an electronic device (101) providing a user experience based on pass-through and / or VST is illustrated, the embodiment is not limited thereto. For example, in one embodiment of an electronic device (101) having a form factor based on AR that transmits external light, the electronic device (101) can project light representing the face of a second user (125) wearing the external electronic device (120) onto a portion (142) corresponding to the external electronic device (120) within a screen (130) based on external light. The first user (110) viewing the projected light can see the face of the second user (125) as if it were not obscured by the external electronic device (120).

[0045] Below, the hardware configuration of the electronic device (101) of FIG. 1 is described with reference to FIG. 2a and / or FIG. 2b.

[0046] FIGS. 2A and 2B illustrate an example of a block diagram of an electronic device (101) according to various embodiments of the present disclosure. The electronic device (101) of FIGS. 2A and / or 2B may include the electronic device (101) of FIG. 1.

[0047] Referring to FIG. 2A, an electronic device (101) according to one embodiment may include at least one of a processor (210), a memory (215), a display (220), a camera (225), a sensor (230), and a communication circuit (235). The processor (210), the memory (215), the display (220), the camera (225), the sensor (230), and / or the communication circuit (235) may be electrically and / or operably coupled with each other by an electronic component such as a communication bus (202). The type and / or number of hardware components included in the electronic device (101) is not limited to those illustrated in FIG. 2A. For example, the electronic device (101) may include only some of the hardware components illustrated in FIG. 2A.

[0048] The processor (210) of the electronic device (101) according to one embodiment may include a hardware component for processing data based on instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a field programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In one embodiment, the electronic device (101) may include one or more processors. The processor (210) may have a multi-core processor structure, such as a dual core, a quad core, a hexa core, and / or an octa core.

[0049] The memory (215) of the electronic device (101) according to one embodiment may include a hardware component for storing data and / or instructions input to and / or output from the processor (210). The memory (215) may include, for example, volatile memory such as random-access memory (RAM) and / or non-volatile memory such as read-only memory (ROM). The volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disc, and embedded multimedia card (eMMC). In one embodiment, the memory (215) may be referred to as storage.

[0050] In one embodiment, the display (220) of the electronic device (101) can output visualized information (e.g., screen (130) of FIG. 1, screen (540) of FIG. 5, screen (630) of FIG. 6A and / or FIG. 6B) to a user of the electronic device (101) (e.g., user (110) of FIG. 1). For example, the display (220) can visualize information provided from a processor (210) that includes a circuit such as a graphic processing unit (GPU). The display (220) can include a flexible display, a flat panel display (FPD), and / or electronic paper. The display (220) can include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light emitting diodes (LEDs). The LEDs can include organic LEDs (OLEDs). Embodiments are not limited thereto, and for example, if the electronic device (101) includes a lens for transmitting external light (or ambient light), the display (220) may include a projector (or projection assembly) for projecting light onto the lens. In one embodiment, the display (220) may be referred to as a display panel and / or a display module.

[0051] In one embodiment, the camera (225) of the electronic device (101) may include optical sensors (e.g., a charged coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor) that generate electrical signals representing the color and / or brightness of light. The camera (225) may be referred to as an image sensor and may be included in the sensor (230) of FIGS. 2A and 2B . The plurality of optical sensors included in the camera (225) may be arranged in the form of a two-dimensional array. The camera (225) may acquire electrical signals of each of the plurality of optical sensors substantially simultaneously, and generate two-dimensional frame data corresponding to light reaching the optical sensors of the two-dimensional array. For example, photographic data captured using the camera (225) may mean one (a) two-dimensional frame data acquired from the camera (225). For example, video data captured using a camera (225) may mean a sequence of two-dimensional frame data obtained from the camera (225) according to a frame rate. The camera (225) may be positioned toward the direction in which the camera (225) receives light and may further include a flash light for outputting light toward the direction.

[0052] According to one embodiment, the electronic device (101) may include a plurality of cameras, as examples of cameras (225), arranged in different directions. Referring to FIGS. 2A and 2B , the camera (225) included in the electronic device (101) may include a gaze tracking camera (225-1) and / or an outward camera (225-2). The gaze tracking camera (225-1) may be arranged toward at least one of the two eyes of a user wearing the electronic device (101). In order to be arranged toward each of the two eyes of the user, the electronic device (101) may include two gaze tracking cameras (225-1). The processor (210) may identify the direction of the user's gaze using images and / or videos acquired from the gaze tracking camera (225-1). The gaze tracking camera (225-1) may include an infrared (IR) sensor. The gaze tracking camera (225-1) may be referred to as an eye sensor, a gaze tracker, and / or an eye tracker.

[0053] Referring to FIGS. 2A and 2B, the external camera (225-2) may be positioned facing the front of a user wearing the electronic device (101) (e.g., a direction in which both eyes may face). The embodiment is not limited thereto, and the external camera (225-2) may be positioned facing an external space. Using images and / or videos acquired from the external camera (225-2), the processor (210) may identify external objects (e.g., the second electronic device (120) and / or the second user (125) of FIG. 1). The processor (210) may display images and / or videos acquired from the external camera (225-2) on the display (220), thereby providing a user experience based on VST and / or pass-through.

[0054] According to one embodiment, a sensor (230) of an electronic device (101) may generate electrical information from non-electronic information related to the electronic device (101), which may be processed and / or stored by a processor (210) and / or a memory (215) of the electronic device (101). The information may be referred to as sensor data. The sensor (230) may include a global positioning system (GPS) sensor for detecting a geographic location of the electronic device (101), an image sensor, an audio sensor (e.g., a microphone and / or a microphone array including a plurality of microphones), a grip sensor, a touch sensor (or a touch sensor panel (TSP)), a proximity sensor, an ambient light sensor, and / or a time-of-flight (ToF) sensor (or a ToF camera).

[0055] In one embodiment, the sensor (230) may include an inertial measurement unit (IMU) for detecting physical motion of the electronic device (101). An acceleration sensor, a geomagnetic sensor, a gravity sensor, or any combination thereof may be referred to as an IMU. The acceleration sensor may output an electrical signal representing gravitational acceleration and / or acceleration along each of a plurality of axes (e.g., the x-axis, the y-axis, and the z-axis) that are perpendicular to each other and based on a designated origin of the electronic device (101). The gyro sensor may output an electrical signal representing angular velocities (e.g., roll, pitch, and / or yaw) of the plurality of axes. The gyro sensor may be referred to as an angular velocity sensor. The geomagnetic sensor may output an electrical signal representing the magnitude of a magnetic field formed in the electronic device (101) along each of the plurality of axes (e.g., the x-axis, the y-axis, and / or the z-axis). For example, an acceleration sensor, a gyro sensor, and / or a geomagnetic sensor may repeatedly output sensor data including accelerations, angular velocities, and / or magnitudes of magnetic fields of a number of axes based on a specified period (e.g., 1 millisecond).

[0056] In one embodiment, the communication circuit (235) of the electronic device (101) may include circuitry for supporting transmission and / or reception of signals between the electronic device (101) and an external electronic device (e.g., the external electronic device (120) of FIG. 1). The communication circuit (235) may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit (235) may support transmission and / or reception of electrical signals based on various types of protocols, such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), 5G new radio (NR), 6G, and / or above-6G. In one embodiment, the communication circuit (235) may be referred to as a communication processor and / or a communication module.

[0057] According to one embodiment, one or more instructions (or commands) representing data to be processed, calculations to be performed, and / or operations to be performed by the processor (210) of the electronic device (101) may be stored in the memory (215). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or a software application (hereinafter, “application”). For example, the electronic device (101) and / or the processor (210) may perform at least one of the operations of FIG. 3 when a set of a plurality of instructions distributed in the form of an operating system, firmware, a driver, a program, and / or an application is executed. Hereinafter, the fact that an application is installed in an electronic device (101) may mean that one or more instructions provided in the form of an application are stored in a memory (215), and that the one or more applications are stored in a format executable by the processor (210) (e.g., a file having an extension specified by the operating system of the electronic device (101). As an example, an application may include a program and / or a library related to a service provided to a user.

[0058] Referring to FIGS. 2A and 2B , programs installed in the electronic device (101) may be included in any one of different layers, including an application layer (240), a framework layer (250), and / or a hardware abstraction layer (HAL) (280), based on a target. For example, programs (e.g., modules or drivers) designed to target the hardware (e.g., a display (220), a camera (225), a sensor (230), and / or a communication circuit (235)) of the electronic device (101) may be included in the hardware abstraction layer (280). The framework layer (250) may be referred to as an XR framework layer in terms of including one or more programs for providing an XR (extended reality) service. For example, the layers illustrated in FIGS. 2A and 2B may be logically (or for convenience of explanation) separated, but this does not mean that the address space of the memory (215) is separated by the layers.

[0059] For example, within the framework layer (250), programs designed to target at least one of the hardware abstraction layer (280) and / or the application layer (240) (e.g., a position tracker (271), a space recognizer (272), a gesture tracker (273), an eye-gaze tracker (274), and / or a face tracker (275)) may be included. The programs included in the framework layer (250) may provide an API (application programming interface) that is executable (or callable) based on other programs.

[0060] For example, a program designed to target users of the electronic device (101) may be included within the application layer (240). As an example of programs included in the application layer (240), an extended reality (XR) system user interface (UI) (241) and / or an XR application (242) are exemplified, but the embodiment is not limited thereto. For example, programs (e.g., software applications) included in the application layer (240) may call an application programming interface (API) to cause execution of functions supported by programs included in the framework layer (250).

[0061] For example, the electronic device (101) may display one or more visual objects on the display (220) for performing interaction with the user based on the execution of the XR system UI (241). A visual object may refer to an object that can be deployed within a screen for transmitting and / or interacting with information, such as text, an image, an icon, a video, a button, a checkbox, a radio button, a text box, a slider, and / or a table. A visual object may be referred to as a visual guide, a virtual object, a visual element, a UI element, a view object, and / or a view element. The electronic device (101) may provide the user with functions available within a virtual space based on the execution of the XR system UI (241).

[0062] Referring to FIGS. 2A and 2B , although the XR system UI (241) is illustrated to include a lightweight renderer (243) and / or an XR plug-in (244), the embodiment is not limited thereto. For example, based on the XR system UI (241), the processor (210) may execute the lightweight renderer (243) and / or the XR plug-in (244) within the framework layer (250).

[0063] For example, the electronic device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute a rendering pipeline that allows partial changes based on the execution of a lightweight renderer (243). The lightweight renderer (243) may be referred to as a lightweight render pipeline from the perspective of defining a rendering pipeline that allows partial changes. The lightweight renderer (243) may include a renderer built prior to the execution of a software application (e.g., a prebuilt renderer). For example, the electronic device (101) may obtain resources (e.g., APIs, system processes, and / or libraries) used to define, create, and / or execute an entire rendering pipeline based on the execution of an XR plug-in (244). The XR plug-in (244) may be referred to as an open XR native client from the perspective of defining (or configuring) an entire rendering pipeline.

[0064] For example, the electronic device (101) may display a screen representing at least a portion of a virtual space on the display (220) based on the execution of the XR application (242). The XR plug-in (244-1) included in the XR application (242) may include instructions that support functions similar to those of the XR plug-in (244) of the XR system UI (241). Descriptions of the XR plug-in (244-1) that overlap with the description of the XR plug-in (244) may be omitted. The electronic device (101) may cause the execution of the virtual space manager (251) based on the execution of the XR application (242).

[0065] According to one embodiment, the electronic device (101) may provide a virtual space service based on the execution of the virtual space manager (251). For example, the virtual space manager (251) may include a platform for supporting the virtual space service. Based on the execution of the virtual space manager (251), the electronic device (101) may identify a virtual space formed based on the user's location indicated by data acquired through the sensor (230), and may display at least a portion of the virtual space on the display (220). The virtual space manager (251) may be referred to as a composition presentation manager (CPM).

[0066] For example, the virtual space manager (251) may include a runtime service (252). As an example, the runtime service (252) may be referred to as an OpenXR runtime module (or an OpenXR runtime program). The electronic device (101) may execute at least one of a user's pose prediction function, a frame timing function, and / or a spatial input function based on the execution of the runtime service (252). As an example, the electronic device (101) may perform rendering for a virtual space service for the user based on the execution of the runtime service (252). For example, a function related to a virtual space, executable by the application layer (240), may be supported based on the execution of the runtime service (252).

[0067] For example, the virtual space manager (251) may include a pass-through manager (253). Based on the execution of the pass-through manager (253), the electronic device (101) may display a screen representing a virtual space (e.g., screen (130) of FIG. 1) on the display (220), while displaying an image and / or video representing an actual space acquired through an external camera (225-2) by overlaying it on at least a portion of the screen.

[0068] For example, the virtual space manager (251) may include an input manager (254). The electronic device (101) may identify data (e.g., sensor data) acquired by executing one or more programs included in the recognition service layer (270) based on the execution of the input manager (254). The electronic device (101) may use the acquired data to identify a user input related to the electronic device (101). The user input may be related to a motion (e.g., a hand gesture), a gaze, and / or a speech of the user identified by the sensor (230) and / or the camera (225) (e.g., an external camera (225-2). The user input may be identified based on an external electronic device (e.g., a remote controller provided to control the electronic device (101)) connected (or paired) via the communication circuit (235).

[0069] For example, the perception abstract layer (260) can be used for data exchange between the virtual space manager (251) and the perception service layer (270). From the perspective of being used for data exchange between the virtual space manager (251) and the perception service layer (270), the perception abstract layer (260) can be referred to as an interface. For example, the perception abstract layer (260) can be referenced as OpenPX. The perception abstract layer (260) can be used for a perception client and a perception service.

[0070] According to one embodiment, the recognition service layer (270) may include one or more programs for processing data acquired from a sensor (230) and / or a camera (225). The one or more programs may include at least one of a position tracker (271), a space recognizer (272), a gesture tracker (273), an eye tracker (274), and / or a face tracker (275). The type and / or number of the one or more programs included in the recognition service layer (270) are not limited to those illustrated in FIGS. 2A and 2B .

[0071] For example, the electronic device (101) can identify the pose of the electronic device (101) using the sensor (230) based on the execution of the position tracker (271). The electronic device (101) can identify the 6 degrees of freedom pose (6 dof pose) of the electronic device (101) using data acquired using the external camera (225-2) and / or the IMU (e.g., a gyro sensor, an acceleration sensor, and / or a geomagnetic sensor) based on the execution of the position tracker (271). The position tracker (271) may be referred to as a head tracking (HeT) module (or head tracker, a head tracking program).

[0072] For example, the electronic device (101) may obtain information for providing a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of the electronic device (101) (or the user of the electronic device (101)) based on the execution of the space recognizer (272). The electronic device (101) may reconstruct the surrounding environment of the electronic device (101) in three dimensions using data obtained using the external camera (225-2) based on the execution of the space recognizer (272). The electronic device (101) may identify at least one of a plane, a slope, and stairs based on the surrounding environment of the electronic device (101) reconstructed in three dimensions based on the execution of the space recognizer (272). The space recognizer (272) may be referred to as a scene understanding (SU) module (or a scene understanding program).

[0073] For example, the electronic device (101) can identify (or recognize) a pose and / or gesture of a hand of a user of the electronic device (101) based on the execution of the gesture tracker (273). As an example, the electronic device (101) can identify a pose and / or gesture of a hand of a user using data acquired from an external camera (225-2) based on the execution of the gesture tracker (273). As an example, the electronic device (101) can identify a pose and / or gesture of a hand of a user based on data (or images) acquired using the external camera (225-2) based on the execution of the gesture tracker (273). The gesture tracker (273) may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or a gesture tracking module.

[0074] For example, the electronic device (101) can identify (or track) eye movements of a user of the electronic device (101) based on the execution of the gaze tracker (274). As an example, the electronic device (101) can identify eye movements of the user using data acquired from the gaze tracking camera (225-1) based on the execution of the gaze tracker (274). The gaze tracker (274) may be referred to as an eye tracking (ET) module (or eye tracking program) and / or a gaze tracking module.

[0075] For example, the recognition service layer (270) of the electronic device (101) may further include a face tracker (275) for tracking the user's face. For example, the electronic device (101) may identify (or track) the movement of the user's face and / or the user's expression based on the execution of the face tracker (275). The electronic device (101) may estimate the user's expression based on the movement of the user's face based on the execution of the face tracker (275). As an example, the electronic device (101) may identify the movement of the user's face and / or the user's expression based on data (e.g., an image) acquired using a sensor (230) (e.g., an image sensor directed at at least a portion of the user's face) based on the execution of the face tracker (275).

[0076] According to one embodiment, the processor (210) of the electronic device (101) may remove the appearance of an external electronic device (e.g., the external electronic device (120) of FIG. 1) connected to the electronic device (101) via the communication circuit (235) while displaying a screen based on pass-through. Removing the appearance of the external electronic device may include an operation of synthesizing a virtual object, image, and / or video representing a body part covered by the external electronic device. The body part may include both eyes of a user wearing the external electronic device.

[0077] Referring to FIG. 2b, one or more programs among the programs of FIG. 2a are exemplarily illustrated to be executed to remove the appearance of an external electronic device (120). The embodiment is not limited thereto.

[0078] According to one embodiment, the processor (210) of the electronic device (101) may conditionally perform an operation of removing the appearance of the external electronic device (120) based on an interaction between the electronic device (101) and the external electronic device (120). The interaction may include a positional relationship between the electronic device (101) and the external electronic device (120), and / or an interaction (e.g., conversation) between users wearing the electronic device (101) and the external electronic device (120). The interaction may be detected based on information included in the electronic device (101), such as an address book. The interaction may be detected based on a usage history of the electronic device (101). The interaction may be detected based on whether the external electronic device (120) is an electronic device associated with a user registered in the address book and / or a list pre-specified by the user. The operation of the processor (210) of the electronic device (101) to identify the above interaction is described with reference to FIG. 4.

[0079] The processor (210) that detects an interaction between the electronic device (101) and the external electronic device (120) (e.g., the external electronic device (120) of FIG. 1) can detect a portion corresponding to the external electronic device (120) within an image and / or video of the external environment displayed through the display (220). Hereinafter, an embodiment based on the external electronic device (120), such as an HMD device, is described, but the embodiment is not limited thereto, and the external electronic device (120) may include a portable electronic device such as a smartphone and / or other real objects. The processor (210) can control the external camera (225-2) to acquire the image and / or the video. The processor (210) that detects a portion corresponding to the external electronic device (120) within the image and / or the video can execute a function to infer an appearance of a body part covered by the external electronic device (120) within the portion.

[0080] The processor (210) can receive information (e.g., information (150) of FIG. 1) about a body part covered by the external electronic device (120) from the external electronic device (120) connected via the communication circuit (235). The information can be received via the communication link after a communication link between the electronic device (101) and the external electronic device (120) is established. For example, the information can be received before detecting an interaction between the electronic device (101) and the external electronic device (120), substantially coincident with the timing of detecting the interaction, or after detecting the interaction.

[0081] Referring to FIG. 2B, one or more programs (e.g., a gaze information exchanger (292), an eye image generator (294), and / or an eye image synthesizer (or compositor) (296)) included in the eye tracker (274) are illustrated. The processor (210) may receive information from the external electronic device (120) based on the execution of the gaze information exchanger (292). The processor (210) may control the communication circuit (235) to transmit information related to the gaze tracking camera (225-1) to the external electronic device (120) based on the execution of the gaze information exchanger (292). The information related to the gaze tracking camera (225-1) may include information related to the gaze of a user wearing the electronic device (101) (e.g., the first user (110) of FIG. 1). The information related to the gaze of a user wearing the electronic device (101) may include an image and / or a video related to at least one of the user's two eyes. The embodiment is not limited thereto, and the information transmitted to the external electronic device (120) may include parameters (e.g., the direction and / or angle at which the two eyes are pointed) calculated by the processor (210) based on the image and / or the video. For example, the processor (210) may transmit the information indicating the direction of the user's gaze to the external electronic device (120) via the communication circuit (235).

[0082] According to one embodiment, the processor (210) of the electronic device (101) may perform rendering of a body part of a user wearing an external electronic device (120) based on execution of the eye image generator (294). For example, the processor (210) may perform the rendering based on information received from the external electronic device (120). The processor (210) that executes the eye image generator (294) may perform the rendering using information acquired from the external electronic device (120) based on the gaze information exchanger (292). Based on the rendering of the body part, the processor (210) may obtain a virtual object representing the body part (e.g., a three-dimensional graphical object including a mesh representing a head and / or eyes), an image, and / or a video. The operation of the processor (210) that executes the eye image generator (294) is described with reference to FIG. 5 and / or FIG. 6a and / or FIG. 6b.

[0083] According to one embodiment, the processor (210) of the electronic device (101) may display a composite image on the display (220) based on the execution of the eye image synthesizer (296). For example, the processor (210) may display, on the display (220), a composite image generated based on a first image acquired from an external camera (225-2) and a second image acquired from an eye image generator (294). The composite image may be an image in which at least a portion of the first image is replaced with the second image. The composite image may include the first image and the second image superimposed (or positioned) on at least a portion of the first image. A graphical object (e.g., the virtual object, the image, and / or the video) generated by the eye image generator (294) may be displayed. While displaying a first image acquired from an external camera (225-2), the processor (210) may display the first image including a replaced portion (e.g., a portion corresponding to the external electronic device (120)) based on a second image acquired from an eye image generator (294). The embodiment is not limited thereto, and the processor (210) may display the second image by overlapping and / or updating it on the first image displayed on the display (220). The embodiment is not limited thereto, and the processor (210) may synthesize the first image and the second image to synthesize a new image in which a portion of the first image captured by the external electronic device (120) is changed to the second image. The portion of the first image overlapping with the second image may be a portion of the external electronic device (120) captured. The second image may be acquired by performing rendering based on a direction of gaze indicated by information received from the external electronic device (120). The second image may include at least a portion representing the eyes of a user wearing the external electronic device.The above second image can be generated based on the execution of the eye image generator (294).

[0084] According to one embodiment, the processor (210) of the electronic device (101) may display a first image on which a second image is superimposed on the display (220) based on the execution of the eye image synthesizer (296) and / or the pass-through manager (253). Based on the second image, the electronic device (101) may provide a user experience, such as viewing a body part of another user covered by an external electronic device (120), to a user wearing the electronic device (101). For example, based on the execution of the eye image synthesizer (296), the processor (210) may determine a point on the first image at which the second image generated by the eye image generator (294) is to be superimposed. Based on the execution of the pass-through manager (253), the processor (210) may display the first image on which the second image is superimposed on the display (220).

[0085] As described above, according to one embodiment, the electronic device (101) may perform an operation to remove the appearance of the external electronic device (120) from the display (220) by using the gaze of the user wearing the external electronic device (120) tracked by the external electronic device (120) and the face of the user tracked based on the external camera (225-2). For example, the electronic device (101) may display a virtual object (e.g., an avatar) representing the gaze as an overlay on a portion corresponding to the external electronic device (120) within an image and / or video displayed on the display (220). In order to assist the external electronic device (120) in removing the appearance of the electronic device (101) from the display of the external electronic device (120), the electronic device (101) may transmit information related to the gaze of the user tracked based on the gaze tracking camera (225-1) to the external electronic device (120).

[0086] Hereinafter, with reference to FIG. 3, the operation of the electronic device (101) and / or the processor (210) described with reference to FIG. 1, FIG. 2a and / or FIG. 2b is described.

[0087] FIG. 3 illustrates an example of a flowchart of an electronic device according to one embodiment of the present disclosure. The electronic device (101) and / or processor (210) of FIG. 1 , FIG. 2A , and / or FIG. 2B may perform at least one of the operations described with reference to FIG. 3 .

[0088] Referring to FIG. 3, in operation (310), according to one embodiment, a processor of an electronic device may obtain first information related to a face of a first user. The first user of operation (310) may include a person identified as a user wearing the electronic device, such as the first user (110) of FIG. 1. The first information of operation (310) may include a photo (e.g., a selfie) related to the face of the first user.

[0089] In one embodiment, the first information of the action (310) may include data (e.g., mesh, vertices, voxels, fragments and / or textures associated with at least a portion of the face) used to render a face of the first user in three dimensions, acquired by a time-of-flight (ToF) and / or stereo camera. The first information may be acquired at the time the first user wears the electronic device. The embodiment is not limited thereto, and the processor of the electronic device may acquire the first information of the action (310) from images and / or videos stored in a memory (e.g., memory 215 of FIGS. 2A and / or 2B ).

[0090] Referring to FIG. 3, in operation (320), according to one embodiment, a processor of an electronic device may connect with an external electronic device adjacent to the electronic device while the electronic device is worn by a first user. The electronic device may determine whether the electronic device is worn by the first user based on sensor data of a sensor (e.g., sensor 230 of FIGS. 2A and 2B ). While worn by the first user, the electronic device may search for or identify an external electronic device (e.g., external electronic device 120 of FIG. 1 ) adjacent to the electronic device using a communication circuit (e.g., communication circuit 235 of FIGS. 2A and 2B ). The search for the external electronic device may be performed based on a wireless communication protocol such as Bluetooth low energy (BLE) and / or ultra-wideband (UWB). Based on the search for the external electronic device, the processor may control the communication circuit to establish a communication link between the electronic device and the external electronic device. The above communication link may be established based on a wireless communication protocol such as Wi-Fi Direct and / or Bluetooth. For example, a Wi-Fi Direct discovery procedure may be performed to connect with an external electronic device in operation (320). Based on the discovery procedure, a communication link between the electronic device and the external electronic device may be established. For example, a wireless signal, such as an advertising signal, for detecting the external electronic device may be transmitted by the electronic device performing operation (320). Based on a response to the advertising signal, the electronic device may connect with the external electronic device.

[0091] In one embodiment, a processor of an electronic device may perform operation (320) based on whether an interaction between the electronic device and an external electronic device is detected. The interaction may be detected based on at least one of a distance between the electronic device and the external electronic device, whether the external electronic device is facing the electronic device, and / or a utterance of at least one of users wearing the electronic device and the external electronic device. In one embodiment, the processor that detects the interaction may display a screen and / or a visual object confirming the connection with the external electronic device. In response to a user input confirming the connection with the external electronic device, the processor may perform operation (320).

[0092] While connected to an external electronic device based on operation (320), the processor of the electronic device may enter a reverse pass-through mode. Within the reverse pass-through mode, the electronic device may communicate with the external electronic device to perform a function for removing the appearance of the external electronic device from a screen displayed to a first user, and may provide information to the external electronic device that is used to remove the appearance of the electronic device from a screen displayed to a second user of the external electronic device. For example, within the reverse pass-through mode, the processor may repeatedly perform operations (330, 340, 350) of FIG. 3 using the communication link established based on operation (320).

[0093] Referring to FIG. 3, in operation (330), according to one embodiment, a processor of an electronic device may obtain second information related to a gaze of a second user (e.g., the second user (125) of FIG. 1) from an external electronic device (e.g., the external electronic device (120) of FIG. 1 and / or FIG. 2B). The second information of operation (330) may include information (150) of FIG. 1. Through the communication link established based on operation (320), the processor may repeatedly (or continuously or periodically) receive the second information. The second information may include images and / or videos of both eyes of the second user captured by the external electronic device. The second information may include a numeric value indicating a direction of the gaze of the second user calculated by the external electronic device. The second information of operation (330) may be referred to as eye tracking information identified by the external electronic device. The embodiment is not limited thereto, and the second information may include an image synthesized by the external electronic device, representing a body part of the user covered by the external electronic device (e.g., a portion of the face including the user's two eyes). For example, the external electronic device may generate an image to be transmitted to the electronic device using an image and / or a video of the user's two eyes captured by the external electronic device. As another example, the second information may include an image synthesized by the external electronic device, representing the entire face of the user wearing the external electronic device. For example, the external electronic device may combine or synthesize another image generated by the eye tracking information identified by the external electronic device onto an image of the face of the user wearing the external electronic device, to generate an image to be transmitted to the electronic device.

[0094] In one embodiment, the period at which the processor receives the second information of operation (330) may be adaptively adjusted depending on the status of the electronic device and / or the external electronic device. For example, the processor may change the period at which the second information is received depending on the direction of the gaze of the user of the electronic device. For example, if the gaze of the user of the electronic device is directed toward the external electronic device, the electronic device may transmit a signal to the external electronic device to reduce the period at which the external electronic device transmits the second information. For example, if the gaze of the user of the electronic device is directed in a different direction from the external electronic device, the electronic device may transmit a signal to the external electronic device to increase the period at which the second information is received.

[0095] In one embodiment, an electronic device receives second information based on data communication, although the embodiment is not limited thereto. For example, the processor may transmit the second information using other media, such as infrared communication and / or acoustic waves.

[0096] The size of the second information obtained from the external electronic device based on the action (330) may be related to (e.g., inversely proportional to) the distance between the electronic device (101) and the external electronic device. For example, as the external electronic device moves away from the user, the size of the portion covered by the external electronic device within the screen displayed by the electronic device (101) may decrease. In the above example, the electronic device may request the external electronic device to transmit the second information in a size based on the distance between the external electronic device and the electronic device.

[0097] Within the state of acquiring second information based on operation (330), the processor of the electronic device may transmit information related to the gaze of the first user to an external electronic device. For example, based at least on operation (330), eye tracking information of the first user and the second user may be exchanged between the electronic device and the external electronic device. The exchange of eye tracking information may be performed via a communication link established based on operation (320).

[0098] The information exchanged via the communication link of operation (320) is not limited to eye tracking information. For example, within a state in which a communication link is established, the electronic device may transmit the first information of operation (310) to an external electronic device. The first information transmitted to the external electronic device may be used to render at least a portion of the face of the first user covered by the electronic device. For example, within a state in which a communication link is established, the electronic device may obtain third information related to the face of the second user from the external electronic device. The third information may be obtained by the external electronic device performing an operation similar to operation (310). The third information may include a photograph related to the face of the second user.

[0099] Referring to FIG. 3, in operation (340), according to one embodiment, a processor of an electronic device may change a portion corresponding to an external electronic device within an image and / or video acquired from a camera based on second information. The camera of operation (340) may include the camera (225) and / or the external camera (225-2) of FIGS. 2A and 2B. The processor may determine or identify the portion corresponding to the external electronic device within the image and / or video based on object recognition. The object recognition may be performed based on one or more feature points or key points extracted from the image and / or video. The object recognition may be performed based on a model (e.g., a neural network such as a convolutional neural network (CNN)) into which the image and / or the video is input.

[0100] For example, the processor may obtain the coordinates of the vertices of a polygon (e.g., a rectangle such as a bounding box) representing the portion. For example, the processor may obtain pixel-wise probabilities that each pixel of the image and / or the video corresponds to the external electronic device. At least one pixel corresponding to at least one probability exceeding a threshold among the probabilities may be included in the portion of the image and / or the video captured by the external electronic device.

[0101] In operation (340), the processor of the electronic device may obtain a virtual object representing a body part covered by the external electronic device based on the second information received from the external electronic device via the communication circuit. The virtual object may be obtained based on at least one of the third information and / or the second information related to the face of the second user received from the external electronic device via the communication link. The operation of the processor for modifying at least a portion of the image and / or video of operation (340) based on the three-dimensional virtual object is described with reference to FIG. 5.

[0102] In operation (340), the processor of the electronic device may infer the shape of a body part covered by the external electronic device based on the second information received from the external electronic device via the communication circuit. Inferring the shape of the body part may be performed by a model driven by the processor. The model may include hardware for performing computations based on a mathematical model (e.g., a neural network) that mathematically models the neural activity of a human and / or animal. The operation of the processor for modifying at least a portion of an image and / or video of operation (340) based on the result of inferring the shape of the body part is described with reference to FIG. 6A and / or FIG. 6B.

[0103] Referring to FIG. 3 , in operation (350), according to one embodiment, the processor of the electronic device may display, on the display, an image and / or video that has been modified based on the second information. For example, while displaying the image and / or video on the display (e.g., display (220) of FIG. 2A ), the processor may display, on at least a portion of the display where the portion of operation (340) is displayed, at least a portion of the virtual object obtained based on operation (340). For example, the processor may superimpose the appearance of the body part inferred by the model on at least a portion of the display. The processor may align or synthesize, based on operation (350), the image and / or video obtained from the camera with the image and / or video obtained based on the second information.

[0104] In one embodiment, obtaining the second information based on operation (330) may be delayed or interrupted depending on the communication status between the electronic device and the external electronic device. In one embodiment, the processor of the electronic device may play a designated animation related to the appearance of a body part covered by the external electronic device when receiving the second information is interrupted or delayed. The designated animation may include an image and / or video that was displayed before receiving the second information was interrupted. The embodiment is not limited thereto, and the processor of the electronic device may stop receiving the second information using the communication link when the state of charge (SOC) of the battery decreases below a designated SOC, and display an image and / or video that was displayed before the SOC decreased below the designated SOC.

[0105] In one embodiment, a processor of an electronic device may repeatedly perform operations (330, 340, 350). While repeatedly performing operations (330, 340, 350) based on a reverse pass-through mode, the processor may determine whether interaction between the electronic device and an external electronic device is interrupted. If interaction between the electronic device and the external electronic device is interrupted, the processor may stop repeatedly performing operations (330, 340, 350). If interaction between the electronic device and the external electronic device is interrupted, the processor may display a screen and / or a virtual object on a display to determine whether to stop performing operations (330, 340, 350) performed to display a body part covered by the external electronic device. In response to an input to the screen and / or the virtual object, the processor of the electronic device may stop repeatedly performing operations (330, 340, 350). Because the user wearing the electronic device is stopped from performing the actions (330, 340, 350), the user can view the external electronic device through images and / or videos of the external environment displayed on the display.

[0106] In one embodiment, the processor of the electronic device may reduce or optimize the amount of computation for modifying an image and / or video based on operation (340) while repeatedly performing operations (330, 340, 350). For example, the processor may adjust the frequency with which it infers the appearance of a body part of a second user covered by an external electronic device based on the size of a portion of a display corresponding to the external electronic device. The operations performed by the processor of the electronic device to optimize the amount of computation are described with reference to FIGS. 7A and 7B .

[0107] Hereinafter, with reference to FIG. 4, the operation of an electronic device for detecting interaction between an electronic device and an external electronic device is described.

[0108] FIG. 4 illustrates an example of an operation of an electronic device (101) based on an image (410) and / or a video acquired from a camera according to an embodiment of the present disclosure. The electronic device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIGS. 2A and 2B may perform the operation of the electronic device (101) described with reference to FIG. 4. The operation of the electronic device (101) described with reference to FIG. 4 may be related to at least one of the operations of FIG. 3.

[0109] Referring to FIG. 4, a state (401) of an electronic device (101) adjacent to an external electronic device (120) is illustrated. Within the state (401), the electronic device (101) can detect the external electronic device (120) using an image (410) and / or a video of a camera (e.g., the external camera (225-2) of FIG. 2A and / or FIG. 2B). Referring to FIG. 4, an example of an image (410) acquired by the electronic device (101) within the state (401) is illustrated. The image (410) can correspond to a single image frame of a video acquired by the camera of the electronic device (101). Within the image (410), the electronic device (101) can determine a portion corresponding to the external electronic device (120) based on object recognition. The embodiment is not limited thereto, and the electronic device (101) can identify an external electronic device (120) adjacent to the electronic device (101) based on a wireless communication protocol such as BLE, NFC (near field communication) and / or UWB.

[0110] In state (401) of FIG. 4, the electronic device (101) can identify an interaction between a first user (110) wearing the electronic device (101) and a second user (125) wearing an external electronic device (120). For example, the electronic device (101) can determine or identify a direction (d1) of the external electronic device (120) using an image (410). Based on whether the direction (d1) is directed toward the electronic device (101), the electronic device (101) can detect the interaction. For example, when the second user (125) looks at the first user (110) wearing the electronic device (101), the electronic device (101) can detect a direction (d1) of the external electronic device (120) directed toward the electronic device (101). Based on detecting a direction (d1) toward the electronic device (101) or toward a point adjacent to the electronic device (101), the electronic device (101) can detect an interaction between the electronic device (101) and an external electronic device (120).

[0111] In one embodiment, the electronic device (101) can detect an interaction between the electronic device (101) and the external electronic device (120) based on a communication link established between the electronic device (101) and the external electronic device (120). For example, the electronic device (101) can receive an input for executing a reverse pass-through mode from the first user (110) based on a user interface (UI) (e.g., screen (130) of FIG. 1) provided to the first user (110). The input received by the electronic device (101) is not limited to the UI. For example, the electronic device (101) can receive an input for executing the reverse pass-through mode through a button and / or a dial. Based on the input, the electronic device (101) can establish the communication link. Based on the establishment of the above communication link, the electronic device (101) may execute a reverse pass-through mode and perform an operation to remove the appearance of the external electronic device (120) displayed on the display, or to display the appearance of the second user (125) covered by the external electronic device (120) on the display.

[0112] In one embodiment, the electronic device (101) can detect an interaction between the electronic device (101) and an external electronic device (120) based on voice recognition and / or speech-to-text (STT). For example, the electronic device (101) can detect the interaction based on a speech of a first user (110) and / or a second user (125). To identify the speech, the electronic device (101) can include one or more microphones. For example, the electronic device (101) can recognize a natural language sentence including the name and / or pronoun of the first user (110) from an audio signal output from the one or more microphones. In response to recognizing the natural language sentence, the electronic device (101) can detect the interaction. For example, the electronic device (101) can recognize a natural language sentence including the name and / or pronoun of the second user (125) from the audio signal. In response to recognition of the above natural language sentence, the electronic device (101) can detect interaction between the electronic device (101) and an external electronic device (120).

[0113] The electronic device (101) that detects an interaction between the electronic device (101) and the external electronic device (120) may perform the operations described above with reference to FIG. 3 (e.g., operations (320, 330, 340, 350) of FIG. 3) to enter a reverse pass-through mode. Within the reverse pass-through mode, the electronic device (101) may replace the appearance of the external electronic device (120) displayed on the display with a virtual object, image, and / or video representing the face of a second user (125) wearing the external electronic device (120). Within the reverse pass-through mode, the electronic device (101) may display the entire face of the second user (125) that is covered by the external electronic device (120), and provide a nonverbal expression based on the facial expression of the second user (125) to the first user (110) wearing the electronic device (101). In reverse pass-through mode, to support the external electronic device (120) to display the entire face of the first user (110), the electronic device (101) can transmit information related to at least a portion of the face of the first user (110) to the external electronic device (120).

[0114] Within the reverse pass-through mode, the electronic device (101) can detect a cessation of interaction between the external electronic device (120) and the electronic device (101). Based on the cessation of interaction, the electronic device (101) can switch to a mode other than the reverse pass-through mode. For example, the electronic device (101) can stop displaying the entire face of the second user (125) that is obscured by the external electronic device (120).

[0115] A disruption in the interaction between the electronic device (101) and the external electronic device (120) may be identified or determined based on the direction (d1) of the external electronic device (120). For example, if the direction (d1) of the external electronic device (120) is directed toward another point away from the point where the electronic device (101) is placed (e.g., away from the point by more than a specified distance), the electronic device (101) may detect the disruption in the interaction. For example, if the direction (d1) is directed toward another point away from the point where the electronic device (101) is placed for a period exceeding a specified duration, the electronic device (101) may detect the disruption in the interaction.

[0116] In one embodiment, the electronic device (101) may, in response to an input for stopping and / or terminating the reverse pass-through mode, stop displaying a body part (e.g., at least a portion of the face) of the second user (125) covered by the external electronic device (120). For example, the electronic device (101) may receive or identify the input for stopping and / or terminating the reverse pass-through mode through a visual object provided through a screen and / or a button (or dial) of the electronic device (101). In response to the input, the electronic device (101) may switch to a mode other than the reverse pass-through mode for providing the entire face of the second user (125) and stop providing the entire face.

[0117] In one embodiment, the electronic device (101) may switch from the reverse pass-through mode to another mode based on the speech of users wearing the electronic device (101) and the external electronic device (120) (e.g., the first user (110) and / or the second user (125)). For example, if the users remain silent for a specified period of time or if the electronic device (101) recognizes a natural language sentence indicating a break in conversation between the users (e.g., “See you later!”), the electronic device (101) may switch from the reverse pass-through mode to another mode. The electronic device (101) that has switched to the other mode may resume displaying the external electronic device (120) within the screen, or may display a UI different from the pass-through mode, such as VR.

[0118] As described above, according to one embodiment, the electronic device (101) may enter the reverse pass-through mode based on an input, speech, and / or location of a first user (110) wearing the electronic device (101). The electronic device (101) may enter the reverse pass-through mode based on a location and / or orientation of an external electronic device (120) with respect to the electronic device (101). The electronic device (101) may enter the reverse pass-through mode based on a speech, location, and / or orientation of a second user (125) wearing the external electronic device (120) (e.g., an orientation of the face of the second user (125) wearing the external electronic device (120). An electronic device (101) that has entered reverse pass-through mode can provide at least a portion of the face of a second user (125) covered by the external electronic device (120) using an image (410) and information received from an external electronic device (120) (e.g., information (150) of FIG. 1).

[0119] Hereinafter, with reference to FIG. 5, FIG. 6a and / or FIG. 6b, the operation of an electronic device (101) for generating a virtual object, image and / or video representing the face of a second user (125) covered by an external electronic device (120) is described.

[0120] FIG. 5 illustrates an example of an operation of an electronic device (101) that processes information received from an external electronic device (120) according to an embodiment of the present disclosure. The electronic device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIGS. 2A and 2B may perform the operation of the electronic device (101) described with reference to FIG. 5. The operation of the electronic device (101) described with reference to FIG. 5 may be related to the operations of FIG. 3.

[0121] Referring to FIG. 5, the electronic device (101) can obtain an image (410) using a camera for capturing an external environment (e.g., an external camera (225-2) of FIG. 2A and / or FIG. 2B). In a state (501) adjacent to an external electronic device (120), the electronic device (101) can recognize the external electronic device (120) from the image (410). Based on at least one of a direction (d1) of the external electronic device (120), a distance from the external electronic device (120) (e.g., a distance between the external electronic device (120) and the electronic device (101), and a speech of a first user (110) and a second user (125) wearing the electronic device (101) and the external electronic device (120), the electronic device (101) can determine whether to enter a reverse pass-through mode. Before entering the reverse pass-through mode, the electronic device (101) may display an image (410) including an image of the external electronic device (120) based on the VST and / or pass-through.

[0122] Referring to FIG. 5, a state (501) in which a communication link is established between an external electronic device (120) and an electronic device (101) is illustrated. The electronic device (101) may establish the communication link in response to detecting an interaction between the electronic device (101) and the external electronic device (120) by performing the operation described above with reference to FIG. 4. Through the communication link, the electronic device (101) may receive information related to a second user (125) wearing the external electronic device (120) from the external electronic device (120).

[0123] For example, the electronic device (101) may receive three-dimensional mesh information (520) about the second user (125) from the external electronic device (120) via a communication link. The three-dimensional mesh information (520) may include information required for three-dimensional rendering of a body part (e.g., face and / or head) of the second user (125). For example, the three-dimensional mesh information (520) may include vertices (or voxels) corresponding to each point (or landmark) of the face of the second user (125). The three-dimensional mesh information (520) may include information for forming fragments (or unit planes) based on connections of at least three or more vertices. The three-dimensional mesh information (520) may include information about textures to be filled in each of the fragments. The above textures may be generated based on a three-dimensional scan of an external electronic device (120) of a body part of a second user (125).

[0124] Similar to the 3D mesh information (520) received from the external electronic device (120), the electronic device (101) may obtain or generate 3D mesh information about the first user (110) based on a 3D scan of the first user (110) wearing the electronic device (101). For example, the electronic device (101) may obtain 3D mesh information about the first user (110) by performing operation (310) of FIG. 3. The electronic device (101) may obtain 3D mesh information about the first user (110) from images and / or videos of a camera (e.g., camera (225) and / or gaze tracking camera (225-1) of FIG. 2A and / or FIG. 2B) of the electronic device (101) using a deep-learning model. The above three-dimensional mesh information may include the geometry and / or texture of a body part (e.g., head) of the first user (110). The three-dimensional mesh information about the first user (110) may be transmitted to an external electronic device (120) via a communication link.

[0125] Referring to FIG. 5, the electronic device (101) may receive three-dimensional mesh information (520) and gaze information (510) of a second user (125) from an external electronic device (120). The gaze information (510) may include images (511, 512) of two eyes of the second user (125). An image (511) corresponding to the right eye of the second user (125) and an image (512) corresponding to the left eye of the second user (125) are illustrated as examples of the gaze information (510), but the embodiment is not limited thereto. For example, the gaze information (510) may include parameters indicating directions of two eyes of the second user (125), calculated by the external electronic device (120).

[0126] According to one embodiment, the electronic device (101) may change a portion corresponding to the gaze information (510) within a three-dimensional mesh generated (or reconstructed) by three-dimensional mesh information (520) received from an external electronic device (120). For example, the electronic device (101) may change vertices and / or textures associated with the two eyes of the second user (125) within the three-dimensional mesh. For example, an image (512) corresponding to the left eye within the gaze information (510) may be determined as at least one texture corresponding to the left eye within the three-dimensional mesh. For example, an image (511) corresponding to the right eye within the gaze information (510) may be determined as at least one texture corresponding to the right eye within the three-dimensional mesh. The three-dimensional mesh generated based on the gaze information (510) and the three-dimensional mesh information (520) may be referred to as a three-dimensional avatar and / or virtual object for the second user (125).

[0127] According to one embodiment, the electronic device (101) may display at least a portion of a virtual object to a second user (125) based on a reverse pass-through mode while displaying an image (410) (or video) on the display based on a VST. The electronic device (101) may display at least a portion of the virtual object based on detecting an interaction between the electronic device (101) and an external electronic device (120). For example, the electronic device (101) may display on the display a portion (530) of the virtual object, in which eyes having a direction indicated by the gaze information (510) are arranged, within the virtual object representing a head wearing the external electronic device (120).

[0128] Referring to FIG. 5, the electronic device (101) can display a portion (530) of the virtual object as an overlay on a portion (505) corresponding to the external electronic device (120) within the image (410). The electronic device (101) can display a portion (530) of the virtual object that is rotated (or aligned) based on the direction (d1) of the external electronic device (120) identified from the image (410) on the portion (505) of the image (410). Since the portion (530) of the virtual object is overlaid on the portion (505) of the image (410) within the screen (540) displayed on the display of the electronic device (101), the entire facial expression of the second user (125) can be reproduced.

[0129] The electronic device (101) can compare landmarks included in the virtual object with landmarks of a face recognized in the image (410) to match a portion (530) of the virtual object and a portion (505) within the image (410). The landmarks can include designated points within a body part such as a face (e.g., both ends of the eyes). The electronic device (101) can determine a portion (505) associated with a portion (530) of the virtual object based on feature points (or key points) extracted from the image (410).

[0130] Although an embodiment of restoring the appearance of a body part covered by an external electronic device (120) using gaze information (510) has been described, the embodiment is not limited thereto. For example, the electronic device (101) may generate and / or display a virtual object for at least a portion of a body part of the second user (125) covered by the external electronic device (120) using at least one of the direction of the gaze of a user wearing the electronic device (101), the distance between the electronic device (101) and the external electronic device (120), the appearance of the body part (e.g., two eyes of the second user (125)) covered by the external electronic device (120) estimated using a microphone of the electronic device (101), and / or images and / or videos previously stored in the electronic device (101).

[0131] As described above, according to one embodiment, the electronic device (101) can render a virtual object for a second user (125) wearing the external electronic device (120) in three dimensions to provide an image of a body part covered by the external electronic device (120) within a screen (540). The electronic device (101) can perform rendering of the virtual object based on mesh information (520) including the face of the second user (125) and gaze information (510) repeatedly (or periodically) transmitted from the external electronic device (120). By displaying a portion (530) of the virtual object, the electronic device (101) can provide an image of a body part covered by the external electronic device (120).

[0132] In one embodiment, the electronic device (101) can use a neural network to infer the shape of a body part beyond a portion (505) corresponding to an external electronic device (120). Hereinafter, with reference to FIGS. 6A and / or 6B, an operation of the electronic device (101) for inferring the shape of a body part of a second user (125) covered by an external electronic device (120) based on a neural network is described.

[0133] FIGS. 6A and 6B illustrate an example of an operation of an electronic device (101) that processes information received from an external electronic device (120) according to various embodiments of the present disclosure. The electronic device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIGS. 2A and 2B may perform the operation of the electronic device (101) described with reference to FIGS. 6A and / or 6B. The operation of the electronic device (101) described with reference to FIGS. 6A and / or 6B may be related to the operations of FIG. 3.

[0134] Referring to state (601) of FIG. 6A, the electronic device (101) can acquire an image (410) of an external environment using a camera (e.g., an external camera (225-2) of FIG. 2A and / or FIG. 2B). The image (410) can correspond to any one of image frames continuously acquired through the camera. While providing a user experience based on VST and / or pass-through, the electronic device (101) can display at least a portion of the image (410) on a display (e.g., a display (220) of FIG. 2A and / or FIG. 2B). The electronic device (101), which detects an interaction between the electronic device (101) and the external electronic device (120), can identify a portion (505) corresponding to the external electronic device (120) within the image (410).

[0135] In one embodiment, the electronic device (101) may perform a synthesis (e.g., inpainting) of a portion (505) corresponding to the external electronic device (120) within the image (410). The synthesis may include an operation of combining, replacing, or superimposing a virtual object, image, and / or video of a body part (e.g., at least a portion of a face including both eyes) of the second user (125) covered by the external electronic device (120) onto the portion (505). The synthesis may be performed based on the execution of a facial image generation model (620). The electronic device (101) executing the eye image generator (294) of FIG. 2B may generate or obtain an image and / or video to be synthesized onto the portion (505) of the image (410) using the facial image generation model (620).

[0136] Referring to FIG. 6A, the electronic device (101) may input at least one of an image (410) obtained from an external camera of the electronic device (101), gaze information (510), and / or an image (610) related to a second user (125) into a facial image generation model (620). The image (610) may be received from the external electronic device (120) via a communication link established between the electronic device (101) and the external electronic device (120). The image (610) may include at least a portion corresponding to the face of the second user (125) wearing the external electronic device (120). The gaze information (510) may be repeatedly and / or continuously received from the external electronic device (120) via the communication link. Gaze information (510) including images (511, 512) of two eyes of a second user (125) is exemplarily illustrated, but the embodiment is not limited thereto. Gaze information (510) may be referred to as gaze tracking information.

[0137] According to one embodiment, the electronic device (101) may obtain an image (630) related to the face of a second user (125) wearing the external electronic device (120) from a facial image generation model (620) into which at least one of the images (410, 610) and the gaze information (510) is input. The facial image generation model (620) may include a neural network for rendering the face. For example, the facial image generation model (620) may be trained to output an image (630) related to a face that has the appearance of the face indicated by the image (610), includes eyes having a direction indicated by the gaze information (510), and faces the direction (d1) of the face expressed in the image (410). The facial image generation model (620) may include a generative model, such as a generative adversarial network (GAN). The face image generation model (620) may include a model for generating graphical information (e.g., an image (630)) from non-graphical information, such as text, such as a diffusion model (e.g., a stable diffusion model). The face image generation model (620) may be trained to output an image that includes a background as well as a face, such as an image (630).

[0138] Referring to FIG. 6A, an electronic device (101) that obtains an image (630) from a facial image generation model (620) can segment or extract a portion (635) corresponding to an external electronic device (120) within the image (630). The electronic device (101) can replace the portion (505) corresponding to the external electronic device (120) within an image (410) to be displayed on a display with the portion (635) extracted from the image (630). The electronic device (101) can perform alignment between the image (410) and the portion (635) based on feature points (or landmarks of the face indicated by the feature points) of the face of the second user (125) identified from the image (410).

[0139] In one embodiment, by displaying an image (410) including a portion (505) replaced with a portion (635), the electronic device (101) can provide the first user (110) with a full face of a second user (125) wearing an external electronic device (120). Based on the full face, the electronic device (101) can provide a nonverbal expression of the second user (125) to the first user (110). To support the external electronic device (120) in providing the nonverbal expression of the first user (110) to the second user (125), the electronic device (101) can transmit to the external electronic device (120) at least one of an image of the face of the first user (110) and / or gaze information related to both eyes of the first user (110).

[0140] Referring to state (602) of FIG. 6B, an image (652) of an external environment acquired by an electronic device (101) using a camera is illustrated. When a mobile phone (654) is placed between the electronic device (101) and an external electronic device (120), an image (652) may be acquired in which the external electronic device (120) has an appearance covered by the mobile phone (654) within the image (652). In one embodiment of acquiring the image (652) of FIG. 6B, the electronic device (101) may not be able to identify the external electronic device (120) using the image (652).

[0141] In one embodiment of FIG. 6B, the electronic device (101) can detect an external electronic device (120) using an image (660) of a mobile phone (654) connected to the electronic device (101) (e.g., account information of the first user (110) is logged in). For example, the electronic device (101) can detect a mobile phone (654) located in front of the electronic device (101) using a positioning protocol such as UWB. To the detected mobile phone (654), the electronic device (101) can transmit a signal to acquire an image from a camera of the mobile phone (654). Based on the signal, the electronic device (101) that acquires the image (660) captured by the mobile phone (654) can detect a portion (505) corresponding to the external electronic device (120) within the image (660).

[0142] In one embodiment of FIG. 6B, the electronic device (101) that detects a portion (505) covered by an external electronic device (120) from an image (660) acquired from a mobile phone (654) may execute a function to synthesize an image and / or video representing a body part (e.g., at least a portion of a face including both eyes) of a second user (125) wearing the external electronic device (120) with the portion (505) corresponding to the portion in the image (652). The synthesis may be performed based on the execution of a facial image generation model (620). For example, the electronic device (101) may input at least one of the image (660) acquired from the mobile phone (654), gaze information (510), and / or an image (610) related to the second user (125) into the facial image generation model (620). The image (610) and / or gaze information (510) may be transmitted to the electronic device (101) based on the operation of the electronic device (101) and / or the external electronic device (120) described above with reference to FIG. 6A.

[0143] Referring to FIG. 6B, the electronic device (101) can obtain an image (630) representing the face of the second user (125) using the facial image generation model (620). Within the image (630), using a portion (635) corresponding to a portion (505) of the image (660), the electronic device (101) can replace, change, or update a portion corresponding to the external electronic device (120) within the image (652).

[0144] As described above, according to one embodiment, the electronic device (101) can visualize a body part (e.g., a part of the face where both eyes are arranged) of the counterpart (e.g., the second user (125)) covered by the external electronic device (120) by using the three-dimensional graphical object (e.g., a virtual object based on mesh information (520)) of FIG. 5 and / or an image (630) obtained from the neural network (e.g., the facial image generation model (620)) of FIG. 6A and / or FIG. 6B. The electronic device (101) can optimize the amount of computation for visualizing the body part of the counterpart. For example, if an interaction (e.g., conversation) between a first user (110) and a second user (125) is interrupted, and / or the size of a portion (e.g., a portion (505) of an image (410)) corresponding to an external electronic device (120) on the display is reduced, the electronic device (101) may reduce the amount of computation required to infer the face of the second user (125) covered by the external electronic device (120).

[0145] Below, the operation of the electronic device (101) for optimizing the amount of calculation is described with reference to FIGS. 7a and 7b.

[0146] FIGS. 7A and 7B illustrate an example of an operation of an electronic device (101) according to various embodiments of the present disclosure. The electronic device (101) of FIGS. 1, 2A, and 2B and / or the processor (210) of FIGS. 2A and 2B may perform the operations of the electronic device (101) described with reference to FIGS. 7A and 7B. The operations of the electronic device (101) described with reference to FIGS. 7A and 7B may be related to the operations of FIG. 3.

[0147] Referring to FIG. 7A, a state (701) of an electronic device (101) providing a screen (130) including an image of a face of a second user (125) covered by the external electronic device (120) is illustrated. The state (701) of the electronic device (101) may be related to a reverse pass-through mode. The electronic device (101) may display a screen (130) including an image of the entire face of the second user (125) based on at least one of three-dimensional rendering using the mesh information (520) and / or the gaze information (510) of FIG. 5 and / or the face image generation model (620) of FIGS. 6A and 6B.

[0148] Referring to FIG. 7A, within the screen (130), the position of the visual object (710) corresponding to the second user (125) may correspond to the position (L1) of the second user (125) and / or the external electronic device (120). Based on the direction (d1) of the second user (125) and / or the external electronic device (120), the electronic device (101) may display a virtual object, image and / or video representing at least a portion of the face of the second user (125) covered by the external electronic device (120) on a portion (720) corresponding to the external electronic device (120) within the screen (130). As in the screen (130) of FIG. 7A, the first user (110) wearing the electronic device (101) may view an external environment from which the external electronic device (120) is removed, based on the virtual object, the image and / or the video.

[0149] In the state (701) of FIG. 7A, the positional relationship between the electronic device (101) and the external electronic device (120) may change based on movement and / or rotation of the first user (110) wearing the electronic device (101) and / or the second user (125) wearing the external electronic device (120). The change in the positional relationship may cause a change in the size of a portion (720) corresponding to the external electronic device (120) within the screen (130).

[0150] For example, when the second user (125) turns his head, the direction of the external electronic device (120) may change from the direction (d1) of FIG. 7A to the direction (d2). When the external electronic device (120) faces the direction (d2), since the back of the second user (125) faces the electronic device (101), the size of the portion (720) corresponding to the external electronic device (120) may be substantially reduced to 0. For example, when the second user (125) moves, the position and / or size of the portion (720) corresponding to the external electronic device (120) within the screen (130) may change. For example, when the position of the external electronic device (120) moves from position (L1) to position (L2) as the second user (125) moves, the size of the portion (720) corresponding to the external electronic device (120) within the screen (130) may be substantially reduced to 0 because the second user (125) moves out of the field-of-view (FoV) of the camera (e.g., the external camera (225-2) of FIG. 2A and / or FIG. 2B) of the electronic device (101). For example, when the position of the external electronic device (120) moves from position (L1) to position (L3) in a direction away from the electronic device (101) (e.g., in the +z-axis direction) as the second user (125) moves, the size of the portion (720) corresponding to the external electronic device (120) within the screen (130) may be reduced.

[0151] According to one embodiment, the electronic device (101) can determine the size of a portion (720) corresponding to the external electronic device (120) within the screen (130) based on at least one of a distance from the external electronic device (120) or a direction (e.g., an azimuth angle) of the external electronic device (120) with respect to the electronic device (101). The distance from the external electronic device (120) can include a distance between the electronic device (101) and the external electronic device (120). The distance can be obtained or calculated based on object recognition based on an image acquired from a camera. The electronic device (101) can calculate the distance based on a wireless communication protocol such as BLE and / or UWB.

[0152] According to one embodiment, the electronic device (101) may determine the size and / or resolution of an image (e.g., image (630) of FIGS. 6A and / or 6B) to be synthesized to represent a body part (e.g., at least a portion of a face including both eyes) of a second user (125) covered by the external electronic device (120) based on the size of the portion (720) corresponding to the external electronic device (120). For example, the electronic device (101) may selectively operate any one of different models (e.g., face image generation model (620) of FIGS. 6A and 6B) for synthesizing images of different sizes (or resolutions) to synthesize an image having the determined size and / or resolution. The amount of computation required to operate the models may be related to the size and / or resolution of the image to be synthesized by each of the models. For example, as the size and / or resolution of the image to be synthesized by the model increases, the amount of computation performed to operate the model may increase. The number of weights and / or parameters included in the model may also increase as the size and / or resolution of the image to be synthesized by the model increases.

[0153] An electronic device (101) including models corresponding to different sizes can generate an image to be synthesized on the part (720) by driving one model corresponding to the size and / or resolution of the part (720). Since the size of the part (720) changes depending on the distance and / or direction between the electronic device (101) and the external electronic device (120), the electronic device (101) can select or identify one model from among a plurality of models based at least on the size of the part (720). Using the selected model, the electronic device (101) can acquire or generate at least a portion of an image to be displayed on the part (720) within the screen (130). The electronic device (101) selecting a different model based on the size of the part (720) can adaptively adjust the amount of calculation performed to synthesize or generate the image to be displayed on the part (720) based on the selection of the model.

[0154] According to one embodiment, the electronic device (101) may adjust the amount of graphical resources of a three-dimensional virtual object corresponding to a second user (125) based on the size of a portion (720) corresponding to an external electronic device (120). The three-dimensional virtual object may include a virtual object rendered by mesh information (520) and / or gaze information (510) of FIG. 5. The graphical resources may include vertices and / or textures included in the mesh information (520). For example, the electronic device (101) may adjust the number of vertices and / or textures of the virtual object based on the size of the portion (720). For example, the electronic device (101) may adjust the size of each texture based on the size of the portion (720). For example, the electronic device (101) can reduce the amount of graphical resources for a virtual object based on texture regression when the size of the portion (720) is reduced.

[0155] For example, the electronic device (101) can change the frequency and / or cycle for performing rendering of a three-dimensional virtual object corresponding to the second user (125) based on the size of the portion (720). For example, when the size of the portion (720) is reduced, the electronic device (101) can increase the cycle for performing the rendering. Independently of the cycle, the electronic device (101) can maintain the cycle for displaying a portion (e.g., a portion (530) of FIG. 5) of the three-dimensional virtual object on the portion (720) of the screen (130). For example, the electronic device (101) can maintain the cycle for aligning a portion of the three-dimensional virtual object and the portion (720) independently from the cycle for performing the rendering of the three-dimensional virtual object.

[0156] According to one embodiment, the electronic device (101) can change the period (e.g., frames per second (FPS)) of a camera (e.g., the gaze tracking camera (225-1) and / or the external camera (225-2) of FIGS. 2A and / or 2B ) based on at least one of a distance from the external electronic device (120) and / or a direction of the external electronic device (120) with respect to the electronic device (101). For example, as the distance between the electronic device (101) and the external electronic device (120) increases and / or the direction of the external electronic device (120) changes (e.g., directions (d1, d2)), the size of a portion (720) corresponding to the external electronic device (120) within the screen (130) can change. The electronic device (101) that identifies that the size of the portion (720) has changed can control the camera based on the period corresponding to the changed size. For example, as the size of the portion (720) decreases, the cycle associated with controlling the camera may increase. As the size of the portion (720) decreases, the FPS of the video acquired from the camera may decrease, or the number of images (e.g., the number per unit time) may decrease. Because the number of images is reduced, the frequency with which an operation to replace the appearance of the external electronic device (120) is performed may decrease.

[0157] According to one embodiment, the electronic device (101) may change the cycle for receiving information from the external electronic device (120) based on at least one of a distance from the external electronic device (120) and / or a direction of the external electronic device (120) with respect to the electronic device (101). The information received from the external electronic device (120) may include line of sight information (510) of FIGS. 5, 6A, and / or 6B. For example, if the size of a portion (720) corresponding to the external electronic device (120) within the screen (130) decreases based on the distance and / or the direction, the electronic device (101) may increase the cycle. Since the electronic device (101) receives information based on the increased cycle, the current consumption of a circuit (e.g., a communication circuit (235) of FIGS. 2A and / or 2B) activated to receive the information may be reduced.

[0158] The conditions under which the operation of the electronic device (101) described above with reference to FIG. 7A is performed are not limited to the positional relationship (e.g., distance and / or direction) between the electronic device (101) and the external electronic device (120). For example, the electronic device (101) may stop or limit the operation for replacing the corresponding part (720) of the external electronic device (120) based on a user input indicating the termination and / or cessation of the reverse pass-through mode. For example, the electronic device (101) may reduce the size (or resolution) of a virtual object, image and / or video to be synthesized into the portion (720) or increase a cycle for acquiring information required for synthesizing the virtual object, image and / or video based on information related to speech (e.g., frequency, speed and / or number of times) of the speech of users (e.g., first user (110) and / or second user (125)) wearing the electronic device (101) and the external electronic device (120).

[0159] The operation of the electronic device (101) based on the positional relationship between the electronic device (101) and the external electronic device (120) is not limited to the above-described embodiment. For example, while displaying a screen (130) based on a reverse pass-through mode, the electronic device (101) may provide a visual effect indicating whether a second user (125) wearing the external electronic device (120) is gazing at the electronic device (101) and / or the first user (110) wearing the electronic device (101), based on the direction (d1) of the external electronic device (120). For example, when the direction (d1) of the external electronic device (120) is directed toward the electronic device (101), the electronic device (101) may highlight a visual object (710) corresponding to the second user (125) within the screen (130). The action of highlighting a visual object (710) may include an action of adjusting the color, brightness, and / or saturation of the visual object (710). The action of highlighting a visual object (710) may include an action of displaying a shape having a shape that surrounds at least a portion of the visual object (710) (e.g., a shape having a specified transparency and / or a specified color).

[0160] Referring to FIG. 7B, a state (702) of an electronic device (101) providing a screen (130) including an image of a face of a second user (125) covered (or obscured, or hidden) by an external electronic device (120) is illustrated. Within the state (702), the electronic device (101) may display a screen (130) associated with a reverse pass-through mode.

[0161] Referring to FIG. 7B, an external image of a user (110) wearing an electronic device (101) may include a visual object (710) corresponding to a second user (125) wearing an external electronic device (120), as well as a visual object (752) corresponding to a third user (750) in front of the user (110). The electronic device (101) may use information transmitted from the external electronic device (120) to display an image representing at least a portion of the face of the second user (125) hidden by the external electronic device (120) (e.g., a portion of the face including both eyes of the second user (125)) on a portion (720) corresponding to the external electronic device (120).

[0162] Referring to the screen (130) of FIG. 7B, while displaying the faces of all of the second user (125) wearing the external electronic device (120) and the third user (750) through the screen (130), the electronic device (101) may display a visual object (760) to represent the second user (125) wearing the external electronic device (120) among the users (e.g., the second user (125) and / or the third user (750)) visible through the screen (130). For example, a visual object (760) including designated text such as “VR user” may be displayed on the visual object (710) corresponding to the second user (125) wearing the external electronic device (120) within the screen (130). By displaying a visual object (760) having a location within a screen (130) linked to a visual object (710), the electronic device (101) can guide or inform a second user (125) corresponding to the visual object (710) that the external electronic device (120) is worn.

[0163] As described above, according to one embodiment, the electronic device (101) can operate in a reverse pass-through mode to support the transmission of nonverbal expressions based on facial expressions. For example, electronic devices (e.g., the electronic device (101) and / or the external electronic device (120)) positioned toward each other can exchange information to swap the appearances of the electronic devices within their respective screens (e.g., the screen (130)) based on the reverse pass-through mode. The information can be generated to visualize body parts covered by each of the electronic devices (e.g., parts of the faces of users wearing each of the electronic devices).

[0164] Below, with reference to FIG. 8, the operations of the electronic devices performed for exchanging the information are described.

[0165] FIG. 8 illustrates a signal flow diagram between electronic devices (e.g., a first electronic device (801) and / or a second electronic device (802)) according to one embodiment of the present disclosure. Each of the first electronic device (801) and the second electronic device (802) of FIG. 8 may correspond to each of the electronic device (101) and the external electronic device (120) described with reference to FIGS. 1, 2A, 2B, 3 to 5, 6A, 6B, 7A, and / or 7B. The operations of the first electronic device (801) and / or the second electronic device (802) described with reference to FIG. 8 may be performed by the processor (210) of FIG. 2A and / or 2B.

[0166] Referring to FIG. 8, within operation (812), a processor of a first electronic device (801) according to an embodiment may detect an interaction with a second electronic device (802). Similar to operation (812), within operation (814), a processor of a second electronic device (802) according to an embodiment may detect an interaction with the first electronic device (801). The interaction may be detected based on a user input for executing a reverse pass-through mode, as described above with reference to FIG. 4.

[0167] Referring to FIG. 8, within operation (820), according to one embodiment, the first electronic device (801) and the second electronic device (802) may enter a reverse pass-through mode. The first electronic device (801) and the second electronic device (802) may enter the reverse pass-through mode of operation (820) while displaying images and / or videos of the external environment based on the pass-through. Based on entering the reverse pass-through mode, a communication link may be established between the first electronic device (801) and the second electronic device (802). Through the communication link, the processor of the first electronic device (801) can transmit, to the second electronic device (802), information (e.g., 3D mesh information and / or one or more photographs) related to the face of a first user (e.g., the first user (110) of FIGS. 1 , 2A, 2B, 3 to 5, 6A, 6B, 7A, and / or 7B) wearing the first electronic device (801). Similarly, through the communication link, the processor of the second electronic device (802) can transmit, to the first electronic device (801), information related to the face of a second user (e.g., the second user (125) of FIGS. 1 , 2A, 2B, 3 to 5, 6A, 6B, 7A, and / or 7B) wearing the second electronic device (802). Within the operation (820), information for rendering the faces of users can be exchanged between the first electronic device (801) and the second electronic device (802).

[0168] Within the reverse pass-through mode, based on the operation (832), the first electronic device (801) can acquire a first image through an external camera (e.g., the external camera (225-2) of FIG. 2A and / or FIG. 2B). The first image can correspond to the external environment of the front of the first electronic device (801). Within the first image, the processor of the first electronic device (801) can identify a portion corresponding to the second electronic device (802).

[0169] Similarly, within the operation (834), the second electronic device (802) can acquire a second image through an external camera. The second image can correspond to the external environment of the front of the second electronic device (802). In one embodiment where the first electronic device (801) and the second electronic device (802) are positioned facing each other, the second electronic device (802) can identify or confirm a portion corresponding to the first electronic device (801) within the second image.

[0170] Referring to FIG. 8, in operation (842), according to one embodiment, the processor of the first electronic device (801) may obtain a third image through a gaze tracking camera (e.g., the gaze tracking camera (225-1) of FIG. 2A and / or FIG. 2B). The third image may be an image of at least one of the two eyes of the first user wearing the first electronic device (801). Based on the third image, the processor of the first electronic device (801) may obtain or calculate a direction of the gaze of the first user.

[0171] Referring to FIG. 8, in operation (844), according to one embodiment, the processor of the second electronic device (802) may acquire a fourth image through a gaze tracking camera. The fourth image may correspond to at least one of the two eyes of the second user wearing the second electronic device (802). Based on the fourth image, the processor of the second electronic device (802) may generate or acquire information indicating the direction of the second user's gaze.

[0172] Referring to FIG. 8, within operation (852), according to an embodiment, the processor of the first electronic device (801) may transmit the third image of operation (842) to the second electronic device (802) through a communication link established based on the reverse pass-through mode of operation (820). The embodiment is not limited thereto, and the first electronic device (801) may transmit information indicating the direction of the first user's gaze, obtained based on the third image, to the second electronic device (802). Within the reverse pass-through mode, the first electronic device (801) may repeatedly perform operation (852). Based on the repeatedly performed operation (852), the first electronic device (801) may notify the second electronic device (802) of information related to the first user's gaze in real time. The embodiment is not limited thereto, and the processor of the first electronic device (801) may transmit an image representing the entire face of the first user. The image transmitted to the second electronic device may at least partially include a third image acquired based on operation (842).

[0173] Referring to FIG. 8, in operation (854), according to one embodiment, the processor of the second electronic device (802) may transmit the fourth image of operation (844) to the first electronic device (801) via a communication link. What is transmitted to the first electronic device (801) is not limited to the fourth image. For example, the processor of the second electronic device (802) may transmit information related to the gaze of the second user, obtained from the fourth image, to the first electronic device (801). The second electronic device (802) may periodically (or repeatedly) perform operation (854) within a reverse pass-through mode. The embodiment is not limited thereto, and the processor of the second electronic device (802) may transmit an image representing the entire face of the second user. The image transmitted to the first electronic device may at least partially include the fourth image obtained based on operation (844).

[0174] The information transmitted by the first electronic device (801) and the second electronic device (802) within the actions (852, 854) may not only transmit images acquired through the gaze tracking camera, but may also transmit images of the entire face of the user of the electronic devices. For example, the first electronic device (801) may transmit images and / or videos representing the user's face covered by the first electronic device (801) to the second electronic device (802).

[0175] Referring to FIG. 8, in operation (862), according to one embodiment, the processor of the first electronic device (801) may synthesize the first image and the fourth image. For example, the first electronic device (801) may combine the fourth image with a portion of the first image corresponding to the second electronic device (802). As the first image and the fourth image are combined, the fourth image representing the two eyes of the second user covered by the second electronic device (802) may be placed on the portion of the first image corresponding to the second electronic device (802). The embodiment is not limited thereto, and the first electronic device (801) may generate a virtual image of at least a portion of the face including both eyes of the second user from the fourth image using a three-dimensional virtual object and / or a neural network (e.g., the facial image generation model (620) of FIG. 6A and / or FIG. 6B), as described above with reference to FIGS. 5, 6A, and 6B. The first electronic device (801) may combine the generated virtual image with the first image.

[0176] Referring to FIG. 8, in operation (864), according to one embodiment, the processor of the second electronic device (802) may synthesize the second image and the third image. For example, the second electronic device (802) may perform operation (864) to reconstruct a face of the first user covered by the first electronic device (801) (e.g., a portion of the face including both eyes) within the second image associated with the external environment. The processor of the second electronic device (802) may perform the operation of the electronic device (101) described above with reference to FIGS. 5, 6A, and / or 6B to replace the portion covered by the first electronic device (801) within the second image with a virtual object, image, and / or video representing the face of the first user.

[0177] Referring to FIG. 8, in operation (872), the processor of the first electronic device (801) according to one embodiment may display the synthesized result based on operation (862) on the display of the first electronic device (801) (e.g., the display (220) of FIG. 2A and / or FIG. 2B). For example, the first electronic device (801) may display the first image (e.g., an image of an external environment including the second electronic device (802)) synthesized with the fourth image (e.g., an image representing two eyes of a second user wearing the second electronic device (802). The first electronic device (801) may perform operations (862, 872) of FIG. 8 for image frames continuously acquired through an external camera in a reverse pass-through mode.

[0178] Referring to FIG. 8, in operation (874), the processor of the second electronic device (802) according to one embodiment may display a synthesized result based on operation (864) on a display. The second electronic device (802) may display a second image (e.g., an image corresponding to an external environment including the first electronic device (801)) synthesized with the third image (e.g., an image related to two eyes of a first user wearing the first electronic device (801). The second electronic device (802) may perform operations (864, 874) of FIG. 8 for each of the image frames sequentially acquired through an external camera in a reverse pass-through mode.

[0179] Referring to FIG. 8, each of the first electronic device (801) and the second electronic device (802) may repeatedly perform operations (832, 842, 852, 862, 872) and operations (834, 844, 854, 864, 874) after entering the reverse pass-through mode of operation (820). When switching to another mode different from the reverse pass-through mode, the first electronic device (801) may stop repeatedly performing operations (832, 842, 852, 862, 872). The other mode may include a mode for providing a user experience disconnected from the external environment, such as a VR mode, for example. The first electronic device (801) may switch from the reverse pass-through mode to the other mode in response to the cessation of the interaction detected based on operation (812). As described above with reference to FIG. 4, the first electronic device (801) can switch from the reverse pass-through mode to the other mode based on the distance and / or direction between the first electronic device (801) and the second electronic device (802).

[0180] With reference to FIGS. 1, 2a, 2b, 3 to 5, 6a, 6b, 7a, 7b, and 8, two electronic devices are described, but the embodiment is not limited thereto. For example, each of the three electronic devices can perform a function to remove another electronic device from the field-of-view (FoV) that obscures a body part within the FoV using a reverse pass-through mode. With reference to FIGS. 1, 2a, 2b, 3 to 5, 6a, 6b, 7a, 7b, and 8, the operation of a first electronic device (801) to synthesize a face of a second user wearing a second electronic device (802) is described, but the embodiment is not limited thereto. For example, the second electronic device (802) may transmit a virtual object, an image, and / or a video of a face of a second user wearing the second electronic device (802) to the first electronic device (801). The first electronic device (801) may display the virtual object, the image, and / or the video transmitted from the second electronic device (802) as an overlay on the first image of the operation (832).

[0181] Hereinafter, with reference to FIGS. 9A, 9B, 10A and / or 10B, an appearance of an electronic device (101) described with reference to FIGS. 1, 2A, 2B, 3 to 5, 6A, 6B, 7A, and / or 7B is illustrated. The wearable device (900) of FIG. 9A and / or 9B and / or the wearable device (1000) of FIG. 10A and / or 10B may be an example of the electronic device (101) of FIG. 1.

[0182] FIG. 9A illustrates an example of a perspective view of a wearable device according to one embodiment of the present disclosure.

[0183] According to one embodiment, the wearable device (900) may have the form of glasses that are wearable on a body part of the user (e.g., the head). The wearable device (900) may include a head-mounted display (HMD). For example, the housing of the wearable device (900) may include a flexible material, such as rubber and / or silicone, that is configured to fit closely to a portion of the user's head (e.g., a portion of the face surrounding both eyes). For example, the housing of the wearable device (900) may include one or more straps that are capable of being twined around the user's head, and / or one or more temples that are detachably attachable to the ears of the head.

[0184] Referring to FIG. 9A, according to one embodiment, a wearable device (900) may include at least one display (950) and a frame supporting at least one display (950).

[0185] FIG. 9b illustrates an example of one or more hardware elements arranged in a wearable device, according to one embodiment of the present disclosure.

[0186] According to one embodiment, a wearable device (900) can be worn on a part of a user's body. The wearable device (900) can provide augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality to a user wearing the wearable device (900). For example, the wearable device (900) can display a virtual reality image provided from at least one optical device (982, 984) of FIG. 9B on at least one display (950) in response to a user's designated gesture acquired through the motion recognition cameras (960-2, 960-3) of FIG. 9B.

[0187] According to one embodiment, at least one display (950) may provide visual information to a user. For example, at least one display (950) may include a transparent or translucent lens. At least one display (950) may include a first display (950-1) and / or a second display (950-2) spaced apart from the first display (950-1). For example, the first display (950-1) and the second display (950-2) may be positioned at positions corresponding to the user's left and right eyes, respectively.

[0188] Referring to FIG. 9B, at least one display (950) can provide visual information transmitted from external light to a user through a lens included in at least one display (950), and other visual information distinct from the visual information. The lens can be formed based on at least one of a Fresnel lens, a pancake lens, or a multi-channel lens. For example, at least one display (950) can include a first surface (931) and a second surface (932) opposite to the first surface (931). A display area can be formed on the second surface (932) of at least one display (950). When a user wears the wearable device (900), external light can be transmitted to the user by being incident on the first surface (931) and transmitted through the second surface (932). As another example, at least one display (950) can display an augmented reality image combined with a virtual reality image provided from at least one optical device (982, 984) on a real screen transmitted through external light, in a display area formed on the second surface (932).

[0189] In one embodiment, at least one display (950) may include at least one waveguide (933, 934) that diffracts light emitted from at least one optical device (982, 984) and transmits the diffracted light to a user. The at least one waveguide (933, 934) may be formed based on at least one of glass, plastic, or polymer. A nano-pattern may be formed on at least a portion of the exterior or interior of the at least one waveguide (933, 934). The nano-pattern may be formed based on a grating structure having a polygonal and / or curved shape. Light incident on one end of the at least one waveguide (933, 934) may be propagated to the other end of the at least one waveguide (933, 934) by the nano-pattern. At least one waveguide (933, 934) may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) and at least one reflective element (e.g., a reflective mirror). For example, at least one waveguide (933, 934) may be arranged within the wearable device (900) to guide a screen displayed by at least one display (950) to the user's eyes. For example, the screen may be transmitted to the user's eyes based on total internal reflection (TIR) ​​occurring within the at least one waveguide (933, 934).

[0190] The wearable device (900) can analyze an object included in a real image collected through a shooting camera (960-4), combine a virtual object corresponding to an object to be provided with augmented reality among the analyzed objects, and display the virtual object on at least one display (950). The virtual object can include at least one of text and an image regarding various information related to the object included in the real image. The wearable device (900) can analyze the object based on a multi-camera such as a stereo camera. For the object analysis, the wearable device (900) can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) using the multi-camera and / or time-of-flight (ToF). A user wearing the wearable device (900) can view an image displayed on at least one display (950).

[0191] According to one embodiment, the frame may be formed as a physical structure that allows the wearable device (900) to be worn on the user's body. According to one embodiment, the frame may be configured so that, when the user wears the wearable device (900), the first display (950-1) and the second display (950-2) can be positioned corresponding to the user's left and right eyes. The frame may support at least one display (950). For example, the frame may support the first display (950-1) and the second display (950-2) to be positioned corresponding to the user's left and right eyes.

[0192] Referring to FIG. 9A, the frame may include a region (920) that at least partially contacts a portion of the user's body when the user wears the wearable device (900). For example, the region (920) of the frame that contacts a portion of the user's body may include a region that contacts a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that the wearable device (900) makes contact with. According to one embodiment, the frame may include a nose pad (910) that contacts a portion of the user's body. When the wearable device (900) is worn by the user, the nose pad (910) may contact a portion of the user's nose. The frame may include a first temple (904) and a second temple (905) that contact a portion of the user's body that is distinct from the portion of the user's body.

[0193] For example, the frame may include a first rim (901) that surrounds at least a portion of the first display (950-1), a second rim (902) that surrounds at least a portion of the second display (950-2), a bridge (903) that is disposed between the first rim (901) and the second rim (902), a first pad (911) that is disposed along a portion of an edge of the first rim (901) from one end of the bridge (903), a second pad (912) that is disposed along a portion of an edge of the second rim (902) from the other end of the bridge (903), a first temple (904) that extends from the first rim (901) and is secured to a portion of an ear of the wearer, and a second temple (905) that extends from the second rim (902) and is secured to a portion of an ear opposite the ear. The first pad (911) and the second pad (912) may be in contact with a portion of the user's nose, and the first temple (904) and the second temple (905) may be in contact with a portion of the user's face and a portion of the user's ear. The temples (904, 905) may be rotatably connected to the rim through the hinge units (906, 907) of FIG. 9B. The first temple (904) may be rotatably connected to the first rim (901) through the first hinge unit (906) disposed between the first rim (901) and the first temple (904). The second temple (905) may be rotatably connected to the second rim (902) through the second hinge unit (907) disposed between the second rim (902) and the second temple (905). According to one embodiment, the wearable device (900) can identify an external object (e.g., a user's fingertip) touching the frame and / or a gesture performed by the external object by using a touch sensor, a grip sensor, and / or a proximity sensor formed on at least a portion of a surface of the frame.

[0194] According to one embodiment, the wearable device (900) may include hardwares (e.g., hardwares described above based on the block diagrams of FIGS. 2A and / or 2B) that perform various functions. For example, the hardwares may include a battery module (970), an antenna module (975), at least one optical device (982, 984), speakers (e.g., speakers 955-1, 955-2), microphones (e.g., microphones 965-1, 965-2, 965-3), a light-emitting module, and / or a printed circuit board (PCB) (990) (e.g., a printed circuit board). The various hardware components may be arranged within a frame.

[0195] According to one embodiment, the microphones (e.g., microphones 965-1, 965-2, 965-3) of the wearable device (900) may be disposed on at least a portion of the frame to acquire sound signals. A first microphone (965-1) disposed on the bridge (903), a second microphone (965-2) disposed on the second rim (902), and a third microphone (965-3) disposed on the first rim (901) are illustrated in FIG. 9B , but the number and arrangement of the microphones (965) are not limited to the embodiment of FIG. 9B . When the number of microphones (965) included in the wearable device (900) is two or more, the wearable device (900) may identify the direction of the sound signal by using a plurality of microphones disposed on different portions of the frame.

[0196] In one embodiment, at least one optical device (982, 984) can project a virtual object onto at least one display (950) to provide various image information to a user. For example, at least one optical device (982, 984) can be a projector. At least one optical device (982, 984) can be positioned adjacent to at least one display (950) or can be included within at least one display (950) as a part of at least one display (950). In one embodiment, the wearable device (900) can include a first optical device (982) corresponding to a first display (950-1) and a second optical device (984) corresponding to a second display (950-2). For example, at least one optical device (982, 984) may include a first optical device (982) disposed at an edge of a first display (950-1) and a second optical device (984) disposed at an edge of a second display (950-2). The first optical device (982) may transmit light to a first waveguide (933) disposed on the first display (950-1), and the second optical device (984) may transmit light to a second waveguide (934) disposed on the second display (950-2).

[0197] In one embodiment, the camera (960) may include a recording camera (960-4), an eye tracking camera (ET CAM) (960-1), and / or motion recognition cameras (960-2, 960-3). The recording camera (960-4), the eye tracking camera (960-1), and the motion recognition cameras (960-2, 960-3) may be positioned at different locations on the frame and may perform different functions. The eye tracking camera (960-1) may output data indicating the position or gaze of the eyes of a user wearing the wearable device (900). For example, the wearable device (900) may detect the gaze from an image including the user's pupils obtained through the eye tracking camera (960-1).

[0198] The wearable device (900) can identify an object (e.g., a real object and / or a virtual object) focused on by the user using the user's gaze acquired through the gaze tracking camera (960-1). The wearable device (900) that has identified the focused object can execute a function (e.g., gaze interaction) for interaction between the user and the focused object. The wearable device (900) can express a part corresponding to the eye of an avatar representing the user in a virtual space using the user's gaze acquired through the gaze tracking camera (960-1). The wearable device (900) can render an image (or screen) displayed on at least one display (950) based on the position of the user's eyes.

[0199] For example, the visual quality of a first region related to the gaze within an image and the visual quality (e.g., resolution, brightness, saturation, grayscale, PPI (pixels per inch)) of a second region distinct from the first region may be different from each other. The wearable device (900) may obtain an image having the visual quality of the first region matching the user's gaze and the visual quality of the second region using foveated rendering. For example, if the wearable device (900) supports an iris recognition function, user authentication may be performed based on iris information obtained using the gaze tracking camera (960-1). Although an example in which the gaze tracking camera (960-1) is positioned toward the user's right eye is illustrated in FIG. 9B, the embodiment is not limited thereto, and the gaze tracking camera (960-1) may be positioned solely toward the user's left eye, or toward both eyes.

[0200] In one embodiment, the capturing camera (960-4) can capture an actual image or background to be aligned with a virtual image to implement augmented reality or mixed reality content. The capturing camera (960-4) can be used to acquire a high-resolution image based on HR (high resolution) or PV (photo video). The capturing camera (960-4) can capture an image of a specific object existing at a location viewed by the user and provide the image to at least one display (950). The at least one display (950) can display a single image in which information about an actual image or background including the image of the specific object acquired using the capturing camera (960-4) is superimposed on a virtual image provided through at least one optical device (982, 984). The wearable device (900) can compensate for depth information (e.g., the distance between the wearable device (900) and an external object acquired through a depth sensor) using the image acquired through the capturing camera (960-4). The wearable device (900) can perform object recognition through an image acquired using the capturing camera (960-4). The wearable device (900) can perform a function of focusing on an object (or subject) in an image (e.g., auto focus) and / or an optical image stabilization (OIS) function (e.g., anti-shake function) using the capturing camera (960-4). The wearable device (900) can perform a pass-through function to display an image acquired through the capturing camera (960-4) by overlapping at least a portion of a screen representing a virtual space on at least one display (950). In one embodiment, the capturing camera (960-4) can be disposed on a bridge (903) disposed between the first rim (901) and the second rim (902).

[0201] The gaze tracking camera (960-1) can implement more realistic augmented reality by tracking the gaze of a user wearing the wearable device (900) and matching the user's gaze with visual information provided to at least one display (950). For example, when the wearable device (900) looks straight ahead, the wearable device (900) can naturally display environmental information related to the user's front at a location where the user is located on at least one display (950). The gaze tracking camera (960-1) can be configured to capture an image of the user's pupil to determine the user's gaze. For example, the gaze tracking camera (960-1) can receive gaze detection light reflected from the user's pupil and track the user's gaze based on the position and movement of the received gaze detection light. In one embodiment, the gaze tracking camera (960-1) can be positioned at positions corresponding to the user's left and right eyes. For example, the gaze tracking camera (960-1) may be positioned within the first rim (901) and / or the second rim (902) to face the direction in which the user wearing the wearable device (900) is positioned.

[0202] The gesture recognition camera (960-2, 960-3) can recognize the movement of the user's entire body, such as the user's torso, hand, or face, or a part of the body, and thereby provide a specific event on a screen provided on at least one display (950). The gesture recognition camera (960-2, 960-3) can recognize the user's gesture (gesture recognition), obtain a signal corresponding to the gesture, and provide a display corresponding to the signal on at least one display (950). The processor can identify the signal corresponding to the gesture, and perform a designated function based on the identification. The gesture recognition camera (960-2, 960-3) can be used to perform a spatial recognition function using SLAM and / or a depth map for 6 degrees of freedom pose (6 dof pose). The processor can perform a gesture recognition function and / or an object tracking function using the gesture recognition camera (960-2, 960-3). In one embodiment, the motion recognition cameras (960-2, 960-3) may be positioned on the first rim (901) and / or the second rim (902).

[0203] The camera (960) included in the wearable device (900) is not limited to the above-described gaze tracking camera (960-1) and motion recognition cameras (960-2, 960-3). For example, the wearable device (900) can identify an external object included in the FoV using a camera positioned toward the user's FoV. The wearable device (900) can identify an external object based on a sensor for identifying the distance between the wearable device (900) and the external object, such as a depth sensor and / or a time of flight (ToF) sensor. The camera (960) positioned toward the FoV can support an autofocus function and / or an optical image stabilization (OIS) function. For example, the wearable device (900) may include a camera (960) (e.g., a face tracking (FT) camera) positioned toward the face to obtain an image including the face of a user wearing the wearable device (900).

[0204] Although not shown, in one embodiment, the wearable device (900) may further include a light source (e.g., an LED) that emits light toward a subject (e.g., a user's eyes, face, and / or an external object within the FoV) being captured using the camera (960). The light source may include an infrared wavelength LED. The light source may be disposed on at least one of the frame and hinge units (906, 907).

[0205] In one embodiment, the battery module (970) may supply power to the electronic components of the wearable device (900). In one embodiment, the battery module (970) may be disposed within the first temple (904) and / or the second temple (905). For example, the battery module (970) may be a plurality of battery modules (970). The plurality of battery modules (970) may be disposed within each of the first temple (904) and the second temple (905). In one embodiment, the battery module (970) may be disposed at an end of the first temple (904) and / or the second temple (905).

[0206] The antenna module (975) can transmit signals or power to the outside of the wearable device (900), or receive signals or power from the outside. In one embodiment, the antenna module (975) can be positioned within the first temple (904) and / or the second temple (905). For example, the antenna module (975) can be positioned close to one surface of the first temple (904) and / or the second temple (905).

[0207] The speaker (955) can output an audio signal to the outside of the wearable device (900). The audio output module may be referred to as a speaker. In one embodiment, the speaker (955) may be positioned within the first temple (904) and / or the second temple (905) so as to be positioned adjacent to the ear of a user wearing the wearable device (900). For example, the speaker (955) may include a second speaker (955-2) positioned within the first temple (904) and thus adjacent to the user's left ear, and a first speaker (955-1) positioned within the second temple (905) and thus adjacent to the user's right ear.

[0208] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state or emit light with an action corresponding to a specific state in order to visually provide information regarding a specific state of the wearable device (900) to the user. For example, when the wearable device (900) requires charging, it may emit red light at a regular cycle. In one embodiment, the light-emitting module may be disposed on the first rim (901) and / or the second rim (902).

[0209] Referring to FIG. 9B, according to one embodiment, a wearable device (900) may include a printed circuit board (PCB) (990). The PCB (990) may be included in at least one of the first temple (904) or the second temple (905). The PCB (990) may include an interposer positioned between at least two sub-PCBs. One or more hardwares included in the wearable device (900) (e.g., hardwares illustrated by different blocks in FIG. 2A and / or FIG. 2B) may be positioned on the PCB (990). The wearable device (900) may include a flexible PCB (FPCB) for interconnecting the hardwares.

[0210] According to one embodiment, a wearable device (900) may include at least one of a gyro sensor, a gravity sensor, and / or an acceleration sensor for detecting a posture of the wearable device (900) and / or a posture of a body part (e.g., a head) of a user wearing the wearable device (900). Each of the gravity sensor and the acceleration sensor may measure gravitational acceleration and / or acceleration based on mutually perpendicular designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). The gyro sensor may measure an angular velocity of each of the designated three-dimensional axes (e.g., the x-axis, the y-axis, and the z-axis). At least one of the gravity sensor, the acceleration sensor, and the gyro sensor may be referred to as an inertial measurement unit (IMU). According to one embodiment, the wearable device (900) may identify a user's motion and / or gesture performed to execute or terminate a specific function of the wearable device (900) based on the IMU.

[0211] FIGS. 10A and 10B illustrate examples of the appearance of a wearable device according to various embodiments of the present disclosure.

[0212] The wearable device (1000) of FIGS. 10A and 10B may include at least a portion of the hardware of the wearable device (900) described with reference to FIGS. 9A and / or 9B. According to one embodiment, an example of the appearance of a first side (1010) of a housing of the wearable device (1000) is illustrated in FIG. 10A, and an example of the appearance of a second side (1020) opposite to the first side (1010) is illustrated in FIG. 10B.

[0213] Referring to FIG. 10A, according to one embodiment, a first surface (1010) of a wearable device (1000) may have a form attachable to a body part of a user (e.g., the face of the user). Although not shown, the wearable device (1000) may further include a strap for fixing to a body part of a user, and / or one or more temples (e.g., the first temple (904) and / or the second temple (905) of FIGS. 9A and 9B). A first display (950-1) for outputting an image to a left eye among the user's two eyes, and a second display (950-2) for outputting an image to a right eye among the user's two eyes may be disposed on the first surface (1010). The wearable device (1000) is formed on the first surface (1010) and may further include a rubber or silicone packing to prevent interference by light (e.g., ambient light) different from the light emitted from the first display (950-1) and the second display (950-2).

[0214] According to one embodiment, a wearable device (1000) may include cameras (960-1) for photographing and / or tracking both eyes of a user adjacent to each of the first display (950-1) and the second display (950-2). The cameras (960-1) may be referred to as the gaze tracking camera (960-1) of FIG. 9B. According to one embodiment, a wearable device (1000) may include cameras (960-5, 960-6) for photographing and / or recognizing a face of a user. The cameras (960-5, 960-6) may be referred to as FT cameras. The wearable device (1000) may control an avatar representing the user in a virtual space based on the motion of the user's face identified using the cameras (960-5, 960-6). For example, the wearable device (1000) may change the texture and / or shape of a portion of an avatar (e.g., a portion of an avatar representing a human face) using information obtained by cameras (960-5, 960-6) (e.g., FT cameras) and representing a facial expression of a user wearing the wearable device (1000).

[0215] Referring to FIG. 10B, a camera (e.g., cameras (960-7, 960-8, 960-9, 960-10, 960-11, 960-12)) and / or a sensor (e.g., a depth sensor (1030)) for obtaining information related to the external environment of the wearable device (1000) may be disposed on a second surface (1020) opposite to the first surface (1010) of FIG. 10A. For example, the cameras (960-7, 960-8, 960-9, 960-10) may be disposed on the second surface (1020) for recognizing external objects. The cameras (960-7, 960-8, 960-9, 960-10) of FIG. 10b can correspond to the motion recognition cameras (960-2, 960-3) of FIG. 9b.

[0216] For example, using cameras (960-11, 960-12), the wearable device (1000) can obtain images and / or videos to be transmitted to each of the user's eyes. The camera (960-11) can be placed on the second face (1020) of the wearable device (1000) to obtain an image to be displayed through the second display (950-2) corresponding to the right eye among the two eyes. The camera (960-12) can be placed on the second face (1020) of the wearable device (1000) to obtain an image to be displayed through the first display (950-1) corresponding to the left eye among the two eyes. The cameras (960-11, 960-12) can correspond to the shooting camera (960-4) of FIG. 9B.

[0217] According to one embodiment, the wearable device (1000) may include a depth sensor (1030) disposed on the second face (1020) to identify a distance between the wearable device (1000) and an external object. Using the depth sensor (1030), the wearable device (1000) may obtain spatial information (e.g., a depth map) for at least a portion of the FoV of a user wearing the wearable device (1000). Although not illustrated, a microphone may be disposed on the second face (1020) of the wearable device (1000) to obtain a sound output from an external object. The number of microphones may be one or more, depending on the embodiment.

[0218] The Metaverse is a portmanteau of the English word "meta," meaning "virtual" or "transcendence," and "universe," and refers to a three-dimensional virtual world where social, economic, and cultural activities, like those in the real world, take place. The Metaverse is a concept that is one step more advanced than virtual reality (VR, a cutting-edge technology that allows people to have realistic experiences in a computer-generated virtual world), and has the characteristic of allowing people to engage in social and cultural activities similar to those in the real world, rather than simply enjoying games or virtual reality using avatars. The Metaverse service can provide media content to enhance immersion in the virtual world based on augmented reality (AR), virtual reality environment (VR), mixed reality (MR), and / or extended reality (XR).

[0219] For example, media content provided by a metaverse service may include social interaction content, including avatar-based games, concerts, parties, and / or conferences. For example, the media content may include advertisements, user-created content, and / or information for economic activities such as the sale and / or shopping of products. Ownership of the user-created content may be proven by a blockchain-based non-fungible token (NFT). The metaverse service may support economic activities based on real-world currency and / or cryptocurrency. The metaverse service may provide virtual content linked to the real world, such as digital twins or life logging.

[0220] FIG. 11 is a diagram of a network environment (1101) that provides a metaverse service through a server (1110) according to one embodiment of the present disclosure.

[0221] Referring to FIG. 11, a network environment (1101) may include a server (1110), a user terminal (1120) (e.g., a first terminal (1120-1) and a second terminal (1120-2)), and a network connecting the server (1110) and the user terminal (1120). Within the network environment (1101), the server (1110) may provide a metaverse service to the user terminal (1120). The network may be formed by at least one intermediate node (1130) including an access point (AP) and / or a base station. The user terminal (1120) may connect to the server (1110) through the network and output a user interface (UI) related to the metaverse service to the user of the user terminal (1120). Based on the above UI, the user terminal (1120) can obtain information to be input into the metaverse service from the user, or output information related to the metaverse service (e.g., multimedia content) to the user.

[0222] At this time, the server (1110) provides a virtual space so that the user terminal (1120) can be active in the virtual space. In addition, the user terminal (1120) installs a software agent for accessing the virtual space provided by the server (1110) to express information provided by the server (1110) to the user or transmit information that the user wishes to express in the virtual space to the server. The software agent may be provided directly through the server (1110), downloaded from a public server, or embedded and provided when the terminal is purchased.

[0223] In one embodiment, the metaverse service may be provided to a user terminal (1120) and / or a user using a server (1110). The embodiment is not limited thereto, and the metaverse service may be provided through individual contact between users. For example, within a network environment (1101), the metaverse service may be provided independently from the server (1110) through a direct connection between a first terminal (1120-1) and a second terminal (1120-2). Referring to FIG. 11, within the network environment (1101), the first terminal (1120-1) and the second terminal (1120-2) may be connected to each other through a network formed by at least one intermediate node (1130). In one embodiment where the first terminal (1120-1) and the second terminal (1120-2) are directly connected, either of the user terminals of the first terminal (1120-1) and the second terminal (1120-2) may perform the role of the server (1110). For example, a metaverse environment may be configured solely through device-to-device connections (e.g., peer-to-peer (P2P) connections).

[0224] In one embodiment, the user terminal (1120) (or the user terminal (1120) including the first terminal (1120-1) and the second terminal (1120-2)) may be made in various form factors and is characterized by including an output device for providing images and / or sounds to the user and an input device for inputting information into a metaverse service. Examples of various form factors of the user terminal (1120) include a smartphone (e.g., the second terminal (1120-2)), an AR device (e.g., the first terminal (1120-1)), a VR device, an MR device, a VST (Video See Through) device, an OST (Optical See Through) device, a smart lens, a smart mirror, an input / output capable TV or projector.

[0225] The network (e.g., a network formed by at least one intermediate node (1130)) includes various broadband networks including 3G (third generation), 4G (fourth generation), and 5G (fifth generation), and short-range networks including Wifi and BT (e.g., a wired network or a wireless network directly connecting the first terminal (1120-1) and the second terminal (1120-2)). The user terminal (1120) of FIG. 11 may include the HMD device (101) described with reference to FIG. 1, FIG. 2A, FIG. 2B, FIG. 3 to FIG. 5, FIG. 6A, FIG. 6B, FIG. 7A, and / or FIG. 7B.

[0226] In one embodiment, a method may be required to display a screen that includes an image of a body part (e.g., a user's face) covered by a wearable device, such as an HMD device. According to one embodiment, a head-mounted display (HMD) device (e.g., the HMD device (101) of FIGS. 1, 2A, 2B, 3-5, 6A, 6B, 7A, and / or 7B, the first HMD device (801) and / or the second HMD device (802) of FIG. 8) as described above may include a camera (e.g., the camera (225) of FIGS. 2A and / or 2B), a communication circuit (e.g., the communication circuit (235) of FIGS. 2A and / or 2B), at least one display (e.g., the display (220) of FIGS. 2A and / or 2B), a memory storing one or more computer programs, and one or more processors (e.g., the processor (210) of FIGS. 2A and / or 2B). The one or more computer programs may include computer-executable instructions that, when executed individually or collectively by the one or more processors, cause the HMD device to control the camera to acquire a first image external to the HMD device (e.g., image (410) of FIGS. 4, 5, 6A, and 6B). The one or more computer programs may include computer-executable instructions that, when executed individually or collectively by the one or more processors, cause the HMD device to detect a portion of the first image (e.g., portion (142) of FIG. 1) corresponding to an external HMD device (e.g., external HMD device (120) of FIGS. 1, 2A, 2B, 3-5, 6A, 6B, 7A, and / or 7B).The one or more computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the HMD device to receive, from the external HMD device via the communication circuitry, information about a body part covered by the external HMD device (e.g., information (150) of FIG. 1 ). The one or more computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the HMD device to perform rendering of the body part based on the information to obtain a second image representing the body part. The one or more computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the HMD device to display, via the at least one display, a composite image including the first image and the second image, the second image being positioned on the portion of the first image. According to one embodiment, the HMD device may display images and / or videos representing the appearance of a body part covered by the external HMD device.

[0227] For example, the one or more computer programs may include computer-executable instructions that, when executed individually or collectively by the one or more processors, cause the HMD device to superimpose the second image on the portion of the first image to generate the composite image.

[0228] For example, the one or more computer programs may include computer-executable instructions that, when executed individually or collectively by the one or more processors, cause the HMD device to detect an interaction between the HMD device and the external HMD device based on whether the direction of the external HMD device, determined using the first image, is toward the HMD device.

[0229] For example, the one or more computer programs may include computer-executable instructions that, when executed individually or collectively by the one or more processors, cause the HMD device to detect an interaction between the HMD device and the external HMD device based on a communication link established between the HMD device and the external HMD device.

[0230] For example, the one or more computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the HMD device to perform rendering based on a direction of gaze indicated by the information to obtain the second image representing the eyes of the user wearing the external HMD device. The information about the body part may indicate a direction of the eyes of the user wearing the external HMD device.

[0231] For example, the one or more computer programs may include computer-executable instructions that, when executed individually or collectively by the one or more processors, cause the HMD device to input the information and the first image into a neural network for rendering to obtain the second image.

[0232] For example, the one or more computer programs may include computer-executable instructions that, when executed individually or collectively by the one or more processors, cause the HMD device to modify a portion of a three-dimensional mesh corresponding to the body part within the three-dimensional mesh received from the external HMD device via the communication circuitry, based on the information.

[0233] For example, the one or more computer programs may include computer-executable instructions that, when executed individually or collectively by the one or more processors, cause the HMD device to determine a size of the second image based on at least one of a distance from the external HMD device or an orientation of the external HMD device relative to the HMD device.

[0234] For example, the one or more computer programs may include computer-executable instructions that, when executed individually or collectively by the one or more processors, cause the HMD device to control the camera using a cycle based on at least one of a distance from the external HMD device or an orientation of the external HMD device relative to the HMD device, thereby acquiring the first image.

[0235] For example, the one or more computer programs may include computer-executable instructions that, when executed individually or collectively by the one or more processors, cause the HMD device to change a frequency at which it receives the information from the external HMD device based on at least one of a distance from the external HMD device or an orientation of the external HMD device relative to the HMD device.

[0236] For example, the HMD device may include a second camera communicatively coupled to the one or more processors. The information about the body part may be first information. The one or more computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the HMD device to obtain second information about the body part of a user wearing the HMD device (e.g., a direction of a gaze of the user wearing the HMD device) based on a third image of the second camera. The one or more computer programs may include computer-executable instructions that, when individually or collectively executed by the one or more processors, cause the HMD device to transmit the second information to the external HMD device via the communication circuit.

[0237] According to one embodiment, a method is provided, which is performed by a head-mounted display (HMD) device including a camera, a communication circuit, at least one display, and one or more processors as described above. The method may include an operation of controlling the camera to obtain a first image of an exterior of the HMD device. The method may include an operation of detecting a portion of the first image corresponding to the external HMD device. The method may include an operation of receiving, from the external HMD device via the communication circuit, information about a body part covered by the external HMD device. The method may include an operation of performing rendering on the body part based on the information to obtain a second image representing the body part. The method may include an operation of displaying, via the at least one display, a composite image of the first image and the second image, wherein the second image may be positioned over the portion of the first image.

[0238] For example, the method may include an operation of superimposing the second image on the portion of the first image to generate the composite image.

[0239] For example, the action of controlling the camera may include an action of detecting an interaction between the HMD device and the external HMD device based on whether the direction of the external HMD device, determined using the first image, is toward the HMD device.

[0240] For example, the action of controlling the camera may include an action of detecting an interaction between the HMD device and the external HMD device based on a communication link established between the HMD device and the external HMD device.

[0241] For example, the operation of obtaining the second image may include an operation of obtaining the second image representing the eyes of the user wearing the external HMD device by performing rendering based on the direction of the gaze indicated by the information. The information about the body part may indicate the direction of the eyes of the user wearing the external HMD device.

[0242] For example, the operation of obtaining the second image may include an operation of obtaining the second image by inputting the information and the first image into a neural network for the rendering.

[0243] For example, the operation of acquiring the second image may include an operation of changing a portion of the three-dimensional mesh corresponding to the body part, based on the information, within the three-dimensional mesh received from the external HMD device through the communication circuit.

[0244] For example, the act of acquiring the second image may include an act of determining a size of the second image based on at least one of a distance from the external HMD device or an orientation of the external HMD device with respect to the HMD device.

[0245] For example, the controlling operation may include an operation of controlling the camera to acquire the first image using a cycle based on at least one of a distance from the external HMD device or an orientation of the external HMD device with respect to the HMD device.

[0246] For example, the receiving operation may include an operation of changing a period for receiving the information from the external HMD device based on at least one of a distance from the external HMD device or an orientation of the external HMD device with respect to the HMD device.

[0247] For example, the method may include an operation of obtaining second information about a body part of a user wearing the HMD device (e.g., a direction of a gaze of the user wearing the HMD device) based on a third image of a second camera of the HMD device. The method may include an operation of transmitting the second information to the external HMD device via the communication circuit.

[0248] According to one embodiment, as described above, an electronic device may include a camera, a communication circuit, at least one display, and a processor. The processor may be configured to detect a portion corresponding to a head-mounted device (HMD) device within a video acquired through the camera. The processor may be configured to acquire a virtual object representing a body part covered by the HMD device based on information received from the HMD device through the communication circuit. The processor may be configured to display at least a portion of the virtual object on at least a portion of the at least one display on which the detected portion is displayed, while displaying the video on the at least one display.

[0249] For example, the processor may be configured to display at least a portion of the virtual object based on detecting an interaction between the electronic device and the HMD device while displaying the video on the at least one display based on video-see through (VST).

[0250] For example, the processor may be configured to display, on the at least one display, at least a portion of the virtual object representing a head wearing the HMD device, in which eyes having a direction indicated by the information are positioned.

[0251] According to one embodiment of the present invention, a method of an electronic device including a camera, a communication circuit, at least one display, and a processor may include an operation of detecting a portion corresponding to a head-mounted display (HMD) device within a video acquired through the camera. The method may include an operation of acquiring a virtual object representing a body part covered by the HMD device based on information received from the HMD device via the communication circuit. The method may include an operation of displaying at least a portion of the virtual object on at least a portion of the at least one display on which the detected portion is displayed, while displaying the video on the at least one display.

[0252] For example, the displaying action may include displaying at least a portion of the virtual object based on detecting an interaction between the electronic device and the HMD device while displaying the video on the at least one display based on a VST.

[0253] For example, the displaying action may include an action of displaying, on at least one display, at least a portion of the virtual object representing a head wearing the HMD device, in which eyes having a direction indicated by the information are arranged.

[0254] According to one embodiment, a non-transitory computer-readable storage medium as described above may include instructions. The instructions, when executed by a processor of an electronic device, may cause the electronic device to perform operations of the electronic device and / or HMD device as described above.

[0255] The devices and components described in this disclosure may be implemented using one or more general-purpose computers or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used singly; however, those skilled in the art will appreciate that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors, or one processor and one controller. Other processing configurations, such as parallel processors, are also possible.

[0256] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may independently or collectively command the processing device. The software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0257] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may be one that continuously stores a computer-executable program or one that temporarily stores it for execution or download. In addition, the medium may be various recording or storage means in the form of a single or multiple hardware combinations, and is not limited to a medium directly connected to a computer system, but may also be distributed over a network. Examples of the medium may include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs (compact disc read only memory) and DVDs (digital versatile discs), magneto-optical media such as floptical disks, and those configured to store program commands, including ROMs, RAMs, and flash memory. In addition, examples of other media may include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0258] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0259] While the present disclosure has been illustrated and described with reference to various embodiments, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. In a head-mounted display (HMD) device, camera; communication circuit; At least one display; a memory storing one or more computer programs; and comprising one or more processors communicatively coupled with the camera, the communication circuit, the at least one display, and the memory; The one or more computer programs comprise computer-executable instructions, which, when individually or collectively executed by the one or more processors, cause the HMD device to: To obtain a first image outside of the HMD device, the camera is controlled, Detecting a portion of the first image corresponding to an external HMD device, Receive information about a body part covered by the external HMD device from the external HMD device through the communication circuit, Based on the above information, rendering is performed on the body part to obtain a second image representing the body part, and Causing to display a composite image including the second image of the first image through the at least one display, wherein the second image is positioned on the portion of the first image. HMD device.

2. In claim 1, the one or more computer programs, when individually or collectively executed by the one or more processors, the HMD device, Further comprising computer executable instructions causing the second image to be superimposed on the portion of the first image to generate the composite image. HMD device.

3. In claim 1, the instructions, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the HMD device to: Further comprising computer executable instructions causing the computer to detect an interaction between the HMD device and the external HMD device based on whether the direction of the external HMD device, determined using the first image, is directed toward the HMD device. HMD device.

4. In claim 1, the instructions, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the HMD device to: Further comprising computer executable instructions causing the HMD device and the external HMD device to detect an interaction between the HMD device and the external HMD device based on a communication link established between the HMD device and the external HMD device. HMD device.

5. In claim 1, the instructions, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the HMD device to: Further comprising computer-executable instructions that cause rendering to be performed based on the direction of gaze indicated by said information, thereby obtaining said second image representing the eyes of a user wearing said external HMD device; and The information about the said body part indicates the direction of the eyes of the user wearing the said external HMD device. HMD device.

6. In claim 1, the instructions, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the HMD device to: Further comprising computer executable instructions causing the neural network for rendering to input the information and the first image to obtain the second image. HMD device.

7. In claim 1, the instructions, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the HMD device to: Further comprising computer executable instructions causing a portion of the 3D mesh corresponding to the body part to be changed based on the information received from the external HMD device through the communication circuit. HMD device.

8. In claim 1, the instructions, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the HMD device to: Further comprising computer-executable instructions causing the computer to determine a size of the second image based on at least one of a distance from the external HMD device or an orientation of the external HMD device relative to the HMD device. HMD device.

9. In claim 1, the instructions, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the HMD device to: Further comprising computer executable instructions causing the camera to acquire the first image by controlling the camera using a period based on at least one of a distance from the external HMD device or an orientation of the external HMD device relative to the HMD device. HMD device.

10. In claim 1, the instructions, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the HMD device to: Further comprising computer executable instructions causing the computer to change the period of receiving the information from the external HMD device based on at least one of a distance from the external HMD device or an orientation of the external HMD device relative to the HMD device. HMD device.

11. In claim 1, further comprising a second camera communicatively coupled with said one or more processors, The above information about the above body part is first information, and The above instructions, the one or more computer programs, when individually or collectively executed by the one or more processors, cause the HMD device to: Based on the third image of the second camera, second information about a body part of a user wearing the HMD device is acquired; and Further comprising computer executable instructions causing the second information to be transmitted to the external HMD device via the communication circuit. HMD device.

12. A method performed by a head-mounted display (HMD) device including a camera, communication circuitry, at least one display, and one or more processors, An operation of controlling the camera to obtain a first image of the exterior of the HMD device by the HMD device; An operation of detecting a portion of the first image corresponding to an external HMD device by the HMD device; An operation of receiving, by the HMD device, information about a body part covered by the external HMD device from the external HMD device through the communication circuit; An operation of obtaining a second image representing the body part by performing rendering on the body part based on the information by the HMD device; and An operation of displaying a composite image including the first image and the second image through the at least one display by the HMD device, wherein the second image is positioned on the portion of the first image. method.

13. In claim 12, Further comprising an action of generating the composite image by superimposing the second image on the portion of the first image. method.

14. In claim 12, the operation of controlling the camera comprises: An operation of detecting an interaction between the HMD device and the external HMD device based on whether the direction of the external HMD device, determined using the first image, is directed toward the HMD device, method.

15. In claim 12, the operation of controlling the camera comprises: An operation for detecting interaction between the HMD device and the external HMD device based on a communication link established between the HMD device and the external HMD device, method.

Images