Electronic device for synthesizing images obtained from cameras and method thereof

Abstract

According to one embodiment, a processor of a head-mounted display (HMD) device can control a camera to obtain a first image of an outside of the HMD device. The processor can detect a portion of the first image corresponding to an outside HMD device. The processor can receive information about a body part covered by the outside HMD device from the outside HMD device via a communication circuit. The processor can obtain a second image representing the body part by performing rendering on the body part based on the information. The processor can display the second image located in the portion of the first image and a composite image including the first image through at least one display.

Description

Technical Field

[0001] This disclosure relates to an electronic device (e.g., a head-mounted display (HMD) device) for synthesizing images obtained from a camera and a method thereof. Background Technology

[0002] To provide an enhanced user experience, an electronic device is being developed that displays computer-generated information in association with external objects in the real world, providing an augmented reality (AR) service. The electronic device can be a wearable device that can be worn by the user. For example, the electronic device can be AR glasses and / or a head-mounted display (HMD) device.

[0003] The above information is provided for background information purposes only to aid in understanding this disclosure. No determination or assertion is made regarding whether any of the above content can be used as prior art with respect to this disclosure. Summary of the Invention

[0004] Technical solution

[0005] The aspects of this disclosure will at least address the aforementioned problems and / or disadvantages, and provide at least the following advantages. Therefore, one aspect of this disclosure is to provide an electronic device (e.g., a head-mounted display (HMD) device) and a method thereof for synthesizing images obtained from a camera.

[0006] Other aspects will be set forth in part in the description which follows, and will be apparent in part from the description, or may be learned by practicing the rendered embodiments.

[0007] According to one aspect of this disclosure, a head-mounted display (HMD) device is provided. The HMD device may include a camera, communication circuitry, at least one display, a memory storing one or more computer programs, and one or more processors communicatively coupled to the camera, communication circuitry, at least one display, and memory. The one or more computer programs include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the HMD device to control the camera to acquire a first image outside a forward direction of the HMD device. The one or more computer programs may include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the HMD device to detect a portion of the first image corresponding to an external HMD device. The one or more computer programs may include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the HMD device to receive information about a body part covered by the external HMD device via communication circuitry. The one or more computer programs may include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the HMD device to obtain a second image representing a body part by performing rendering relative to the body part based on the information. One or more computer programs may include computer-executable instructions that, when executed individually or jointly by one or more processors, cause an HMD device to display a composite image comprising a first image and a second image via at least one display, wherein the second image is located at that portion of the first image.

[0008] According to another aspect of this disclosure, a method is provided performed by a head-mounted display (HMD) device including a camera, communication circuitry, at least one display, and one or more processors. The method may include controlling the camera by the HMD device to obtain a first image of an area outside the HMD device. The method may include detecting a portion of the first image corresponding to the external HMD device by the HMD device. The method may include receiving information about a body part covered by the external HMD device via the communication circuitry by the HMD device. The method may include obtaining a second image representing the body part by performing rendering relative to the body part based on the information by the HMD device. The method may include displaying a composite image including the first image and the second image via at least one display by the HMD device, wherein the second image is located on the portion of the first image.

[0009] According to another aspect of this disclosure, an electronic device is provided. The electronic device may include a camera, communication circuitry, at least one display, a memory storing one or more computer programs, and one or more processors communicatively coupled to the camera, communication circuitry, at least one display, and memory. The one or more computer programs may include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to detect portions corresponding to a head-mounted display (HMD) device in video acquired by the camera. The one or more computer programs may include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to obtain, based on information received from the HMD device via the communication circuitry, a virtual object representing a body part covered by the HMD device. The one or more computer programs may include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to display at least a portion of the virtual object on at least a portion of the portion detected on at least one display while simultaneously displaying video on at least one display.

[0010] According to another aspect of this disclosure, a method is provided for an electronic device, the electronic device including a camera, communication circuitry, at least one display, and one or more processors. The method may include detecting a portion corresponding to a head-mounted display (HMD) device in video acquired by the camera. The method may include obtaining a virtual object representing a body part covered by the HMD device based on information received from the HMD device via the communication circuitry. The method may include displaying at least a portion of the virtual object on at least a portion of the portion detected by the display on at least one display while the video is displayed on at least one display.

[0011] According to another aspect of this disclosure, one or more non-transitory computer-readable storage media are provided, storing one or more computer programs including computer-executable instructions that, when executed individually or jointly by one or more processors of a head-mounted display (HMD) device, cause the HMD device to perform operations. The operations may include: controlling a camera of the HMD device to obtain a first image of an area outside the HMD device; detecting a portion of the first image corresponding to the external HMD device; receiving information about a body part covered by the external HMD device from the external HMD device via communication circuitry of the HMD device; obtaining a second image representing the body part by performing rendering of the body part based on the information; and displaying a composite image including the first image and the second image via at least one display of the HMD device, wherein the second image is located on the portion of the first image.

[0012] Other aspects, advantages, and salient features of this disclosure will become apparent to those skilled in the art from the following detailed description of various embodiments disclosed in conjunction with the accompanying drawings. Attached Figure Description

[0013] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0014] Figure 1 An example of a screen displayed by a head-mounted display (HMD) device according to an embodiment of the present disclosure is shown;

[0015] Figure 2a and Figure 2b Examples showing block diagrams of HMD devices according to various embodiments of the present disclosure;

[0016] Figure 3 An example flowchart illustrating an HMD device according to an embodiment of the present disclosure;

[0017] Figure 4 Examples of operation of an HMD device based on images and / or videos obtained from a camera, according to embodiments of the present disclosure, are shown.

[0018] Figure 5 An example of the operation of an HMD device that processes information received from an external HMD device according to an embodiment of the present disclosure is shown;

[0019] Figure 6a and Figure 6b Examples illustrating the operation of an HMD device that processes information received from an external HMD device according to various embodiments of the present disclosure;

[0020] Figure 7a and Figure 7b Examples of operation of an HMD device according to various embodiments of the present disclosure are shown;

[0021] Figure 8 A signal flow diagram between HMD devices according to an embodiment of the present disclosure is shown;

[0022] Figure 9a An example perspective view of a wearable device according to an embodiment of the present disclosure;

[0023] Figure 9b Examples of one or more hardware components arranged in a wearable device according to embodiments of the present disclosure are shown;

[0024] Figure 10a and Figure 10b Examples of the exterior of a wearable device according to various embodiments of the present disclosure are shown; and

[0025] Figure 11 This is a diagram of a network environment related to metaverse services according to embodiments of this disclosure.

[0026] Throughout the accompanying drawings, it should be noted that the same reference numerals are used to depict the same or similar elements, features, and structures. Detailed Implementation

[0027] The following description with reference to the accompanying drawings is intended to aid in a full understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. It includes various specific details to aid understanding, but these are considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope of this disclosure. Additionally, for clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0028] The terms and words used in the following description and claims are not limited to their literal meaning, but are used by the inventors only to enable a clear and consistent understanding of this disclosure. Therefore, those skilled in the art should understand that the following description of various embodiments of this disclosure is for illustrative purposes only and is not intended to limit the scope of this disclosure as defined by the appended claims and their equivalents.

[0029] It should be understood that, unless the context clearly specifies otherwise, the singular forms “a,” “an,” and “the” include plural indicators. Thus, for example, a reference to “component surface” includes a reference to one or more such surfaces.

[0030] The various embodiments and terminology used herein are not intended to limit the technology described herein to the specific embodiments, and should be understood to include various modifications, equivalents, or alternatives to the corresponding embodiments. Regarding the description of the drawings, reference numerals may be used for similar parts. In the documentation, expressions such as "A or B," "at least one of A and / or B," "A, B, or C," or "at least one of A, B, and / or C" may include all possible combinations of items listed together. Expressions such as "first," "second," "first," or "second" may modify the corresponding components, regardless of order or importance, and are used only to distinguish one component from another, but do not limit the corresponding components. When (e.g., a first) component is referred to as "connected (functionally or communicatively)" or "accessed" to another (e.g., a second) component, that component may be directly connected to the other component, or may be connected via another component (e.g., a third component).

[0031] For example, the term "module" as used in this document can include a unit configured with hardware, software, or firmware, and can be used interchangeably with terms such as logic, logic block, component, or circuit. A module can be a component or the smallest unit or part of a whole configuration that performs one or more functions. For example, a module can be configured with an application-specific integrated circuit (ASIC).

[0032] It should be understood that the boxes in each flowchart and the combination of flowcharts can be executed by one or more computer programs including instructions. The entirety of one or more computer programs can be stored in a single memory device, or one or more computer programs can be divided into different parts stored in multiple different memory devices.

[0033] Any function or operation described herein can be processed by a processor or a combination of processors. A processor or a combination of processors is a circuit that performs processing and includes, for example, an application processor (AP, such as a central processing unit (CPU)), a communication processor (CP, such as a modem), a graphics processing unit (GPU), a neural processing unit (NPU) (e.g., an artificial intelligence (AI) chip), a Wi-Fi chip, Bluetooth, etc. ® Chips, Global Positioning System (GPS) chips, Near Field Communication (NFC) chips, connectivity chips, sensor controllers, touch controllers, fingerprint sensor controllers, display driver integrated circuits (ICs), CODEC chips, Universal Serial Bus (USB) controllers, camera controllers, image processing ICs, microprocessor units (MPUs), system-on-a-chip (SoCs), integrated circuits (ICs), etc.

[0034] Figure 1An example of a screen 130 displayed by an electronic device (e.g., a head-mounted display (HMD) device) 101 according to an embodiment of the present disclosure is shown.

[0035] Electronic device 101 may include a head-mounted display (HMD) that can be worn on the head of user 110. Electronic device 101 may be referred to as a wearable device, head-mounted device (HMD) (or HMD device), head-mounted electronics, eyeglasses-type (or goggle-type) electronics, video see-through or visible see-through (VST) device, extended reality (XR) device, virtual reality (VR) device, and / or augmented reality (AR) device. Although the appearance of electronic device 101 in the form of eyeglasses is shown, the embodiments are not limited thereto. Reference will be made to... Figure 2a and Figure 2b An example of the hardware configuration included in electronic device 101 is described exemplarily. (Refer to...) Figure 8 A, 8B, 9A and / or 9B describe structural examples of an electronic device 101 that can be worn on the head of user 110. The electronic device 101 may be referred to as an electronic device. For example, the electronic device may include accessories (e.g., a strap) for attaching to the head of user 110.

[0036] According to embodiments, electronic device 101 can perform functions related to augmented reality (AR) and / or mixed reality (MR). For example, when user 110 wears electronic device 101, electronic device 101 may include at least one lens arranged adjacent to the user's eyes. Electronic device 101 can combine light emitted from the display of electronic device 101 with ambient light passing through the lens. The display area of ​​the display may be formed in the lens through which the ambient light passes. Because electronic device 101 combines ambient light and light emitted from the display, user 110 can see an image in which real objects identified by ambient light and virtual objects formed by light emitted from the display are mixed. The aforementioned augmented reality, mixed reality, and / or virtual reality may be referred to as extended reality (XR).

[0037] According to embodiments, electronic device 101 can perform functions related to video perspective or visible perspective (VST) and / or virtual reality (VR). For example, when user 110 is wearing electronic device 101, electronic device 101 may include a housing covering user 110's eyes. Electronic device 101 may include a display disposed on a first surface of the housing facing the eyes in this state. Electronic device 101 may include a camera disposed on a second surface opposite the first surface. Using the camera, electronic device 101 can acquire images and / or video representing ambient light. Electronic device 101 can output images and / or video to the display disposed on the first surface, allowing user 110 to recognize ambient light through the display. The display area or display region (or active area or active region) of the display disposed on the first surface may be formed by one or more pixels included in the display. Electronic device 101 can composite virtual objects with images and / or video output through the display, allowing user 110 to recognize virtual objects together with real objects recognized by ambient light.

[0038] Reference Figure 1 An example of a screen 130 displayed by an electronic device 101 is shown. The electronic device 101 can display images and / or videos of the external environment, including the direction in front of a user 110 wearing the electronic device 101, based on VST (or transmission). Using the screen 130 including images and / or videos, the electronic device 101 can provide a user experience, such as the user 110 wearing the electronic device 101 viewing the external environment (or such as not wearing the electronic device 101). In the following, VST and / or transmission are functions supported by the electronic device 101 and can be used to direct the user 110 wearing the electronic device 101 to display images and / or videos of the external environment on the screen 130.

[0039] Reference Figure 1 This illustrates an environment including a user 110 wearing an electronic device 101 and another user 125 wearing an external electronic device 120. For ease of illustration, user 110 is sometimes referred to as the first user 110, and the other user 125 as the second user 125. Similarly, electronic device 101 may be referred to as the first electronic device 101, and external electronic device 120 may be referred to as the second electronic device 120. Figure 1In such an environment, because electronic device 101 covers both eyes of the first user 110 and external electronic device 120 covers both eyes of the second user 125, the first user 110 may not be able to see a portion of the face including the eyes of the second user 125, and the second user 125 may not be able to see a portion of the face including the eyes of the first user 110. When the first user 110 and the second user 125 are talking to each other, nonverbal expressions, including facial expressions, may be blocked by electronic device 101 and / or external electronic device 120.

[0040] According to one embodiment, electronic device 101 can change at least a portion of the face of second user 125 covered by second electronic device 120 on screen 130 displayed to first user 110 to support the exchange of nonverbal expressions between first user 110 and second user 125. (See also...) Figure 1 In the images and / or videos displayed on the screen 130 based on the transmission display, the electronic device 101 can display a visual object 140 corresponding to the second user 125.

[0041] In an embodiment, electronic device 101 can couple virtual objects (e.g., 3D graphical objects), images, and / or videos representing body parts of a second user 125 covered by second electronic device 120 to a portion 142 of visual object 140 covered (or hidden) by second electronic device 120. The virtual objects, images, and / or videos coupled to portion 142 can be determined based on information 150 received from second electronic device 120. For example, information 150 can be related to the two eyes of the second user 125 detected by second electronic device 120. Information 150 can include images and / or videos of the two eyes of the second user 125. Information 150 can include parameters indicating the orientation of the two eyes of the second user 125.

[0042] Reference Figure 1In the screen 130 displayed by the electronic device 101, the first user 110 wearing the electronic device 101 can see the entire face of the second user 125 because the portion 142 corresponding to the second electronic device 120 is replaced by a virtual object, image, and / or video representing the face of the second user 125. Seeing the face of the second user 125, the first user 110 can recognize the nonverbal expressions (e.g., facial expressions) of the second user 125. The electronic device 101 can support interaction between the first user 110 and the second user 125 by using the screen 130, which includes the entire face of the second user 125. Similarly, the electronic device 101 can send information related to the facial expressions of the first user 110 wearing the electronic device 101 to the external electronic device 120. This information may include images, videos, and / or parameters related to the two eyes of the first user 110, similar to the information 150 transmitted from the external electronic device 120 to the electronic device 101.

[0043] As described above, according to the embodiment, electronic device 101 can provide a screen 130 related to the external environment to a first user 110 of wearable electronic device 101 based on MR, AR, and / or XR. To support interaction between users (e.g., first user 110 and / or second user 125) of wearable electronic devices (e.g., electronic device 101 and / or external electronic device 120), electronic device 101 can display screen 130 based on transmission. When a visual object 140 of the corresponding party (e.g., second user 125) is displayed on screen 130, electronic device 101 can perform partial rendering on the portion 142 of the visual object 140 covered by external electronic device 120. (Refer to...) Figure 2a and / or Figure 2b Describes the hardware configuration of the electronic device 101 used for partial rendering.

[0044] Based on partial rendering, electronic device 101 can at least partially display the face of the other party, which is covered (or hidden) by external electronic device 120, in portion 142, rather than representing a visual object of external electronic device 120. (Refer to...) Figures 3 to 5 , Figure 6a , Figure 6b , Figure 7a and / or Figure 7b Describes the operation of electronic device 101 displaying the face of another person covered by external electronic device 120. (Refer to...) Figure 8 Describes the operation of electronic device 101 receiving information 150 from external electronic device 120 and similarly sending other information to external electronic device 120.

[0045] The operation of an electronic device 101 providing a delivery and / or VST-based user experience is illustrated, but embodiments are not limited thereto. For example, in an embodiment of the electronic device 101 with an AR-based shape factor that transmits ambient light, the electronic device 101 may project light representing the face of a second user 125 wearing an external electronic device 120 onto a portion 142 of a screen 130 corresponding to the external electronic device 120, based on ambient light. A first user 110 viewing the projected light can see the face of the second user 125, such as when it is not covered by the external electronic device 120.

[0046] In the following text, reference will be made to Figure 2a and / or Figure 2b describe Figure 1 Hardware configuration of electronic device 101.

[0047] Figure 2a and Figure 2b Examples of block diagrams illustrating an electronic device 101 according to various embodiments of the present disclosure. Figure 2a and / or Figure 2b The electronic device 101 may include Figure 1 Electronic equipment 101.

[0048] Reference Figure 2a The electronic device 101 according to an embodiment may include at least one of a processor 210, a memory 215, a display 220, a camera 225, a sensor 230, and a communication circuit 235. The processor 210, memory 215, display 220, camera 225, sensor 230, and / or communication circuit 235 may be electrically coupled and / or operatively coupled to each other via electronic components such as a communication bus 202. The type and / or number of hardware components included in the electronic device 101 are not limited to... Figure 2a As shown. For example, electronic device 101 may include only Figure 2a Some of the hardware components shown.

[0049] The processor 210 of the electronic device 101 according to an embodiment may include hardware components for processing data based on instructions. The hardware components for processing data may include, for example, an arithmetic and logic unit (ALU), a field-programmable gate array (FPGA), a central processing unit (CPU), and / or an application processor (AP). In an embodiment, the electronic device 101 may include one or more processors. The processor 210 may have a multi-core processor architecture such as a dual-core, quad-core, hexa-core, and / or octa-core processor.

[0050] The memory 215 of the electronic device 101 according to the embodiment may include hardware components for storing data and / or instructions input to or output from the processor 210. The memory 215 may include, for example, volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM)). Volatile memory may include at least one of, for example, dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include at least one of, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, optical disk, and embedded multimedia card (eMMC). In one embodiment, the memory 215 may be referred to as a storage device.

[0051] In an embodiment, the display 220 of the electronic device 101 can display information to the user of the electronic device 101 (e.g., Figure 1 User 110) outputs visual information (e.g., Figure 1 130 screen Figure 5 The screen 540 Figure 6a and / or Figure 6b The display 220 may visualize information provided by a processor 210, which includes circuitry such as a graphics processing unit (GPU). The display 220 may include a flexible display, a flat panel display (FPD), and / or electronic paper. The display 220 may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or one or more light-emitting diodes (LEDs). LEDs may include organic LEDs (OLEDs). Embodiments are not limited thereto, and for example, where the electronic device 101 includes a lens for transmitting external light (or ambient light), the display 220 may include a projector (or projection assembly) for projecting light onto the lens. In embodiments, the display 220 may be referred to as a display panel and / or a display module.

[0052] In an embodiment, the camera 225 of the electronic device 101 may include an optical sensor (e.g., a charge-coupled device (CCD) sensor, a complementary metal-oxide-semiconductor (CMOS) sensor) that generates electrical signals indicating the color and / or brightness of light. The camera 225 may be referred to as an image sensor and may be included in... Figure 2a and Figure 2bThe sensor 230 includes multiple optical sensors in the camera 225, which can be arranged in a two-dimensional array. The camera 225 can generate two-dimensional frame data corresponding to the light arriving at the optical sensors in the two-dimensional array by substantially simultaneously acquiring electrical signals from each of the multiple optical sensors. For example, photographic data captured using the camera 225 can represent two-dimensional frame data obtained from the camera 225. Similarly, video data captured using the camera 225 can represent a sequence of two-dimensional frame data obtained from the camera 225 according to the frame rate. The camera 225 can be positioned in the direction in which it receives light and may also include a flash for outputting light in that direction.

[0053] According to an embodiment, electronic device 101 is an example of camera 225, and may include multiple cameras arranged in different directions. (See also...) Figure 2a and Figure 2b The camera 225 included in the electronic device 101 may include an eye-tracking camera 225-1 and / or an external camera 225-2. The eye-tracking camera 225-1 may be positioned facing at least one of the eyes of a user wearing the electronic device 101. For positioning facing each of the user's eyes, the electronic device 101 may include two eye-tracking cameras 225-1. The processor 210 can identify the direction of the user's gaze using images and / or video obtained from the eye-tracking cameras 225-1. The eye-tracking camera 225-1 may include an infrared (IR) sensor. The eye-tracking camera 225-1 may be referred to as an eye sensor, a gaze tracker, and / or an eye tracker.

[0054] Reference Figure 2a and Figure 2b The external camera 225-2 can be positioned facing in front of the user of the wearable electronic device 101 (e.g., in the direction the two eyes can be pointing). Embodiments are not limited thereto, and the external camera 225-2 can also be positioned facing external space. Using images and / or video obtained from the external camera 225-2, the processor 210 can identify external objects (e.g., Figure 1 The processor 210 can provide a VST-based and / or transmitted user experience by displaying images and / or videos obtained from the external camera 225-2 on the display 220.

[0055] According to an embodiment, the sensor 230 of the electronic device 101 can generate electrical information from non-electronic information associated with the electronic device 101, which can be processed and / or stored by the processor 210 and / or memory 215 of the electronic device 101. This information may be referred to as sensor data. The sensor 230 may include a Global Positioning System (GPS) sensor, an image sensor, an audio sensor (e.g., a microphone array including a microphone and / or multiple microphones), a grip sensor, a touch sensor (or a touch sensor panel (TSP)), a proximity sensor, an illuminance sensor, and / or a time-of-flight (ToF) sensor (or ToF camera) for detecting the geographical location of the electronic device 101.

[0056] In an embodiment, sensor 230 may include an inertial measurement unit (IMU) for detecting physical motion of electronic device 101. An accelerometer, a geomagnetic sensor, a gravity sensor, or any combination thereof may be referred to as an IMU. An accelerometer may output an electrical signal indicating gravitational acceleration and / or acceleration of each of multiple axes (e.g., x-axis, y-axis, and z-axis), said multiple axes being perpendicular to each other and based on a designated origin of electronic device 101. A gyroscope sensor may output an electrical signal indicating angular velocities (e.g., roll, pitch, and / or yaw) of multiple axes. A gyroscope sensor may be referred to as an angular velocity sensor. A geomagnetic sensor may output an electrical signal indicating the magnitude of a magnetic field formed in electronic device 101 along each of the multiple axes (e.g., x-axis, y-axis, and / or z-axis). For example, the accelerometer, gyroscope, and / or geomagnetic sensor may repeatedly output sensor data including the magnitude of acceleration, angular velocity, and / or magnetic field along multiple axes based on a specified period (e.g., 1 millisecond).

[0057] In an embodiment, the communication circuit 235 of the electronic device 101 may include a function to support communication between the electronic device 101 and external electronic devices (e.g., Figure 1 The communication circuit 235 is a circuit for transmitting and / or receiving signals between external electronic devices 120 and other devices. The communication circuit 235 may include at least one of, for example, a modem, an antenna, and an optical / electronic (O / E) converter. The communication circuit 235 may support the transmission and / or reception of electrical signals based on various types of protocols such as Ethernet and Local Area Network (LAN), Wide Area Network (WAN), WiFi, Bluetooth, Bluetooth Low Energy (BLE), ZigBee, Long Term Evolution (LTE), 5G New Radio (NR), 6G, and / or higher. In embodiments, the communication circuit 235 may be referred to as a communication processor and / or a communication module.

[0058] According to an embodiment, the memory 215 of the electronic device 101 may store data to be processed by the processor 210 of the electronic device 101, as well as one or more instructions (or commands) indicating the calculations and / or operations to be performed. The set of one or more instructions may be referred to as a program, firmware, operating system, process, routine, subroutine, and / or software application (hereinafter referred to as an application). For example, when executing a set of multiple instructions distributed in the form of an operating system, firmware, driver, program, and / or application, the electronic device 101 and / or processor 210 may perform... Figure 3 At least one of the operations. In the following, the fact that an application is installed in electronic device 101 may mean that one or more instructions provided in the form of an application are stored in memory 215, and that one or more applications are stored by processor 210 in an executable format (e.g., a file with an extension specified by the operating system of electronic device 101). For example, an application may include programs and / or libraries related to services provided to the user.

[0059] Reference Figure 2a and Figure 2b Depending on the objective, programs installed in electronic device 101 can be included in any of the different layers including application layer 240, frame layer 250, and / or hardware abstraction layer (HAL) 280. For example, programs (e.g., modules or drivers) designed for hardware of electronic device 101 (e.g., display 220, camera 225, sensor 230, and / or communication circuitry 235) can be included in hardware abstraction layer (HAL) 280. Frame layer 250 can be referred to as an XR frame layer for including one or more programs for providing extended reality (XR) services. For example, Figure 2a and Figure 2b The layers shown are logically (or for ease of interpretation) divided, and may not mean that the address space of memory 215 is divided into layers.

[0060] For example, a program designed for at least one of the Hardware Abstraction Layer (HAL) 280 and / or Application Layer 240 (e.g., position tracker 271, spatial recognizer 272, gesture tracker 273, gaze tracker 274, and / or face tracker 275) may be included in the Frame Layer 250. The program included in the Frame Layer 250 may provide an Application Programming Interface (API) that can be executed (or invoked) based on another program.

[0061] For example, application layer 240 may include programs designed for users of electronic device 101. An example of a program included in application layer 240 is an extended reality (XR) system user interface (UI) 241 and / or an XR application 242, but embodiments are not limited thereto. For example, a program included in application layer 240 (e.g., a software application) can perform functions supported by a program included in framework layer 250 by calling an application programming interface (API).

[0062] For example, electronic device 101 can display one or more visual objects on display 220 for interacting with the user, based on the execution of XR system UI 241. Visual objects can refer to objects that can be deployed on a screen for information transmission and / or interaction, such as text, images, icons, videos, buttons, checkboxes, radio buttons, text boxes, sliders, and / or tables. Visual objects can be referred to as visual guides, virtual objects, visual elements, UI elements, view objects, and / or view elements. Electronic device 101 can provide the user with functionalities available in a virtual space based on the execution of XR system UI 241.

[0063] Reference Figure 2a and Figure 2b Lightweight renderer 243 and / or XR plugin 244 are shown as included in XR system UI 241, but are not limited thereto. For example, based on XR system UI 241, processor 210 can execute lightweight renderer 243 and / or XR plugin 244 in frame layer 250.

[0064] For example, based on the execution of lightweight renderer 243, electronics 101 can obtain resources (e.g., APIs, system processes, and / or libraries) for defining, creating, and / or executing a rendering pipeline that allows for partial modification. In defining a rendering pipeline that allows for partial modification, lightweight renderer 243 can be referred to as a lightweight rendering pipeline. Lightweight renderer 243 may include renderers built before the execution of the software application (e.g., pre-built renderers). For example, electronics 101 can obtain resources (e.g., APIs, system processes, and / or libraries) for defining, creating, and / or executing the entire rendering pipeline based on the execution of XR plugin 244. In defining (or setting up) the entire rendering pipeline, XR plugin 244 can be referred to as an open XR native client.

[0065] For example, electronic device 101 can display a screen indicating at least a portion of the virtual space on display 220 based on the execution of XR application 242. XR plugin 244-1 included in XR application 242 may include instructions supporting functions similar to those of XR plugin 244 in XR system UI 241. In the description of XR plugin 244-1, descriptions overlapping with those of XR plugin 244 may be omitted. Electronic device 101 can execute virtual space manager 251 based on the execution of XR application 242.

[0066] According to an embodiment, electronic device 101 can provide virtual space services based on the execution of virtual space manager 251. For example, virtual space manager 251 may include a platform for supporting virtual space services. Based on the execution of virtual space manager 251, electronic device 101 can identify a virtual space based on a user location indicated by data obtained through sensor 230, and can display at least a portion of the virtual space on display 220. Virtual space manager 251 may be referred to as compositional rendering manager (CPM).

[0067] For example, the virtual space manager 251 may include a runtime service 252. For example, the runtime service 252 may be referred to as an OpenXR runtime module (or OpenXR runtime program). The electronic device 101 may, based on the execution of the runtime service 252, perform at least one of the following: user pose prediction functions, frame timing functions, and / or spatial input functions. For example, the electronic device 101 may, based on the execution of the runtime service 252, perform a service to render virtual space to the user. For example, based on the execution of the runtime service 252, functions related to virtual space that can be executed by the application layer 240 may be supported.

[0068] For example, the virtual space manager 251 may include a transfer manager 253. Based on the execution of the transfer manager 253, the electronic device 101 can display images and / or videos indicating the actual space obtained by the external camera 225-2 by overlaying them on at least a portion of the screen, while simultaneously displaying a screen indicating the virtual space (e.g., ...) on the display 220. Figure 1 Screen 130).

[0069] For example, the virtual space manager 251 may include an input manager 254. Electronic device 101 can identify data (e.g., sensor data) obtained by executing one or more programs included in the perception service layer 270, based on the execution of the input manager 254. Electronic device 101 can identify user input associated with electronic device 101 by using the acquired data. User input may be related to user movements (e.g., gestures), gaze, and / or speech identified by sensor 230 and / or camera 225 (e.g., external camera 225-2). User input may be identified based on an external electronic device (e.g., a remote control provided for controlling electronic device 101) connected (or paired) via communication circuitry 235.

[0070] For example, the Perception Abstraction Layer 260 can be used for data exchange between the Virtual Space Manager 251 and the Perception Service Layer 270. In terms of data exchange between the Virtual Space Manager 251 and the Perception Service Layer 270, the Perception Abstraction Layer 260 can be referred to as an interface. For example, the Perception Abstraction Layer 260 can be referred to as OpenPX. The Perception Abstraction Layer 260 can be used for both perception clients and perception services.

[0071] According to an embodiment, the perception service layer 270 may include one or more programs for processing data obtained from the sensor 230 and / or the camera 225. The one or more programs may include at least one of a position tracker 271, a spatial recognizer 272, a gesture tracker 273, a gaze tracker 274, and / or a face tracker 275. The type and / or number of programs included in the perception service layer 270 are not limited to... Figure 2a and Figure 2b As shown.

[0072] For example, electronic device 101 can identify its posture by using sensor 230 based on the execution of position tracker 271. Based on the execution of position tracker 271, electronic device 101 can identify its 6-DOF posture by using data obtained using external camera 225-2 and / or IMU (e.g., gyroscope sensor, accelerometer sensor, and / or magnetometer sensor). Position tracker 271 may be referred to as a head tracking (HeT) module (or head tracker, head tracking program).

[0073] For example, electronic device 101 can obtain information for providing a three-dimensional virtual space corresponding to the surrounding environment (e.g., external space) of electronic device 101 (or its user) based on the execution of spatial recognizer 272. Electronic device 101 can reconstruct its surrounding environment in three dimensions using data obtained from external camera 225-2 based on the execution of spatial recognizer 272. Electronic device 101 can identify at least one of planes, inclinations, and stairs based on the three-dimensionally reconstructed surrounding environment of electronic device 101 according to the execution of spatial recognizer 272. Spatial recognizer 272 may be referred to as a scene understanding (SU) module (or scene understanding (SU) program).

[0074] For example, electronic device 101 can recognize (or identify) the hand gestures and / or hand postures of its user based on the execution of gesture tracker 273. For example, electronic device 101 can recognize the hand gestures and / or hand postures of its user by using data obtained from external camera 225-2, based on the execution of gesture tracker 273. For example, electronic device 101 can recognize the hand gestures and / or hand postures of its user based on data (or images) obtained using external camera 225-2, based on the execution of gesture tracker 273. Gesture tracker 273 may be referred to as a hand tracking (HaT) module (or hand tracking program) and / or gesture tracking module.

[0075] For example, electronic device 101 can identify (or track) the movement of a user's eyes based on the execution of eye tracker 274. For example, electronic device 101 can identify the movement of a user's eyes based on the execution of eye tracker 274 by using data obtained from eye-tracking camera 225-1. Eye tracker 274 may be referred to as an eye-tracking (ET) module (or eye-tracking program) and / or eye-tracking module.

[0076] For example, the perception service layer 270 of electronic device 101 may also include a face tracker 275 for tracking a user's face. For example, electronic device 101 may identify (or track) movement of the user's face and / or the user's facial expressions based on the execution of face tracker 275. Electronic device 101 may estimate the user's facial expressions based on movement of the user's face based on the execution of face tracker 275. For example, electronic device 101 may identify movement of the user's face and / or the user's facial expressions based on data (e.g., images) obtained using sensor 230 (e.g., an image sensor facing at least a portion of the user's face) based on the execution of face tracker 275.

[0077] According to an embodiment, the processor 210 of the electronic device 101 can remove external electronic devices (e.g., those connected to the electronic device 101 via the communication circuit 235) while simultaneously transmitting the display screen. Figure 1 The appearance of the external electronic device (120) on the screen. Removing the appearance of the external electronic device may include actions that synthesize virtual objects, images, and / or videos representing body parts covered by the external electronic device. Body parts may include the two eyes of the user wearing the external electronic device.

[0078] Reference Figure 2b ,exist Figure 2a The program exemplarily illustrates one or more programs executed to remove the appearance of the external electronic device 120. The embodiments are not limited thereto.

[0079] According to an embodiment, the processor 210 of the electronic device 101 can conditionally perform the operation of removing the appearance of the external electronic device 120 based on the interaction between the electronic device 101 and the external electronic device 120. The interaction may include the positional relationship between the electronic device 101 and the external electronic device 120 and / or the interaction (e.g., a conversation) between the electronic device 101 and the user wearing the external electronic device 120. The interaction may be detected based on information included in the electronic device 101, such as an address book. The interaction may be detected based on the usage history of the electronic device 101. The interaction may be detected based on whether the external electronic device 120 is an electronic device associated with a user registered in the address book and / or a list previously specified by the user. (See reference...) Figure 4 The processor 210 of the electronic device 101 is described as recognizing interactive operations.

[0080] Electronic device 101 was detected to be connected to external electronic device 120 (e.g., Figure 1 The processor 210, which interacts with the external electronic device 120, can detect portions corresponding to the external electronic device 120 in images and / or videos of the external environment displayed on the display 220. Hereinafter, embodiments based on external electronic devices 120, such as HMD devices, are described; however, the embodiments are not limited thereto, and external electronic devices 120 may include portable electronic devices such as smartphones and / or other physical objects. The processor 210 can control an external camera 225-2 to acquire images and / or videos. In the images and / or videos, the processor 210, which detects portions corresponding to the external electronic device 120, can perform functions for inferring the appearance of body parts covered by the external electronic device 120 within those portions.

[0081] Processor 210 can receive information about body parts covered by external electronic device 120 (e.g., connected via communication circuit 235) from external electronic device 120. Figure 1(Information 150). After establishing a communication link between electronic device 101 and external electronic device 120, information can be received through the communication link. For example, information can be received before the interaction between electronic device 101 and external electronic device 120 is detected, at a timing substantially consistent with the detection of the interaction, or after the interaction is detected.

[0082] Reference Figure 2b This illustrates one or more programs (e.g., eye information exchanger 292, eye image generator 294, and / or eye image synthesizer (or synthesizer) 296) included in eye tracker 274. Processor 210 can receive information from external electronic device 120 based on the execution of eye information exchanger 292. Based on the execution of eye information exchanger 292, processor 210 can control communication circuitry 235 to send information related to eye-tracking camera 225-1 to external electronic device 120. Information related to eye-tracking camera 225-1 may include information related to the user of wearable electronic device 101 (e.g., ...). Figure 1 Information related to the gaze of the user (first user 110). Information related to the gaze of the user of the wearable electronic device 101 may include images and / or videos associated with at least one of the user's two eyes. Embodiments are not limited thereto, and the information sent to the external electronic device 120 may include parameters calculated by the processor 210 based on the images and / or videos (e.g., the direction and / or angle of the two eyes). For example, the processor 210 may send information indicating the direction of the user's gaze to the external electronic device 120 via communication circuitry 235.

[0083] According to one embodiment, the processor 210 of the electronic device 101 can perform rendering of body parts for a user wearing an external electronic device 120, based on the execution of an eye image generator 294. For example, the processor 210 can perform rendering based on information received from the external electronic device 120. The processor 210 executing the eye image generator 294 can perform rendering by using information obtained from the external electronic device 120 using a gaze-based information exchanger 292. Based on the rendering of the body parts, the processor 210 can obtain virtual objects representing the body parts (e.g., 3D graphical objects including meshes representing the head and / or eyes), images, and / or videos. (See also...) Figure 5 and / or Figure 6a and / or Figure 6b Describes the operation of the processor 210 that executes the eye image generator 294.

[0084] According to an embodiment, the processor 210 of the electronic device 101 can display a composite image on the display 220 based on the execution of the eye image synthesizer (or synthesizer) 296. For example, the processor 210 can display a composite image generated on the display 220 based on a first image obtained from an external camera 225-2 and a second image obtained from an eye image generator 294. The composite image can be an image in which at least a portion of the first image is replaced by the second image. The composite image can include a first image and a second image superimposed (or positioned) on at least a portion of the first image. Graphical objects (e.g., virtual objects, images, and / or videos) generated by the eye image generator 294 can be displayed. When displaying the first image obtained from the external camera 225-2, the processor 210 can display the first image including the replaced portion (e.g., the portion corresponding to the external electronic device 120) based on the second image obtained from the eye image generator 294. The embodiment is not limited thereto, and the processor 210 can display by superimposing and / or updating the second image on the first image displayed on the display 220. The embodiments are not limited thereto, and the processor 210 can synthesize a new image by combining a first image and a second image, in which the portion of the first image capturing the external electronic device 120 is changed to the second image. The overlapping portion of the first image and the second image may be the portion of the external electronic device 120 that was captured. The second image can be obtained by performing rendering based on the gaze direction indicated by information received from the external electronic device 120. The second image may include at least a portion representing the eye of a user wearing the external electronic device. The second image may be generated based on the execution of the eye image generator 294.

[0085] According to an embodiment, the processor 210 of the electronic device 101 can display a first image on the display 220 in which a second image is superimposed, based on the execution of the eye image synthesizer (or synthesizer) 296 and / or the transfer manager 253. Based on the second image, the electronic device 101 can provide a user experience to a user wearing the electronic device 101, such as viewing a body part of another user covered by an external electronic device 120. For example, based on the execution of the eye image synthesizer (or synthesizer) 296, the processor 210 can determine the points on the first image to overlap with the second image generated by the eye image generator 294. Based on the operation of the transfer manager 253, the processor 210 can display the first image superimposed on the second image on the display 220.

[0086] As described above, according to the embodiment, electronic device 101 can perform operations to remove the appearance of external electronic device 120 from display 220 by using the gaze of a user wearing external electronic device 120 tracked by external electronic device 120 and the user's face tracked by external camera 225-2. For example, electronic device 101 can display a virtual object representing the gaze (e.g., an avatar) overlaid on the portion of an image and / or video displayed on display 220 that corresponds to external electronic device 120. To assist external electronic device 120 in removing the appearance of electronic device 101 from the display of external electronic device 120, electronic device 101 can send information related to the user's gaze tracked by eye-tracking camera 225-1 to external electronic device 120.

[0087] In the following text, reference will be made to Figure 3 Description Reference Figure 1 , Figure 2a and / or Figure 2b The operation of the described electronic device 101 and / or processor 210.

[0088] Figure 3 An example flowchart illustrating an electronic device according to an embodiment of the present disclosure. Figure 1 , Figure 2a and / or Figure 2b The electronic device 101 and / or processor 210 can perform the reference Figure 3 At least one of the operations described.

[0089] Reference Figure 3 In operation 310, according to an embodiment, the processor of the electronic device can obtain first information related to the face of a first user. The first user in operation 310 can include a person identified as a user wearing the electronic device, such as... Figure 1 The first user 110. The first information of operation 310 may include a photo (e.g., a selfie) related to the first user's face.

[0090] In an embodiment, the first information for operation 310 may include time-of-flight (ToF) and / or data (e.g., meshes, vertices, voxels, fragments, and / or textures associated with at least a portion of the face) obtained via a time-of-flight (ToF) and / or stereo camera for rendering the first user's face in three dimensions. The first information may be obtained at timing when the first user wears the electronic device. Embodiments are not limited thereto, and the processor of the electronic device may access the data from memory (e.g., Figure 2a and / or Figure 2b The image and / or video in the memory (215) obtain the first information for operation 310.

[0091] Reference Figure 3 In operation 320, according to an embodiment, the processor of the electronic device can be connected to an external electronic device adjacent to the electronic device while being worn by a first user. The electronic device can be based on sensors (e.g., Figure 2a and Figure 2b The electronic device determines whether the device is worn by a first user by using sensor data from sensor 230. When worn by a first user, the electronic device can determine whether the device is worn by a communication circuit (e.g., ...). Figure 2a and Figure 2b The communication circuit 235) is used to search for or identify external electronic devices adjacent to the electronic device (e.g., Figure 1 External electronic device 120. The search for external electronic devices can be performed based on wireless communication protocols such as Bluetooth Low Energy (BLE) and / or Ultra Wideband (UWB). Based on the search for external electronic devices, the processor can establish a communication link between the electronic device and the external electronic device by controlling the communication circuitry. The communication link can be established based on wireless communication protocols such as Wi-Fi Direct and / or Bluetooth. For example, a Wi-Fi Direct discovery process can be performed to connect to the external electronic device operating 320. Based on the discovery process, a communication link can be established between the electronic device and the external electronic device. For example, a wireless signal used to detect the external electronic device—such as an advertising signal—can be sent by the electronic device performing operation 320. Based on the response to the advertising signal, the electronic device can connect to the external electronic device.

[0092] In an embodiment, the processor of the electronic device may perform operation 320 based on whether an interaction between the electronic device and an external electronic device is detected. The interaction may be detected based on the distance between the electronic device and the external electronic device, whether the external electronic device is facing the electronic device, and / or utterances from at least one of the users wearing the electronic device and the user wearing the external electronic device. In an embodiment, the processor that detects the interaction may display a screen and / or visual object confirming the connection to the external electronic device. In response to user input confirming the connection to the external electronic device, the processor may perform operation 320.

[0093] While connected to an external electronic device based on operation 320, the processor of the electronic device can enter a reverse pass-through mode. In reverse pass-through mode, the electronic device can perform the function of removing the appearance of the external electronic device from the screen displayed to a first user, and can provide information to the external electronic device for removing the appearance of the electronic device from the screen displayed to a second user via communication with the external electronic device. For example, in reverse pass-through mode, the processor can repeatedly perform the function using the communication link established based on operation 320. Figure 3 Operations 330, 340, and 350.

[0094] Reference Figure 3 In operation 330, according to an embodiment, the processor of the electronic device can be accessed from an external electronic device (e.g., Figure 1 and / or Figure 2b External electronic device 120) obtains communication with a second user (e.g., Figure 1 The second information is related to the gaze of the second user (125). The second information of operation 330 may include... Figure 1 Information 150. Through the communication link established based on operation 320, the processor can repeatedly (or continuously or periodically) receive second information. The second information may include images and / or videos of the two eyes of a second user captured by an external electronic device. The second information may include numerical values ​​calculated by the external electronic device indicating the direction of the second user's gaze. The second information of operation 330 may be referred to as eye-tracking information recognized by the external electronic device. Embodiments are not limited thereto, and the second information may include an image synthesized by the external electronic device representing a body part of the user covered by the external electronic device (e.g., a portion of the user's face including both eyes). For example, the external electronic device can generate an image to be sent to the electronic device using images and / or videos of the user's two eyes captured by the external electronic device. As another example, the second information is an image synthesized by the external electronic device and may include an image representing the entire face of the user wearing the external electronic device. For example, the external electronic device can generate an image to be sent to the electronic device by combining or synthesizing another image generated by eye-tracking information recognized by the external electronic device onto an image of the user's face wearing the external electronic device.

[0095] In this embodiment, the time period for the processor to receive the second information in operation 330 can be adaptively adjusted according to the state of the electronic device and / or the external electronic device. For example, the processor can change the time period for receiving the second information based on the direction of the user's gaze on the electronic device. For example, if the user's gaze on the electronic device is directed towards the external electronic device, the electronic device can send a signal to the external electronic device to reduce the time period for the external electronic device to receive the second information. For example, if the user's gaze on the electronic device is directed in a direction different from that of the external electronic device, the electronic device can send a signal to the external electronic device to increase the time period for receiving the second information.

[0096] In this embodiment, an example of an electronic device receiving second information based on data communication is described, but the embodiment is not limited thereto. For example, the processor may transmit the second information using another medium such as infrared communication and / or sound waves.

[0097] The size of the second information obtained from the external electronic device based on operation 330 can be related to (e.g., inversely proportional to) the distance between electronic device 101 and the external electronic device. For example, when the external electronic device moves away from the user, the size of the portion covered by the external electronic device can decrease on the screen displayed by electronic device 101. In the above example, the electronic device can request the external electronic device to send second information, the size of which is based on the distance between the external electronic device and the electronic device.

[0098] In the state where the second information is obtained based on operation 330, the processor of the electronic device can send information related to the gaze of the first user to an external electronic device. For example, at least based on operation 330, the exchange of eye-tracking information between the first user and the external electronic device can be performed. The exchange of eye-tracking information can be performed through a communication link established based on operation 320.

[0099] The information exchanged via the communication link in operation 320 is not limited to eye-tracking information. For example, when the communication link is established, the electronic device can send the first information from operation 310 to an external electronic device. The first information sent to the external electronic device can be used to render at least a portion of the face of a first user covered by the electronic device. For example, when the communication link is established, the electronic device can obtain third information related to the face of a second user from the external electronic device. The third information can be obtained by the external electronic device that has performed an operation similar to operation 310. The third information may include a photograph related to the face of the second user.

[0100] Reference Figure 3 In operation 340, according to an embodiment, the processor of the electronic device can modify the portion corresponding to the external electronic device in the image and / or video acquired from the camera based on the second information. The camera in operation 340 may include... Figure 2a and Figure 2b The processor can determine or identify portions of an image and / or video corresponding to an external electronic device based on object recognition. Object recognition can be performed based on one or more feature points (or key points) extracted from the image and / or video. Object recognition can also be performed based on a model of the input image and / or video (e.g., a neural network such as a convolutional neural network (CNN)).

[0101] For example, the processor can obtain the coordinates of the vertices of the polygon representing the portion (e.g., a quadrilateral such as a bounding box). For example, the processor can obtain the pixel-by-pixel probability of each pixel in the image and / or video corresponding to an external electronic device. At least one pixel corresponding to at least one probability exceeding a threshold can be included in a portion of the image and / or video from which the external electronic device is captured.

[0102] In operation 340, the processor of the electronic device can obtain a virtual object representing a body part covered by the external electronic device based on second information received from the external electronic device via a communication circuit. The virtual object can also be obtained based on at least one of third information related to the second user's face and / or the second information received from the external electronic device via a communication link. (See also...) Figure 5 Describes the processor's operation 340 based on a 3D virtual object, which alters at least a portion of an image and / or video.

[0103] In operation 340, the processor of the electronic device can infer the appearance of a body part covered by the external electronic device based on second information received from the external electronic device via a communication circuit. Inferring the appearance of the body part can be performed using a processor-driven model. The model may include hardware for performing calculations based on a mathematical model (e.g., a neural network) that mathematically models the neural activity of humans and / or animals. (See reference...) Figure 6a And / or 6B describes the processor's operation 340, which alters at least a portion of an image and / or video based on the result of inferring the appearance of body parts.

[0104] Reference Figure 3 In operation 350, according to an embodiment, the processor of the electronic device can display images and / or videos on a display that have changed based on the second information. For example, when on a display (e.g., Figure 2a While displaying images and / or videos on the display (220), the processor can display at least a portion of a virtual object obtained based on operation 340 on at least a portion of the display, which is part of the display operation 340. For example, the processor can display the appearance of body parts inferred from the model by overlaying them on at least a portion of the display. Based on operation 350, the processor can align or synthesize images and / or videos obtained from the camera with images and / or videos obtained based on second information.

[0105] In an embodiment, the acquisition of second information based on operation 330 may be delayed or stopped depending on the communication status between the electronic device and the external electronic device. According to an embodiment, in the case of stopping or delaying the reception of second information, the processor of the electronic device may play a specified animation related to the appearance of a body part covered by the external electronic device. The specified animation may include images and / or videos displayed before the cessation of receiving second information. The embodiment is not limited thereto, and when the battery's state of charge (SOC) drops to a specified SOC or lower, the processor of the electronic device may stop receiving second information via the communication link and may display images and / or videos displayed before the SOC dropped to the specified SOC or lower.

[0106] According to an embodiment, the processor of the electronic device can repeatedly execute operations 330, 340, and 350. When operations 330, 340, and 350 are repeatedly executed based on a reverse transmission mode, the processor can determine whether the interaction between the electronic device and an external electronic device is interrupted. If the interaction between the electronic device and the external electronic device stops, the processor can stop repeatedly executing operations 330, 340, and 350. If the interaction between the electronic device and the external electronic device stops, the processor can display a screen and / or virtual objects based on a display to confirm whether operations 330, 340, and 350 for displaying body parts covered by the external electronic device have stopped. In response to input to the screen and / or virtual objects, the processor of the electronic device can stop repeatedly executing operations 330, 340, and 350. Because operations 330, 340, and 350 have stopped, the user of the wearable electronic device can view the external electronic device through images and / or videos of the external environment displayed on the display.

[0107] According to an embodiment, the processor of the electronic device can reduce or optimize the computational load based on operation 340 for changing images and / or video while repeatedly executing operations 330, 340, and 350. For example, the processor can adjust the frequency of inferring the appearance of a second user's body part covered by the external electronic device based on the size of a portion of the display corresponding to the external electronic device. (See also...) Figure 7a and Figure 7b Describes operations performed by the processor of an electronic device to optimize computational load.

[0108] In the following text, reference will be made to Figure 4 Describe the operation of an electronic device that detects interactions between itself and external electronic devices.

[0109] Figure 4 An example of the operation of an electronic device 101 based on images 410 and / or video obtained from a camera, according to an embodiment of the present disclosure, is shown. Figure 1 , Figure 2a and Figure 2b Electronic devices 101 and / or Figure 2a and Figure 2b The processor 210 can execute reference Figure 4 The operation of the described electronic device 101. (Refer to...) Figure 4 The operation of the described electronic device 101 can be with Figure 3 At least one of the operations is related.

[0110] Reference Figure 4 This illustrates state 401 of electronic device 101 adjacent to external electronic device 120. In state 401, electronic device 101 can be accessed via a camera (e.g., ...). Figure 2a and / or Figure 2b Images 410 and / or video from an external camera 225-2) are used to detect external electronic devices 120. (Refer to...) Figure 4 In state 401, an example of image 410 obtained by electronic device 101 is shown. Image 410 may correspond to a single image frame of video obtained by the camera of electronic device 101. In image 410, electronic device 101 may determine the portion corresponding to external electronic device 120 based on object recognition. Embodiments are not limited thereto, and electronic device 101 may identify external electronic device 120 adjacent to electronic device 101 based on wireless communication protocols such as BLE, Near Field Communication (NFC), and / or UWB.

[0111] exist Figure 4 In state 401, electronic device 101 can recognize the interaction between a first user 110 wearing electronic device 101 and a second user 125 wearing external electronic device 120. For example, electronic device 101 can determine or recognize the orientation d1 of external electronic device 120 by using image 410. Based on whether orientation d1 is facing electronic device 101, electronic device 101 can detect the interaction. For example, if the second user 125 is looking at the first user 110 wearing electronic device 101, electronic device 101 can detect that orientation d1 of external electronic device 120 is facing electronic device 101. Based on detecting that orientation d1 is facing electronic device 101 or a point adjacent to electronic device 101, electronic device 101 can detect the interaction between electronic device 101 and external electronic device 120.

[0112] In an embodiment, electronic device 101 can detect the interaction between electronic device 101 and external electronic device 120 based on a communication link established between electronic device 101 and external electronic device 120. For example, electronic device 101 can detect the interaction based on a user interface (UI) provided to first user 110 (e.g., Figure 1The screen 130 receives input from the first user 110 for executing the reverse transmission mode. The input received by the electronic device 101 is not limited to the UI. For example, the electronic device 101 can receive input for executing the reverse transmission mode via buttons and / or a dial pad. Based on this input, the electronic device 101 can establish a communication link. Based on the establishment of the communication link, the electronic device 101 can execute the reverse transmission mode, which can remove the appearance of the external electronic device 120 displayed on the screen, or can perform operations to display the appearance of the second user 125 covered by the external electronic device 120 on the screen.

[0113] In an embodiment, electronic device 101 may detect interactions between electronic device 101 and external electronic device 120 based on speech recognition and / or speech-to-text (STT). For example, electronic device 101 may detect interactions based on utterances from a first user 110 and / or a second user 125. To recognize utterances, electronic device 101 may include one or more microphones. For example, electronic device 101 may recognize natural language sentences including the name and / or pronouns of the first user 110 from audio signals output from one or more microphones. In response to the recognition of natural language sentences, electronic device 101 may detect interactions. For example, electronic device 101 may recognize natural language sentences including the name and / or pronouns of the second user 125 from audio signals. In response to the recognition of natural language sentences, electronic device 101 may detect interactions between electronic device 101 and external electronic device 120.

[0114] Electronic device 101, which detects the interaction between electronic device 101 and external electronic device 120, can refer to Figure 3 Perform the above operations (for example, Figure 3 Operations 320, 330, 340, and 350 are used to enter the reverse transmission mode. In the reverse transmission mode, the electronic device 101 can replace the appearance of the external electronic device 120 displayed on the monitor with virtual objects, images, and / or videos representing the face of a second user 125 wearing the external electronic device 120. In the reverse transmission mode, by displaying the entire face of the second user 125 covered by the external electronic device 120, the electronic device 101 can provide non-verbal expressions based on the facial expressions of the second user 125 to the first user 110 wearing the electronic device 101. In the reverse transmission mode, in order to support the external electronic device 120 displaying the entire face of the first user 110, the electronic device 101 can send information related to at least a portion of the face of the first user 110 to the external electronic device 120.

[0115] In reverse transmission mode, electronic device 101 can detect the cessation of interaction between external electronic device 120 and electronic device 101. Based on the cessation of interaction, electronic device 101 can switch to a mode different from reverse transmission mode. For example, electronic device 101 can stop displaying the entire face of the second user 125 that was covered by external electronic device 120.

[0116] The cessation of interaction between electronic device 101 and external electronic device 120 can be identified or determined based on the direction d1 of external electronic device 120. For example, electronic device 101 can detect the cessation of interaction if the direction d1 of external electronic device 120 is directed to another point spaced apart from the point where electronic device 101 is located (e.g., the distance exceeds a specified distance). For example, electronic device 101 can detect the cessation of interaction if the direction d1 is directed to another point spaced apart from the point where electronic device 101 is located for a period of time exceeding a specified duration.

[0117] In one embodiment, electronic device 101 may stop displaying body parts (e.g., at least a portion of the face) of a second user 125 covered by external electronic device 120 in response to an input to stop and / or terminate the reverse transmission mode. For example, electronic device 101 may receive or recognize input for stopping and / or terminating the reverse transmission mode via visual objects and / or buttons (or dial pads) provided by the electronic device 101 via a screen. In response to this input, electronic device 101 may switch to a mode different from the reverse transmission mode used to provide the entire face of the second user 125, and may stop providing the entire face.

[0118] In one embodiment, electronic device 101 can switch from a reverse transmission mode to another mode based on the speech of users (e.g., first user 110 and / or second user 125) wearing electronic device 101 and external electronic device 120. For example, electronic device 101 can switch from a reverse transmission mode to another mode if a user remains silent for more than a specified period of time, or if a natural language sentence indicating the end of the conversation between the users is recognized (e.g., “See you next time!”). Electronic device 101, after switching to another mode, can either resume displaying external electronic device 120 on the screen, or it can display a different UI than the transmission, such as VR.

[0119] As described above, according to the embodiments, electronic device 101 can enter a reverse transmission mode based on the input, speech, and / or location of a first user 110 wearing the electronic device 101. Electronic device 101 can enter a reverse transmission mode based on the position and / or orientation of external electronic device 120 relative to electronic device 101. Electronic device 101 can enter a reverse transmission mode based on the speech, location, and / or orientation (e.g., the orientation of the face of the second user 125 wearing external electronic device 120) of a second user 125 wearing external electronic device 120. Electronic device 101 entering the reverse transmission mode can use images 410 and information received from external electronic device 120 (e.g., ... Figure 1 Information 150) is used to provide at least a portion of the face of the second user 125 covered by the external electronic device 120.

[0120] In the following text, reference will be made to Figure 5 , Figure 6a and / or Figure 6b Describes the operation of electronic device 101 generating virtual objects, images, and / or videos representing the face of a second user 125 covered by external electronic device 120.

[0121] Figure 5 An example of the operation of an electronic device 101 according to an embodiment of the present disclosure processing information received from an external electronic device 120 is shown. Figure 1 , Figure 2a and Figure 2b Electronic devices 101 and / or Figure 2a and Figure 2b The processor 210 can execute reference Figure 5 The operation of the described electronic device 101. (Refer to...) Figure 5 The operation of the described electronic device 101 can be with Figure 3 Related to the operation.

[0122] Reference Figure 5 Electronic device 101 can use a camera for capturing the external environment (e.g., Figure 2a and / or Figure 2bAn external camera 225-2 is used to acquire an image 410. In a state 501 adjacent to the external electronic device 120, the electronic device 101 can identify the external electronic device 120 from the image 410. Based on at least one of the orientation d1 of the external electronic device 120, the distance from the external electronic device 120 (e.g., the distance between the external electronic device 120 and the electronic device 101), and the speech of the first user 110 and the second user 125 of the external electronic device 120, the electronic device 101 can determine whether to enter a reverse transmission mode. Before entering the reverse transmission mode, the electronic device 101 can display an image 410 including the appearance of the external electronic device 120 based on VST and / or pass-through.

[0123] Reference Figure 5 This shows state 501, indicating the establishment of a communication link between external electronic device 120 and electronic device 101. Electronic device 101 can perform the above-mentioned reference... Figure 4 The operation is to establish a communication link in response to the detection of interaction between electronic device 101 and external electronic device 120. Through the communication link, electronic device 101 can receive information related to a second user 125 wearing external electronic device 120 from external electronic device 120.

[0124] For example, electronic device 101 can receive 3D mesh information 520 about the second user 125 from external electronic device 120 via a communication link. The 3D mesh information 520 may include information required for 3D rendering of body parts of the second user 125 (e.g., face and / or head). For example, the 3D mesh information 520 may include vertices (or voxels) corresponding to each point (or keypoint) of the second user 125's face. The 3D mesh information 520 may include information for forming segments (or unit planes) based on connections of at least three or more vertices. The 3D mesh information 520 may include information about the texture to be filled in each segment. The texture may be generated based on a 3D scan of the body parts of the second user 125 by external electronic device 120.

[0125] Similar to the 3D mesh information 520 received from the external electronic device 120, the electronic device 101 can obtain or generate 3D mesh information about the first user 110 based on a 3D scan of the first user 110 wearing the electronic device 101. For example, the electronic device 101 can perform... Figure 3 Operation 310 is used to obtain 3D mesh information about the first user 110. Electronic device 101 can obtain this information from its camera (e.g., using a deep learning model). Figure 2a and / or Figure 2bThe camera 225 and / or eye-tracking camera 225-1 obtains images and / or videos of the first user 110 to obtain three-dimensional mesh information about the first user 110. The three-dimensional mesh information may include the geometry and / or texture of body parts (e.g., head) of the first user 110. The three-dimensional mesh information about the first user 110 can be transmitted to an external electronic device 120 via a communication link.

[0126] Reference Figure 5 Electronic device 101 can receive three-dimensional mesh information 520 and gaze information 510 from external electronic device 120 of second user 125. The gaze information 510 may include images 511 and 512 of the two eyes of second user 125. Image 511 corresponding to the right eye of second user 125 and image 512 corresponding to the left eye of second user 125 are illustrated as examples of gaze information 510, but embodiments are not limited thereto. For example, gaze information 510 may include parameters calculated by external electronic device 120 indicating the direction of the two eyes of second user 125.

[0127] According to an embodiment, electronic device 101 can modify the portion of a 3D mesh corresponding to gaze information 510 generated (or reconstructed) from 3D mesh information 520 received from external electronic device 120. For example, electronic device 101 can modify vertices and / or textures in the 3D mesh associated with the two eyes of the second user 125. For example, image 512 corresponding to the left eye in gaze information 510 can be determined as at least one texture corresponding to the left eye in the 3D mesh. For example, image 511 corresponding to the right eye in gaze information 510 can be determined as at least one texture corresponding to the right eye in the 3D mesh. The 3D mesh generated based on gaze information 510 and 3D mesh information 520 can be referred to as a 3D avatar and / or virtual object of the second user 125.

[0128] According to an embodiment, when an image 410 (or video) is displayed on a display based on VST, electronic device 101 can display at least a portion of a virtual object belonging to the second user 125 based on a reverse transmission mode. Electronic device 101 can display at least a portion of the virtual object based on detected interaction between electronic device 101 and external electronic device 120. For example, in a virtual object representing a head wearing external electronic device 120, electronic device 101 can display a portion 530 of the virtual object on the display, wherein eyes are arranged with directions indicated by gaze information 510.

[0129] Reference Figure 5Electronic device 101 can display a portion 530 of a virtual object by overlapping it with the portion 505 of the image 410 corresponding to the external electronic device 120. Electronic device 101 can display the portion 530 of the virtual object, rotated (or aligned) based on the direction d1 of the external electronic device 120 identified from the image 410, on the portion 505 of the image 410. In the screen 540 displayed on the monitor of electronic device 101, the entire facial expression of the second user 125 can be reconstructed because the portion 530 of the virtual object overlaps with the portion 505 of the image 410.

[0130] Electronic device 101 can compare key points contained in a virtual object with key points of a face identified in image 410 to match a portion 530 of the virtual object with a portion 505 in image 410. Key points may include specified points (e.g., the corners of the eyes) in body parts such as the face. Electronic device 101 can determine the portion 505 associated with the portion 530 of the virtual object based on feature points (or key points) extracted from image 410.

[0131] An embodiment of restoring the appearance of a body part covered by an external electronic device 120 using gaze information 510 has been described, but the embodiment is not limited thereto. For example, electronic device 101 can generate and / or display a virtual object of at least a portion of the body part of a second user 125 covered by external electronic device 120 by using the gaze direction of a user wearing electronic device 101, the distance between electronic device 101 and external electronic device 120, the appearance of the body part covered by external electronic device 120 (e.g., the two eyes of a second user 125) estimated using the microphone of electronic device 101, and / or at least one of the images and / or videos pre-stored in electronic device 101.

[0132] As described above, according to one embodiment, electronic device 101 can render a virtual object of a second user 125 wearing an external electronic device 120 in three dimensions to provide the appearance of body parts covered by the external electronic device 120 on screen 540. Electronic device 101 can perform rendering of the virtual object based on mesh information 520 and gaze information 510 of the second user 125's face, which are repeatedly (or periodically) transmitted from the external electronic device 120. By displaying portions 530 of the virtual object, electronic device 101 can provide the appearance of body parts covered by the external electronic device 120.

[0133] In one embodiment, electronic device 101 can infer the appearance of body parts beyond the portion 505 corresponding to external electronic device 120 by using a neural network. Reference will be made below. Figure 6aAnd / or 6B describes the operation of electronic device 101 to infer the appearance of body parts of a second user 125 covered by external electronic device 120 based on neural network.

[0134] Figure 6a and Figure 6b Examples of the operation of an electronic device 101 according to various embodiments of the present disclosure, which processes information received from an external electronic device 120, are shown. Figure 1 , Figure 2a and Figure 2b Electronic devices 101 and / or Figure 2a and Figure 2b The processor 210 can execute reference Figure 6a and Figure 6b The operation of the described electronic device 101. (Refer to...) Figure 6a and / or Figure 6b The operation of the described electronic device 101 can be with Figure 3 Related to the operation.

[0135] Reference Figure 6a In state 601, electronic device 101 can be operated using a camera (e.g., Figure 2a and / or Figure 2b An external camera 225-2 is used to acquire images 410 of the external environment. Image 410 may correspond to any of the image frames acquired sequentially by the camera. While providing a user experience based on VST and / or delivery, the electronic device 101 can display (e.g., on a monitor) to provide a user experience. Figure 2a At least a portion of image 410 is displayed on a display 220 (or a 2B display). Electronic device 101, which detects the interaction between electronic device 101 and external electronic device 120, can identify the portion 505 in image 410 corresponding to external electronic device 120.

[0136] According to an embodiment, electronic device 101 can perform compositing (e.g., repair) on a portion 505 in image 410 corresponding to external electronic device 120. Compositing may include combining, replacing, or overlaying virtual objects, images, and / or videos of body parts of a second user 125 (e.g., at least a portion of a face including both eyes) covered by external electronic device 120 on portion 505. Compositing may be performed based on the operation of facial image generation model 620. Figure 2b The electronic device 101 of the eye image generator 294 can generate or obtain an image and / or video to be synthesized on a portion 505 of the image 410 by using the facial image generation model 620.

[0137] Reference Figure 6aElectronic device 101 can input at least one of image 410, gaze information 510, and / or image 610 related to the second user 125 obtained from an external camera of electronic device 101 into facial image generation model 620. Image 610 can be received from external electronic device 120 via a communication link established between electronic device 101 and external electronic device 120. Image 610 may include at least a portion corresponding to the face of the second user 125 wearing external electronic device 120. Gaze information 510 can be repeatedly and / or continuously received from external electronic device 120 via the communication link. Gaze information 510 including images 511 and 512 of the two eyes of the second user 125 is shown exemplary, but embodiments are not limited thereto. Gaze information 510 may be referred to as gaze tracking information.

[0138] According to one embodiment, electronic device 101 can obtain an image 630 related to the face of a second user 125 wearing an external electronic device 120 from a face image generation model 620 that is input to at least one of images 410 and 610 and gaze information 510. The face image generation model 620 may include a neural network for rendering a face. For example, the face image generation model 620 may be trained to output an image 630 related to a face having the appearance of the face indicated by image 610, including eyes having a direction indicated by gaze information 510 and facing a direction d1 towards the face represented by image 410. The face image generation model 620 may include a generative model, such as a generative adversarial network (GAN). The face image generation model 620 may include a model for generating graphical information (e.g., image 630) from non-graphical information such as text, such as a diffusion model (e.g., a stable diffusion model). Similar to image 630, the face image generation model 620 may be trained to output an image that includes not only the face but also the background.

[0139] Reference Figure 6a The electronic device 101, which obtains image 630 from facial image generation model 620, can segment or extract the portion 635 in image 630 corresponding to external electronic device 120. The electronic device 101 can replace the portion 505 corresponding to external electronic device 120 in image 410 with the portion 635 extracted from image 630 for display on a monitor. The electronic device 101 can perform alignment between image 410 and portion 635 based on feature points (or key points of the face indicated by feature points) of the second user 125's face identified from image 410.

[0140] In an embodiment, by displaying an image 410 including portion 505 replaced by portion 635, electronic device 101 can provide a first user 110 with the entire face of a second user 125 wearing external electronic device 120. Based on the entire face, electronic device 101 can provide the first user 110 with nonverbal expressions of the second user 125. To support external electronic device 120 in providing the second user 125 with the nonverbal expressions of the first user 110, electronic device 101 can send at least one of an image of the first user 110's face and / or gaze information associated with the first user 110's two eyes to external electronic device 120.

[0141] Reference Figure 6b State 602 shows an image 652 of the external environment obtained by the electronic device 101 using a camera. With the mobile phone 654 positioned between the electronic device 101 and the external electronic device 120, image 652 can be obtained showing the appearance of the external electronic device 120 being covered by the mobile phone 654. Figure 6b In the embodiment of image 652, electronic device 101 may identify external electronic device 120 without using image 652.

[0142] exist Figure 6b In this embodiment, electronic device 101 can detect external electronic device 120 by using an image 660 of a mobile phone 654 connected to electronic device 101 (e.g., account information of a first user 110 is logged in). For example, electronic device 101 can detect mobile phone 654 located on the front surface of electronic device 101 by using a positioning protocol such as UWB. Electronic device 101 can send a signal to the detected mobile phone 654 for obtaining an image from the camera of mobile phone 654. Based on this signal, electronic device 101, having obtained the image 660 captured by mobile phone 654, can detect a portion 505 in image 660 corresponding to external electronic device 120.

[0143] exist Figure 6b In an embodiment, electronic device 101, which detects a portion 505 covered by external electronic device 120 in image 660 obtained from mobile phone 654, can perform a function to synthesize an image and / or video in image 652 on the portion corresponding to portion 505, indicating a body part (e.g., at least a portion of the face including both eyes) of a second user 125 wearing external electronic device 120. The synthesis can be performed based on the execution of facial image generation model 620. For example, electronic device 101 can input at least one of image 660, gaze information 510, and / or image 610 related to the second user 125 obtained from mobile phone 654 into facial image generation model 620. Based on the above reference... Figure 6aThe operation of the described electronic device 101 and / or external electronic device 120 can send an image 610 and / or line-of-sight information 510 to the electronic device 101.

[0144] Reference Figure 6b Electronic device 101 can obtain an image 630 representing the face of the second user 125 by using facial image generation model 620. In image 630, electronic device 101 can replace, change or update the portion in image 652 corresponding to external electronic device 120 by using portion 635 corresponding to portion 505 of image 660.

[0145] As described above, according to the embodiment, the electronic device 101 can be used by using... Figure 5 Three-dimensional graphical objects (e.g., virtual objects based on mesh information 520) and / or Figure 6a and / or Figure 6b Image 630 obtained by the neural network (e.g., facial image generation model 620) visualizes body parts (e.g., the portion of the face with the two eyes) of the other party (e.g., the second user 125) covered by the external electronic device 120. Electronic device 101 can optimize the computational load used to visualize the other party's body parts. For example, in the event of an interruption of interaction (e.g., a conversation) between the first user 110 and the second user 125 and / or a reduction in the size of the portion on the display corresponding to the external electronic device 120 (e.g., portion 505 of image 410), electronic device 101 can reduce the computational load used to infer the face of the second user 125 covered by the external electronic device 120.

[0146] In the following text, reference will be made to Figure 7a and Figure 7b Describes the operation of electronic device 101 for optimizing computational load.

[0147] Figure 7a and Figure 7b Examples of operation of an electronic device 101 according to various embodiments of the present disclosure are shown. Figure 1 , Figure 2a and Figure 2b Electronic devices 101 and / or Figure 2a and Figure 2b The processor 210 can execute reference Figure 7a and Figure 7b The operation of the described electronic device 101. (Refer to...) Figure 7a and Figure 7b The operation of the described electronic device 101 can be with Figure 3 Related to the operation.

[0148] Reference Figure 7aThis illustrates state 701 of electronic device 101, where the appearance of external electronic device 120 is removed and a screen 130 is provided that includes the facial appearance of a second user 125 covered by external electronic device 120. State 701 of electronic device 101 can be associated with a reverse transmission mode. Based on usage Figure 5 The grid information 520 and / or the line-of-sight information 510 Figure 6a and Figure 6b At least one of the three-dimensional rendering and / or facial image generation models 620, the electronic device 101 can display a screen 130 including the appearance of the entire face of the second user 125.

[0149] Reference Figure 7a The position of the visual object 710 corresponding to the second user 125 on screen 130 can correspond to the position L1 of the second user 125 and / or the external electronic device 120. Based on the orientation d1 of the second user 125 and / or the external electronic device 120, the electronic device 101 can display virtual objects, images, and / or videos representing at least a portion of the face of the second user 125 covered by the external electronic device 120 on the portion 720 of screen 130 corresponding to the external electronic device 120. Figure 7a As shown on screen 130, a first user 110 of the wearable electronic device 101 can view the external environment in which the external electronic device 120 is removed, based on virtual objects, images and / or videos.

[0150] Based on Figure 7a In state 701, the movement and / or rotation of the first user 110 wearing the electronic device 101 and / or the second user 125 wearing the external electronic device 120 can change the positional relationship between the electronic device 101 and the external electronic device 120. This change in positional relationship can cause a change in the size of the portion 720 on the screen 130 corresponding to the external electronic device 120.

[0151] For example, if the second user 125 rotates his or her head, the orientation of the external electronic device 120 can be adjusted from [the direction of rotation]. Figure 7a The direction d1 changes to direction d2. When the external electronic device 120 faces direction d2, since the rear surface of the second user 125 faces the electronic device 101, the size of the portion 720 corresponding to the external electronic device 120 can be reduced to essentially zero. For example, when the second user 125 moves, the position and / or size of the portion 720 in the screen 130 corresponding to the external electronic device 120 can change. For example, when the second user 125 moves, if the position of the external electronic device 120 moves from position L1 to position L2, since the second user 125 moves out of the camera of the electronic device 101 (e.g., Figure 2a and / or Figure 2bThe field of view (FoV) of the external camera 225-2 is reduced, so the size of the portion 720 in the screen 130 corresponding to the external electronic device 120 can be reduced to essentially zero. For example, as the second user 125 moves, and the position of the external electronic device 120 moves from position L1 to position L3 in a direction away from the electronic device 101 (e.g., the +z axis direction), the size of the portion 720 in the screen 130 corresponding to the external electronic device 120 can be reduced.

[0152] According to an embodiment, electronic device 101 can determine the size of the portion 720 in screen 130 corresponding to external electronic device 120 based on at least one of the distance to external electronic device 120 or the orientation of external electronic device 120 relative to electronic device 101 (e.g., azimuth angle). The distance to external electronic device 120 may include the distance between electronic device 101 and external electronic device 120. The distance may be obtained or calculated based on object recognition from images obtained from a camera. Electronic device 101 may calculate the distance based on wireless communication protocols such as BLE and / or UWB.

[0153] According to an embodiment, electronic device 101 can determine, based on the size of the portion 720 corresponding to external electronic device 120, an image to be synthesized to represent a body portion (e.g., at least a portion of the face including both eyes) of a second user 125 covered by external electronic device 120, to be synthesized to represent the body portion of the second user 125 covered by external electronic device 120. Figure 6a and / or Figure 6b The size and / or resolution of the image (630). For example, the electronic device 101 can selectively drive different models for synthesizing images of different sizes (or resolutions) (e.g., Figure 6a and Figure 6b The model can be any one of the facial image generation models (620) to synthesize an image with a defined size and / or resolution. The computational cost required to drive the model may be related to the size and / or resolution of the image to be synthesized by each model. For example, the computational cost required to drive the model may increase as the size and / or resolution of the image to be synthesized by the model increases. The number of weights and / or parameters included in the model may increase as the size and / or resolution of the image to be synthesized by the model increases.

[0154] Electronic device 101, which includes models corresponding to each of different sizes, can generate an image to be synthesized on portion 720 by driving any one of the models corresponding to the size and / or resolution of portion 720. Since the size of portion 720 changes according to the distance and / or orientation between electronic device 101 and external electronic device 120, electronic device 101 can select or identify any one of the multiple models, at least based on the size of portion 720. By using the selected model, electronic device 101 can obtain or generate at least a portion of an image to be displayed on portion 720 in screen 130. Electronic device 101, which selects different models according to the size of portion 720, can adaptively adjust the computational load performed on synthesizing or generating the image to be displayed on portion 720 based on the model selection.

[0155] According to one embodiment, electronic device 101 can adjust the amount of graphics resources of a three-dimensional virtual object corresponding to the second user 125 based on the size of the portion 720 corresponding to the external electronic device 120. The three-dimensional virtual object may include graphics resources of... Figure 5 The virtual object is rendered using mesh information 520 and / or view information 510. Graphics resources may include vertices and / or textures included in the mesh information 520. For example, the electronic device 101 may adjust the number of vertices and / or textures of the virtual object based on the size of portion 720. For example, the electronic device 101 may adjust the size of each texture based on the size of portion 720. For example, if the size of portion 720 is reduced, the electronic device 101 may reduce the amount of graphics resources for the virtual object based on texture regression.

[0156] For example, electronic device 101 can change the frequency and / or period of rendering the 3D virtual object corresponding to the second user 125 based on the size of portion 720. For example, if the size of portion 720 decreases, electronic device 101 can increase the rendering period. Independent of the period, electronic device 101 can maintain display of a portion of the 3D virtual object on portion 720 of screen 130 (e.g., Figure 5 For example, independent of the cycle used to perform rendering of the 3D virtual object, the electronic device 101 can maintain a portion of the 3D virtual object and a time period for aligning that portion 720.

[0157] According to one embodiment, electronic device 101 can change the camera based on at least one of the distance to external electronic device 120 and / or the orientation of external electronic device 120 relative to electronic device 101 (e.g., Figure 2aThe period (e.g., frames per second (FPS)) of the eye-tracking camera 225-1 and / or external camera 225-2 (and / or 2B). For example, the size of the portion 720 in the screen 130 corresponding to the external electronic device 120 may change when the distance between the electronic device 101 and the external electronic device 120 increases, and / or the orientation (e.g., orientations d1 and d2) of the external electronic device 120 changes. The electronic device 101, which recognizes that the size of the portion 720 has changed, can control the camera based on the time period corresponding to the changed size. For example, when the size of the portion 720 decreases, the time period associated with camera control may increase. When the size of the portion 720 decreases, the FPS of the video obtained from the camera may decrease, or the number of images (e.g., the number per unit time) may decrease. Due to the reduction in the number of images, the frequency of operations performed to replace the appearance of the external electronic device 120 may be reduced.

[0158] According to one embodiment, electronic device 101 may vary the period at which it receives information from external electronic device 120 based on at least one of distance from external electronic device 120 and / or orientation of external electronic device 120 relative to electronic device 101. The information received from external electronic device 120 may include... Figure 5 , Figure 6a and / or Figure 6b The gaze information 510. For example, if the size of the portion 720 in screen 130 corresponding to external electronic device 120 is reduced based on distance and / or orientation, electronic device 101 can increase the cycle. Since electronic device 101 receives information based on the increased cycle, the number of circuits activated to receive information (e.g., Figure 2a and / or Figure 2b The current consumption of the communication circuit 235.

[0159] Reference Figure 7a The conditions for performing the above-described operations of electronic device 101 are not limited to the positional relationship (e.g., distance and / or orientation) between electronic device 101 and external electronic device 120. For example, electronic device 101 may stop or limit the operation of replacing part 720 corresponding to external electronic device 120 based on user input indicating termination and / or cessation of reverse transmission mode. For example, electronic device 101 may reduce the size (or resolution) of the virtual object, image, and / or video to be synthesized in part 720, or increase the period of obtaining the information required to synthesize the virtual object, image, and / or video, based on information related to the speech of the user (e.g., first user 110 and / or second user 125) of each of the wearable electronic device 101 and external electronic device 120.

[0160] The operation of electronic device 101 based on the positional relationship between electronic device 101 and external electronic device 120 is not limited to the embodiments described above. For example, when displaying screen 130 based on a reverse transmission mode, electronic device 101 can provide a visual effect indicating whether a second user 125 wearing external electronic device 120 is looking at electronic device 101 and / or a first user 110 wearing electronic device 101, based on the orientation d1 of external electronic device 120. For example, when the orientation d1 of external electronic device 120 faces electronic device 101, electronic device 101 can emphasize visual object 710 corresponding to the second user 125 on screen 130. The operation of emphasizing visual object 710 may include adjusting the color, brightness, and / or saturation of visual object 710. The operation of emphasizing visual object 710 may include displaying a graphic (e.g., a graphic with specified transparency and / or specified color) having a shape surrounding at least a portion of visual object 710.

[0161] Reference Figure 7b The image shows a state 702 of an electronic device 101 providing a screen 130, wherein the screen 130 includes the appearance of the face of a second user 125 covered (hidden or concealed) by an external electronic device 120. In state 702, the electronic device 101 may display the screen 130 in relation to a reverse transmission mode.

[0162] Reference Figure 7b The external image of the user 110 wearing the electronic device 101 may include a visual object 710 corresponding to a second user 125 wearing the external electronic device 120 and a visual object 752 corresponding to a third user 750 in the direction in front of the user 110. The electronic device 101 may use information transmitted from the external electronic device 120 to display an image representing at least a portion of the face of the second user 125 hidden by the external electronic device 120 (e.g., a portion of the face including the two eyes of the second user 125) on the portion 720 corresponding to the external electronic device 120.

[0163] Reference Figure 7bOn screen 130, while displaying all the faces of the second user 125 and the third user 750 of the wearable external electronic device 120, electronic device 101 can display a visual object 760 for indicating the second user 125 of the wearable external electronic device 120 among the users visible through screen 130 (e.g., the second user 125 and / or the third user 750). For example, on screen 130, a visual object 760 including designated text such as "VR user" can be displayed on a visual object 710 corresponding to the second user 125 of the wearable external electronic device 120. By displaying a visual object 760 with a position linked to visual object 710 on screen 130, electronic device 101 can guide or notify the second user 125 corresponding to visual object 710 to wear the external electronic device 120.

[0164] As described above, according to the embodiment, electronic device 101 can operate in a reverse transmission mode to support the transmission of nonverbal expressions based on facial expressions. For example, electronic devices arranged toward each other (e.g., electronic device 101 and / or external electronic device 120) can exchange information on the appearance of the electronic device in a screen (e.g., screen 130) for replacing each of the electronic devices based on the reverse transmission mode. Information can be generated to visualize the body parts covered by each electronic device (e.g., portions of the face of a user wearing each electronic device).

[0165] In the following text, reference will be made to Figure 8 Described as the operations performed by electronic devices in order to exchange information.

[0166] Figure 8 A signal flow diagram is shown between electronic devices (e.g., a first electronic device 801 and / or a second electronic device 802) according to embodiments of the present disclosure. Figure 8 Each of the first electronic device 801 and the second electronic device 802 may correspond to a reference. Figure 1 , Figure 2a , Figure 2b , Figures 3 to 5 , Figure 6a , Figure 6b , Figure 7a and / or Figure 7b Each of the described electronic device 101 and external electronic device 120. (Refer to...) Figure 8 The operation of the first electronic device 801 and / or the second electronic device 802 described herein can be performed by Figure 2a and / or Figure 2b The processor 210 executes.

[0167] Reference Figure 8In operation 812, according to an embodiment, the processor of the first electronic device 801 can detect interaction with the second electronic device 802. Similar to operation 812, in operation 814, according to an embodiment, the processor of the second electronic device 802 can detect interaction with the first electronic device 801. Interaction can be detected based on user input for executing the reverse pass-through mode, as referred to above. Figure 4 As stated above.

[0168] Reference Figure 8 In operation 820, according to an embodiment, the first electronic device 801 and the second electronic device 802 can enter a reverse transmission mode. When transmitting images and / or videos of the external environment, the first electronic device 801 and the second electronic device 802 can enter the reverse transmission mode of operation 820. Based on entering the reverse transmission mode, a communication link can be established between the first electronic device 801 and the second electronic device 802. Through the communication link, the processor of the first electronic device 801 can send messages to the second electronic device 802 from a first user wearing the first electronic device 801 (e.g., [user name]). Figure 1 , Figure 2a , Figure 2b , Figures 3 to 5 , Figure 6a , Figure 6b , Figure 7a and / or Figure 7b The first electronic device 802 can send facial-related information (e.g., 3D mesh information and / or one or more photos) to the first electronic device 801 via a communication link to the second user (e.g., a person wearing the second electronic device 802). Figure 1 , Figure 2a , Figure 2b , Figures 3 to 5 , Figure 6a , Figure 6b , Figure 7a and / or Figure 7b The second user (125) has facial-related information. In operation 820, information for rendering the user's face can be exchanged between the first electronic device 801 and the second electronic device 802.

[0169] In the reverse transmission mode, based on operation 832, the first electronic device 801 can use an external camera (e.g., Figure 2a and / or Figure 2b An external camera 225-2 acquires a first image. The first image can correspond to the external environment of the front surface of the first electronic device 801. In the first image, the processor of the first electronic device 801 can identify the part corresponding to the second electronic device 802.

[0170] Similarly, in operation 834, the second electronic device 802 can acquire a second image via an external camera. The second image can correspond to the external environment of the front surface of the second electronic device 802. In embodiments where the first electronic device 801 and the second electronic device 802 are arranged facing each other, the second electronic device 802 can identify or confirm the portion in the second image corresponding to the first electronic device 801.

[0171] Reference Figure 8 In operation 842, according to an embodiment, the processor of the first electronic device 801 can use an eye-tracking camera (e.g., Figure 2a and / or Figure 2b The eye-tracking camera 225-1 acquires a third image. The third image may be an image of at least one of the two eyes of the first user wearing the first electronic device 801. Based on the third image, the processor of the first electronic device 801 can acquire or calculate the direction of the first user's gaze.

[0172] Reference Figure 8 In operation 844, according to an embodiment, the processor of the second electronic device 802 can acquire a fourth image via an eye-tracking camera. The fourth image can correspond to at least one of the two eyes of a second user wearing the second electronic device 802. Based on the fourth image, the processor of the second electronic device 802 can generate or obtain information indicating the direction of the second user's gaze.

[0173] Reference Figure 8 In operation 852, according to an embodiment, the processor of the first electronic device 801 can send the third image of operation 842 to the second electronic device 802 via a communication link established based on the reverse transmission mode of operation 820. The embodiment is not limited thereto, and the first electronic device 801 can send information indicating the direction of the first user's gaze based on the third image to the second electronic device 802. In the reverse transmission mode, the first electronic device 801 can repeatedly execute operation 852. Based on the repeatedly executed operation 852, the first electronic device 801 can notify the second electronic device 802 in real time of information related to the first user's gaze. The embodiment is not limited thereto, and the processor of the first electronic device 801 can send an image representing the entire face of the first user. The image sent to the second electronic device can at least partially include the third image obtained based on operation 842.

[0174] Reference Figure 8In operation 854, according to an embodiment, the processor of the second electronic device 802 can send the fourth image from operation 844 to the first electronic device 801 via a communication link. Sending to the first electronic device 801 is not limited to sending a fourth image. For example, the processor of the second electronic device 802 can send information related to the second user's gaze obtained from the fourth image to the first electronic device 801. The second electronic device 802 can periodically (or repeatedly) perform operation 854 in a reverse transmission mode. The embodiment is not limited to this, and the processor of the second electronic device 802 can send an image representing the entire face of the second user. The image sent to the first electronic device can at least partially include the fourth image obtained based on operation 844.

[0175] The information transmitted by the first electronic device 801 and the second electronic device 802 in operations 852 and 854 may include an image of the user's entire face and an image obtained via an eye-tracking camera. For example, the first electronic device 801 may transmit an image and / or video representing the user's face covered by the first electronic device 801 to the second electronic device 802.

[0176] Reference Figure 8 In operation 862, according to an embodiment, the processor of the first electronic device 801 can synthesize the first image and the fourth image. For example, the first electronic device 801 can combine the fourth image with the portion of the first image corresponding to the second electronic device 802. When the first image and the fourth image are combined, the fourth image representing the two eyes of the second user covered by the second electronic device 802 can be arranged on the portion of the first image corresponding to the second electronic device 802. The embodiment is not limited to this, and the first electronic device 801 can be used with reference to the above. Figure 5 , Figure 6a and Figure 6b The aforementioned three-dimensional virtual objects and / or neural networks (e.g., Figure 6a and / or Figure 6b A facial image generation model 620 generates a virtual image of at least a portion of the face of the second user, including both eyes, from a fourth image. A first electronic device 801 can combine the generated virtual image with the first image.

[0177] Reference Figure 8 In operation 864, according to an embodiment, the processor of the second electronic device 802 can synthesize the second image and the third image. For example, the second electronic device 802 can perform operation 864 to reconstruct the face of the first user covered by the first electronic device 801 (e.g., a portion of the face including both eyes) in the second image related to the external environment. The processor of the second electronic device 802 can perform the above-mentioned operation. Figure 5 , Figure 6aand / or Figure 6b The described operation of the electronic device 101 involves replacing the portion of a second image covered by the first electronic device 801 with a virtual object, image, and / or video representing the face of a first user.

[0178] Reference Figure 8 In operation 872, according to an embodiment, the processor of the first electronic device 801 can be located at the display of the first electronic device 801 (e.g., Figure 2a and / or Figure 2b The result synthesized based on operation 862 is displayed on the display 220. For example, the first electronic device 801 may display a first image (e.g., an image including the external environment of the second electronic device 802) synthesized from a fourth image (e.g., an image representing the two eyes of a second user wearing the second electronic device 802). The first electronic device 801 may perform operations on image frames continuously acquired by an external camera in reverse transmission mode. Figure 8 Operations 862 and 872.

[0179] Reference Figure 8 In operation 874, according to an embodiment, the processor of the second electronic device 802 can display on a display the result synthesized based on operation 864. The second electronic device 802 can display a second image (e.g., an image corresponding to the external environment including the first electronic device 801) synthesized from the third image (e.g., an image related to the two eyes of a first user wearing the first electronic device 801). The second electronic device 802 can execute on each of the image frames continuously acquired via an external camera in reverse transmission mode. Figure 8 Operations 864 and 874.

[0180] Reference Figure 8 Each of the first electronic device 801 and the second electronic device 802 can repeatedly execute operations 832, 842, 852, 862, and 872, and operations 834, 844, 854, 864, and 874 after entering the reverse transmission mode of operation 820. Upon switching to a mode different from the reverse transmission mode, the first electronic device 801 can stop repeatedly executing operations 832, 842, 852, 862, and 872. Other modes may include, for example, modes for providing a user experience disconnected from the external environment, such as a VR mode. In response to the cessation of interaction detected based on operation 812, the first electronic device 801 can switch from the reverse transmission mode to another mode. (Refer to the above) Figure 4 The first electronic device 801 can switch from a reverse transmission mode to another mode based on the distance and / or direction between the first electronic device 801 and the second electronic device 802.

[0181] Reference Figure 1 , Figure 2a , Figure 2b , Figures 3 to 5 , Figure 6a , Figure 6b , Figure 7a , Figure 7b and Figure 8 The text describes two electronic devices, but the embodiments are not limited to this. For example, each of three electronic devices can perform the function of removing a body part covering the field of view (FoV) by using a reverse transmission mode. See also... Figure 1 , Figure 2a , Figure 2b , Figures 3 to 5 , Figure 6a , Figure 6b , Figure 7a , Figure 7b and Figure 8 The operation of the first electronic device 801 for synthesizing the face of a second user wearing the second electronic device 802 has been described, but the embodiments are not limited thereto. For example, the second electronic device 802 may send a virtual object, image, and / or video of the face of the second user wearing the second electronic device 802 to the first electronic device 801. The first electronic device 801 may display the virtual object, image, and / or video sent from the second electronic device 802 overlaid on the first image of operation 832.

[0182] In the following text, refer to Figure 9a , Figure 9b , Figure 10a and / or Figure 10b Show reference Figure 1 , Figure 2a , Figure 2b , Figures 3 to 5 , Figure 6a , Figure 6b , Figure 7a and / or Figure 7b The appearance of the described electronic device 101. Figure 9a and / or Figure 9b Wearable devices 900 and / or Figure 10a and / or Figure 10b The wearable device 1000 can be Figure 1 Example of electronic device 101.

[0183] Figure 9a An example perspective view of a wearable device according to an embodiment of the present disclosure is shown.

[0184] According to an embodiment, wearable device 900 may take the form of glasses that can be worn on a part of a user's body (e.g., the head). Wearable device 900 may include a head-mounted display (HMD). For example, the housing of wearable device 900 may include a flexible material such as rubber and / or silicone, having the form of being tightly adhered to a portion of the user's head (e.g., a portion of the face surrounding the two eyes). For example, the housing of wearable device 900 may include one or more straps capable of wrapping around the user's head and / or one or more temples that can be attached to the ears of the head.

[0185] Reference Figure 9a According to an embodiment, the wearable device 900 may include at least one display 950 and a frame supporting the at least one display 950.

[0186] Figure 9b Examples of one or more hardware components arranged in a wearable device according to embodiments of the present disclosure are shown.

[0187] According to an embodiment, the wearable device 900 can be worn on a part of a user's body. The wearable device 900 can provide the user wearing the wearable device 900 with augmented reality (AR), virtual reality (VR), or mixed reality (MR) that combines augmented reality and virtual reality. For example, the wearable device 900 can respond to... Figure 9b The user's specified gestures are obtained from motion recognition cameras 960-2 and 960-3 and displayed on at least one display 950. Figure 9b Virtual reality images provided by at least one optical device 982 and 984.

[0188] According to an embodiment, at least one display 950 can provide visual information to a user. For example, at least one display 950 may include a transparent or semi-transparent lens. At least one display 950 may include a first display 950-1 and / or a second display 950-2 spaced apart from the first display 950-1. For example, the first display 950-1 and the second display 950-2 may be arranged at positions corresponding to the user's left and right eyes, respectively.

[0189] Reference Figure 9bAt least one display 950 can provide a user with visual information transmitted from ambient light and other visual information different from the visual information through lenses included in at least one display 950. The lenses can be formed based on at least one of Fresnel lenses, pancake lenses, or multi-channel lenses. For example, at least one display 950 can include a first surface 931 and a second surface 932 opposite to the first surface 931. A display area can be formed on the second surface 932 of at least one display 950. When a user wears the wearable device 900, ambient light can be transmitted to the user by incident on the first surface 931 and passing through the second surface 932. As another example, at least one display 950 can display augmented reality images on a display area formed on the second surface 932, wherein virtual reality images provided from at least one optical device 982 and 984 are combined with a real-world screen transmitted via ambient light.

[0190] In one embodiment, at least one display 950 may include at least one waveguide 933 and 934 that diffracts light emitted from at least one optical device 982 and 984 and transmits the diffracted light to a user. At least one waveguide 933 and 934 may be formed based on at least one of glass, plastic, or polymer. Nanopatterns may be formed on at least a portion of the exterior or interior of at least one waveguide 933 and 934. The nanopatterns may be formed based on a grating structure having a polygonal and / or curved surface shape. Light incident on one end of at least one waveguide 933 and 934 can be propagated through the nanopattern to the other end of at least one waveguide 933 and 934. At least one waveguide 933 and 934 may include at least one of at least one diffractive element (e.g., a diffractive optics element (DOE), a holographic optics element (HOE)) and a reflective element (e.g., a mirror). For example, at least one waveguide 933 and 934 may be disposed in a wearable device 900 to guide the screen displayed by at least one display 950 toward the user's eyes. For example, the screen can be transmitted to the user's eyes based on total internal reflection (TIR) ​​generated in at least one waveguide 933 and 934.

[0191] Wearable device 900 can analyze objects included in real-world images collected by camera 960-4, combine virtual objects corresponding to objects among the analyzed objects that become the subject of augmented reality, and display them on at least one display 950. The virtual objects can include at least one of text and images containing various information related to the objects included in the real-world images. Wearable device 900 can analyze objects based on multiple cameras, such as stereo cameras. For object analysis, wearable device 900 can perform spatial recognition (e.g., simultaneous localization and mapping (SLAM)) by using multiple cameras and / or time-of-flight (ToF). A user wearing wearable device 900 can view the images displayed on at least one display 950.

[0192] According to an embodiment, the frame may be configured with a physical structure that allows the wearable device 900 to be worn on a user's body. According to an embodiment, the frame may be configured such that when the user wears the wearable device 900, the first display 950-1 and the second display 950-2 are positioned corresponding to the user's left and right eyes. The frame may support at least one display 950. For example, the frame may support the first display 950-1 and the second display 950-2 positioned corresponding to the user's left and right eyes.

[0193] Reference Figure 9a The frame may include a region 920, at least a portion of which contacts a part of the user's body when the user wears the wearable device 900. For example, the region 920 in contact with a part of the user's body may include a portion of the user's nose, a portion of the user's ear, and a portion of the side of the user's face that contacts the wearable device 900. According to an embodiment, the frame may include a nose pad 910 that contacts said portion of the user's body. When the wearable device 900 is worn by the user, the nose pad 910 may contact the portion of the user's nose. The frame may include a first temple 904 and a second temple 905, the second temple 905 contacting a different portion of the user's body than said portion of the user's body.

[0194] For example, the frame may include a first frame 901 surrounding at least a portion of a first display 950-1, a second frame 902 surrounding at least a portion of a second display 950-2, a bridge 903 disposed between the first frame 901 and the second frame 902, a first pad 911 disposed from one end of the bridge 903 along a portion of the edge of the first frame 901, a second pad 912 disposed from the other end of the bridge 903 along a portion of the edge of the second frame 902, a first temple 904 extending from the first frame 901 and fixed to a portion of the wearer's ear, and a second temple 905 extending from the second frame 902 and fixed to a portion of the ear opposite the ear. The first pad 911 and the second pad 912 may contact a portion of the user's nose, and the first temple 904 and the second temple 905 may contact a portion of the user's face and a portion of the user's ear. Temples 904 and 905 can be... Figure 9b Hinges 906 and 907 are rotatably connected to the edge. A first temple 904 is rotatably connected relative to a first frame 901 via a first hinge unit 906 disposed between the first frame 901 and the first temple 904. A second temple 905 is rotatably connected relative to a second frame 902 via a second hinge unit 907 disposed between the second frame 902 and the second temple 905. According to an embodiment, the wearable device 900 can identify external objects touching the frame (e.g., a user's fingertips) and / or gestures performed by external objects by using touch sensors, grip sensors, and / or proximity sensors formed on at least a portion of the surface of the frame.

[0195] According to an embodiment, the wearable device 900 may include hardware that performs various functions (e.g., based on the above). Figure 2a and / or Figure 2b The block diagram describes the hardware. For example, the hardware may include a battery module 970, an antenna module 975, at least one optical device 982 and 984, a speaker (e.g., speaker 955-1 and 955-2), a microphone (e.g., microphone 965-1, 965-2 and 965-3), a light-emitting module and / or a printed circuit board (PCB) 990 (e.g., a printed circuit board). Various hardware components may be arranged in the frame.

[0196] According to an embodiment, the microphones of the wearable device 900 (e.g., microphones 965-1, 965-2, and 965-3) can acquire sound signals by being arranged on at least a portion of the frame. Figure 9b The image shows a first microphone 965-1 positioned on the bridge of the nose 903, a second microphone 965-2 positioned on the second frame 902, and a third microphone 965-3 positioned on the first frame 901. However, the number and arrangement of the microphones 965 are not limited to these specifications. Figure 9bIn an embodiment where the number of microphones 965 included in the wearable device 900 is two or more, the wearable device 900 can identify the direction of the sound signal by using multiple microphones arranged on different parts of the frame.

[0197] According to an embodiment, at least one optical device 982 and 984 can project virtual objects onto at least one display 950 to provide various image information to a user. For example, at least one optical device 982 and 984 can be a projector. At least one optical device 982 and 984 can be arranged near at least one display 950, or can be included as part of at least one display 950. According to an embodiment, wearable device 900 can include a first optical device 982 corresponding to a first display 950-1 and a second optical device 984 corresponding to a second display 950-2. For example, at least one optical device 982 and 984 can include a first optical device 982 arranged at the edge of the first display 950-1 and a second optical device 984 arranged at the edge of the second display 950-2. The first optical device 982 can transmit light to a first waveguide 933 arranged on the first display 950-1, and the second optical device 984 can transmit light to a second waveguide 934 arranged on the second display 950-2.

[0198] In an embodiment, camera 960 may include a capturing camera 960-4, an eye-tracking camera (ET CAM) 960-1, and / or motion recognition cameras 960-2 and 960-3. The capturing camera 960-4, the eye-tracking camera (ET CAM) 960-1, and the motion recognition cameras 960-2 and 960-3 may be positioned at different locations on the frame and may perform different functions. The eye-tracking camera (ET CAM) 960-1 may output data indicating the eye position or gaze of a user wearing the wearable device 900. For example, the wearable device 900 may detect gaze from an image including the user's pupils obtained through the eye-tracking camera (ET CAM) 960-1.

[0199] Wearable device 900 can identify objects (e.g., real and / or virtual objects) focused by the user using the user's gaze obtained through an eye-tracking camera (ET CAM) 960-1. Wearable device 900, which identifies the focused object, can perform functions for interaction between the user and the focused object (e.g., gaze interaction). Wearable device 900 can represent portions corresponding to the eyes of the user's avatar in a virtual space using the user's gaze obtained through the eye-tracking camera (ET CAM) 960-1. Wearable device 900 can render an image (or screen) displayed on at least one display 950 based on the position of the user's eyes.

[0200] For example, the visual quality of a first region associated with the user's gaze in an image and the visual quality of a second region different from the first region (e.g., resolution, brightness, saturation, grayscale, and pixels per inch (PPI)) can differ from each other. Wearable device 900 can obtain an image with the visual quality of the first region and the second region matching the user's gaze by using foveal rendering. For example, if wearable device 900 supports iris recognition, user authentication can be performed based on iris information obtained using an eye-tracking camera (ET CAM) 960-1. Figure 9b An example of an eye-tracking camera (ET CAM) 960-1 being positioned toward the user's right eye is shown, but the embodiment is not limited thereto, and the eye-tracking camera (ET CAM) 960-1 may be positioned toward the user's left eye alone, or it may be positioned toward both eyes.

[0201] In an embodiment, the capturing camera 960-4 can capture a real image or background to be matched with a virtual image to enable augmented reality or mixed reality content. The capturing camera 960-4 can be used to obtain images with high resolution based on high resolution (HR) or photo-video (PV). The capturing camera 960-4 can capture images of specific objects present at the user's viewing location and can provide the images to at least one display 950. At least one display 950 can display an image in which the virtual image provided by at least one optical device 982 and 984 overlaps with information including the real image or background containing the image of the specific object obtained using the capturing camera 960-4. The wearable device 900 can compensate for depth information (e.g., the distance between the wearable device 900 and external objects obtained by a depth sensor) by using the images obtained by the capturing camera 960-4. The wearable device 900 can perform object recognition by using the images obtained by the capturing camera 960-4. The wearable device 900 can perform a function of focusing on an object (or subject) in an image (e.g., autofocus) by using a camera 960-4 and / or optical image stabilization (OIS) functions (e.g., image stabilization). The wearable device 900 can perform a transfer function to display an image acquired by the camera 960-4 that overlaps with at least a portion of a screen indicating a virtual space on at least one display 950. In an embodiment, the camera 960-4 may be mounted on a bridge of the nose 903, which is positioned between a first frame 901 and a second frame 902.

[0202] An eye-tracking camera (ET CAM) 960-1 can achieve more realistic augmented reality by tracking the gaze of a user wearing the wearable device 900 and matching the user's gaze with visual information provided on at least one display 950. For example, when the user's gaze is forward, the wearable device 900 can naturally display environmental information relevant to the user's position on at least one display 950 at the user's location. The eye-tracking camera (ET CAM) 960-1 can be configured to capture images of the user's pupils to determine the user's gaze. For example, the eye-tracking camera (ET CAM) 960-1 can receive gaze detection light reflected from the user's pupils and can track the user's gaze based on the position and movement of the received gaze detection light. In an embodiment, the eye-tracking camera (ET CAM) 960-1 can be positioned corresponding to the user's left and right eyes. For example, the eye-tracking camera (ET CAM) 960-1 can be positioned in a first edge 901 and / or a second edge 902, facing the direction in which the user wearing the wearable device 900 is located.

[0203] Motion recognition cameras 960-2 and 960-3 can provide specific events to a screen provided on at least one display 950 by recognizing the movement of the whole or part of a user's body (e.g., the user's torso, hands, or face). Motion recognition cameras 960-2 and 960-3 can obtain signals corresponding to user gestures (gesture recognition) and can provide a display corresponding to the signals to at least one display 950. A processor can recognize signals corresponding to gestures and can perform specified functions based on this recognition. Motion recognition cameras 960-2 and 960-3 can be used to perform spatial recognition functions using SLAM and / or depth maps for 6-DOF poses. A processor can perform gesture recognition and / or object tracking functions using motion recognition cameras 960-2 and 960-3. In an embodiment, motion recognition cameras 960-2 and 960-3 can be arranged on a first frame 901 and / or a second frame 902.

[0204] The camera 960 included in the wearable device 900 is not limited to the eye-tracking camera (ET CAM) 960-1 and motion recognition cameras 960-2 and 960-3 described above. For example, the wearable device 900 can identify external objects included in the FoV by using a camera set towards the user's FoV. The identification of external objects by the wearable device 900 can be performed based on sensors (such as depth sensors and / or time-of-flight (ToF) sensors) used to identify the distance between the wearable device 900 and the external object. The camera 960 set towards the FoV can support autofocus and / or optical image stabilization (OIS) functions. For example, the wearable device 900 may include a camera 960 set towards the face (e.g., a face-tracking (FT) camera) to obtain an image including the face of the user wearing the wearable device 900.

[0205] Although not shown, according to an embodiment, the wearable device 900 may also include a light source (e.g., an LED) that emits light toward an object captured by the camera 960 (e.g., the user's eyes, face, and / or an external object in FoV). The light source may include an LED having an infrared wavelength. The light source may be disposed on at least one of the frame and hinge units 906 and 907.

[0206] According to an embodiment, the battery module 970 can power the electronic components of the wearable device 900. In one embodiment, the battery module 970 may be disposed in the first temple 904 and / or the second temple 905. For example, there may be multiple battery modules 970. The multiple battery modules 970 may be respectively arranged on each of the first temple 904 and the second temple 905. In one embodiment, the battery module 970 may be disposed at the end of the first temple 904 and / or the second temple 905.

[0207] The antenna module 975 can transmit signals or power to the outside of the wearable device 900, or can receive signals or power from the outside. In an embodiment, the antenna module 975 may be disposed in the first temple 904 and / or the second temple 905. For example, the antenna module 975 may be arranged close to a surface of the first temple 904 and / or the second temple 905.

[0208] The speaker 955 can output sound signals to the outside of the wearable device 900. The sound output module may be referred to as a speaker. In an embodiment, the speaker 955 may be arranged in the first temple 904 and / or the second temple 905 so as to be adjacent to the ear of the user wearing the wearable device 900. For example, the speaker 955 may include a second speaker 955-2 arranged in the first temple 904 and adjacent to the user's left ear, and a first speaker 955-1 arranged in the second temple 905 and adjacent to the user's right ear.

[0209] The light-emitting module (not shown) may include at least one light-emitting element. The light-emitting module may emit light of a color corresponding to a specific state, or may emit light during operations corresponding to a specific state, in order to visually provide the user with information about a specific state of the wearable device 900. For example, when the wearable device 900 needs charging, it may emit red light at regular intervals. In an embodiment, the light-emitting module may be arranged on a first edge 901 and / or a second edge 902.

[0210] Reference Figure 9b According to an embodiment, the wearable device 900 may include a printed circuit board (PCB) 990. The PCB 990 may be contained in at least one of a first temple 904 or a second temple 905. The PCB 990 may include an interposer layer disposed between at least two sub-PCBs. One or more hardware components (e.g., those included in the wearable device 900) may be disposed on the PCB 990. Figure 2a and / or Figure 2b (The different boxes illustrate the hardware). The wearable device 900 may include a flexible PCB (FPCB) for interconnecting the hardware.

[0211] According to an embodiment, the wearable device 900 may include at least one of a gyroscope sensor, a gravity sensor, and / or an accelerometer sensor for detecting the posture of the wearable device 900 and / or the posture of a body part (e.g., head) of the user wearing the wearable device 900. Each of the gravity sensor and the accelerometer sensor may measure gravitational acceleration and / or acceleration based on specified 3D axes (e.g., x-axis, y-axis, and z-axis) perpendicular to each other. The gyroscope sensor may measure the angular velocity of each of the specified 3D axes (e.g., x-axis, y-axis, and z-axis). At least one of the gravity sensor, accelerometer sensor, and gyroscope sensor may be referred to as an inertial measurement unit (IMU). According to an embodiment, the wearable device 900 may, based on the IMU, identify user movements and / or gestures performed to perform or stop specific functions of the wearable device 900.

[0212] Figure 10a and Figure 10b Examples of the exterior of a wearable device according to various embodiments of this disclosure are shown.

[0213] Figure 10a and Figure 10b Wearable device 1000 may include reference Figure 9a and Figure 9b At least a portion of the hardware of the described wearable device 900. According to an embodiment, in Figure 10a An example of the exterior of the first surface 1010 of the housing of the wearable device 1000 can be shown, and... Figure 10b An example of the exterior of the second surface 1020, which is opposite to the first surface 1010, can be shown.

[0214] Reference Figure 10a According to an embodiment, the first surface 1010 of the wearable device 1000 may have an attachable form on a user's body part (e.g., the user's face). Although not shown, the wearable device 1000 may also include features for securing to the user's body part and / or one or more temples (e.g., Figure 9a and Figure 9b The first temple 904 and / or the second temple 905) have stripes. A first display 950-1 for outputting an image to the left eye of the user's eyes and a second display 950-2 for outputting an image to the right eye of the user's eyes can be provided on the first surface 1010. The wearable device 1000 may also include rubber or silicone filler formed on the first surface 1010 to prevent interference from light different from the light emitted from the first display 950-1 and the second display 950-2 (e.g., ambient light).

[0215] According to an embodiment, the wearable device 1000 may include a camera 960-1 for capturing and / or tracking the eyes of a user adjacent to each of the first display 950-1 and the second display 950-2. The camera 960-1 may be referred to as... Figure 9b The wearable device 1000 may include an eye-tracking camera 960-1. According to an embodiment, the wearable device 1000 may include cameras 960-5 and 960-6 for capturing and / or recognizing a user's face. Cameras 960-5 and 960-6 may be referred to as FT cameras. The wearable device 1000 may control an avatar representing the user in virtual space based on the movement of the user's face recognized using cameras 960-5 and 960-6. For example, the wearable device 1000 may alter the texture and / or shape of a portion of the avatar (e.g., a portion representing a face) by using information obtained by cameras 960-5 and 960-6 (e.g., FT cameras) and representing the facial expressions of the user wearing the wearable device 1000.

[0216] Reference Figure 10b Cameras (e.g., cameras 960-7, 960-8, 960-9, 960-10, 960-11, and 960-12) and / or sensors (e.g., depth sensor 1030) used to acquire information related to the external environment of the wearable device 1000 can be arranged in conjunction with... Figure 10a The first surface 1010 is on the opposite second surface 1020. For example, cameras 960-7, 960-8, 960-9 and 960-10 can be arranged on the second surface 1020 to identify external objects. Figure 10b The cameras 960-7, 960-8, 960-9, and 960-10 can correspond to Figure 9b The motion recognition cameras 960-2 and 960-3.

[0217] For example, using cameras 960-11 and 960-12, wearable device 1000 can acquire images and / or videos to be sent to each of the user's eyes. Camera 960-11 can be arranged on the second surface 1020 of wearable device 1000 to acquire an image to be displayed on a second display 950-2 corresponding to the right eye. Camera 960-12 can be arranged on the second surface 1020 of wearable device 1000 to acquire an image to be displayed on a first display 950-1 corresponding to the left eye above the eyes. Cameras 960-11 and 960-12 can correspond to... Figure 9b The camera used for taking pictures is the 960-4.

[0218] According to an embodiment, the wearable device 1000 may include a depth sensor 1030 disposed on a second surface 1020 to identify the distance between the wearable device 1000 and an external object. Using the depth sensor 1030, the wearable device 1000 can obtain spatial information (e.g., a depth map) of at least a portion of the FoV of the user wearing the wearable device 1000. Although not shown, a microphone for obtaining sound output from an external object may be disposed on the second surface 1020 of the wearable device 1000. According to an embodiment, the number of microphones may be one or more.

[0219] The metaverse is a combination of the English words "Meta" (meaning "virtual" and "transcendent") and "universe" (meaning space), referring to a three-dimensional virtual world in which social, economic, and cultural activities, including those in the real world, take place. The metaverse is a more advanced concept than virtual reality (VR, the most advanced technology that allows people to experience real life in computer-created virtual worlds), characterized by the ability to use avatars not only to enjoy games or virtual reality but also to engage in real-world social and cultural activities. Based on augmented reality (AR), virtual reality environments (VR), mixed environments (MR), and / or extended reality (XR), metaverse services can provide media content to enhance immersion in the virtual world.

[0220] For example, media content provided by Metaverse services can include socially interactive content, such as avatar-based games, concerts, parties, and / or conferences. Media content can also include information for economic activities, such as advertising, user-created content, and / or the sale and / or purchase of products. Ownership of user-created content can be proven using blockchain-based non-fungible tokens (NFTs). Metaverse services can support economic activities based on real-world currencies and / or cryptocurrencies. Through Metaverse services, virtual content linked to the real world can be provided, such as digital twins or lifestyle logs.

[0221] Figure 11 This is a diagram of a network environment 1101 that receives metaverse services through server 1110 according to an embodiment of the present disclosure.

[0222] Reference Figure 11Network environment 1101 may include server 1110, user terminals 1120 (e.g., first terminal 1120-1 and second terminal 1120-2), and a network connecting server 1110 and user terminals 1120. In network environment 1101, server 1110 may provide metaverse services to user terminals 1120. The network may be formed by at least one intermediate node 1130 including access points (APs) and / or base stations. User terminals 1120 can output a user interface (UI) related to metaverse services to their users by accessing server 1110 via the network. Based on the UI, user terminals 1120 can obtain information from users to input into metaverse services, or can output information related to metaverse services (e.g., multimedia content) to users.

[0223] In this configuration, server 1110 provides a virtual space, allowing user terminal 1120 to perform activities within it. Furthermore, user terminal 1120 communicates information provided by server 1110 to the user, or sends information the user wishes to communicate within the virtual space to the server, by installing a service proxy for accessing the virtual space provided by server 1110. The service proxy can be provided directly by server 1110, downloaded from a public server, or embedded when the terminal is purchased.

[0224] In this embodiment, the metaverse service can be provided to user terminal 1120 and / or users using server 1110. The embodiment is not limited to this, and the metaverse service can be provided through personal connections between users. For example, in network environment 1101, the metaverse service can be provided independently of server 1110 via a direct connection between first terminal 1120-1 and second terminal 1120-2. (See also...) Figure 11 In network environment 1101, the first terminal 1120-1 and the second terminal 1120-2 can be connected to each other through a network formed by at least one intermediate node 1130. In embodiments where the first terminal 1120-1 and the second terminal 1120-2 are directly connected, either the first terminal 1120-1 or the second terminal 1120-2 can perform the role of server 1110. For example, the metaverse environment can be configured solely by device-to-device connections (e.g., peer-to-peer (P2P) connections).

[0225] In an embodiment, user terminal 1120 (or user terminal 1120 including first terminal 1120-1 and second terminal 1120-2) can be made of various shape factors and is characterized by including output devices for providing images and / or sound to the user and input devices for inputting information into a metaverse service. For example, various shape factors of user terminal 1120 may include a smartphone (e.g., second terminal 1120-2), an AR device (e.g., first terminal 1120-1), a VR device, an MR device, a video see-through (VST) device, an optical see-through (OST) device, a smart lens, a smart mirror, a television (TV), or a projector capable of input and output.

[0226] The network (e.g., a network formed by at least one intermediate node 1130) includes all kinds of broadband networks, including third-generation (3G), fourth-generation (4G) and fifth-generation (5G), as well as short-range networks including WiFi and BT (e.g., wired or wireless networks that directly connect the first terminal 1120-1 and the second terminal 1120-2). Figure 11 User terminal 1120 may include reference Figure 1 , Figure 2a , Figure 2b , Figures 3 to 5 , Figure 6a , Figure 6b , Figure 7a and / or Figure 7b HMD device 101 as described.

[0227] In embodiments, a method may be needed to display the appearance of a screen including body parts (e.g., a user's face) covered by a wearable device such as an HMD device. As described above, according to embodiments, head-mounted display (HMD) devices (e.g., Figure 1 , Figure 2a , Figure 2b , Figures 3 to 5 , Figure 6a , Figure 6b , Figure 7a and / or Figure 7b HMD device 101, Figure 8 The first HMD device 801 and / or the second HMD device 802 may include a camera (e.g., Figure 2a and / or Figure 2b Camera 225), communication circuits (e.g., Figure 2a and / or Figure 2b The communication circuit 235), and at least one display (e.g., Figure 2a and / or Figure 2b The display 220), memory storing one or more computer programs, and one or more processors (e.g., Figure 2a and / or Figure 2b The processor 210). One or more computer programs may include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to control the camera to acquire a first image outside the HMD device (e.g., ...). Figure 4 , 5 Images 410 of 6a and 6b). One or more computer programs may include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to detect external HMD devices (e.g., Figure 1 , Figure 2a , Figure 2b , Figures 3 to 5 , Figure 6a , Figure 6b , Figure 7a and / or Figure 7b A portion of the first image corresponding to the external HMD device 120 (e.g., Figure 1 Part 142). One or more computer programs may include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to receive information about body parts covered by the external HMD device via communication circuitry (e.g., Figure 1 (Information 150). One or more computer programs may include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to obtain a second image representing a body part by performing rendering relative to the body part based on information. One or more computer programs may include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to display, via at least one display, a composite image including a first image and a second image, wherein the second image is located at that portion of the first image. According to an embodiment, the HMD device may display images and / or videos representing the appearance of a body part covered by an external HMD device.

[0228] For example, one or more computer programs may also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to generate a composite image by overlaying a second image onto a portion of a first image.

[0229] For example, one or more computer programs may further include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to detect interaction between the HMD device and the external HMD device based on whether the orientation of the external HMD device, determined by using the first image, is toward the HMD device.

[0230] For example, one or more computer programs may also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to detect interactions between the HMD device and the external HMD device based on a communication link established between the HMD device and the external HMD device.

[0231] For example, one or more computer programs also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to obtain a second image representing the eyes of a user wearing the external HMD device by performing rendering based on a gaze direction indicated by information. Information about body parts can indicate the direction of the user's eyes while wearing the external HMD device.

[0232] For example, one or more computer programs may also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to obtain a second image by inputting information and the first image into a neural network for rendering.

[0233] For example, one or more computer programs further include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to modify a portion of the three-dimensional mesh corresponding to a body part based on the information received from an external HMD device via communication circuitry.

[0234] For example, one or more computer programs also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to determine the size of the second image based on at least one of the distance from the external HMD device or the orientation of the external HMD device relative to the HMD device.

[0235] For example, one or more computer programs also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to control the camera by using a period based on at least one of the distance from the external HMD device or the orientation of the external HMD device relative to the HMD device, thereby acquiring a first image.

[0236] For example, one or more computer programs may also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to change the period at which it receives information from the external HMD device based on at least one of the distance from the external HMD device or the orientation of the external HMD device relative to the HMD device.

[0237] For example, an HMD device may include a second camera communicatively coupled to one or more processors. Information about body parts may be the first information. One or more computer programs may also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to obtain second information about body parts of a user wearing the HMD device (e.g., the user's gaze direction) based on a third image from the second camera. One or more computer programs may also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to transmit the second information to an external HMD device via communication circuitry.

[0238] As described above, according to embodiments, a method is provided performed by a head-mounted display (HMD) device including a camera, communication circuitry, at least one display, and one or more processors. The method may include controlling the camera to obtain a first image of the exterior of the HMD device. The method may include detecting a portion of the first image corresponding to the exterior HMD device. The method may include receiving information about a body part covered by the exterior HMD device via the communication circuitry. The method may include obtaining a second image representing the body part by performing rendering relative to the body part based on the information. The method may include displaying a composite image including the first image and the second image via at least one display, wherein the second image is located on the portion of the first image.

[0239] For example, the method may include generating a composite image by overlaying a second image onto that portion of a first image.

[0240] For example, controlling the camera may include detecting the interaction between the HMD device and the external HMD device based on whether the orientation of the external HMD device, determined by using a first image, is toward the HMD device.

[0241] For example, controlling the camera may include detecting interactions between the HMD device and the external HMD device based on a communication link established between the HMD device and the external HMD device.

[0242] For example, obtaining the second image may include: performing rendering based on the gaze direction indicated by information to obtain a second image representing the eyes of a user wearing an external HMD device. Information about body parts can indicate the direction of the user's eyes while wearing the external HMD device.

[0243] For example, obtaining a second image may include: obtaining a second image by inputting information and a first image into a neural network for rendering.

[0244] For example, obtaining a second image may include: modifying the portion of a 3D mesh corresponding to a body part based on information received from an external HMD device via a communication circuit.

[0245] For example, acquiring the second image may include determining the size of the second image based on at least one of the distance from the external HMD device or the orientation of the external HMD device relative to the HMD device.

[0246] For example, control may include using a period based on at least one of the distance from the external HMD device or the direction of the external HMD device relative to the HMD device to control the camera to obtain a first image.

[0247] For example, receiving may include changing the period of receiving information from the external HMD device based on at least one of the distance from the external HMD device or the orientation of the external HMD device relative to the HMD device.

[0248] For example, the method may include obtaining second information about body parts of the user wearing the HMD device (e.g., the user's gaze direction) based on a third image from a second camera of the HMD device. The method may also include transmitting the second information to an external HMD device via a communication circuit.

[0249] As described above, according to an embodiment, an electronic device may include a camera, communication circuitry, at least one display, and a processor. The processor may be configured to detect a portion corresponding to a head-mounted display (HMD) device in video acquired via the camera. The processor may be configured to obtain a virtual object representing a body part covered by the HMD device based on information received from the HMD device via the communication circuitry. The processor may be configured to display at least a portion of the virtual object on at least a portion of the detected portion on at least one display while simultaneously displaying the video on at least one display.

[0250] For example, the processor can be configured to display at least a portion of a virtual object based on detected interaction between an electronic device and an HMD device while simultaneously displaying video on at least one display based on video perspective (VST).

[0251] For example, the processor can be configured to display, on at least one display, at least a portion of a virtual object representing a head wearing an HMD device, in which eyes with directions indicated by information are arranged.

[0252] As described above, according to embodiments, a method for an electronic device including a camera, communication circuitry, at least one display, and a processor may include detecting a portion corresponding to a head-mounted display (HMD) device in video acquired by the camera. The method may include obtaining a virtual object representing a body part covered by the HMD device based on information received from the HMD device via the communication circuitry. The method may include displaying at least a portion of the virtual object on at least a portion of the detected portion on at least one display while simultaneously displaying the video on at least one display.

[0253] For example, the display may include: displaying at least a portion of a virtual object based on the detection of interaction between an electronic device and an HMD device while simultaneously displaying video on at least one display based on video perspective (VST).

[0254] For example, the display may include displaying at least a portion of the eyes arranged with directions indicated by information in a virtual object representing a head wearing an HMD device on at least one display.

[0255] As described above, according to an embodiment, a non-transitory computer-readable storage medium may include instructions. When executed by a processor of an electronic device, the instructions can cause the electronic device to perform the operations described above for the electronic device and / or HMD device.

[0256] The devices and components described in this disclosure can be implemented using one or more general-purpose or special-purpose computers, such as processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field-programmable gate arrays (FPGAs), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The processing device can execute an operating system (OS) and one or more software applications running on that operating system. Furthermore, the processing device can access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, there are cases where a single processing device is described, but those skilled in the art will recognize that a processing device can include multiple processing elements and / or various types of processing elements. For example, a processing device can include multiple processors or one processor and one controller. Furthermore, another processing configuration, such as a parallel processor, is also possible.

[0257] Software may include computer programs, code, instructions, or combinations thereof, and may configure a processing device to operate as needed or to command the processing device independently or jointly. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or apparatus to be interpreted by or to provide commands or data to the processing device. Software may be distributed across network-connected computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0258] The method according to the embodiments can be implemented in the form of program commands, which can be executed by various computer devices and recorded on a computer-readable medium. In this case, the medium can continuously store a computer-executable program, or it can temporarily store a program for execution or download. Furthermore, the medium can be various recording or storage means in the form of a combination of single or several hardware components, but is not limited to media directly connected to a computer system, and can be distributed across a network. Examples of media can include those configured to store program instructions, including magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as optical disc read-only memories (CD-ROMs) and digital versatile optical discs (DVDs); magneto-optical media such as optical floppy disks; and ROM, RAM, flash memory, etc. Additionally, other examples of media can include recording or storage media managed by application stores that distribute applications, sites that supply or distribute various software, servers, etc.

[0259] As described above, although embodiments have been described with limited examples and figures, those skilled in the art will be able to make various modifications and variations from the above description. For example, suitable results may be achieved even if the described techniques are performed in a different order than the described methods, and / or the components of the described systems, structures, devices, circuits, etc., are coupled or combined in a different form than the described methods, or are replaced or substituted by other components or equivalents.

[0260] Although this disclosure has been shown and described with reference to various embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the scope of this disclosure as defined by the appended claims and their equivalents.

Claims

1. A head-mounted display (HMD) device, comprising: camera; Communication circuits; At least one display; A memory that stores one or more computer programs; as well as One or more processors are communicatively coupled to a camera, communication circuitry, at least one display, and memory. One or more computer programs include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to: Control the camera to obtain the first image of the exterior of the HMD device. Detect the portion of the first image that corresponds to the external HMD device. Information about body parts covered by an external HMD device is received via a communication circuit. A second image representing the body part is obtained by performing rendering relative to the body part based on the information, and A composite image comprising a first image and a second image is displayed via at least one display, wherein the second image is located at said portion of the first image.

2. The HMD device according to claim 1, wherein, One or more computer programs also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to: A composite image is generated by overlaying a second image onto a portion of a first image.

3. The HMD device according to claim 1, wherein, One or more computer programs also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to: The interaction between the HMD device and the external HMD device is detected based on whether the orientation of the external HMD device, determined by using the first image, is facing the HMD device.

4. The HMD device according to claim 1, wherein, One or more computer programs also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to: Based on the communication link established between the HMD device and the external HMD device, the interaction between the HMD device and the external HMD device is detected.

5. The HMD device according to claim 1, in, One or more computer programs also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to: A second image representing the eyes of a user wearing an external HMD device is obtained by rendering based on the gaze direction indicated by the information. The information about the body parts indicates the direction of the user's eyes when wearing an external HMD device.

6. The HMD device according to claim 1, wherein, One or more computer programs also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to: A second image is obtained by inputting information and a first image into a neural network used for rendering.

7. The HMD device according to claim 1, wherein, One or more computer programs also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to: In a 3D mesh received from an external HMD device via a communication circuit, the portion of the 3D mesh corresponding to the body part is changed based on the information received.

8. The HMD device according to claim 1, wherein, One or more computer programs also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to: The size of the second image is determined based on at least one of the distance from the external HMD device or the orientation of the external HMD device relative to the HMD device.

9. The HMD device according to claim 1, wherein, One or more computer programs also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to: The camera is controlled to obtain the first image by using a cycle based on at least one of the distance from the external HMD device or the direction of the external HMD device relative to the HMD device.

10. The HMD device according to claim 1, wherein, One or more computer programs also include computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to: The period for receiving information from the external HMD device is changed based on at least one of the distance from the external HMD device or the orientation of the external HMD device relative to the HMD device.

11. The HMD device according to claim 1, further comprising: The second camera is communicatively coupled to one or more processors. Among them, the information about the body parts is the first information, and The computer program further includes computer-executable instructions that, when executed individually or jointly by one or more processors, cause the HMD device to: Based on the third image from the second camera, second information about the body parts of the user wearing the HMD device is obtained, and The second message is sent to an external HMD device via a communication circuit.

12. A method performed by a head-mounted display (HMD) device, the HMD device including a camera, communication circuitry, at least one display, and one or more processors, the method comprising: The camera is controlled by the HMD device to obtain the first image of the outside of the HMD device; The HMD device detects the portion of the first image that corresponds to the external HMD device; The HMD device receives information about the body parts covered by the external HMD device via a communication circuit. The HMD device performs rendering relative to the body part based on the information to obtain a second image representing the body part; as well as A composite image comprising a first image and a second image is displayed by an HMD device via at least one display, wherein the second image is located on said portion of the first image.

13. The method of claim 12, further comprising: A composite image is generated by overlaying a second image onto a portion of a first image.

14. The method according to claim 12, wherein, Controlling the camera includes: The interaction between the HMD device and the external HMD device is detected based on whether the orientation of the external HMD device, determined by using the first image, is facing the HMD device.

15. The method according to claim 12, wherein, Controlling the camera includes: Based on the communication link established between the HMD device and the external HMD device, the interaction between the HMD device and the external HMD device is detected.

Images