Hmd device for supporting subject photographing

Abstract

The electronic device wearable on the face of a user comprises: a first display disposed to illuminate the eyes of the user located within the electronic device when the electronic device is worn on the face; a first camera disposed on the front surface of the electronic device and configured to photograph the outside; a sensor configured to generate data regarding the direction in which the first camera faces; a communication circuit for communicating with an external electronic device; a memory storing instructions; and a processor. The instructions, when executed by the processor, may instruct the electronic device to: acquire, by means of the sensor, information indicating the direction in which the first camera faces; receive, from the external electronic device by means of the communication circuit, information regarding the direction in which a second camera included in the external electronic device faces and an angle of view of the second camera; determine, on the basis of the angle of view of the first camera, the angle of view of the second camera, the angle of view of the first display, the direction in which the first camera faces, and the direction in which the second camera faces, an overlapping area in the first display that overlaps a visible area entering a field of view of the second camera; receive an image from the external electronic device by means of the communication circuit; and display the image in the overlapping area.

Description

HMD device that supports subject photography

[0001] The present disclosure relates to a head mounted display (HMD) device that supports photographing a subject.

[0002] A head mounted display (HMD) device can implement augmented reality (AR) and / or virtual reality (VR). One type of HMD device is a so-called AR headset (or, optical see-through (OST) device). When worn, the AR headset can be configured to allow external light to reach the user's eyes through the glasses. One type of HMD device is a so-called VR headset (or, video see-through (VST) device). When worn, the VR headset can be configured to allow light emitted from the display to reach the user's eyes, but block external light so that external light does not reach the user's eyes. AR headsets and VR headsets can be collectively referred to as mixed reality (MR) devices or extended reality (XR) devices.

[0003] The above information is provided as background information to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] An HMD device may include a camera. This camera may be referred to as a see-through camera (STC). For example, a VR headset may offer a VR mode and a see-through (ST) mode. In ST mode, images captured by the STC may be transmitted in real time to the HMD device's display. This allows the user to observe the surroundings while wearing the HMD.

[0005] The angle of view (AOV) of the STC can be aligned with the angle of view of the display in the HMD device. Here, the angle of view in the STC may refer to an angle that can be captured through a camera lens. In the display, the angle of view may refer to an angle of the field of view (FOV) that can be seen by the human eye when the area (display area) where visual information is displayed on the display is incident on the eye of the person wearing the headset. Depending on the alignment of the two angles of view, at least a portion of the display area of ​​the display can be used as a view finder.

[0006] The HMD device allows the user to find a desired subject (region of interest) through the display area (viewfinder) and then captures the found subject using STC and provides the captured image to the user. When capturing a subject using STC, the field of view is focused on the display area (viewfinder). This may result in a narrow field of view. This narrowing of the field of view can directly lead to stability issues. For example, it may be difficult to designate and confirm a region of interest while the user is actively engaged in an activity. Consequently, when capturing a subject using STC during an activity, an area undesired by the user may be designated and captured.

[0007] The image quality of images captured by an HMD device depends on the performance of the camera mounted on the device. Cameras mounted on HMDs can have a wide field of view (e.g., over 100 degrees). Consequently, this limits the ability to capture detailed information about the subject (area of ​​interest) being captured.

[0008] According to various embodiments of the present disclosure, an HMD device can be provided that can precisely designate a region of interest and support capturing high-quality images in the designated region of interest. The technical challenges addressed by the present disclosure are not limited to the technical challenges mentioned above, and other technical challenges not mentioned will be readily apparent to those skilled in the art, based on the description below.

[0009] According to one embodiment, a wearable electronic device on a user's face includes a first display arranged to shine light into the user's eyes located inside the electronic device when the electronic device is worn on the face; a first camera arranged on a front surface of the electronic device and configured to photograph the outside; a sensor arranged to generate data regarding a direction in which the first camera faces; a communication circuit for communicating with an external electronic device; a memory for storing instructions; and at least one processor. The above instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain information indicating a direction in which the first camera faces through the sensor, receive information regarding a direction in which a second camera included in the external electronic device faces and an angle of view of the second camera from the external electronic device through the communication circuit, determine an overlapping area that overlaps a visible area within the field of view of the second camera on the first display based on the angle of view of the first camera, the angle of view of the second camera, the angle of view of the first display, the direction in which the first camera faces, and the direction in which the second camera faces, receive an image from the external electronic device through the communication circuit, and display the image in the overlapping area.

[0010] According to one embodiment, a method for operating an electronic device wearable on a user's face may include: acquiring information indicating a direction in which a first camera of the electronic device faces through a sensor of the electronic device; receiving information regarding a direction in which a second camera included in an external electronic device faces and an angle of view of the second camera from the external electronic device; determining an overlapping area that overlaps a visible area within a field of view of the second camera on the first display based on the angle of view of the first camera, the angle of view of the second camera, the angle of view of a first display of the electronic device, the direction in which the first camera faces, and the direction in which the second camera faces; receiving an image from the external electronic device; and displaying the image in the overlapping area.

[0011] According to one embodiment, a recording medium is provided that stores instructions readable by at least one processor in an electronic device wearable on a user's face. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to perform the following operations: obtaining information indicating a direction in which the first camera faces through the sensor; receiving information regarding a direction in which a second camera included in the external electronic device faces and an angle of view of the second camera from the external electronic device through the communication circuit; determining an overlapping area that overlaps a visible area within the field of view of the second camera on the first display based on the angle of view of the first camera, the angle of view of the second camera, the angle of view of the first display, the direction in which the first camera faces, and the direction in which the second camera faces; receiving an image from the external electronic device through the communication circuit; and displaying the image in the overlapping area.

[0012] According to an embodiment of the present disclosure, a wearable electronic device on a user's face can, when worn, capture a desired subject using a camera of an external electronic device. In addition, various effects, directly or indirectly identified through this document, may be provided.

[0013] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0014] FIG. 2 is a block diagram illustrating a camera module according to various embodiments.

[0015] FIG. 3 is a diagram illustrating an electronic device wearable on a user's face, according to one embodiment.

[0016] FIGS. 4A and 4B are perspective views schematically illustrating the front and back of an electronic device wearable on a user's face, according to one embodiment of the present disclosure.

[0017] FIGS. 5a, 5b, 5c, and 5d are diagrams illustrating various examples of camera angles and camera facing directions in a VR headset and an external electronic device according to one embodiment.

[0018] FIGS. 6A, 6B, and 6C are diagrams illustrating various examples of camera angles and camera facing directions in an AR headset and an external electronic device according to one embodiment.

[0019] FIG. 7A is a diagram illustrating an example of a field of view of a display in a VR headset according to one embodiment. FIG. 7B is a diagram illustrating an example of a field of view of a display in an AR headset according to one embodiment.

[0020] FIGS. 8A, 8B, and 8C are diagrams illustrating various examples of a viewable area and an overlapping area determined by a field of view and direction in a VR headset and an external electronic device according to one embodiment.

[0021] FIGS. 9a, 9b, 9c, and 9d are diagrams illustrating various examples of a viewable area and an overlapping area determined by a field of view and direction in an AR headset and an external electronic device according to one embodiment.

[0022] FIGS. 10a, 10b, 10c, 10d, and 10e are diagrams illustrating various examples of overlapping areas in a display of a VR headset according to one embodiment.

[0023] FIGS. 11a, 11b, 11c, and 11d are diagrams illustrating various examples of overlapping areas in a display of an AR headset according to one embodiment.

[0024] FIG. 12 is a block diagram of a first electronic device and a second electronic device according to one embodiment.

[0025] FIG. 13 is a flowchart illustrating operations performed in the first electronic device (1201) of FIG. 12, which is wearable on a user's face, according to one embodiment.

[0026] FIG. 14 is a flowchart illustrating operations performed in the first electronic device of FIG. 12, which is wearable on a user's face, according to one embodiment.

[0027] FIGS. 15a, 15b, and 15c are diagrams showing examples of images captured by an external electronic device being displayed on a display of a VR headset according to one embodiment.

[0028] FIG. 16 is a diagram showing an example of an image captured by an external electronic device being displayed on a display of an AR headset according to one embodiment.

[0029] FIG. 17 is a flowchart illustrating operations performed in the first electronic device of FIG. 12, which is wearable on a user's face, according to one embodiment.

[0030] FIG. 18 is a flowchart illustrating operations performed in the first electronic device of FIG. 12, which is wearable on a user's face, according to one embodiment.

[0031] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0032] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0033] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0034] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0035] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0036] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0037] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0038] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0039] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0040] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0041] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0042] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0043] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0044] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0045] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0046] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0047] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0048] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0049] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0050] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0051] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0052] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0053] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0054] FIG. 2 is a block diagram (200) illustrating a camera module (180) according to various embodiments. Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

[0055] The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

[0056] The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect the movement of the camera module (180) or the electronic device (101) by using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display module (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

[0057] The image signal processor (260) can perform one or more image processing operations on an image acquired through the image sensor (230) or an image stored in the memory (250). The one or more image processing operations can include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) can perform control (e.g., exposure time control, readout timing control, etc.) on at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display module (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may be configured as at least a part of the processor (120) or may be configured as a separate processor that operates independently of the processor (120). When the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display module (160) by the processor (120) as is or after undergoing additional image processing.

[0058] According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

[0059] According to various embodiments of the present disclosure, the above-described components of the electronic device (101) can be applied to an HMD device that implements AR and / or VR. The HMD device can be referred to as an AR headset or a VR headset depending on whether it blocks external light. The VR headset can implement a VR mode that blocks external light and displays a virtual image through a display, and an ST (see through) mode that displays an external real image captured using a camera through a display. Here, the camera can include a camera mounted on an external electronic device (e.g., a smartphone, a digital camera) and / or a camera mounted on the HMD device. In the ST mode, only a real image can be provided, or a real image and a virtual image can be provided together.

[0060] In this document, 'inside' may refer to the inner space of the HMD device (301a, referring to FIG. 3, described below) where the user's eyes are located when the HMD device is worn on the user's face. 'Outside' may refer to the outer space of the HMD device (301b, referring to FIG. 3, described below) when the HMD device is worn on the user's face. In this document, 'front' may refer to the front of the display formed on the outside when the HMD device is worn on the user's face (e.g., the side facing the -y-axis direction, referring to FIGS. 3 and 4a, described below). 'Side' may refer to the side of the display formed on the outside when the HMD device is worn on the user's face. 'Rear' may refer to the side facing the user's eyes when the HMD device is worn on the user's face (e.g., the side facing the y-axis direction, referring to FIGS. 3 and 4b, described below).

[0061] FIG. 3 is a diagram illustrating an electronic device (300) wearable on a user's face according to one embodiment. Referring to FIG. 3, the electronic device (300) (e.g., the electronic device (101) of FIG. 1) may be classified as an AR headset that, when worn, allows external light to reach the user's eyes through the glasses. The electronic device (300) may include glasses (310), a display module (320), a first camera (or a first front camera) (325), a second camera (or a second front camera) (330), a third camera (or a rear camera) (340), an LED light (350), a printed circuit board unit (PCB) (360), a battery (370), a speaker (380), and a microphone (390).

[0062] The glass (310) may include a first glass (311) (e.g., glass for the right eye) and a second glass (312) (e.g., glass for the left eye). For example, the glass (310) may be positioned on the front of the display module (320) to protect the display module (320). For example, the first glass (311) and / or the second glass (312) may be formed of a glass plate or a polymer, and may be manufactured to be transparent or translucent. According to one embodiment, the first glass (311) and the second glass (312) may be connected to form an integral body. For example, the glass (310) may control the transmission of external light incident on the display module (320).

[0063] A display module (320) (e.g., display module (160)) may include a first display (321) for the right eye, a second display (322) for the left eye, and a frame (e.g., a glasses frame) (355) for fixing the displays (311, 322). The first display (321) and the second display (322) may each include a display panel, a projection lens, combiner optics (or waveguide) (366), and an optical barrier (e.g., a tube). For example, the first display (321) and the second display (322) may be composed of substantially the same elements. Therefore, when describing the components and operating methods of the displays (321, 322), a single display may be described without distinction between the right eye and the left eye.

[0064] The display (321, 322) may include a liquid crystal on silicon (LCoS), a light emitting diode (LED) on silicon (LEDoS), an organic light emitting diode (OLED), a micro light emitting diode (micro LED), or a digital mirror device (DMD). When the display (321, 322) is formed of either a digital mirror display or a silicon liquid crystal display, the electronic device (300) may include a light source that irradiates light to a screen output area of ​​the display. For example, when the display (321, 322) is formed of either an organic light emitting diode or a micro LED that can generate light on its own, it may provide a good quality AR image to the user even without including a separate light source. According to one embodiment, if the display (321, 322) is implemented with an organic light emitting diode or a micro LED, a light source is unnecessary, and thus the electronic device (300) may be lightweight. The user can use the electronic device (300) while wearing it on his or her face.

[0065] The first camera (325) may be positioned on the front of the display module (320) with its lens facing the outside (301b). For example, the first camera (325) may be positioned at the exact center of the frame (355). The first camera (325) may photograph a subject, generate an image corresponding to the subject, and provide the image to a processor (e.g., processor (120)) of the electronic device (300). The processor may display the provided image on a display (321, 322). The first camera (325) may also be referred to as HR (high resolution) or PV (photo video) and may include a high-resolution camera. The first camera (325) may include a color camera equipped with functions for obtaining high-quality images, such as an AF (auto focus) function and an optical image stabilizer (OIS). Not limited thereto, the first camera (325) may include a GS (global shutter) camera or an RS (rolling shutter) camera.

[0066] The first camera (325) can be used as a sensor for measuring the brightness (illuminance) around the electronic device (300). According to one embodiment, the processor (e.g., processor (120)) of the electronic device (300) can obtain an exposure value representing the time for which the light-receiving unit (e.g., image sensor (230)) of the first camera (325) is exposed to light from the settings of the first camera (325) for capturing an image. External light can be collected by the light-receiving unit of the first camera (325) through a lens (e.g., lens assembly (210)) of the first camera (325). The light collected by the light-receiving unit is converted into an electrical analog signal by the light-receiving unit, and the analog signal can be converted into a digital signal by an analog-to-digital converter (ADC). The processor can obtain a brightness value representing the brightness of the light collected by the light-receiving unit from the digital signal. The processor can obtain an illuminance value representing the brightness around the electronic device (300) based on the exposure value and the brightness value.

[0067] The second camera (330) may be positioned on the side of the display module (320) with its lens facing the outside (301b). The second camera (330) may photograph a subject, generate an image corresponding to the subject, and provide the image to a processor (e.g., processor (120)) of the electronic device (300). The second camera (330) may be configured as a stereo camera to support at least one of head tracking, hand detection, and spatial recognition. For example, the second camera (330) may include a right-eye camera (331) located on the right side with respect to the user's eyes when worn, and a left-eye camera (332) located on the left side. The electronic device (300) may obtain information indicating depth of field (e.g., 3DOF (degrees of freedom) or 6DOF) by using an image obtained from at least one of the left-eye camera (331) and the right-eye camera (332). The electronic device (300) can perform various functions such as head tracking, hand tracking, hand gesture recognition, object location recognition, or space recognition using depth of field. The second camera (330) may include, for example, a global shutter (GS) camera or a rolling shutter (RS) camera. Bluetooth (e.g., Bluetooth low energy (BLE)) may be used as an auxiliary means for recognizing the location of an object. For example, the electronic device (300) may include a Bluetooth circuit for short-range wireless communication with an external electronic device. The electronic device (300) can determine the location of the external electronic device using the signal strength of an RF signal received from the external electronic device (e.g., received signal strength indicator (RSSI)) and an image acquired from the second camera (330). The second camera (330) may replace the first camera (325).Accordingly, the first camera (325) may be omitted from the electronic device (300). For example, the electronic device (300) may measure the ambient illuminance using an image received from the second camera (330).

[0068] The third camera (340) may be positioned on the display module (320) such that its lens faces inward (301a) (e.g., in the y-axis direction as shown in FIG. 3). The third camera (340) may be used as a sensor for detecting and tracking pupils (ET). For example, a flash (e.g., the flash (220) of FIG. 2) may be positioned on the display module (320) such that light (e.g., infrared) emitted from the flash faces inward. The third camera (340) may detect and track pupils using light (e.g., infrared) reflected from a subject (pupil). The third camera (340) may include a first internal camera (341) disposed on the first display (321) and used for detecting and tracking the right eye, and a second internal camera (342) disposed on the second display (322) and used for detecting and tracking the left eye. A processor (e.g., processor (120)) of the electronic device (300) may receive an image from the third camera (340) and detect the position and size of the pupil in the image. The processor may track the position of the pupil in the image received from the third camera (340) so that the center of the image displayed on the display (321, 322) is located on the side where the user is looking.

[0069] The above-described cameras (325, 331, 332, 341, 342) may each include components of the camera module (180) of FIG. 2.

[0070] The processor can use the third camera (340) to detect the size (e.g., diameter) of the pupil and, based on the detected size of the pupil, adjust the brightness (luminance) of the display (321, 322). The size of the pupil changes depending on the illumination. In a high-illumination environment, the size of the pupil is small, and in a low-illumination environment, the size of the pupil is relatively large so that the optic nerve receives more light. When the size of the pupil is small, the processor can increase the brightness of the display (321, 322) to improve visibility, and when the size of the pupil is large, the processor can decrease the brightness to prevent glare.

[0071] The LED light (350) may be attached to the electronic device (300) (e.g., on both sides of the frame (355). The LED light (350) may emit infrared wavelengths to facilitate detection of the pupil when capturing the user's pupil with the third camera (340). In one embodiment, the LED light (350) may also be used as a means to supplement ambient brightness when capturing the surroundings of the electronic device (300) with the second camera (330).

[0072] The display module (320) may include a first display driving unit (323) for driving a right-eye display (321) and a second display driving unit (324) for driving a left-eye display (322).

[0073] The PCB (360) may be placed on the leg portions (302, 303) of the electronic device (300). The PCB (360) may include a first PCB (361) placed on the right leg (302) and a second PCB (362) placed on the left leg (303). For example, the PCB (360) may include at least one control unit (e.g., a processor (120)) and a memory (e.g., a memory (130)) for controlling the glass (310), the display module (320), the first camera (325), the second camera (330), the third camera (340), the LED light (350), the speaker (380), and the microphone (390).

[0074] The battery (370) may be placed in the leg portions (302, 303) of the electronic device (300). The battery (370) may include a first battery (371) placed in the right leg (302) and a second battery (372) placed in the left leg (303). Power for driving the glass (310), the display module (320), the first camera (325), the second camera (330), the third camera (340), the LED light (350), the PCB (360), the speaker (380), and the microphone (390) may be supplied from the battery (370).

[0075] The speaker (380) may include a first speaker (381) (right speaker) and a second speaker (382) (left speaker). For example, the speaker (380) may output sound according to control by the driving unit of the PCB (360).

[0076] The microphone (390) may include a first microphone (391) (e.g., a top microphone), a second microphone (392) (e.g., a right microphone), and a third microphone (393) (e.g., a left microphone). For example, the user's voice and external sounds may be converted into electrical signals through the microphone (390). For example, the first microphone (391) (e.g., a top microphone), the second microphone (392) (e.g., a right microphone), and the third microphone (393) (e.g., a left microphone) may include microphones of a condenser, a dynamic (moving coil and ribbon), a piezoelectric element, or a MEMS (micro-electro mechanical systems) type.

[0077] Although not shown, the electronic device (300) may further include a sensor (e.g., a proximity sensor) that generates data used to determine whether the electronic device (300) is worn on the face and provides it to the processor of the electronic device (300).

[0078] FIGS. 4A and 4B are perspective views schematically illustrating the front and back of an electronic device (400) wearable on a user's face according to one embodiment of the present disclosure. Referring to FIGS. 4A and 4B , the electronic device (400) (e.g., the electronic device (101) of FIG. 1 ) may be classified as a VR headset that prevents external light from reaching the user's eyes through the front when worn. The electronic device (400) may include a housing (410), a first camera (421, 422), a display (431, 432), a second camera (441, 442), and a third camera (451, 452). The aforementioned cameras constituting the electronic device (400) may each include components of the camera module (180) of FIG. 2 .

[0079] The housing (410) may include a front (or first side) (411) (e.g., the front) that is exposed to the external environment and a back (or second side) (412) that is not exposed to the external environment and adheres closely to the face when worn. When the electronic device is worn on the user's face, the front (411) may be exposed to the external environment and the back (412) may adhere at least partially to the user's face. The back (412) may adhere closely to the face through various components. For example, the back (412) may adhere closely to the face around the eyes by a band that has elasticity and is connected to the housing (410). The electronic device (400) may also be worn on the face through eyeglass temples, helmets, or straps. The housing (410) may have a shape or structure that is easy to be worn on the user's face. For example, the back (412) may be formed in a streamlined shape so as to cover the user's eyes and part of the nose. A nose recess (413) may be formed on the back (412) so that it can be supported on the user's nose.

[0080] The first camera (421, 422) may be arranged on the front (411) and configured such that its lens faces outward. The first camera (421, 422) may include a first front camera (421) corresponding to the user's right eye and a second front camera (422) corresponding to the user's left eye. The electronic device may use at least one of the first front camera (421) and the second front camera (422) to capture an image of a subject located in a direction toward which the front of the electronic device (400) faces (the -y-axis direction, referring to FIG. 4A), generate an image corresponding to the subject, and provide the image to a processor (e.g., an image signal processor (260) or a processor (120)) of the electronic device (400).

[0081] The first camera (421, 422) can be used as a sensor for measuring depth of field (DOF). The electronic device (400) can obtain information indicating depth of field (e.g., 3DOF (degrees of freedom) or 6DOF) using an image obtained from at least one of the left-eye camera (421) and the right-eye camera (422). The electronic device (400) can perform various functions, such as head tracking, hand detection or tracking, gesture recognition, location recognition, or spatial recognition, using the depth of field.

[0082] The first camera (421, 422) can be used as a sensor to recognize the surrounding space of the electronic device. The electronic device (400) can detect a gesture of a user wearing the electronic device (400) using an image acquired from at least one of the right-eye camera (421) and the left-eye camera (422). At least one of the right-eye camera (421) and the left-eye camera (422) can include a GS (global shutter) camera capable of reducing the RS (rolling shutter) phenomenon in order to detect and track the user's rapid hand movements and / or fine movements of the fingers.

[0083] Bluetooth (e.g., BLE (Bluetooth low energy)) may be used as an auxiliary means for recognizing the location of an object. For example, the electronic device (400) may include a Bluetooth circuit for short-range wireless communication with an external electronic device. The electronic device (400) may determine the location of the external electronic device by using the strength of an RF signal received from the external electronic device (e.g., RSSI (received signal strength indicator)) and an image acquired from the first camera (421, 422).

[0084] The screens in the displays (431, 432) may be positioned on the back (412) so as to face the user's eyes when worn. For example, a first display (431) corresponding to the user's right eye and a second display (432) corresponding to the user's left eye may be positioned on the back (412) of the electronic device (400).

[0085] The second camera (441, 442) may be arranged on the rear side (412) and configured such that its lens faces inward. The second camera (441, 442) may include a first rear camera (441) for tracking the movement of the right eye and a second rear camera (442) for tracking the movement of the left eye. The electronic device may use at least one of the first rear camera (441) and the second rear camera (442) to capture a subject (pupil) located in a direction toward which the rear side of the electronic device (400) faces (e.g., the y direction as shown in FIG. 4B), generate an image corresponding to the subject, and provide the image to a processor (e.g., an image signal processor (260) or a processor (120)) of the electronic device (400). An electronic device (400) (e.g., an image signal processor (260) or a processor (120)) can track the movement of the pupil (left eye, right eye) using an image generated by a second camera (441, 442) and determine the display area where the pupil is looking at on the display (431, 432).

[0086] The third camera (451, 452) may be arranged on the rear side (412) and configured such that its lens faces inward. The third camera (451, 452) may include a third rear camera (451) and a fourth rear camera (452) used for facial recognition. The electronic device (400) (e.g., the image signal processor (260) or the processor (120)) may determine whether the electronic device (400) is worn on the face using images generated by the third rear camera (451) and the fourth rear camera (452).

[0087] The electronic device (400) may include at least one sensor. For example, a plurality of sensors (461, 462, 463, 464) may be arranged on the front surface (411) of the electronic device (400). The electronic device (400) may measure a distance between an object located around the electronic device (400) and the electronic device (400) using data generated by at least one sensor among the plurality of sensors (461, 462, 463, 464). At least one of the plurality of sensors (461, 462, 463, 464) may include an infrared sensor, an ultrasonic sensor, a time of flight (ToF) sensor, and / or a light detection and ranging (LiDAR) sensor. For example, the electronic device (400) can measure the time difference between the time when light is emitted from the ToF sensor and the time when the light is reflected and returned, and / or the phase difference between the emitted light and the reflected light, and calculate the distance between the subject and the electronic device (400) based on the time difference. Examples of the ToF sensor include an i(indirect)-ToF sensor and a d(direct)-ToF sensor. The electronic device (400) can measure the time it takes for the emitted light (e.g., infrared ray) to return using the d-ToF sensor, and calculate the distance based on the measured time. The electronic device (400) can measure the phase difference that occurs when the emitted light returns using the i-ToF sensor, calculate the time based on the measured phase difference, and calculate the distance based on the calculated time. Although the sensor is illustrated as being arranged on the front (411) of the electronic device (400), it is not limited thereto. A plurality of sensors (461, 462, 463, 464) may be omitted from the electronic device (400). For example, the electronic device (400) may measure the distance between an object and the electronic device (400) using an image acquired from the first camera (421, 422).A sensor corresponding to at least one of the plurality of sensors (461, 462, 463, 464) may also be provided in an AR headset (e.g., electronic device (300) of FIG. 3).

[0088] FIGS. 5A, 5B, 5C, and 5D are diagrams showing various examples of camera angles and camera facing directions in a VR headset (500) and an external electronic device (520) according to one embodiment. FIGS. 5A, 5B, and 5C illustrate a case where the VR headset (500) and the external electronic device (520) are viewed from above and below (in the -z-axis direction, referring to the drawing). FIG. 5D illustrates a case where the VR headset (500) and the external electronic device (520) are viewed from the side (in the -x-axis direction, referring to the drawing).

[0089] Referring to FIGS. 5A, 5B, and 5C, the VR headset (500) may include a first camera (501) corresponding to the right eye (e.g., the first front camera (421) of FIG. 4A) and a second camera (502) corresponding to the left eye (e.g., the second front camera (422) of FIG. 4A). The third camera (521) refers to a camera mounted on an external electronic device (e.g., a smartphone) (520). In the drawings, “θ_VRCRH (VR camera right horizontal)” refers to a horizontal angle of view of the right eye camera (the first camera (501)) mounted on the VR headset (500). “θ_VRCLH (VR camera left horizontal)” refers to a horizontal angle of view of the left eye camera (the second camera (502)) mounted on the VR headset (500). “θ_EH (external horizontal)” refers to the horizontal angle of view of the third camera (521) mounted on the external electronic device (520). The horizontal angle of view (θ_VRCRH) of the first camera (501) may be the same as the horizontal angle of view (θ_VRCLH) of the second camera (502). The horizontal angles of view (θ_VRCLH, θ_VRCRH) of the cameras mounted on the VR headset (500) (e.g., the first camera (501) and / or the second camera (502)) may be implemented as wide angles (e.g., about 100 degrees). The horizontal angle of view (θ_EH) of the third camera (521) may be implemented to be narrower than the horizontal angles of view (θ_VRCLH, θ_VRCRH) of the cameras mounted on the VR headset (500).

[0090] Referring to FIG. 5A, reference numeral 531 denotes a first horizontal viewable area within the field of view of the first camera (501). The first horizontal viewable area (531) may be determined based on a horizontal direction toward which the first camera (501) faces (e.g., a horizontal direction toward which the center of the corresponding lens faces) and a horizontal field of view (θ_VRCRH) of the first camera (501) in the VR headset (500). Here, the horizontal direction corresponds to a direction perpendicular to the z-axis with reference to the three axes illustrated. In the same manner, a second horizontal viewable area (532) within the field of view of the second camera (502) may be determined. In addition, a third horizontal viewable area (533) within the field of view of the third camera (521) may be determined in the same manner. As illustrated in FIG. 5a, the third horizontal visible area (533) may be spatially included in the first horizontal visible area (531) and the second horizontal visible area (532), respectively.

[0091] Referring to FIGS. 5A and 5B, in FIG. 5A, reference numeral 541 indicates a first horizontal direction toward which the third camera (521) faces. In FIG. 5B, reference numeral 542 indicates a second horizontal direction toward which the third camera (521) faces. As the direction toward which the third camera (521) faces changes from the first horizontal direction (541) to the second horizontal direction (542), the horizontal visible area within the field of view of the third camera (521) also changes from the third horizontal visible area (533) to the fourth horizontal visible area (534). As illustrated in FIG. 5B, the fourth horizontal visible area (534) may be included in the second horizontal visible area (532). Although not illustrated, the fourth horizontal visible area (534) may be included in the first horizontal visible area (531).

[0092] Referring to FIG. 5C, reference numeral 543 in FIG. 5C indicates a third horizontal direction toward which the third camera (521) faces. As the direction toward which the third camera (521) faces changes from the first horizontal direction (541; see FIG. 5A) or the second horizontal direction (542; see FIG. 5B) to the third horizontal direction (543), the horizontal visible area that enters the field of view of the third camera (521) also changes from the third horizontal visible area (533) (or the fourth horizontal visible area (534)) to the fifth horizontal visible area (535). As illustrated in FIG. 5C, a part of the fifth horizontal visible area (535) may not be spatially included in the second horizontal visible area (532). Although not illustrated, the first horizontal visible area (531) may also not include a part of the fifth horizontal visible area (535).

[0093] Referring to FIG. 5d, “θ_VRRV (VR camera right vertical)” in the drawing means the vertical angle of view of the first camera (501) mounted on the VR headset (500). “θ_EV (external vertical)” refers to the vertical angle of view of the third camera (520) mounted on the external electronic device (520). The vertical angle of view (θ_VRCRV) of the first camera (501) may be the same as the vertical angle of view (θ_VRCLV (VR camera left vertical)) of the second camera (502). The vertical angles of view (θ_VRCLV, θ_VRCRV) of the cameras mounted on the VR headset (500) (e.g., the first camera (501) and / or the second camera (502)) may be implemented as a wide angle (e.g., about 100 degrees). The vertical angle of view (θ_EV) of the third camera (521) may be implemented to be narrower than the vertical angles of view (θ_VRCLV, θ_VRCRV) of the cameras mounted on the VR headset (500).

[0094] Referring back to FIG. 5d, reference numeral 551 denotes a first vertical viewable area within the field of view of the first camera (501). The first vertical viewable area (551) may be determined based on a vertical direction (e.g., a vertical direction toward which the center of the corresponding lens is directed) (561) toward which the first camera (501) faces in the VR headset (500) and a vertical field of view of the first camera (501). Here, the vertical direction corresponds to a direction perpendicular to the x-axis with reference to the illustrated three-axis. In the same manner, the second vertical viewable area within the field of view of the second camera (502) may be determined. In addition, the third vertical viewable area (553) within the field of view of the third camera (521) may be determined in the same manner. As illustrated in FIG. 5d, the third vertical viewable area (553) may be spatially included in the first vertical viewable area (551). Although not shown, the third vertical visible area (553) may be included in the second vertical visible area.

[0095] As illustrated in FIGS. 5A, 5B, and 5D, a subject captured by a third camera (521) may be included in an image of a visible area acquired using the first camera (501) and the second camera (502) in a VR headset (500). Accordingly, when a user views the external surroundings through the first camera (501) and / or the second camera (502), all subjects captured by the third camera (503) may be included in the user's field of view.

[0096] As illustrated in FIG. 5c, the subject captured by the third camera (521) may not be included in the image of the visible area acquired using the first camera (501) and the second camera (502) in the VR headset (500). Accordingly, when the user views the external surroundings through the first camera (501) and / or the second camera (502), some of the subjects captured by the third camera (503) may be out of the user's field of view.

[0097] FIGS. 6A, 6B, and 6C are diagrams illustrating various examples of camera angles and camera facing directions in an AR headset (600) and an external electronic device (620) according to one embodiment. FIGS. 6A, 6B, and 6C illustrate a case where the VR headset (600) and the external electronic device (620) are viewed from above and below (in the -z-axis direction, referring to the drawings).

[0098] Referring to FIGS. 6a, 6b, and 6c, the AR headset (600) may include a first camera (601) corresponding to the right eye (e.g., the right-eye camera (331) of FIG. 3), a second camera (602) corresponding to the left eye (e.g., the left-eye camera (332) of FIG. 3), a first display (603) corresponding to the right eye (e.g., the first display (321) of FIG. 3), and a second display (604) corresponding to the right eye (e.g., the second display (322) of FIG. 3). The third camera (621) refers to a camera mounted on an external electronic device (e.g., a smartphone) (620). In the drawings, “θ_ARCRH (AR camera right horizontal)” refers to a horizontal field of view of the right-eye camera (the first camera (601)) mounted on the AR headset (600). “θ_ARCLH (AR camear left horizontal)” refers to the horizontal angle of view of the left eye camera (second camera (602)) mounted on the AR headset (600). In the drawing, “θ_EH (external horizontal)” refers to the horizontal angle of view of the third camera (621) mounted on the external electronic device (620). The horizontal angle of view (θ_ARRH) of the first camera (601) may be the same as the horizontal angle of view (θ_ARCLH) of the second camera (602). The horizontal angles of view (θ_ARCLH, θ_ARCRH) of the cameras (e.g., the first camera (601) and / or the second camera (602)) mounted on the AR headset (600) may be implemented as a wide angle (e.g., about 100 degrees). The horizontal angle of view (θ_EH) of the third camera (621) can be implemented to be narrower than the horizontal angle of view (θ_ARCLH, θ_ARCRH) of the camera mounted on the AR headset (600).

[0099] Referring to FIG. 6A, reference numeral 631 denotes a first horizontal viewable area within the field of view of the first camera (601). The first horizontal viewable area (631) may be determined based on a vertical direction toward which the first camera (601) faces (e.g., a vertical direction toward which the center of the corresponding lens faces) and a horizontal field of view (θ_ARCRH) of the first camera (601) in the electronic AR headset (600). In the same manner, a second horizontal viewable area (632) within the field of view of the second camera (602) may be determined. In addition, a third horizontal viewable area (633) within the field of view of the third camera (621) may be determined in the same manner. As illustrated in FIG. 6A, the third horizontal viewable area (633) may be spatially included in each of the first horizontal viewable area (631) and the second horizontal viewable area (632).

[0100] In the AR headset (600) of FIG. 6a, the first camera (601) or the second camera (602) may be omitted from the AR headset (600).

[0101] Referring to FIG. 6b, “θ_ARCRV (AR camera right vertical)” in the drawing means the vertical angle of view of the first camera (501) mounted on the AR headset (600). “θ_EV(external vertical)” refers to the vertical angle of view of the third camera (520) mounted on the external electronic device (620). The vertical angle of view (θ_ARCRV) of the first camera (601) may be the same as the vertical angle of view (θ_ARCLV(AR camear left vertical)) of the second camera (602). The vertical angles of view (θ_VRCLV, θ_VRCRV) of the cameras mounted on the AR headset (600) (e.g., the first camera (601) and / or the second camera (602)) may be implemented as a wide angle (e.g., about 100 degrees). The vertical angle of view (θ_EV) of the third camera (621) may be implemented to be narrower than the vertical angles of view (θ_ARCLV, θ_ARCRV) of the cameras mounted on the AR headset (600).

[0102] Referring back to FIG. 6B, reference numeral 651 denotes a first vertical viewable area within the field of view of the first camera (601). The first vertical viewable area (651) may be determined based on the vertical direction in which the first camera (601) faces (e.g., the vertical direction in which the center of the corresponding lens faces) and the vertical field of view of the first camera (601) in the AR headset (600). In the same manner, the second vertical viewable area within the field of view of the second camera (602) may be determined. In addition, the third vertical viewable area (653) within the field of view of the third camera (621) may be determined in the same manner. As illustrated in FIG. 6B, the third vertical viewable area (653) may be spatially included in the first vertical viewable area (651). Although not illustrated, the third vertical viewable area (653) may be included in the second vertical viewable area.

[0103] As illustrated in FIGS. 6A and 6B, the subject captured by the third camera (621) may be included in the image of the visible area acquired using the first camera (601) and the second camera (602) in the AR headset (600). When the angles of view of the first camera (601) and the second camera (602) are configured to match or be narrower than the user's field of view, it can be seen that the subject captured by the third camera (503) is all included in the user's field of view when the user views the external surroundings.

[0104] Unlike as illustrated in FIGS. 6A and 6B , the third horizontal visible area (633) may not coincide with the first horizontal visible area (631) and / or the second horizontal visible area (632). The third vertical visible area (653) may not coincide with the first vertical visible area (651) and / or the second vertical visible area. In such a case of inconsistency, the subject captured by the third camera (621) may not be included in the image of the visible area acquired using the first camera (601) and the second camera (602) in the AR headset (600). Accordingly, when the user views the external surroundings, some of the subjects captured by the third camera (503) may be out of the user's field of view.

[0105] Referring to FIG. 6C, the AR headset (600) may include a sensor (e.g., an inertial sensor) (610) for detecting the direction in which the AR headset (600) is facing. Some of the components illustrated in FIG. 6A may be omitted. For example, compared to FIG. 6A, the first display (603) and the cameras (601, 602) may be omitted from the AR headset (600). The AR headset (600) (e.g., a processor) may determine the vertical / horizontal direction in which the AR headset (600) is facing using data acquired from the sensor (610). The AR headset (600) (e.g., a processor) may determine a visible area (e.g., a horizontal visible area (661) and a vertical viewing angle) that enters the user's eyes based on the determined direction and the user's viewing angle (e.g., a horizontal viewing angle of 180 degrees and a vertical viewing angle of 120 degrees). When the visible area (633, 653) of the third camera (621) is included in the user's field of view, it can be considered that all subjects captured by the third camera (621) are included in the user's field of view. Accordingly, based on the fact that the visible area (633, 653) of the third camera (621) is included in the user's field of view, the AR headset (600) can display an image acquired using the third camera (621) on the display (603).

[0106] FIG. 7A is a diagram illustrating an example of the field of view of a display (710) in a VR headset according to one embodiment. FIG. 7B is a diagram illustrating an example of the field of view of a display (720) in an AR headset according to one embodiment.

[0107] Referring to FIG. 7A, a VR headset display (710) (e.g., the first display (431) or the second display (432) of FIG. 4B) may include a display panel (or a light-emitting panel) (e.g., an LED (light emitting diode) panel) (711) and an optical system (e.g., a lens) (712). Light output from the display panel (711) may be incident on the user's pupil (701) through the optical system (712). The angle of view of the VR headset display (710) may be determined by a property of the optical system (e.g., a refractive index of the lens). In the drawing, “θ_VRD (VR display)” may mean a range (display area) of the VR headset display (710) that a human eye can see through the optical system (712) and its angle. For example, when the field of view (θ_VRD) is about 100 degrees or more, the entire display area of ​​the VR headset display (710) can be incident on the user's eyes. The field of view (θ_VRD) may be a concept including a horizontal field of view (θ_VRDH (VR display horizontal)) and a vertical field of view (θ_VRDV (VR display vertical)), as described with reference to FIGS. 5A and 5D.

[0108] In one embodiment, the VR headset can track where the eye (701) is looking based on images acquired using an internal camera (e.g., the second camera (441, 442) of FIG. 4B). The AR headset can display images received from an external electronic device (e.g., the external electronic device (520) of FIG. 5A) at the location where the user is looking on the VR headset display (710).

[0109] According to one embodiment, the VR headset display (710) may be configured such that the angle of view (e.g., θ_VRCLH, θ_VRCLV) of a camera mounted on the VR headset matches or is greater than the display angle of view (θ_VRD).

[0110] According to one embodiment, the VR headset may be configured to display images acquired using an onboard camera (e.g., the first camera (501) and / or the second camera (502) of FIG. 5A) or images received from an external electronic device (e.g., the external electronic device (520) of FIG. 5A) aligned with the central axis (713) of the display field of view (θ_VRD).

[0111] Referring to FIG. 7B, the AR headset display (720) (e.g., the first display (321) or the second display (322) of FIG. 3) may include a display panel (721), an optical system (e.g., a lens) (722), and a waveguide (723). The optical system (722) may allow light output from the panel (721) to be incident on the waveguide (723). Although not shown, an optical barrier (e.g., a lens tube) may be included in the AR headset display (720). The optical barrier may prevent light output from the panel (721) from leaking out through the glass (730) and block external light from being incident on the panel (721). The waveguide (723) may guide light incident from the panel (721) to the user's eye (701) through the optical system (722). The waveguide (723) can guide externally incident light (real image) through the glass (730) to the user's eyes (701). The user can perceive an augmented reality (AR) image that combines a virtual image and a real image through the optical coupler (722).

[0112] The angle of view of the AR headset display (720) may be determined by the properties of the optical system (722) (e.g., the refractive index of the lens) and / or the properties of the waveguide (723) (e.g., the refractive index). In the drawing, “θ_ARD (VR display)” may refer to the range (display area) of the AR headset display (720) that can be seen by the human eye and the angle thereof. For example, the angle of view (θ_ARD) may have a maximum value of 50 degrees depending on the properties of the waveguide (723). A portion of the display area of ​​the AR headset display (720) may be incident on the user’s eye. The angle of view (θ_ARD) may be a concept including a horizontal angle of view (θ_ARDH (AR display horizontal)) and a vertical angle of view (θ_ARDV (AR display vertical)), as described with reference to FIGS. 6A and 6B.

[0113] In one embodiment, the AR headset can track where the user's eyes (701) are looking based on an image acquired using an internal camera (e.g., the third camera (340) of FIG. 3). The AR headset can display an image received from an external electronic device (e.g., the external electronic device (620) of FIG. 6A) at the location where the user is looking on the AR headset display (720).

[0114] According to one embodiment, the AR headset may be configured to display an image received from an external electronic device (e.g., the external electronic device (620) of FIG. 6A) aligned with the central axis (724) of the display field of view (θ_ARD).

[0115] FIGS. 8A, 8B, and 8C are diagrams illustrating various examples of a viewable area and an overlapping area determined by a field of view and direction in a VR headset and an external electronic device according to one embodiment.

[0116] Referring to FIG. 8A, “θ_VRC” refers to the angle of view (horizontal angle of view and vertical angle of view) of a camera mounted on a VR headset (e.g., the first camera (501) or the second camera (502) of FIG. 5A). “θ_VRD” refers to the angle of view (horizontal angle of view and vertical angle of view) of a display mounted on a VR headset (e.g., the display (710) of FIG. 7A). “θ_E” refers to the angle of view (horizontal angle of view and vertical angle of view) of a camera mounted on an external electronic device (e.g., the third camera (521) of FIG. 5A).

[0117] A VR headset may have a camera having a “θ_VRC” greater than “θ_VRD”. The camera and the display may be configured in the VR headset such that the direction in which the central axis of “θ_VRC” faces is aligned with the direction in which the central axis A of “θ_VRD” (e.g., the central axis (713) of FIG. 7A) faces. Both the horizontal and vertical directions may be aligned. Accordingly, the second viewable area (820) of the display determined by “θ_VRD” may be spatially included in the first viewable area (810) of the VR headset camera determined by “θ_VRC”.

[0118] The external electronic device may have a camera with “θ_E” smaller than “θ_VRD”. The third visible area (831) determined by “θ_E” and the direction in which the external camera faces may be spatially included in the second visible area (820). Accordingly, a first overlapping area (821) that overlaps the entire third visible area (831) may be formed in the second visible area (820).

[0119] In one embodiment, the VR headset can display an image received from an external electronic device in a first overlapping area (821). The user can view the subject to be photographed through the first overlapping area (821) (i.e., the viewfinder).

[0120] Referring to FIG. 8B, the direction in which the external camera faces may change, and accordingly, the visible area within the field of view of the external camera may change from the third visible area (831; see FIG. 8A) to the fourth visible area (832). Only a portion of the fourth visible area (832) may be spatially included in the second visible area (820). A second overlapping area (822) that overlaps a portion of the fourth visible area (832) may be formed in the second visible area (821). Accordingly, a user may identify only a portion of the fourth visible area (832) through the second overlapping area (822).

[0121] According to one embodiment, the VR headset receives information about the field of view of the external camera (e.g., “θ_E” and the direction in which the external camera is facing) from an external electronic device, determines a fourth visible area (832) that falls within the field of view of the external camera based on the received information, and recognizes that at least a portion of the fourth visible area (832) is not spatially included in the second visible area (821). Based on this recognition, the VR headset may display a visual notification requesting orientation adjustment on the VR headset display or output an audible notification through a speaker. The user may recognize this notification and adjust the position and direction of the external camera so that the entire field of view of the external camera can be confirmed on the display.

[0122] When the subject is distant, the spatial distance between the headset camera (e.g., the user's head) and the external camera (e.g., the user's hand) can be ignored. Therefore, as shown in Figures 8a and 8b, the center of the VR headset camera can be assumed to be aligned with the center of the external camera. Conversely, when the subject is close, the distance between the two cameras can be an important parameter in determining the overlap area.

[0123] Referring to FIG. 8C, the VR headset can receive information about the field of view of the external camera (e.g., “θ_E” and the direction in which the external camera is facing) from an external electronic device. The VR headset can calculate a distance (851) between the external camera and the VR headset using data received from a sensor (e.g., at least one of the plurality of sensors (461, 462, 463, 464) of FIG. 4A). Based on the information about the received information and the calculated distance (851), the VR headset can determine a fifth visible area (833) that falls within the field of view of the external camera. The VR headset can determine a third overlapping area (823) that overlaps the fifth visible area (833) in the second visible area (820).

[0124] As described above, the size of the overlap area can be determined based on the field of view. For example, the larger “θ_E” is, the larger the overlap area can be. The size of the overlap area can be determined based on the direction in which the external camera is facing. For example, the more the direction of the external camera matches the direction in which the VR headset camera is facing, the larger the overlap area can be. The size of the overlap area can be determined based on the distance between the VR headset and the external camera. For example, the closer the distance between the two is, the larger the overlap area can be. The distance between the VR headset and the external camera can be excluded from the parameters for determining the overlap area. For example, the VR headset can calculate the distance between the subject and the VR headset using data received from a sensor (e.g., at least one of the plurality of sensors (461, 462, 463, 464) of FIG. 4A), and if the calculated distance is greater than a threshold, the distance between the VR headset and the external camera can be excluded from the calculation of the overlap area.

[0125] FIGS. 9a, 9b, 9c, and 9d are diagrams illustrating various examples of a viewable area and an overlapping area determined by a field of view and direction in an AR headset and an external electronic device according to one embodiment.

[0126] Referring to FIG. 9A, “θ_ARC” refers to the angle of view (horizontal angle of view and vertical angle of view) of a camera mounted on an AR headset (e.g., the first camera (601) or the second camera (602) of FIG. 6A). “θ_ARD” refers to the angle of view (horizontal angle of view and vertical angle of view) of a display mounted on an AR headset (e.g., the display (720) of FIG. 7B). “θ_E” refers to the angle of view (horizontal angle of view and vertical angle of view) of a camera mounted on an external electronic device (e.g., the third camera (621) of FIG. 6A).

[0127] The AR headset may have a camera with a “θ_ARC” greater than “θ_ARD”. The camera and the display may be configured in the AR headset such that the direction of the central axis of “θ_VRC” is aligned with the direction of the central axis B of “θ_VRD” (e.g., the central axis (724) of FIG. 7B ). Both the horizontal and vertical directions may be aligned. Accordingly, the second viewable area (920) of the display determined by “θ_ARD” may be spatially included in the first viewable area (910) of the AR headset camera determined by “θ_ARC”.

[0128] The external electronic device may have a camera with “θ_E” greater than “θ_ARD”. For example, “θ_E” may be smaller than “θ_VRD” but greater than “θ_ARD”. A third visible area (931) may be determined based on “θ_E” and the direction in which the external camera faces. The second visible area (920) may be spatially included in the third visible area (931). Accordingly, the entire second visible area (920) may be set as the first overlapping area (921).

[0129] In one embodiment, the AR headset can display an image received from an external electronic device in a first overlapping area (921). The user can view the subject to be photographed through the first overlapping area (921) (i.e., the viewfinder).

[0130] Referring to FIG. 9B, the direction in which the external camera faces may change, and accordingly, the visible area within the field of view of the external camera may change from the third visible area (931; see FIG. 8A) to the fourth visible area (932). Due to this change in direction, only a portion of the second visible area (920) may be set as a second overlapping area (922) that spatially overlaps the fourth visible area (932). For example, the AR headset may compare the second visible area (920) and the fourth visible area (932), and based on the comparison result, divide the second visible area (920) into a second overlapping area (922) and a non-overlapping area (922a). Accordingly, when viewing the AR headset display, the user may identify only a portion of the fourth visible area (932) through the second overlapping area (922).

[0131] According to one embodiment, the AR headset receives information about the field of view of the external camera (e.g., “θ_E” and the direction in which the external camera is facing) from an external electronic device, determines a fourth visible area (932) that falls within the field of view of the external camera based on the received information, and recognizes that at least a portion of the fourth visible area (832) is not spatially included in the second visible area (821). Based on this recognition, the VR headset may display a visual notification requesting orientation adjustment on the VR headset display or output an audible notification through a speaker. The user may recognize this notification and adjust the position and direction of the external camera so that the entire field of view of the external camera can be confirmed on the display.

[0132] When the subject is distant, the spatial distance can be ignored, and the center of the AR headset camera can be assumed to coincide with the center of the external camera, as illustrated in Figures 9a and 9b. Conversely, when the subject is close, the distance between the two cameras can be an important parameter in determining the overlap area.

[0133] Referring to FIG. 9C, the AR headset can receive information about the field of view of the external camera (e.g., “θ_E” and the direction in which the external camera is facing) from an external electronic device. The AR headset can calculate the distance (951) between the external camera and the AR headset using data received from an image sensor (e.g., the first camera (325) and / or the second camera (330) of FIG. 3). The AR headset can also measure time (or phase difference) using data received from a ToF sensor (e.g., an i-ToF sensor or a d-ToF sensor) and calculate the distance (951) based on the measurement result. The AR headset can determine the fifth visible area (933) that falls within the field of view of the external camera based on the information about the received information and the calculated distance (951). The AR headset can compare the second visible area (920) and the fifth visible area (933), and based on the comparison result, divide the second visible area (920) into a third overlapping area (923) and a non-overlapping area (923a).

[0134] Referring to FIG. 9D, an AR headset (e.g., the AR headset (600) of FIG. 6D) may not have a camera and may only include a sensor (e.g., an inertial sensor). The AR headset may determine a direction in which the AR headset is facing based on data received from the sensor, and may determine a second viewable area (920) using the determined direction and the field of view “θ_ARD” of the AR headset display. The AR headset may receive information about the field of view of an external camera (e.g., “θ_E” and the direction in which the external camera is facing) from an external electronic device. The AR headset may calculate a distance (952) between the external camera and the AR headset using data received from a sensor (e.g., the first camera (325) and / or the second camera (330) of FIG. 3). The AR headset may determine a sixth viewable area (934) that falls within the field of view of the external camera based on the received information and the information about the calculated distance (952). The AR headset can compare the second visible area (920) and the sixth visible area (934), and based on the comparison result, divide the second visible area (920) into a fourth overlapping area (924) and a non-overlapping area (924a).

[0135] As described above, the size of the overlapping area can be determined based on the field of view. For example, the larger “θ_E” is, the larger the overlapping area can be. The size of the overlapping area can be determined based on the direction in which the external camera is facing. For example, the more the direction of the external camera matches the direction in which the AR headset camera is facing, the larger the overlapping area can be. The size of the overlapping area can be determined based on the distance between the AR headset and the external camera. For example, the closer the distance between them is, the larger the overlapping area can be. The distance between the AR headset and the external camera can be excluded from the parameters for determining the overlapping area. For example, the AR headset can calculate the distance between the subject and the AR headset using data received from a sensor (e.g., the first camera (325) and / or the second camera (330) of FIG. 3), and if the calculated distance is greater than a threshold, the distance between the AR headset and the external camera can be excluded from the calculation of the overlapping area.

[0136] FIGS. 10a, 10b, 10c, 10d, and 10e are diagrams illustrating various examples of overlapping areas in a display of a VR headset according to one embodiment.

[0137] Referring to FIG. 10A, the VR headset may determine a direction in which a camera of the VR headset faces using data received from a sensor (e.g., an inertial sensor), and, based on the determined direction and the angle of view of the VR headset camera, determine a first visible area (1010) (e.g., the first visible area (810)) that falls within the field of view of the VR headset camera. The VR headset may determine, based on the angle of view of the VR headset display, a second visible area (1020) (e.g., the second visible area (820)) that falls within the user's eye on the VR headset display. The VR headset may align the second visible area (1020) with the first visible area (1010) so that the second visible area (1020) is spatially included in the first visible area (1010). The VR headset may receive information about the angle of view of a camera of the external electronic device (hereinafter, the external camera) and the direction in which the external camera is pointed from the external electronic device. The VR headset can determine a third visible area (e.g., the third visible area (831) in FIG. 8A) within the field of view of the external camera based on the received information. The distance between the external camera and the VR headset (e.g., the distance (851) in FIG. 8C) can be used to determine the visible area of ​​the external camera. The VR headset can determine a first overlapping area (1041) (e.g., the first overlapping area (821)) that overlaps the third visible area in the second visible area (1020). Additionally, the VR headset can display an indicator (1050) indicating a central axis of the field of view (e.g., “θ_E” in FIG. 8A) of the external camera within the first overlapping area (1041) so that the user can recognize the direction in which the external camera is facing. The VR headset can display an image received from the external camera in the first overlapping area (1041; viewfinder).

[0138] The first overlapping area (1041) may have a shape other than a rectangle as illustrated (e.g., a trapezoid, a rectangle with rounded corners). For example, when the direction in which the external camera faces is different from the direction in which the VR headset camera faces, the shape of the overlapping area may be a trapezoid. The VR headset may display only an image in the first overlapping area (1041) excluding the indicator (1050). The VR headset may display only the indicator (1050) on the VR headset display.

[0139] The VR headset can display a user interface (UI) (1027) related to an external camera on the VR headset display. For example, the VR headset can display the UI (1027) on the VR headset display so as not to overlap with the first overlapping area (1041). The external camera UI (1027) is intended to allow a user to use the external camera while wearing the VR headset, and may include UI elements for camera operation that are familiar to the user. The UI (1027) may be positioned below the first overlapping area (1041). However, the present invention is not limited thereto, and may also be positioned adjacent to the first overlapping area (1041). The user can check various functions (e.g., taking pictures, recording videos, changing shooting modes, setting auto focus (AF), adjusting zoom ratio, moving to a gallery) that can be executed using a camera on an external electronic device through the external camera UI (1027) displayed on the VR headset display. The VR headset can transmit a notification message to the external electronic device indicating that the external camera UI (1027) is being displayed on the VR headset display. Based on the receipt of such a notification message, the external electronic device can turn off its display. The external electronic device can turn off its display, but still respond to a user's touch input on the display. The user can operate the external electronic device through the external camera UI (1027) without viewing the UI for camera operation on the display of the external electronic device (hereinafter, the external display). When operating an external electronic device (e.g., a smartphone or a digital camera with a touchpad), the user already knows from experience which function icons are placed in which areas of the external display screen. Therefore, the user can operate the external camera by touching a specific location on the external display without viewing the external display.The external electronic device may transmit a notification message to the VR headset indicating which function button has been selected by the user. In response to receiving the notification message, the VR headset may identify a UI element corresponding to the function button selected by the user on the external camera UI (1027). The VR headset may display a visual change for the identified UI element (e.g., a capture button (1027a)). The user may recognize that the external electronic device has responded to the user input on the external display.

[0140] Referring to FIG. 10B, the direction in which the external camera faces can be changed by the user. The VR headset can receive information about the changed direction of the external camera from an external electronic device, and based on the received direction information and the field of view of the external camera, determine a fourth visible area within the field of view of the external camera. The VR headset can determine a second overlapping area (1042) (e.g., the second overlapping area (822)) that overlaps the fourth visible area in the second visible area (1020). The VR headset can display an image received from the external camera in the second overlapping area (1042; viewfinder). The VR headset may recognize that there is a portion (1042a) in the fourth visible area that is spatially outside the second visible area (1020), and accordingly, the VR headset may display a visual notification requesting an adjustment of the direction of the external camera (e.g., an arrow (1060) pointing inward into the second visible area (1020)) on the VR headset display or output an audible notification through the speaker.

[0141] Referring to FIG. 10c, in addition to information regarding orientation and field of view, the VR headset may receive information regarding the posture (e.g., rotation direction) of the external electronic device from the external electronic device. Based on the received information, the VR headset may determine a third overlapping area (1043). Compared to the first overlapping area (1041; FIG. 10a), the third overlapping area (1043) may be a tilted rectangle.

[0142] Referring to FIGS. 10d and 10e, the VR headset displays a fourth overlapping area (1044) and can identify the location of an external electronic device using an image acquired from a VR headset camera. FIG. 10d illustrates a case where an external electronic device (1060) is located in a second visible area (1020). FIG. 10e illustrates a case where an external electronic device (1060) is located outside the second visible area (1020). The VR headset can display guide lines (1071, 1072) indicating the direction in which the external electronic device (1060) is facing on the VR headset display.

[0143] FIGS. 11a, 11b, 11c, and 11d are diagrams illustrating various examples of overlapping areas in a display of an AR headset according to one embodiment.

[0144] Referring to FIGS. 11a, 11b, 11c, and 11d, the viewing angle of the pupil can be considered wider than the viewing angle (horizontal and vertical viewing angles) of the camera, and the AR headset is fixed to the user's head when worn. Accordingly, the first visible area (1110) within the field of view of the AR headset camera can be considered to be included in the visible area (1101) within the field of view of the pupil, and the position of the first visible area (1110) within the visible area (1101) of the pupil can be considered to be fixed.

[0145] Referring to FIG. 11A, the AR headset may determine a direction in which the camera of the AR headset faces using data received from a sensor (e.g., an inertial sensor), and, based on the determined direction and the field of view of the AR headset camera, determine a first visible area (1110) (e.g., the first visible area (910)) that falls within the field of view of the AR headset camera. The AR headset may determine, based on the field of view of the AR headset display, a second visible area (1120) (e.g., the second visible area (920)) that falls within the user's eye on the AR headset display. The AR headset may align the second visible area (1120) with the first visible area (1110) so that the second visible area (1120) is spatially included in the first visible area (1110). The AR headset may receive information about the field of view of a camera of an external electronic device (hereinafter, an external camera) and the direction in which the external camera is pointed from the external electronic device. The AR headset can determine a third visible area (1131) (e.g., the third visible area (931) of FIG. 9A) that falls within the field of view of the external camera based on the received information. As illustrated, the second visible area (1120) is spatially entirely included in the third visible area (1131), and therefore, the entire second visible area (1120) can be set as a first overlapping area. The distance between the external camera and the AR headset (e.g., the distance (951) of FIG. 9C) can be used to determine the visible area of ​​the external camera. The AR headset can display an indicator (1150) indicating the central axis of the angle of view (e.g., “θ_E” of FIG. 9A) of the external camera within the first overlapping area, i.e., the second visible area (1120), so that the user can recognize the direction in which the external camera is facing. The AR headset can display an image received from the external camera in the second visible area (1120) (viewfinder). According to one embodiment, the AR headset may display the same UI (1127) as the UI (1027; see FIG. 10a) described above on the AR headset display.

[0146] Referring to FIG. 11B, the direction in which the external camera faces can be changed by the user. The AR headset can receive information about the changed direction of the external camera from an external electronic device, and based on the information about the received direction and the angle of view of the external camera, determine a fourth visible area (1132) that falls within the field of view of the external camera. The AR headset can divide the second visible area (1120) into a second overlapping area (1121) that overlaps the fourth visible area (1132) and a non-overlapping area (1121a). The VR headset can display an image received from the external camera in the second overlapping area (1121; viewfinder). In order to enlarge the viewfinder (i.e., enlarge the overlap area), the VR headset can display a visual notification (e.g., an arrow (1160) indicating the position of the external camera) requesting an adjustment of the direction of the external camera on the AR headset display or output an audible notification through a speaker.

[0147] Referring to FIG. 11C, the AR headset may receive information about the changed direction of the external camera from an external electronic device, and, based on the information about the received direction and the angle of view of the external camera, determine a fifth visible area (1133) within the field of view of the external camera. The AR headset may use the image acquired from the AR headset camera to identify the location of the external electronic device (1160) existing in the first visible area (1110). The AR headset may display a guide line (1171) indicating the direction in which the external electronic device (1160) is facing on the AR headset display.

[0148] The field of view of the external camera may be wider than the field of view of the AR headset display, and thus, as illustrated in FIG. 11A, the second visible area (1120) may be spatially included in the third visible area (1131). On the other hand, the field of view of the external camera may be narrower than the field of view of the AR headset display. Accordingly, referring to FIG. 11D, the visible area (1123) within the field of view of the AR headset display may spatially include the entire visible area within the field of view of the external camera. The AR headset may determine an overlapping area (1134) that overlaps the visible area within the field of view of the external camera in the visible area (1123). The AR headset may display an image received from the external camera in the overlapping area (1134).

[0149] In addition to information about orientation and field of view, the AR headset can receive information about the posture (e.g., rotation direction) of the external electronic device from the external electronic device. Based on the received information, the AR headset can determine an overlapping area. The overlapping area determined based on the posture of the external electronic device may have a different shape than that illustrated in FIG. 11d (e.g., the shape of the third overlapping area (1043) of FIG. 10c).

[0150] FIG. 12 is a block diagram of a first electronic device (1201) and a second electronic device (1202) according to one embodiment. The components of the first electronic device (1201) in FIG. 12 can be applied to an HMD device (1288) (e.g., a VR headset described with reference to FIGS. 4A and 5A or an AR headset described with reference to FIGS. 3 and 6A). The components of the second electronic device (1202) in FIG. 12 can be applied to a portable electronic device (1299) having a camera (e.g., an external electronic device described with reference to FIGS. 5A and 6A).

[0151] Referring to FIG. 12, a first electronic device (1201) may include a first camera (1211), a third camera (1213), a first communication circuit (1221), a first sensor (1231), a first display (1241), a first connector (1251), a first antenna module (1261), a first memory (1281), and a first processor (1291). The first camera (1211) may include a lens positioned at the front of the first electronic device (1201) and configured to collect light through the front. The third camera (1213) may include a lens positioned at the rear of the first electronic device (1201) and configured to collect light through the rear.

[0152] According to one embodiment, the camera (1211, 12133), the first communication circuit (1221), the first sensor (1231), the first display (1241), the first connector (1251), the first antenna module (1261), the first memory (1281), and the first processor (1291) in the first electronic device (1201) may be implemented substantially identically to the camera module (180), the communication module (190), the sensor module (176), the display module (160), the connection terminal (178), the antenna module (197), the memory (130), and the processor (120) in the electronic device (101) of FIG. 1, thereby performing the same functions.

[0153] According to one embodiment, the first camera (1211) may include the first camera (325) and / or the second camera (330) in FIG. 3. According to one embodiment, the first camera (1211) may include the first camera (421, 422) in FIG. 4A.

[0154] In one embodiment, the third camera (1213) may include the third camera (340) in FIG. 3. In one embodiment, the third camera (1213) may include the second camera (441, 442) and / or the third camera (451, 452) in FIG. 4b.

[0155] Instructions may be stored in the first memory (1281) or the internal memory of the first processor (1291). The instructions may be partially stored in the first memory (1281) and the internal memory of the first processor (1291). When executed by the first processor (1291), the instructions may cause the first electronic device (1201) to perform a series of operations for communicating with the second electronic device (1202) through a communication interface (e.g., the first connector (1251) or the first antenna module (1261)), determining an overlapping area (viewfinder) on the first display (1241), and displaying an image received from the second electronic device (1202) in the overlapping area.

[0156] Referring again to FIG. 12, the second electronic device (1202) may include a second camera (1212), a second communication circuit (1222), a second sensor (1232), a second display (1242), a second connector (1252), a second antenna module (1262), a second memory (1282), and a second processor (1292).

[0157] According to one embodiment, the second camera (1212), the second communication circuit (1222), the second sensor (1232), the second display (1242), the second connector (1252), the second antenna module (1262), the second memory (1282), and the second processor (1292) in the second electronic device (1202) may be implemented substantially identically to the camera module (180), the communication module (190), the sensor module (176), the display module (160), the connection terminal (178), the antenna module (197), the memory (130), and the processor (120) in the electronic device (101) of FIG. 1, thereby performing the same functions.

[0158] Instructions may be stored in the second memory (1282) or the internal memory of the second processor (1292). The instructions may be partially stored in the internal memory of the second memory (1282) and the second processor (1292). When executed by the second processor (1292), the instructions may cause the second electronic device (1202) to perform a communication with the first electronic device (1201) via a communication interface (e.g., the second connector (1252) or the second antenna module (1262)) (e.g., in response to a request from the first electronic device (1201), an operation of transmitting information about a direction in which the third camera (1213) is facing and an angle of view of the third camera (1213) to the first electronic device (1201)) and to respond to a request from the first electronic device (1201) (e.g., in response to a request from the first electronic device (1201), an operation of turning the second display (1242) on or off).

[0159] FIG. 13 is a flowchart illustrating an operation performed in a first electronic device (1201) of FIG. 12 that can be worn on a user's face, according to one embodiment. According to one embodiment, a first processor (1291) in the first electronic device (1201) may be configured to perform the operation of FIG. 13. According to one embodiment, when a command stored in a memory (e.g., the first memory (1281)) in the first electronic device (1201) is executed by the first processor (1291), the first electronic device (1201) may be configured to perform the operation of FIG. 13. The operation of FIG. 13 may be performed while the first electronic device (1201) is connected to the second electronic device (1202) via a wired communication interface (e.g., a USB connector) or a wireless communication interface (e.g., Bluetooth). The operation of FIG. 13 may be performed based on the activation of the second camera (1212) in the second electronic device (1202). The activation of the second camera (1212) may include, for example, performing an operation of periodically generating image data (e.g., generating an image at a specified frame rate) from the second camera (1212) and outputting the generated image data (e.g., providing the generated image to the second processor (1292)). When the second camera (1212) is deactivated, the operation of FIG. 13 may be terminated.

[0160] In operation 1310, the first electronic device (1201) can obtain information about the direction and angle of view of the first camera (1211) and information about the direction and angle of view of the second camera (1212). For example, the first electronic device (1201) can detect the direction of view of the first camera (1211) using the first sensor (1231) (e.g., an inertial sensor). The first electronic device (1201) can determine the angle of view of the first camera (1211) from data representing the characteristics of the first camera (1211). This data can be obtained from the first camera (1211) or from the first memory (1281). The first electronic device (1201) can receive information about the direction of view of the second camera (1212) and the angle of view of the second camera (1212) from the second electronic device (1202).

[0161] In operation 1320, the first electronic device (1201) can determine a first visible area that falls within the field of view of the first camera (1211) based on the direction in which the first camera (1211) is facing and the angle of view of the first camera (1211). In addition, in operation 1320, the first electronic device (1201) can determine a second visible area that falls within the pupil of the first display (1241) based on the angle of view of the first display (1241). In addition, the first electronic device (1201) can determine a third visible area that falls within the field of view of the second camera (1212) based on the direction in which the second camera (1212) is facing and the angle of view of the second camera (1212).

[0162] In operation 1330, the first electronic device (1201) may align the second visible area with the first visible area such that the second visible area is spatially included in the first visible area. As an example, the first visible area may be the first visible area (1010) of the VR headset camera in FIG. 10A. The second visible area may be the second visible area (1020) of the VR headset display in FIG. 10A. As another example, the first visible area may be the first visible area (1110) of the AR headset camera in FIG. 11A. The second visible area may be the second visible area (1120) of the AR headset display in FIG. 11A.

[0163] In operation 1340, the first electronic device (1201) may determine an overlapping area that overlaps the third visible area in the second visible area. For example, the overlapping area may be a first overlapping area (1041) within the second visible area (1020) of the VR headset display. As another example, the overlapping area may correspond to the entire second visible area (1120) of the AR headset display.

[0164] In operation 1350, the first electronic device (1201) can receive an image acquired by the second electronic device (1202) using the second camera (1212) from the second electronic device (1202) and display it in an overlapping area (viewfinder).

[0165] FIG. 14 is a flowchart illustrating an operation performed in a first electronic device (1201) of FIG. 12 that can be worn on a user's face, according to one embodiment. According to one embodiment, a first processor (1291) in the first electronic device (1201) may be configured to perform the operation of FIG. 14. According to one embodiment, when a command stored in a memory (e.g., the first memory (1281)) in the first electronic device (1201) is executed by the first processor (1291), the first electronic device (1201) may be configured to perform the operation of FIG. 14. The operation of FIG. 14 may be performed while the first electronic device (1201) is connected to the second electronic device (1202) via a wired communication interface (e.g., a USB connector) or a wireless communication interface (e.g., Bluetooth). The operation of FIG. 14 may be performed based on the activation of the second camera (1212) in the second electronic device (1202). The operation of FIG. 14 may be performed while an operation of displaying an image of the second camera (1212) in an overlapping area (e.g., operation 1350) is performed. When the second camera (1212) is deactivated, the operation of FIG. 14 may be terminated.

[0166] In operation 1410, the first electronic device (1201) may display a UI related to the second camera (1212) (e.g., the external camera UI (1027) of FIG. 10A or the external camera UI (1127) of FIG. 11A) on the first display (1241). For example, while the image of the second camera (1212) is displayed in the overlapping area, the display of the UI may be set to a default value. As another example, the first electronic device (1201) may display a UI (e.g., a pop-up window) on the first display (1241) that allows the user to select whether to display the UI.

[0167] In operation 1420, the first electronic device (1201) may transmit a request message to the second electronic device (1202) to turn off the second display (1242) but respond to a user's touch input based on the UI being set to be displayed on the first display (1241). In response to the request of the first electronic device (1201), the second electronic device (1202) may turn off the second display (1242) and transmit an image acquired through the second camera (1212) to the first electronic device (1201). In addition, the second electronic device (1202) may perform a given operation (e.g., taking a picture) in response to the user's touch input as requested.

[0168] In one embodiment, not displaying the UI may be the default. Alternatively, the user may not choose to display the UI. In such a case, the first electronic device (1201) may omit operation 1420, and accordingly, the display of the execution screen of the camera application corresponding to the UI may be maintained on the second display (1242).

[0169] FIGS. 15a, 15b, and 15c are diagrams showing examples of images captured by an external electronic device being displayed on a display of a VR headset according to one embodiment.

[0170] Referring to FIGS. 15A, 15B, and 15C, the VR headset may determine a direction in which a camera of the VR headset is facing using data received from a sensor (e.g., an inertial sensor), and, based on the determined direction and the angle of view of the VR headset camera, determine a first visible area (1510) (e.g., the first visible area (1010)) within the field of view of the VR headset camera. The VR headset may display a first image (1501) captured by the camera of the VR headset for the first visible area (1510) in a second visible area (1520) (e.g., the second visible area (1020)) within the field of view of the eye on the VR headset display. For example, the VR headset may align the second visible area (1520) with the first visible area (1510) such that the second visible area (1520) is spatially included in the first visible area (1510). The VR headset can receive information about the angle of view of the camera of the external electronic device (hereinafter, referred to as the external camera) and the direction in which the external camera is pointed from the external electronic device. Based on the received information, the VR headset can determine a third visible area (1530) (e.g., the third visible area (831) of FIG. 8A) that falls within the field of view of the external camera. The distance between the external camera and the VR headset (e.g., the distance (851) of FIG. 8C) can be used to determine the third visible area (1530). As illustrated, the third visible area (1510) can be included in a second visible area (1520) that is spatially aligned with the first visible area (1510). The VR headset can display a UI element (e.g., a window) on the VR headset display that allows a user to identify a portion (the third visible area (1510)) that falls within the field of view of the external camera in the second visible area (1520). For example, a VR headset may display a dotted line around the border of a rectangular window representing the third visible area (1530).

[0171] Referring to FIG. 15A, the VR headset can display a second image (1502) for a third visible area (1510) together with the first image (1501) on the VR headset display. For example, as illustrated, the VR headset display can obtain a third image (synthesized image) by synthesizing the first image (1501) and the second image (1502) so that the second image (1502) is positioned relatively higher, and display the third image on the VR headset display. The second image (1502) can be an enlarged image of a portion included in the third visible area (1530) of the first image (1501). Here, the subject of the enlargement can be the VR headset or an external electronic device. For example, the external electronic device can generate the second image (1502) by enlarging an image obtained from a camera of the external electronic device using a digital zoom or optical zoom method in response to a request from the VR headset and transmit it to the VR headset. As another example, a VR headset may generate a second image (1502) by digitally zooming in on an image received from an external electronic device.

[0172] Referring to Fig. 15b, the transparency of the second image (1502; Fig. 15a) can be set to '0'. The VR headset can generate a fourth image (1504) by adjusting the transparency of the second image (1502) to a high level, and synthesize it with the first image (1501) to obtain a third image and display it on the VR headset display. The location where the second image (1502) (or the fourth image (1504)) is displayed in the second visible area (1520) can be set to a location that does not overlap with the third visible area (1530).

[0173] The resolution of the image acquired from the external camera may be higher than the resolution of the image acquired from the VR headset camera, depending on the performance of the hardware and / or software of the external camera. Referring to FIG. 15C, instead of displaying the second image (1502) or the fourth image (1504) by synthesizing it with the first image (1501), the VR headset may receive an image (1505) acquired using the external camera from an external electronic device and replace a portion of the first image (1501) with a lower resolution, displayed in the third visible area (1530), with the image (1504). Accordingly, the user can identify an object existing within the visible area (1530) (the area of ​​interest to the user) without magnifying the image.

[0174] FIG. 16 is a diagram showing an example of an image captured by an external electronic device being displayed on a display of an AR headset according to one embodiment.

[0175] Referring to FIG. 16, reference numeral 1610 denotes an area within the field of view of a camera equipped in an AR headset. Reference numeral 1630 denotes an area within the field of view of a camera mounted on an external electronic device. According to one embodiment, the AR headset may display a U element on the AR headset display, which allows a user to identify a portion within the field of view of the external camera. For example, the AR headset may display a dotted rectangle (1630) on the AR headset display, so that the user can recognize that the interior thereof is within the field of view of the camera. The AR headset may display an image (1601) of the area (1630) within the field of view of the external camera on the AR headset display. The image (1601) may be an enlarged image of the area (1630) within the field of view of the external camera. Here, the subject of the enlargement may be the AR headset or the external electronic device. For example, in response to a request from the AR headset, the external electronic device may generate an image (1601) by enlarging an image acquired from an external camera using a digital zoom or optical zoom method and transmit the image to the AR headset. As another example, the AR headset may generate an image (1601) by enlarging an image received from the external electronic device using a digital zoom method.

[0176] FIG. 17 is a flowchart illustrating an operation performed in a first electronic device (1201) of FIG. 12 that can be worn on a user's face, according to one embodiment. According to one embodiment, a first processor (1291) in the first electronic device (1201) may be configured to perform the operation of FIG. 17. According to one embodiment, when a command stored in a memory (e.g., the first memory (1281)) in the first electronic device (1201) is executed by the first processor (1291), the first electronic device (1201) may be configured to perform the operation of FIG. 17. The operation of FIG. 17 may be performed while the first electronic device (1201) is connected to the second electronic device (1202) via a wired communication interface (e.g., a USB connector) or a wireless communication interface (e.g., Bluetooth). The operation of FIG. 17 may be performed based on the activation of the second camera (1212) in the second electronic device (1202). If the second camera (1212) is deactivated, the operation of FIG. 17 may be terminated.

[0177] In operation 1710, the first electronic device (1201) may display a window indicating an area (hereinafter, a shooting area) within the field of view of the second camera (1212) on the first display (1241). For example, the window may have a rectangular shape, as illustrated in FIGS. 15A and 16 , and thus, the user can easily identify that the interior thereof is a shooting area.

[0178] In operation 1720, the first electronic device (1201) may receive an image acquired using the second camera (1212) from the second electronic device (1202) based on whether the zoom ratio set for the second camera (1212) is greater than or equal to a specified threshold. For example, the first electronic device (1201) may receive information regarding the zoom ratio set for the second camera (1212) from the second electronic device (1201). If the zoom ratio is greater than or equal to the threshold, the first electronic device (1201) may transmit a message requesting image transmission to the second electronic device (1202). In response to the request, the second electronic device (1202) may transmit the image acquired using the second camera (1212) to the first electronic device (1201). As another example, the second electronic device (1202) may transmit an image acquired using the second camera (1212) to the first electronic device (1201) based on the zoom ratio being greater than or equal to a threshold. As another example, the second electronic device (1202) may adjust the resolution of the acquired image (e.g., adjust it to have a lower resolution than the original acquired by the second camera (1212). The second electronic device (1202) may also transmit the low-resolution image to the first electronic device (1201).

[0179] In operation 1725, the first electronic device (1201) may process an image received from the second electronic device (1202). For example, the image may be distorted (e.g., a square) depending on the direction and angle of view of the second camera (1212). The first electronic device (1201) may process the received image (e.g., warp the image) to generate a square image.

[0180] In operation 1730, the first electronic device (1201) may display an image received from the second electronic device (1202) (e.g., the second image (1502) of FIG. 15A or the image (1601) of FIG. 16) (or an image generated as a result of performing operation 1725) on the first display (1241).

[0181] In one embodiment, when the zoom factor is changed below a threshold, the first electronic device (1201) may terminate display of the image acquired using the second camera (1212).

[0182] FIG. 18 is a flowchart illustrating an operation performed in a first electronic device (1201) of FIG. 12 that can be worn on a user's face, according to one embodiment. The operation of FIG. 18 may be performed in a first electronic device (1201) applied to an AR headset. According to one embodiment, a first processor (1291) in the first electronic device (1201) may be configured to perform the operation of FIG. 18. According to one embodiment, when a command stored in a memory (e.g., the first memory (1281)) in the first electronic device (1201) is executed by the first processor (1291), the first electronic device (1201) may be configured to perform the operation of FIG. 18. The operation of FIG. 18 may be performed while the first electronic device (1201) is connected to a second electronic device (1202) via a wired communication interface (e.g., a USB connector) or a wireless communication interface (e.g., Bluetooth). The operation of FIG. 18 may be performed based on the activation of the second camera (1212) in the second electronic device (1202). If the second camera (1212) is deactivated, the operation of FIG. 18 may be terminated.

[0183] In operation 1810, the first electronic device (1201) may receive an image acquired using the second camera (1212) from the second electronic device (1202). In operation 1820, the first electronic device (1201) may display the received image (e.g., image (1601) of FIG. 16) on the first display (1241).

[0184] According to one embodiment, a wearable electronic device on a user's face (e.g., a first electronic device (1201) of FIG. 12) includes a first display arranged to shine light into a user's eye located inside the electronic device when the electronic device is worn on the face; a first camera arranged on a front side of the electronic device and configured to photograph the outside; a sensor arranged to generate data regarding a direction in which the first camera faces; a communication circuit for communicating with an external electronic device; a memory for storing instructions; and at least one processor. The above instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain information indicating a direction in which the first camera faces through the sensor, receive information regarding a direction in which a second camera included in the external electronic device faces and an angle of view of the second camera from the external electronic device through the communication circuit, determine an overlapping area that overlaps a visible area within the field of view of the second camera on the first display based on the angle of view of the first camera, the angle of view of the second camera, the angle of view of the first display, the direction in which the first camera faces, and the direction in which the second camera faces, receive an image from the external electronic device through the communication circuit, and display the image in the overlapping area.

[0185] The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine a first viewable area within the field of view of the first camera based on an angle of view of the first camera and a direction in which the first camera faces, determine a second viewable area visible to the user's eyes on the first display based on an angle of view of the first display, determine a third viewable area within the field of view of the second camera based on an angle of view of the second camera and a direction in which the second camera faces, align the second viewable area with the first viewable area such that the second viewable area is spatially included in the first viewable area, and determine an area in the second viewable area overlapping the third viewable area as the overlapping area.

[0186] The electronic device may be configured to implement virtual reality. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine the entire display area of ​​the first display as the third visible area.

[0187] The above command, when executed individually or collectively by the at least one processor, may cause the electronic device to display a user interface (UI) related to the second camera on the first display without overlapping the overlapping area.

[0188] The above instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to transmit a request message to the external electronic device through the communication circuitry to turn off a second display of the external electronic device and respond to a user input based on the UI being set to be displayed on the first display.

[0189] The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a notification on the first display requesting adjustment of a direction in which the second camera is facing based on at least a portion of the visible area within the field of view of the second camera not spatially overlapping with a visible area visible to the user on the first display.

[0190] The above instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display an indicator in the overlapping area indicating a central axis in the field of view of the second camera.

[0191] The above instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain a third image by synthesizing a first image obtained using the first camera and a second image obtained using the second camera, and to display the third image on the display.

[0192] The above instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to generate the second image by enlarging a portion included in a visible area within the field of view of the second camera in the first image.

[0193] The above instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain a fourth image by adjusting the transparency of the second image, and to obtain the third image to be displayed on the first display by synthesizing the fourth image with the second image.

[0194] According to one embodiment, a method for operating an electronic device wearable on a user's face (e.g., a first electronic device (1201) of FIG. 1) may include: acquiring information indicating a direction in which a first camera of the electronic device faces through a sensor of the electronic device; receiving information regarding a direction in which a second camera included in an external electronic device faces and an angle of view of the second camera from the external electronic device; determining an overlapping area that overlaps a visible area within a field of view of the second camera on the first display based on the angle of view of the first camera, the angle of view of the second camera, the angle of view of a first display of the electronic device, the direction in which the first camera faces, and the direction in which the second camera faces; receiving an image from the external electronic device; and displaying the image in the overlapping area.

[0195] The operation of determining the overlapping area may include: an operation of determining a first visible area within the field of view of the first camera based on the angle of view of the first camera and the direction in which the first camera is facing; an operation of determining a second visible area visible to the pupil on the first display based on the angle of view of the first display; an operation of determining a third visible area within the field of view of the second camera based on the angle of view of the second camera and the direction in which the second camera is facing; an operation of aligning the second visible area with the first visible area such that the second visible area is spatially included in the first visible area; and an operation of determining an area in the second visible area that overlaps the third visible area as the overlapping area.

[0196] The electronic device may be configured to implement virtual reality, and the operation of determining the third visible area may include an operation of determining the entire display area of ​​the first display as the third visible area.

[0197] The method may further include an action of displaying a user interface (UI) related to the second camera on the first display without overlapping the overlapping area.

[0198] The method may further include an action of transmitting a request message to the external electronic device through the communication circuit to turn off the second display of the external electronic device and react to a user input based on the UI being set to be displayed on the first display.

[0199] According to one embodiment, a recording medium is provided that stores instructions readable by at least one processor in an electronic device wearable on a user's face. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to perform the following operations: obtaining information indicating a direction in which the first camera faces through the sensor; receiving information regarding a direction in which a second camera included in the external electronic device faces and an angle of view of the second camera from the external electronic device through the communication circuit; determining an overlapping area that overlaps a visible area within the field of view of the second camera on the first display based on the angle of view of the first camera, the angle of view of the second camera, the angle of view of the first display, the direction in which the first camera faces, and the direction in which the second camera faces; receiving an image from the external electronic device through the communication circuit; and displaying the image in the overlapping area.

[0200] In the above explanation, the prefixes “first,” “second,” and “third” are only used to distinguish components of the same name, and do not have any special meaning in themselves, such as importance or order.

[0201] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0202] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0203] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. In one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0204] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0205] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0206] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device that can be worn on the user's face, A first display arranged to shine light into the user's eyes located inside the electronic device when the electronic device is worn on the face; A first camera arranged on the front of the electronic device and configured to photograph the exterior; A sensor configured to generate data regarding the direction in which the first camera is facing; Communication circuit for communicating with external electronic devices; memory that stores instructions; and Contains at least one processor, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Information indicating the direction in which the first camera is facing is acquired through the sensor, Receive information about the direction in which the second camera included in the external electronic device faces and the angle of view of the second camera from the external electronic device through the communication circuit, Based on the angle of view of the first camera, the angle of view of the second camera, the angle of view of the first display, the direction in which the first camera faces, and the direction in which the second camera faces, an overlapping area that overlaps with the visible area within the field of view of the second camera is determined on the first display, Receive an image from the external electronic device through the communication circuit, An electronic device that displays the image in the overlapping area.

2. In the first paragraph, when the command is individually or collectively executed by the at least one processor, the electronic device, Based on the angle of view of the first camera and the direction in which the first camera is facing, a first viewable area within the field of view of the first camera is determined, Based on the angle of view of the first display, a second visible area visible to the user's eyes on the first display is determined, Based on the angle of view of the second camera and the direction in which the second camera is facing, a third visible area that falls within the field of view of the second camera is determined, The second visible area is spatially aligned with the first visible area so that the second visible area is included in the first visible area, An electronic device that determines an area overlapping the third visible area in the second visible area as the overlapping area.

3. In the second paragraph, the electronic device is configured to implement virtual reality, and the instructions, when individually or collectively executed by the at least one processor, the electronic device, An electronic device that determines the entire display area of ​​the first display as the third visible area.

4. In the first paragraph, when the command is individually or collectively executed by the at least one processor, the electronic device, An electronic device that displays a UI (user interface) related to the second camera on the first display without overlapping the overlapping area.

5. In the fourth paragraph, when the command is individually or collectively executed by the at least one processor, the electronic device, An electronic device that transmits a request message to the external electronic device through the communication circuit to turn off the second display of the external electronic device and react to a user input based on the UI being set to be displayed on the first display.

6. In the first paragraph, when the instructions are individually or collectively executed by the at least one processor, the electronic device, An electronic device that displays a notification on the first display requesting adjustment of the direction in which the second camera is facing based on the fact that at least a portion of the visible area within the field of view of the second camera does not spatially overlap with the visible area visible to the user on the first display.

7. In the first paragraph, when the instructions are individually or collectively executed by the at least one processor, the electronic device, An electronic device that displays an indicator indicating a central axis in the field of view of the second camera in the overlapping area.

8. In the first paragraph, when the instructions are individually or collectively executed by the at least one processor, the electronic device, A third image is obtained by synthesizing a first image obtained using the first camera and a second image obtained using the second camera, An electronic device that causes the third image to be displayed on the display.

9. In the 8th paragraph, when the command is individually or collectively executed by the at least one processor, the electronic device, An electronic device that generates the second image by enlarging a portion included in the visible area that enters the field of view of the second camera in the first image.

10. In the 8th paragraph, when the command is individually or collectively executed by the at least one processor, the electronic device, By adjusting the transparency of the second image, a fourth image is obtained, An electronic device that obtains the third image to be displayed on the first display by synthesizing the fourth image with the second image.

11. A method for operating an electronic device wearable on a user's face, An operation of acquiring information indicating a direction in which a first camera of the electronic device is facing through a sensor of the electronic device; An operation of receiving information about a direction in which a second camera included in an external electronic device faces and an angle of view of the second camera from the external electronic device; An operation of determining an overlapping area that overlaps with a visible area within the field of view of the second camera on the first display based on the angle of view of the first camera, the angle of view of the second camera, the angle of view of the first display of the electronic device, the direction in which the first camera faces, and the direction in which the second camera faces; An operation of receiving an image from the external electronic device; and A method comprising the action of displaying the image in the overlapping area.

12. In the 11th paragraph, the operation of determining the overlapping area is: An operation of determining a first visible area within the field of view of the first camera based on the angle of view of the first camera and the direction in which the first camera is facing; An operation of determining a second visible area visible to the pupil on the first display based on the angle of view of the first display; An operation of determining a third visible area that falls within the field of view of the second camera based on the angle of view of the second camera and the direction in which the second camera is facing; An operation of aligning the second visible area with the first visible area such that the second visible area is spatially included in the first visible area; and A method comprising an operation of determining an area overlapping the third visible area in the second visible area as the overlapping area.

13. In the 12th paragraph, the electronic device is configured to implement virtual reality, and the operation of determining the third visible area is: A method comprising an operation of determining the entire display area of ​​the first display as the third visible area.

14. In paragraph 11, A method further comprising an action of displaying a UI (user interface) related to the second camera on the first display without overlapping the overlapping area.

15. In paragraph 14, A method further comprising the action of transmitting a request message to the external electronic device through the communication circuit to turn off the second display of the external electronic device and cause it to respond to a user input based on the UI being set to be displayed on the first display.

Images