Method for generating extended reality content and electronic device for performing same

Abstract

An electronic device according to an embodiment may: acquire sensing data about a real space from a sensor of the electronic device; identify a reflector included in the real space on the basis of the sensing data; add a geometric model of the reflector to scene information for generating a content image including a virtual object; generate a reflected image, in which at least a portion of the virtual object is projected, by rendering the geometric model on the basis of the scene information; and generate a content image, in which the reflected image overlaps the reflector in the real space, on the basis of the sensing data.

Description

Method for creating extended reality content and electronic device for performing the same

[0001] Hereinafter, a technology for generating extended reality content including virtual objects is disclosed.

[0002] Recently, extended reality technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) are being developed, utilizing computer graphics technology. VR technology allows users to experience computer-generated virtual spaces that do not exist in the real world as if they were real. AR or MR technologies integrate the real and virtual worlds by overlaying computer-generated information onto the real world, enabling users to interact with both the real and virtual worlds.

[0003] The importance of realistic representations of virtual worlds and virtual objects in users' augmented reality experiences is emphasized. The realism of augmented reality content can be enhanced by incorporating light sources in real-world scenes, such as brightness, glow, shadows, or reflections on virtual objects.

[0004] According to an embodiment, an electronic device may include a sensor. The electronic device may include at least one processor including processing circuitry. The electronic device may include a memory including one or more storage media storing instructions. When the instructions are individually or collectively executed by the at least one processor, the electronic device may: obtain sensing data for a real space from the sensor. When the instructions are individually or collectively executed by the at least one processor, the electronic device may: identify a reflector included in the real space based on the sensing data, and add a geometry model of the reflector to scene information for generating a content image including a virtual object. When the instructions are individually or collectively executed by the at least one processor, the electronic device may: generate a reflection image in which at least a portion of the virtual object is projected by rendering the geometry model based on the scene information. When the above commands are individually or collectively executed by the at least one processor, the electronic device may: generate a content image in which the reflected image is superimposed on the reflector in the real space based on the sensing data.

[0005] According to one embodiment, a method for generating a content image performed by an electronic device may include an operation of acquiring sensing data regarding a real space from a sensor of the electronic device. The method may include an operation of identifying a reflector included in the real space based on the sensing data. The method may include an operation of adding a geometry model of the reflector to scene information for generating a content image including a virtual object. The method may include an operation of generating a reflection image in which at least a portion of the virtual object is projected by rendering the geometry model based on the scene information. The method may include an operation of generating a content image in which the reflection image is superimposed on the reflector in the real space based on the sensing data.

[0006] According to one embodiment, a non-transitory computer-readable recording medium may store one or more programs including instructions. When the instructions are individually or collectively executed by at least one processor of an electronic device, the instructions may cause the electronic device to: obtain sensing data for a real space from a sensor of the electronic device. When the instructions are individually or collectively executed by the at least one processor, the instructions may cause the electronic device to: identify a reflector included in the real space based on the sensing data, and add a geometry model of the reflector to scene information for generating a content image including a virtual object. When the instructions are individually or collectively executed by the at least one processor, the instructions may cause the electronic device to: generate a reflection image in which at least a portion of the virtual object is projected by rendering the geometry model based on the scene information. When the above commands are individually or collectively executed by the at least one processor, the electronic device may: generate a content image in which the reflected image is superimposed on the reflector in the real space based on the sensing data.

[0007] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0008] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to one embodiment.

[0009] FIGS. 3A and 3B are drawings showing the front and back of a wearable electronic device according to one embodiment.

[0010] FIG. 4 is a schematic block diagram of an electronic device according to one embodiment.

[0011] FIG. 5 is a diagram illustrating a method for providing extended reality content according to one embodiment.

[0012] FIG. 6 is a flowchart of a method for generating extended reality content according to one embodiment.

[0013] FIG. 7 is a diagram illustrating scene information according to one embodiment.

[0014] FIGS. 8A and 8B are drawings each illustrating the occupancy rate and position of a reflector according to one embodiment.

[0015] FIG. 9a is a drawing illustrating a method for generating a geometric model of a reflector according to one embodiment.

[0016] FIG. 9b is a flowchart of a method for generating a geometric model of a reflector according to one embodiment.

[0017] FIG. 10a is a drawing illustrating a method for rendering a geometric model according to one embodiment.

[0018] FIG. 10b is a flowchart of a method for rendering a geometric model according to one embodiment.

[0019] FIG. 11a is a drawing illustrating a method for rendering a geometric model according to one embodiment.

[0020] FIG. 11b is a flowchart of a method for rendering a geometric model according to one embodiment.

[0021] FIGS. 12 to 15 are drawings illustrating a method for providing extended reality content according to various examples.

[0022] Hereinafter, an embodiment of the present document may be described with reference to the attached drawings.

[0023] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0024] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0025] The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0026] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0027] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0028] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0029] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0030] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0031] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0032] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0033] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0034] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0035] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0036] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0037] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0038] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0039] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0040] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0041] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0042] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0043] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0044] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0045] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service by itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0046] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to one embodiment.

[0047] Referring to FIG. 2, a wearable electronic device (200) according to an embodiment may include at least one of a light output module (211), a display member (201), and a camera module (250). An electronic device (e.g., an electronic device (101) of FIG. 1) may be implemented in the form of a wearable electronic device (200). A wearable electronic device (200) (e.g., an electronic device (102) of FIG. 1) may be implemented separately from an electronic device (e.g., an electronic device (101) of FIG. 1). A wearable electronic device (200) (e.g., an electronic device (102) of FIG. 1) may be connected to an electronic device (e.g., an electronic device (101) of FIG. 1).

[0048] According to one embodiment, the light output module (211) may include a light source capable of outputting an image and a lens that guides the image to the display member (201). The output module (211) may include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0049] According to one embodiment, the display member (201) may include an optical waveguide (e.g., a waveguide). An output image of an optical output module (211) incident on one end of the optical waveguide may be propagated inside the optical waveguide and provided to a user. The optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). For example, the optical waveguide may guide an output image of the optical output module (211) to a user's eye by using at least one diffractive element or reflective element.

[0050] According to one embodiment, the camera module (250) can capture still images and / or videos. The camera module (250) is disposed within the lens frame and may be disposed around the display member (201). The camera module (250) may include a first camera module (251), a second camera module (253), and a third camera module (255).

[0051] According to one embodiment, the first camera module (251) can capture and / or recognize the trajectory of the user's eye (e.g., pupil, iris) or gaze. The first camera module (251) can periodically or aperiodically transmit information related to the trajectory of the user's eye or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1).

[0052] According to one embodiment, the second camera module (253) can capture an external image.

[0053] According to one embodiment, the third camera module (255) can be used for hand detection and tracking, and user gesture (e.g., hand movement) recognition. The third camera module (255) can be used for 3 degrees of freedom (3DoF), 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. The second camera module (253) can also be used for hand detection and tracking, and user gesture recognition. At least one of the first camera module (551) to the third camera module (255) can be replaced with a sensor module (e.g., a LiDAR sensor). For example, the sensor module can include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0054] FIGS. 3A and 3B are drawings showing the front and back of a wearable electronic device according to one embodiment.

[0055] Referring to FIGS. 3A and 3B, according to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1) may be implemented in the form of a wearable electronic device (300). Camera modules (311, 312, 313, 314, 315, 316) (e.g., camera module (180) of FIG. 1) and / or depth sensor (317) (e.g., sensor module (176) of FIG. 1) for obtaining information related to the surrounding environment of the wearable electronic device (300) may be arranged on a first surface (310) of the housing.

[0056] According to one embodiment, the camera modules (311, 312) can acquire images related to the environment surrounding the wearable electronic device.

[0057] According to one embodiment, the camera modules (313, 314, 315, 316) can acquire images while the wearable electronic device is worn by a user. The camera modules (313, 314, 315, 316) can be used for hand detection, tracking, and recognition of user gestures (e.g., hand movements). The camera modules (313, 314, 315, 316) can be used for 3DoF, 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. The camera modules (311, 312) can also be used for hand detection and tracking, and recognition of user gestures.

[0058] According to one embodiment, the depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Instead of or in addition to the depth sensor (317), the camera modules (313, 314, 315, 316) may determine the distance to an object.

[0059] According to one embodiment, a camera module (325, 326) for facial recognition (e.g., camera module (180) of FIG. 1) and / or a display (321) (e.g., display module (160) of FIG. 1) (and / or a lens) may be disposed on the second side (320) of the housing.

[0060] According to one embodiment, a face recognition camera module (325, 326) adjacent to the display may be used to recognize a user's face, or may recognize and / or track both eyes of the user.

[0061] According to one embodiment, the display (321) (and / or lens) may be disposed on the second side (320) of the wearable electronic device (300). The wearable electronic device (300) may not include camera modules (315, 316) among the plurality of camera modules (313, 314, 315, 316). Although not illustrated in FIGS. 3A and 3B , the wearable electronic device (300) may further include at least one of the configurations illustrated in FIG. 2 .

[0062] As described above, according to one embodiment, the wearable electronic device (300) may have a form factor for being worn on a user's head. The wearable electronic device (300) may further include a strap and / or a wearing member for being secured to a body part of the user. The wearable electronic device (300) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0063] FIG. 4 is a schematic block diagram of an electronic device according to one embodiment.

[0064] According to one embodiment, the electronic device (400) may include at least a part of the configuration of the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, and / or the wearable electronic device (300) of FIGS. 3A and 3B. The electronic device (400) may be implemented in the form of a wearable electronic device (e.g., the wearable electronic device (200) of FIG. 2) (e.g., smart glasses including virtual reality glasses). The electronic device (400) may also be implemented in the form of a wearable electronic device (e.g., the wearable electronic device (300) of FIG. 3) (e.g., a head mounted display (HMD) including an augmented reality (AR) device, a virtual reality (VR) device, and / or a mixed reality (MR) device). The electronic device (400) may be configured to easily control a UI (user interface) component provided in an AR environment, a VR environment, and / or an MR environment.

[0065] According to one embodiment, the electronic device (400) may be a VST (video see-through) type configured to block external light so that, when worn, light emitted from the display reaches the user's eyes but external light does not reach the user's eyes. According to another embodiment, the electronic device (400) may include an OST (optical see-through) type configured to allow external light to reach the user's eyes through the glasses when worn.

[0066] According to one embodiment, an electronic device (400) (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, or the wearable electronic device (300) of FIGS. 3A and 3B) may include a processor (410) (e.g., the processor (120) of FIG. 1), a memory (420) (e.g., the memory (130) of FIG. 1), and a sensor (430) (e.g., the sensor module (176) of FIG. 1, the camera module (180), the camera module (250) of FIG. 2, the first camera module (251), the second camera module (253), the depth sensor (317) of FIG. 3A, or the camera modules (311, 312, 313, 314, 315, 316, 325, 326) of FIGS. 3A and 3B).

[0067] According to one embodiment, the processor (410) may include at least one processor including a processing circuit.

[0068] According to one embodiment, the memory (420) may include one or more storage media for storing instructions. The memory (420) may include an image acquisition module (421), an image analysis module (422), and a rendering module (423). The rendering module (423) may include a reflection map generation module (424) and a reflection image generation module (425). When the instructions stored in the memory (e.g., the image acquisition module (421), the image analysis module (422), or the rendering module (423)) are individually or collectively executed by the processor (410), they may cause the electronic device (400) to perform at least a part of the method for generating extended reality content (or content image) of the present disclosure.

[0069] According to one embodiment, the sensor (430) may convert the measured or sensed information into an electrical signal (or sensed data) by measuring or sensing a physical quantity. For example, the sensor (430) may include at least one camera or image sensor for capturing at least one frame of a still image or video of a real space (or a physical environment). For example, the sensor (430) may include at least one button for touch input, a gesture sensor, a gyroscope, a gyro sensor, a barometric pressure sensor, a magnetic sensor, a magnetometer, an acceleration sensor, an accelerometer, a grip sensor, a proximity sensor, an RGB sensor, a biophysical sensor, a temperature sensor, a humidity sensor, an illuminance sensor, an ultraviolet sensor, an electromyography sensor, an electroencephalography sensor, an electrocardiogram sensor, an infrared sensor, an ultrasonic sensor, an iris sensor, or a fingerprint sensor, but the present disclosure is not limited thereto.

[0070] In one embodiment, the sensor (430) can capture an actual space including an object. The sensor (430) can generate sensing data for generating a point cloud corresponding to the object. For example, the sensor (430) can be configured as at least one of an image sensor, a LiDAR sensor, a red-green-blue depth (RGB-D) sensor, a depth sensor, a time of flight (ToF) sensor, an ultrasonic sensor, a radar sensor, and a stereo camera, but the present disclosure is not limited thereto.

[0071] According to one embodiment, the sensing data may be at least one still image or video of a real space. The sensing data may be an image (or a real space image) of one or more objects contained in the real space. The sensing data may include depth information. For example, the sensing data may be a color image including depth information, such as an RGB-D image.

[0072] According to one embodiment, the electronic device (400) can obtain sensing data from a sensor (430) through an image acquisition module (421). The electronic device (400) can provide extended reality content using the sensing data obtained from the sensor (430). The electronic device (400) can generate an image of the extended reality content (hereinafter, “content image”) by blending a physical environment and a virtual environment based on the sensing data. The content image can include one or more objects included in a physical environment captured in real time by the electronic device (400) and a virtual object such as an avatar, a control element, an interactive element, a user interface element, or any graphic element.

[0073] According to one embodiment, the electronic device (400) can analyze sensing data through the image analysis module (422). The electronic device (400) can identify information such as an object in a real space, an attribute of the object, a location of the object, a light source, and a location (or viewpoint) of the electronic device (400) based on the sensing data through the image analysis module (422). For example, the electronic device (400) can identify a reflector (or a reflective object) in a real space based on the sensing data through the image analysis module (422). The electronic device (400) can determine the reflectivity of the reflector based on the sensing data through the image analysis module (422).

[0074] According to one embodiment, the electronic device (400) can generate a virtual object included in a virtual environment through the rendering module (423). The electronic device (400) can generate a content image in which an image of one or more objects included in a real space is captured and a virtual object are integrated through the rendering module (423).

[0075] According to one embodiment, the electronic device (400) can simulate graphics in which a virtual object is reflected on a reflector included in a real space through the rendering module (423). That is, the electronic device (400) can present a reflection effect of a virtual object on a reflector in a real space through the rendering module (423). The rendering process for presenting the reflection effect (or applying the reflection effect) can be performed in parallel with the main rendering process for generating a content image, including generating a virtual object.

[0076] According to one embodiment, the electronic device (400) may generate a geometry model corresponding to a reflector through the rendering module (423) to present a reflection effect. The geometry model may be a mesh representing the geometric structure of the reflector. The electronic device (400) may generate a reflection image in which at least a portion of a virtual object is projected based on the geometry model of the reflector through the reflection image generation module (425). Specifically, for example, the electronic device (400) may generate a reflection image by calculating a gaze vector toward the geometry model corresponding to the reflector and a reflection vector in which the gaze vector is reflected by the geometry model through the reflection image generation module (425). This method may be by ray tracing. For example, the electronic device (400) may generate a reflection map corresponding to the geometry model of the reflector through the reflection map generation module (424). The electronic device (400) may generate a reflection image based on the reflection map through the reflection image generation module (425). This method may be by projection pipeline. The method of generating a reflected image is described in detail with reference to FIGS. 10a, 10b, 11a, and 11b.

[0077] FIG. 5 is a diagram illustrating a method for providing extended reality content according to one embodiment.

[0078] The screens (510, 520) of FIG. 5 represent extended reality content provided by an electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIGS. 3A and 3B, or the electronic device (400) of FIG. 4). The electronic device can provide the extended reality content through a display (e.g., the display module (160) of FIG. 1).

[0079] Augmented reality content can be presented as multiple views while the user wears an electronic device (e.g., a head-mounted display). The term "view" can have various meanings. For example, if the augmented reality content is an omnidirectional image, the term "view" can refer to a portion of the augmented reality content projected from the user's perspective. The term "view" can also be referred to as a "viewport." For example, the term "view" can refer to an image rendered based on the position (or viewpoint) of a camera in the three-dimensional space of the augmented reality content. For example, the term "view" can refer to a scene or portion of a real space represented by sensing data acquired by a sensor of the electronic device.

[0080] The screens (510, 520) may display images of one or more objects included in a real space, which are captured images, and images of extended reality content (hereinafter, “content images”) that integrate virtual objects. Through the extended reality content, a user may interact with the virtual objects. According to an embodiment, the virtual objects may interact with the user and / or objects included in the real space in the extended reality content. According to an embodiment, the user may interact with objects included in the real space in the extended reality content.

[0081] The screen (510, 520) includes a reflector (51), which is an object contained in a real space, and an avatar (53), which is a virtual object. The reflector (51) may be an object including a surface with specular reflection properties. For example, the reflector (51) may include objects such as a mirror, liquid, metal (e.g., stainless steel, aluminum), or plastic. In general, the reflector (51) contained in a real space can only reflect other objects contained in the real space, and cannot reflect virtual objects. Referring to the screen (510), the reflection of the avatar (53) is not presented within the reflector (51).

[0082] According to one embodiment, the electronic device can simulate a graphic in which an avatar (53) is reflected on a reflector (51) in a real space. That is, the electronic device can present a reflection effect of a virtual object, such as an avatar (53), on the reflector (51) in a real space. The rendering process for presenting the reflection effect (or applying the reflection effect) can be performed in parallel with the main rendering process for generating a content image, including the creation of a virtual object.

[0083] In a rendering process for presenting a reflection effect according to one embodiment, an electronic device may identify a reflector (51) included in a real space based on sensing data acquired by a sensor. The electronic device may generate a geometric model corresponding to the reflector (51) to present a reflection effect of an avatar (53) on the identified reflector (51). The electronic device may generate a reflection image in which at least a portion of the avatar (53) is projected based on the geometric model of the reflector (51).

[0084] According to one embodiment, when a plurality of reflectors are identified, the electronic device can repeatedly perform a rendering process for presenting a reflection effect corresponding to each of the plurality of reflectors.

[0085] According to one embodiment, the electronic device can generate a content image in which a projection of an avatar (53) is superimposed on a reflector (51) in a real space by using sensing data (or an image of a real space) and a generated reflection image. Referring to the screen (520), a reflection of the avatar (53) can be presented within the reflector (51). A method for generating a content image is described in detail with reference to FIG. 6.

[0086] According to an embodiment, an electronic device may determine whether to present a reflection effect of a virtual object based on a user input. The user input may include motions, gestures, voice, touch, gaze, and inputs from an external device connected to the electronic device, such as a controller, of a user wearing the electronic device. The electronic device may set options related to the reflection effect of the virtual object (e.g., whether to apply the reflection effect to the virtual object, selection of a reflector to apply the reflection effect, or a method of generating the reflection effect) based on the user input. According to an embodiment, referring to screen (510), the electronic device may output a virtual object (e.g., interactive contents, a dialog box) including setting options related to the reflection effect of the virtual object. According to an embodiment, the electronic device may switch whether to apply the reflection effect, such as from screen (510) to screen (520), or from screen (520) to screen (510), based on the user input.

[0087] FIG. 6 is a flowchart of a method for generating extended reality content according to one embodiment.

[0088] According to one embodiment, the operations 610 to 650 below may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIGS. 3A and 3B, or the electronic device (400) of FIG. 4). For example, the electronic device may include at least one processor including a processing circuit (e.g., the processor (120) of FIG. 1 or the processor (410) of FIG. 4), a memory (e.g., the memory (130) of FIG. 1 or the memory (420) of FIG. 4), a communication unit (e.g., the communication module (190) of FIG. 1), and a sensor (e.g., the sensor module (176) of FIG. 1, the camera module (180), the camera module (250) of FIG. 2, the first camera module (251), the second camera module (253), the depth sensor (317) of FIG. 3A, or the camera modules (311, 312, 313, 314, 315, 316, 325, 326) of FIGS. 3A and 3B, or the sensor (430) of FIG. 4).

[0089] In operation 610, the electronic device can obtain sensing data about the actual space from the sensor.

[0090] The sensing data may be images of one or more objects within a real space. The sensing data may include depth information. For example, the sensing data may be a color image containing depth information, such as an RGB-D image.

[0091] In operation 620, the electronic device may identify a reflector contained in a real space based on the sensing data. For example, the reflector may include an object such as a mirror, a liquid, a metal (e.g., stainless steel, aluminum), or a plastic.

[0092] According to one embodiment, an electronic device may estimate spatial correspondence and / or temporal correspondence between objects contained in a real space based on sensed data. 'Relationship' may refer to alignment or relationship between elements or features observed in a real space image (or view of a real space) represented by the sensed data.

[0093] According to one embodiment, an electronic device may estimate a spatial relationship between contents inside a reflector and contents outside the reflector included in the same frame of a real space image based on sensing data. The spatial relationship between a reflected object inside the reflector and a real object outside the reflector may exist at an intra-frame level. The electronic device may estimate a spatial relationship between objects included in at least one frame in which the real space is captured.

[0094] According to one embodiment, the electronic device can estimate the temporal relationship between the contents inside reflectors, contents outside reflectors, or contents inside and outside reflectors included in different frames of a real space image based on sensing data. The temporal relationship between reflected objects inside reflectors, real objects outside reflectors, or reflected objects and real objects included in different frames may exist at an inter-frame level. The electronic device can estimate the temporal relationship between objects included in one or more frames in which the real space is captured. If some of the reflected objects inside reflectors or real objects outside reflectors are not included in some frames of the sensing data due to changes in the angle and movement of the electronic device, the accuracy of reflector identification can be improved by estimating the temporal relationship with other frames that include both reflected objects and real objects.

[0095] According to one embodiment, an electronic device may identify a reflector based on at least one of a spatial relationship or a temporal relationship between objects contained in a real space based on sensing data. The electronic device may identify a reflector by matching an arbitrary object in the real space indicated by the temporal relationship and / or the spatial relationship with the reflected appearance of the object.

[0096] In operation 630, the electronic device can add a geometric model of the reflector to scene information for generating a content image including a virtual object.

[0097] The geometric model of a reflector may be a mesh representing the geometric structure of the reflector. As a geometric model of the reflector, the mesh (or mesh data) may contain geometric information of the reflector that serves as the basis for rendering. A method for generating a geometric model of a reflector is described in detail with reference to FIGS. 9A and 9B.

[0098] Scene information may include information describing objects within a scene of augmented reality content, including various attributes such as location, texture, and other information. Scene information may be referred to as a scene graph, scene description, or scene data structure. An electronic device can generate content images based on the scene information. Scene information is described in detail with reference to FIG. 7.

[0099] In operation 640, the electronic device can generate a reflection image in which at least a portion of the virtual object is projected by rendering a geometric model based on scene information. A method for generating the reflection image is described in detail with reference to FIGS. 10A, 10B, 11A, and 11B.

[0100] In operation 650, the electronic device can generate a content image in which a reflected image is superimposed on a reflector in a real space based on the sensing data.

[0101] In one embodiment, the electronic device may generate a mask corresponding to the contour or area of ​​interest of the reflector based on sensing data. Using the generated mask, the electronic device may generate a content image in which the reflected image is superimposed on the reflector in real space.

[0102] According to one embodiment, the electronic device can output a content image through a display (e.g., a display module (160) of FIG. 1, a light output module (211) of FIG. 2, or a display (321) of FIG. 3b).

[0103] According to one embodiment, the aforementioned operations 610 to 650 may be performed to generate an image of mixed reality content. The electronic device may be of the VST type, which is configured to block external light so that, when worn, light emitted from the display reaches the user's eyes, but external light does not reach the user's eyes.

[0104] According to an embodiment, an electronic device may generate an image of augmented reality content through some modifications to the aforementioned operations 610 to 650. For example, the electronic device may obtain sensing data about a real space from a sensor. The electronic device may identify a reflector included in the real space based on the sensing data. The electronic device may add a geometric model of the reflector to scene information. The electronic device may generate a reflected image in which at least a portion of a virtual object is projected by rendering the geometric model based on the scene information. The electronic device may be of the OST type configured to allow external light to reach the user's eyes through glasses. The electronic device may output virtual content including a virtual object and a reflected image. In this case, the electronic device may output the virtual content in real time so that the reflected image is superimposed on the reflector in the real space based on the sensing data.

[0105] According to one embodiment, if a virtual object corresponding to a reflector (e.g., a virtual mirror) (hereinafter, referred to as a virtual reflector) exists among virtual objects included in scene information, the electronic device may present a reflection effect for the virtual reflector. For example, the electronic device may generate a reflection image of the virtual reflector onto which at least a portion of a virtual object other than the virtual reflector is projected. For example, the electronic device may generate a reflection image of the virtual reflector onto which at least a portion of an object included in a real space is projected.

[0106] FIG. 7 is a diagram illustrating scene information according to one embodiment.

[0107] Scene information may include information describing objects in a scene of augmented reality content, including various properties such as position, texture, and other information. Scene information may be referred to as a scene graph, scene description, or scene data structure.

[0108] Scene information can be represented as a scene graph in a format such as glTF (graphics library transmission format) or USD (universal scene description). For example, scene information can be represented in the form of a node tree.

[0109] Scene information may contain multiple nodes and multiple edges.

[0110] Each of the multiple nodes of scene information can represent information such as objects included in the scene, their positions, object animations, visual characteristics of the objects, and light sources included in the scene. Depending on the embodiment, each node may represent not only virtual objects but also objects included in real space.

[0111] Each of the multiple edges of the scene information may represent a relationship characteristic (e.g., spatial relationship, temporal relationship) between nodes. The nodes and edges of the scene information may represent various information and characteristics known in the art, and are not limited to the examples described or illustrated in the present disclosure.

[0112] For example, referring to Figure 7, scene information may include nodes corresponding to cameras. Two camera nodes may be used to simulate a user's binocular vision, capturing the space from slightly different angles. Each camera node can influence factors such as perspective, focus, and field of view by specifying how a portion of the scene is rendered.

[0113] Scene information can include nodes corresponding to light sources. Light source nodes control characteristics such as the location, intensity, and color of the light source, and can influence how objects within the extended reality space are illuminated. For example, light source nodes can represent natural light, lighting, or virtual light sources in real space.

[0114] Scene information can include nodes corresponding to objects. Object nodes can represent virtual objects, such as control elements (e.g., buttons, menu panels, sliders), interactive elements (e.g., avatars, game content, prompts, dialog boxes, tools for interacting with augmented reality), user interface elements (e.g., application launch screens), or any graphical element. Object nodes can represent not only virtual objects but also objects contained in real space.

[0115] Scene information may include nodes corresponding to properties of an object. For example, scene information may include nodes corresponding to the material or other characteristics of an object.

[0116] According to one embodiment, an electronic device (e.g., an electronic device (101) of FIG. 1, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIGS. 3A and 3B, or an electronic device (400) of FIG. 4) can generate an image of extended reality content (hereinafter, a content image) based on scene information.

[0117] As described with reference to FIG. 6, the electronic device can identify a reflector in a real space based on sensing data. The electronic device can generate a geometric model of the identified reflector to apply a reflection effect of a virtual object to the identified reflector. The electronic device can add the generated geometric model of the reflector to scene information. As illustrated in FIG. 7, a node ('reflector') (hereinafter, 'reflector node') corresponding to the geometric model of the reflector can be generated in the scene information. When a plurality of reflectors are identified, the electronic device can add geometric models corresponding to each of the plurality of reflectors to the scene information. Nodes corresponding to each of the plurality of reflectors can be generated in the scene information.

[0118] According to one embodiment, an electronic device may determine the reflectivity (or reflection coefficient) of a reflector based on sensing data. The electronic device may add the determined reflectivity to scene information by associating it with a geometric model of the reflector. A node (e.g., a material node) corresponding to the reflectivity may be created in the scene information. As illustrated in FIG. 7, a reflector node and a node corresponding to the reflectivity of the reflector may be connected by an edge. According to an embodiment, the reflector node may have metadata including the determined reflectivity. The electronic device may add the determined reflectivity to the scene information as metadata of the reflector node. The electronic device may use the reflectivity to adjust the clarity with which a virtual object is projected on a reflected image.

[0119] FIGS. 8A and 8B are drawings each illustrating the occupancy rate and position of a reflector according to one embodiment.

[0120] As described above with reference to FIG. 6, an electronic device (e.g., an electronic device (101) of FIG. 1, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIGS. 3A and 3B, or an electronic device (400) of FIG. 4) can obtain sensing data for a real space from a sensor of the electronic device (e.g., a sensor module (176) of FIG. 1, a camera module (180) of FIG. 1, a camera module (250) of FIG. 2, a first camera module (251), a second camera module (253), a depth sensor (317) of FIG. 3A, or camera modules (311, 312, 313, 314, 315, 316, 325, 326) of FIGS. 3A and 3B, or a sensor (430) of FIG. 4). The electronic device can identify a reflector included in the real space based on the sensing data.

[0121] According to one embodiment, the electronic device can generate a content image with a virtual object's reflection effect applied unconditionally to the identified reflector. If a single reflector or multiple reflectors are identified, the electronic device can enhance realism by applying the virtual object's reflection effect to all reflector(s).

[0122] According to one embodiment, the electronic device may generate a content image with a virtual object's reflection effect applied to the identified reflector if the identified reflector satisfies a specified condition. If a single reflector is identified, the electronic device may generate a content image without the virtual object's reflection effect applied if the reflector does not satisfy the specified condition. If multiple reflectors are identified, the electronic device may generate a content image with the virtual object's reflection effect applied only to reflectors that satisfy the specified condition. The electronic device may save rendering resources by omitting the application of the reflection effect to reflectors with low importance.

[0123] According to one embodiment, the electronic device may add a geometric model of a reflector to scene information if at least one of an occupancy rate and a position of the reflector in a view of a real space satisfies a specified condition based on the sensing data. That is, the electronic device may not generate a geometric model of a reflector if at least one of an occupancy rate and a position in a view of a real space does not satisfy a specified condition based on the sensing data.

[0124] Figures 8a and 8b illustrate reflectors (81, 83, 85, 87, 89) identified in arbitrary views of real space, respectively.

[0125] According to one embodiment, the electronic device may add a geometric model of the reflector to the scene information if the occupancy rate of the reflector in the view of the real space satisfies a specified condition based on the sensing data.

[0126] An electronic device can determine the occupancy (or area) of a reflector in a view of a real space based on sensing data. If multiple reflectors are identified, the electronic device can determine the occupancy of each of the multiple reflectors based on the sensing data. If the determined occupancy of a reflector satisfies a specified condition (e.g., greater than a threshold), the electronic device can add a geometric model of the reflector to the scene information.

[0127] Referring to FIG. 8A, the electronic device can determine the occupancy rate (21%, 8%, 2.1%) of each of the reflectors (81, 83, 85) in the view of the actual space based on the sensing data. The electronic device can add a geometric model of the reflector (81, 83) that satisfies a specified condition (e.g., an occupancy rate of 5% or more) among the reflectors (81, 83, 85) to the scene information.

[0128] According to one embodiment, the electronic device may add a geometric model of the reflector to the scene information if the position of the reflector in the view of the real space satisfies a specified condition based on the sensing data.

[0129] The electronic device can determine the location (e.g., coordinates) of a reflector in a view of a real space based on sensing data. If multiple reflectors are identified, the electronic device can determine the location of each of the multiple reflectors based on the sensing data. If the determined location of the reflector satisfies a specified condition (e.g., a location within the main field of view), the electronic device can add a geometric model of the reflector to the scene information.

[0130] Referring to FIG. 8B, the electronic device can determine the position of each of the reflectors (87, 89) in the view of the real space based on the sensing data. The electronic device can add a geometric model of a reflector (87) that satisfies a specified condition among the reflectors (87, 89) to the scene information. For example, the electronic device can add a geometric model of a reflector (87) whose center point is located within the main field of view, whose center point is located within a certain distance from the center of the view of the real space, or whose area is within a certain degree or more of the main field of view to the scene information.

[0131] FIG. 9a is a diagram illustrating a method for generating a geometric model of a reflector according to an embodiment. FIG. 9b is a flowchart of a method for generating a geometric model of a reflector according to an embodiment.

[0132] According to one embodiment, in FIG. 9B, the operations 910 to 940 below may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIGS. 3A and 3B, or the electronic device (400) of FIG. 4). For example, the electronic device may include at least one processor including a processing circuit (e.g., the processor (120) of FIG. 1 or the processor (410) of FIG. 4), a memory (e.g., the memory (130) of FIG. 1 or the memory (420) of FIG. 4), a communication unit (e.g., the communication module (190) of FIG. 1), and a sensor (e.g., the sensor module (176) of FIG. 1, the camera module (180), the camera module (250) of FIG. 2, the first camera module (251), the second camera module (253), the depth sensor (317) of FIG. 3A, or the camera modules (311, 312, 313, 314, 315, 316, 325, 326) of FIGS. 3A and 3B, or the sensor (430) of FIG. 4).

[0133] According to one embodiment, operation 630 of adding a geometric model of a reflector of FIG. 6 to scene information may include operations 910 to 940 below.

[0134] Referring to (a) of FIG. 9A, according to one embodiment, the electronic device can identify a reflector (51) included in a real space based on sensing data.

[0135] In operation 910, the electronic device can determine the outline or region of interest of the reflector (51) based on the sensing data.

[0136] In one embodiment, the electronic device can determine whether the identified reflector (51) is composed of a simple reflective plane based on the sensing data. If the reflector (51) is a simple reflective plane (e.g., a plane, a cube, or a spherical mirror), the electronic device can determine (or extract) the outline of the reflector (51). If the reflector (51) is an object having a complex shape (e.g., a curved glass bottle, an unstable surface of water), the electronic device can determine (or extract) the region of interest of the reflector (51).

[0137] Referring to (b) of FIG. 9a, according to one embodiment, the electronic device can generate a mask corresponding to the outline or area of ​​interest of the reflector (51). Using the generated mask, the electronic device can generate a content image in which the reflected image is superimposed on the reflector (51).

[0138] At operation 920, the electronic device may generate a point cloud of the reflector (51) in the contour or region of interest based on depth information included in the sensing data.

[0139] A point cloud is a collection of points located at geometric locations in three-dimensional space. The points in the point cloud may contain geometric information of a reflector (51) that can be expressed in coordinates (e.g., coordinate values ​​in a rectangular coordinate system or a polar coordinate system).

[0140] The electronic device can generate a point cloud of the reflector (51) by sampling a plurality of points included within an outline or region of interest based on depth information included in the sensing data. The process of sampling the plurality of points may include a process of determining the coordinates of each point based on the depth information included in the sensing data.

[0141] In operation 930, the electronic device may generate a mesh including the connection relationship of points of a point cloud as a geometric model of the reflector (51). As a geometric model of the reflector, the mesh (or mesh data) may include geometric information of the reflector (51) that serves as the basis for rendering.

[0142] The electronic device can remove duplicate points and noise points contained in the point cloud. The electronic device can reduce the processing complexity and distortion of mesh generation by removing repetitive points and points that do not represent the actual surface of the reflector (51).

[0143] The electronic device can reconstruct points in unnatural areas to compensate for missing or removed points in the point cloud. The electronic device can estimate missing points or adjust vertex normals through vertex normalization to ensure that the mesh accurately represents the original shape of the reflector (51).

[0144] In operation 940, the electronic device may add a geometric model to the scene information. A node corresponding to the geometric model of the reflector (51) may be added to the scene information.

[0145] According to one embodiment, a node corresponding to a geometric model of a reflector (51) may have metadata including information about properties of the reflector (51), such as reflectivity.

[0146] According to one embodiment, the electronic device may add information about the properties of the reflector (51) to the scene information by associating it with a geometric model of the reflector (51). A node corresponding to the geometric model of the reflector (51) may be connected by an edge to a node corresponding to the properties of the reflector (51).

[0147] As described with reference to FIG. 6, the electronic device can generate a reflected image in which at least a portion of a virtual object is projected by rendering a geometric model based on scene information.

[0148] According to one embodiment, the electronic device can generate a rendered geometric model without a virtual object being projected by rendering a geometric model of a reflector (51) based on scene information. The rendered geometric model can be a two-dimensional image of a mesh. The electronic device can generate a reflected image based on the rendered geometric model of the reflector (51), as described with reference to FIGS. 10A, 10B, 11A, and 11B.

[0149] Figure 9a (c) illustrates the geometric model of the reflector (51). The geometric model illustrated in Figure 9a (c) is an intermediate product and may not be included in the actual content image. For example, in the main rendering process, the node corresponding to the geometric model of the reflector (51) may be hidden or deleted from the scene information.

[0150] Figure 9a (d) illustrates the arrangement of an avatar (53) and a geometric model for applying a reflection effect to the avatar (53) based on scene information. When the scene information includes multiple virtual objects, multiple virtual objects to which the reflection effect is to be applied can be considered together with the geometric model.

[0151] Depending on the embodiment, the number of renderings required to generate a reflection image may vary. For example, as described above, the electronic device may generate a rendered geometric model of a reflector (51) on which no virtual object is projected, and generate a reflection image based on the geometric model of the rendered reflector (51), as described with reference to FIGS. 10A, 10B, 11A, and 11B. For example, instead of generating a rendered geometric model of a reflector (51) on which no virtual object is projected, the electronic device may generate a reflection image based directly on the geometric model of the reflector (51) included in the scene information, i.e., mesh data, as described with reference to FIGS. 10A, 10B, 11A, and 11B.

[0152] FIG. 10a is a diagram illustrating a method for rendering a geometric model according to an embodiment. FIG. 10b is a flowchart of a method for rendering a geometric model according to an embodiment.

[0153] According to one embodiment, in FIG. 10b, the operations 1010 and 1020 below may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIGS. 3a and 3b, or the electronic device (400) of FIG. 4). For example, the electronic device may include at least one processor including a processing circuit (e.g., the processor (120) of FIG. 1 or the processor (410) of FIG. 4), a memory (e.g., the memory (130) of FIG. 1 or the memory (420) of FIG. 4), a communication unit (e.g., the communication module (190) of FIG. 1), and a sensor (e.g., the sensor module (176) of FIG. 1, the camera module (180), the camera module (250) of FIG. 2, the first camera module (251), the second camera module (253), the depth sensor (317) of FIG. 3A, or the camera modules (311, 312, 313, 314, 315, 316, 325, 326) of FIGS. 3A and 3B, or the sensor (430) of FIG. 4).

[0154] According to one embodiment, operation 640 of generating the reflected image of FIG. 6 may include operations 1010 and 1020 below.

[0155] In operation 1010, the electronic device can generate a reflection map (55) corresponding to a geometric model of a reflector based on scene information. The reflection map (55) can be referred to as an environment map.

[0156] A reflection map (55) may be a texture or a collection of textures representing reflections of an environment. A texture may represent an image or pattern applied to the surface of an object. An electronic device may generate a reflection map (55) by rendering a texture (or image) representing reflections of a virtual object included in a virtual environment, i.e., scene information.

[0157] The reflection map (55) may be one of a cube, a sphere, or a paraboloid containing a texture onto which at least a portion of the virtual object is projected. A cube reflection map is a virtual hexahedron representing a view along six axes (up, down, left, right, front, back) of the environment. A spherical reflection map is a virtual sphere representing a view along two axes (front, back) of the environment.

[0158] A reflection map (55) can be generated by rendering a virtual object that is visible (reflected) with respect to the projection center. A cube reflection map can be generated by rendering six textures that at least partially contain the texture of the virtual object projected with respect to the projection center. A spherical reflection map can be generated by rendering two textures that at least partially contain the texture of the virtual object projected with respect to the projection center.

[0159] In one embodiment, the projection center of the reflection map (55) may correspond to the center of the geometric model of the reflector. In one embodiment, the projection center of the reflection map (55) may be located outside the geometric model of the reflector.

[0160] According to one embodiment, the electronic device may add a node corresponding to the reflection map (55) of the geometric model to the scene information prior to operation 1010.

[0161] In one embodiment, the electronic device can render a reflection map (55) of a geometric model and a virtual object such as an avatar (53) together based on scene information.

[0162] Figure 10a (a) illustrates a reflection map (55) generated based on scene information and an avatar (53) projected onto the reflection map (55). When the scene information includes multiple virtual objects, multiple virtual objects can be projected onto the reflection map (55).

[0163] In operation 1020, the electronic device can render the geometric model by applying at least a portion of the color of the reflection map (55) to a corresponding location of the geometric model of the reflector.

[0164] According to one embodiment, the electronic device can calculate a light source vector directed toward a geometric model of a reflector and a reflection vector reflected by the light source vector by the geometric model based on scene information. The scene information can include geometric information of an object included in the scene information (e.g., a geometric model of the reflector, an avatar (53), a reflection map (55)). The geometric information of each object can include a position of the object and coordinates of vertices constituting a surface of the object. The electronic device can render the geometric model by applying a color corresponding to a second intersection point of the reflection vector and the reflection map (55) to a first intersection point of the light source vector and the geometric model. By rendering the geometric model, the electronic device can generate a reflection image in which at least a portion of the virtual object is projected.

[0165] Figure 10a (b) illustrates a reflection image in which at least a portion of an avatar (53) generated by rendering a geometric model is projected. When the scene information includes multiple virtual objects, at least a portion of each of the multiple virtual objects may be projected onto the reflection image.

[0166] According to one embodiment, when rendering a geometric model, the electronic device can hide virtual objects other than the geometric model of the reflector in the scene information. Since a virtual object such as an avatar (53) has already been rendered in operation 1010 of generating a reflection map (55), by hiding the nodes corresponding to the virtual objects already created in the process of rendering the geometric model, they can be excluded from the rendering target, thereby saving computing resources.

[0167] In one embodiment, the electronic device can repeat operations 1010 and 1020. The electronic device can repeat operations 1010 and 1020 when a light source, an object in real space, or an object in virtual space changes (e.g., changes in position or shape), or when the position (or viewpoint) of the electronic device changes.

[0168] When operation 1010 of generating a reflection map (55) is performed again after operation 1020 of rendering a geometric model by hiding a node corresponding to a virtual object generated in the scene information is performed, the node corresponding to the virtual object (e.g., avatar (53)) may be included in the scene information again, and the node corresponding to the geometric model of the reflector may be hidden. Then, when operation 1020 of rendering the geometric model is performed, the node corresponding to the virtual object may be hidden, and the node corresponding to the geometric model of the reflector may be included in the frontal information again. The electronic device can improve rendering efficiency by alternately hiding the node corresponding to the virtual object and the node corresponding to the geometric model of the reflector in the scene information.

[0169] According to one embodiment, when scene information includes a plurality of reflectors, the electronic device may perform operations 1010 and 1020 for each of the plurality of reflectors. That is, the electronic device may generate a plurality of reflectance maps corresponding to each of the plurality of reflectors. According to an embodiment, reflectors (or geometric models of reflectors) at adjacent locations may share the same reflectance map.

[0170] FIG. 11a is a diagram illustrating a method for rendering a geometric model according to an embodiment. FIG. 11b is a flowchart of a method for rendering a geometric model according to an embodiment.

[0171] According to one embodiment, in FIG. 11b, the operations 1110 and 1120 below may be performed by an electronic device (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIGS. 3a and 3b, or the electronic device (400) of FIG. 4). For example, the electronic device may include at least one processor including a processing circuit (e.g., the processor (120) of FIG. 1 or the processor (410) of FIG. 4), a memory (e.g., the memory (130) of FIG. 1 or the memory (420) of FIG. 4), a communication unit (e.g., the communication module (190) of FIG. 1), and a sensor (e.g., the sensor module (176) of FIG. 1, the camera module (180), the camera module (250) of FIG. 2, the first camera module (251), the second camera module (253), the depth sensor (317) of FIG. 3A, or the camera modules (311, 312, 313, 314, 315, 316, 325, 326) of FIGS. 3A and 3B, or the sensor (430) of FIG. 4).

[0172] According to one embodiment, operation 640 of generating the reflected image of FIG. 6 may include operations 1110 and 1120 below.

[0173] In operation 1110, the electronic device may calculate a gaze vector toward the geometric model of the reflector and a reflection vector reflected by the geometric model based on scene information. The scene information may include geometric information of an object included in the scene information (e.g., the geometric model of the reflector, an avatar (53)). The geometric information of each object may include the position of the object and the coordinates of vertices constituting the surface of the object.

[0174] Figure 11a (a) illustrates a line of sight vector (solid line) toward a geometric model of a reflector included in scene information and a reflection vector (dotted line) in which the line of sight vector is reflected by the geometric model of the reflector. The reflection vector can be incident on an avatar (53) included in the scene information.

[0175] The electronic device can calculate a geometric model of a reflector and a reflection vector and / or a refraction vector associated with a virtual object such as an avatar (53) based on scene information. For example, when a plurality of reflectors are included in a real space, the electronic device can calculate multiple reflection vectors for geometric models corresponding to each of the plurality of reflectors. For example, the electronic device can calculate a refraction vector of a line of sight vector and / or a reflection vector by the geometric model of the reflector and other objects located around the virtual object. In addition, the electronic device can calculate a reflection vector in which the refraction vector is reflected by the geometric model. The electronic device can present a realistic reflection effect by considering all reflection vectors and refraction vectors associated with the geometric model of the reflector and other objects.

[0176] In operation 1120, the electronic device can render the geometric model by applying a color corresponding to a second intersection point of the reflection vector and the virtual object to a first intersection point of the gaze vector and the geometric model of the reflector. By rendering the geometric model, the electronic device can generate a reflected image in which at least a portion of the virtual object is projected.

[0177] Figure 11a (b) illustrates a reflection image in which at least a portion of an avatar (53) generated by rendering a geometric model is projected. When the scene information includes multiple virtual objects, at least a portion of each of the multiple virtual objects may be projected onto the reflection image.

[0178] In one embodiment, the electronic device may repeat operations 1110 and 1120. The electronic device may repeat operations 1110 and 1120 when a light source, an object in real space, or an object in virtual space changes (e.g., changes in position or shape), or when the position (or viewpoint) of the electronic device changes.

[0179] FIGS. 12 to 15 are drawings illustrating a method for providing extended reality content according to various examples.

[0180] As described with reference to FIGS. 1 to 11B, an electronic device (e.g., an electronic device (101) of FIG. 1, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIGS. 3A and 3B, or an electronic device (400) of FIG. 4) can obtain sensing data for a real space from a sensor of the electronic device (e.g., a sensor module (176) of FIG. 1, a camera module (180) of FIG. 1, a camera module (250) of FIG. 2, a first camera module (251), a second camera module (253), a depth sensor (317) of FIG. 3A, or camera modules (311, 312, 313, 314, 315, 316, 325, 326) of FIGS. 3A and 3B, or a sensor (430) of FIG. 4). The electronic device can identify a reflector included in the real space based on the sensing data. An electronic device can add a geometric model of a reflector to scene information for generating a content image including a virtual object. The electronic device can generate a reflected image in which at least a portion of the virtual object is projected by rendering the geometric model based on the scene information. The electronic device can generate a content image in which the reflected image is superimposed on a reflector in a real space based on sensing data.

[0181] FIG. 12 is a diagram illustrating a method for providing extended reality content when multiple reflectors are included in a real space, according to an example.

[0182] Referring to (a) of FIG. 12, the electronic device can identify a first reflector (1201) and a second reflector (1203) included in a real space based on sensing data. The electronic device can apply a reflection effect of an avatar (1205) to the first reflector (1201) and the second reflector (1203).

[0183] An electronic device can determine reflection information based on user input. The user input may include motions, gestures, voice, touch, gaze, or input from an external device connected to the electronic device, such as a controller. The reflection information may include information regarding whether to apply a reflection effect, the selection of a reflector to apply the reflection effect to, or the method of generating the reflection effect. The electronic device can set options related to the reflection effect of a virtual object (e.g., whether to apply a reflection effect, the selection of a reflector to apply the reflection effect to, or the method of generating the reflection effect) based on the reflection information.

[0184] According to one embodiment, when the reflector includes a plurality of reflectors, the electronic device can determine whether to generate a reflection image for each of the plurality of reflectors based on predetermined reflection information.

[0185] An electronic device may determine to apply a reflection effect to all reflectors or to apply a reflection effect only to a representative reflector based on predetermined reflection information. In one embodiment, the representative reflector may be determined based on user input. In one embodiment, the representative reflector may be determined based on sensing data. For example, the electronic device may determine a reflector having a large reflector occupancy rate in a view of a real space as the representative reflector based on sensing data. For example, the electronic device may determine a reflector whose position in a view of a real space is closest to the center of the view of the real space as the representative reflector based on sensing data.

[0186] Referring to (b) of FIG. 12, the electronic device may determine whether to generate a reflection image for each of the first reflector (1201) and the second reflector (1203) based on predetermined reflection information. The electronic device may apply the reflection effect of the avatar (1205) to the first reflector (1201), which is a representative reflector, based on the predetermined reflection information, and may not apply the reflection effect to the second reflector (1203).

[0187] In one embodiment, the electronic device may determine whether to generate a reflected image based on a real-time view of the real space based on sensing data. The real-time view of the real space may change as the position (or viewpoint) of the electronic device changes.

[0188] Referring to (c) of FIG. 12, unlike (a) of FIG. 12, the electronic device may not apply a reflection effect to the first reflector (1201) if the first reflector (1201) is not located within the main field of view according to the real-time view of the changed actual space.

[0189] FIG. 13 is a diagram illustrating a method for providing extended reality content according to an example.

[0190] As illustrated in FIG. 7, scene information may include nodes corresponding to objects. Object nodes may represent virtual objects, such as control elements (e.g., buttons, menu panels, sliders), interactive elements (e.g., avatars, game content, prompts, dialog boxes, tools for interacting with augmented reality), user interface elements (e.g., application launch screens), or any graphical elements.

[0191] Electronic devices can present reflection effects for all types of virtual objects, including avatars, based on scene information. The electronic device can determine reflection information, including whether to apply a reflection effect to a specific type of virtual object, based on user input. The electronic device can determine whether to apply a reflection effect to each virtual object based on the reflection information. Referring to FIG. 13 (a) and (b), the electronic device can either not present a reflection effect on the application execution screen or present one.

[0192] FIG. 14 is a diagram illustrating a method for providing extended reality content regarding a plurality of virtual objects, according to an example.

[0193] According to one embodiment, when rendering a geometric model of a reflector, an electronic device may determine whether to hide a virtual object in scene information based on attribute information of the virtual object. That is, the electronic device may determine whether to apply a reflection effect to the virtual object based on attribute information of the virtual object. The attribute information of the virtual object may include information regarding whether to apply a reflection effect. The electronic device may determine the attribute information based on user input.

[0194] According to one embodiment, the electronic device may determine whether to apply a reflection effect based on predetermined attribute information of the virtual object, depending on the type of the virtual object. For example, as illustrated in FIG. 14, if the virtual object is of a type that does not reflect on a reflector (e.g., a ghost), the predetermined attribute information may include information instructing not to apply a reflection effect to the virtual object.

[0195] According to one embodiment, when scene information includes a plurality of virtual objects (e.g., avatars) each corresponding to a plurality of users, the electronic device may receive attribute information of a virtual object corresponding to a user other than the user of the electronic device from an external electronic device or server connected to the electronic device.

[0196] Referring to (a) of FIG. 14, the electronic device can apply both the reflection effect of the first virtual object (1401) and the second virtual object (1403).

[0197] Referring to (b) of FIG. 14, the electronic device may apply a reflection effect to the first virtual object (1401) based on the attribute information of the first virtual object (1401) and the second virtual object (1403), and may not apply a reflection effect to the second virtual object (1403). When rendering a geometric model of a reflector, the electronic device may hide a node corresponding to the second virtual object (1403) from scene information based on the attribute information of the first virtual object (1401) and the second virtual object (1403).

[0198] FIG. 15 is a diagram illustrating a method for providing extended reality content regarding a plurality of virtual objects according to an example.

[0199] According to one embodiment, an electronic device can track the gaze of a user wearing the electronic device. The electronic device can obtain sensing data regarding the user's gaze from a sensor (e.g., the sensor module (176) of FIG. 1, the camera module (180), the camera module (250) of FIG. 2, the first camera module (251), the second camera module (253), the depth sensor (317) of FIG. 3A, or the camera modules (311, 312, 313, 314, 315, 316, 325, 326) of FIGS. 3A and 3B, or the sensor (430) of FIG. 4). When rendering a geometric model of a reflector, the electronic device can determine whether to hide a virtual object in scene information based on the tracked gaze of the user.

[0200] Referring to (a) of FIG. 15, when the gaze of a user wearing the electronic device is directed toward the first virtual object (1501), the electronic device may not apply a reflection effect to a virtual object (e.g., a second virtual object (1502)) other than the first virtual object (1501). When rendering a geometric model of a reflector, the electronic device may hide nodes corresponding to virtual objects (e.g., a second virtual object (1502)) other than the first virtual object (1501) from scene information.

[0201] Referring to (b) of FIG. 15, when the gaze of a user wearing the electronic device is directed toward the second virtual object (1502), the electronic device may not apply a reflection effect to a virtual object (e.g., the first virtual object (1501)) other than the second virtual object (1502). When rendering a geometric model of a reflector, the electronic device may hide nodes corresponding to virtual objects (e.g., the first virtual object (1501)) other than the second virtual object (1502) from scene information.

[0202] In one embodiment, an electronic device (101; 200; 300; 400) comprises a sensor (176; 180; 250; 251; 253; 317; 311; 312; 313; 314; 315; 316; 325; 326; 430), at least one processor (120; 410) comprising processing circuitry, and a memory (130; 420) comprising one or more storage media storing instructions, wherein when the instructions are individually or collectively executed by the at least one processor (120; 410), the electronic device (101; 200; 300; 400) causes: the sensor (176; 180; 250; 251; 253; 317; 311; 312; 313; 314; 315; 316; 325; 326; 430), identify a reflector included in the real space based on the sensing data, add a geometry model of the reflector to scene information for generating a content image including a virtual object, render the geometry model based on the scene information to generate a reflection image in which at least a part of the virtual object is projected, and generate a content image in which the reflection image is superimposed on a reflector in the real space based on the sensing data.

[0203] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120; 410), the electronic device (101; 200; 300; 400) may: identify a reflector based on at least one of a spatial correspondence or a temporal correspondence between objects contained in a real space based on sensing data.

[0204] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120; 410), the electronic device (101; 200; 300; 400) may be caused to: add a geometric model of a reflector to scene information if at least one of an occupancy rate of the reflector and a position of the reflector in a view of a real space satisfies a specified condition based on sensing data.

[0205] According to one embodiment, the instructions, when individually or collectively executed by at least one processor (120; 410), may cause the electronic device (101; 200; 300; 400) to: determine reflectivity of a reflector based on sensing data, and associate the reflectivity with a geometric model of the reflector and add the reflectivity to scene information.

[0206] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120; 410), the electronic device (101; 200; 300; 400) may be configured to: determine a contour or region of interest of a reflector based on sensing data; generate a point cloud of the reflector in the contour or region of interest based on depth information included in the sensing data; generate a mesh including connection relationships of points of the point cloud as a geometric model of the reflector; and add the geometric model to scene information.

[0207] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120; 410), the electronic device (101; 200; 300; 400) may be caused to: generate a reflection map corresponding to a geometric model of a reflector based on scene information, and render the geometric model by applying at least a portion of the color of the reflection map to a corresponding location of the geometric model of the reflector.

[0208] In one embodiment, the reflection map corresponding to the geometric model may be one of a cube, a sphere, or a paraboloid containing a texture onto which at least a portion of the virtual object is projected.

[0209] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120; 410), the electronic device (101; 200; 300; 400) may: cause the electronic device to: hide virtual objects other than the geometric model of a reflector in the scene information when rendering the geometric model.

[0210] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120; 410), the electronic device (101; 200; 300; 400) may be caused to: calculate a gaze vector toward a geometric model of a reflector based on scene information and a reflection vector reflected by the gaze vector from the geometric model, and render the geometric model by applying a color corresponding to a second intersection point of the reflection vector and the virtual object to a first intersection point of the gaze vector and the geometric model.

[0211] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120; 410), the electronic device (101; 200; 300; 400) may: determine whether to generate a reflected image based on a real-time view of the real space based on sensing data.

[0212] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120; 410), the electronic device (101; 200; 300; 400) may: determine whether to generate a reflection image for each of the plurality of reflectors based on predetermined reflection information, if the reflector includes a plurality of reflectors.

[0213] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120; 410), the electronic device (101; 200; 300; 400) may: determine whether to hide a virtual object in scene information based on attribute information of the virtual object when rendering a geometric model.

[0214] According to one embodiment, when the instructions are individually or collectively executed by at least one processor (120; 410), the electronic device (101; 200; 300; 400) may: track the gaze of a user wearing the electronic device, and determine whether to hide a virtual object in scene information based on the gaze of the user when rendering a geometric model.

[0215] According to one embodiment, a method for generating a content image performed by an electronic device (101; 200; 300; 400) may include an operation of acquiring sensing data for a real space from a sensor of the electronic device (101; 200; 300; 400), an operation of identifying a reflector included in the real space based on the sensing data, an operation of adding a geometry model of the reflector to scene information for generating a content image including a virtual object, an operation of generating a reflection image in which at least a part of the virtual object is projected by rendering the geometry model based on the scene information, and an operation of generating a content image in which the reflection image is superimposed on a reflector in the real space based on the sensing data.

[0216] According to one embodiment, the operation of identifying a reflector included in a real space based on sensing data may include an operation of identifying the reflector based on at least one of a spatial correspondence or a temporal correspondence between objects included in the real space based on the sensing data.

[0217] According to one embodiment, the operation of adding a geometric model of a reflector to scene information includes an operation of adding a geometric model of a reflector to scene information when at least one of an occupancy rate of the reflector and a position of the reflector in a view of a real space satisfies a specified condition based on sensing data.

[0218] According to one embodiment, the method for generating a content image may further include an operation of determining reflectivity of a reflector based on sensing data, and an operation of adding the reflectivity to scene information by associating it with a geometric model of the reflector.

[0219] According to one embodiment, the operation of adding a geometric model of a reflector to scene information may include an operation of determining a contour or a region of interest of the reflector based on sensing data, an operation of generating a point cloud of the reflector in the contour or region of interest based on depth information included in the sensing data, an operation of generating a mesh including a connection relationship of points of the point cloud as the geometric model of the reflector, and an operation of adding the geometric model to the scene information.

[0220] According to one embodiment, the operation of generating a reflection image by rendering a geometric model based on scene information may include the operation of generating a reflection map corresponding to the geometric model of the reflector based on the scene information, and the operation of rendering the geometric model by applying at least a part of the color of the reflection map to a corresponding location of the geometric model of the reflector.

[0221] In one embodiment, the operation of rendering the geometric model may further include the operation of hiding virtual objects other than the geometric model of the reflector in the scene information.

[0222] According to one embodiment, a non-transitory computer-readable recording medium stores one or more programs including instructions, which, when individually or collectively executed by at least one processor (120; 410) of an electronic device (101; 200; 300; 400), cause the electronic device (101; 200; 300; 400) to: acquire sensing data for a real space from a sensor (176; 180; 250; 251; 253; 317; 311; 312; 313; 314; 315; 316; 325; 326; 430) of the electronic device (101; 200; 300; 400), identify a reflector included in the real space based on the sensing data, and The method may be configured to perform an operation of adding a geometry model to scene information for generating a content image including a virtual object, an operation of generating a reflection image in which at least a portion of the virtual object is projected by rendering the geometry model based on the scene information, and an operation of generating a content image in which the reflection image is superimposed on a reflector in a real space based on sensing data.

[0223] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0224] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0225] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0226] The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0227] Although the embodiments have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0228] Therefore, other implementations, embodiments and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. In electronic devices (101; 200; 300; 400), sensor (176; 180; 250; 251; 253; 317; 311; 312; 313; 314; 315; 316; 325; 326; 430); At least one processor (120; 410) comprising processing circuitry; and A memory (130; 420) comprising one or more storage media for storing instructions, When the above instructions are individually or collectively executed by the at least one processor (120; 410), the electronic device (101; 200; 300; 400) causes: Obtain sensing data for the actual space from the above sensors (176; 180; 250; 251; 253; 317; 311; 312; 313; 314; 315; 316; 325; 326; 430), Identifying a reflector included in the actual space based on the sensing data, Adding the geometry model of the above reflector to the scene information for generating a content image including a virtual object, Generating a reflection image in which at least a portion of the virtual object is projected by rendering the geometric model based on the scene information, Generate a content image in which the reflected image is superimposed on the reflector in the actual space based on the sensing data. To do, Electronic devices (101; 200; 300; 400).

2. In paragraph 1, When the above instructions are individually or collectively executed by the at least one processor (120; 410), the electronic device (101; 200; 300; 400) causes: Identifying the reflector based on at least one of spatial correspondence or temporal correspondence between objects included in the real space based on the sensing data To do, Electronic devices (101; 200; 300; 400).

3. In paragraph 1 or 2, When the above instructions are individually or collectively executed by the at least one processor (120; 410), the electronic device (101; 200; 300; 400) causes: If at least one of the occupancy rate of the reflector and the position of the reflector in the view of the actual space satisfies a specified condition based on the sensing data, the geometric model of the reflector is added to the scene information. To do, Electronic devices (101; 200; 300; 400).

4. In any one of paragraphs 1 to 3, When the above instructions are individually or collectively executed by the at least one processor (120; 410), the electronic device (101; 200; 300; 400) causes: Determine the reflectivity of the reflector based on the sensing data, Add the above reflectivity to the above scene information by associating it with the above geometric model of the reflector. To do, Electronic devices (101; 200; 300; 400).

5. In any one of paragraphs 1 to 4, When the above instructions are individually or collectively executed by the at least one processor (120; 410), the electronic device (101; 200; 300; 400) causes: Determine the contour or area of ​​interest of the reflector based on the sensing data, Generating a point cloud of the reflector in the contour or the region of interest based on the depth information included in the sensing data, Generating a mesh including the connection relationship of points of the point cloud as the geometric model of the reflector, Add the above geometric model to the above scene information To do, Electronic devices (101; 200; 300; 400).

6. In any one of paragraphs 1 to 5, When the above instructions are individually or collectively executed by the at least one processor (120; 410), the electronic device (101; 200; 300; 400) causes: Generating a reflection map corresponding to the geometric model of the reflector based on the scene information, Rendering the geometric model by applying at least a part of the color of the reflection map to the corresponding location of the geometric model of the reflector. To do, Electronic devices (101; 200; 300; 400).

7. In any one of paragraphs 1 to 6, The reflection map corresponding to the above geometric model is one of a cube, a sphere, or a paraboloid containing a texture onto which at least a portion of the virtual object is projected. Electronic devices (101; 200; 300; 400).

8. In any one of paragraphs 1 to 7, When the above instructions are individually or collectively executed by the at least one processor (120; 410), the electronic device (101; 200; 300; 400) causes: When rendering the above geometric model, virtual objects other than the geometric model of the reflector are hidden in the scene information. To do, Electronic devices (101; 200; 300; 400).

9. In any one of paragraphs 1 to 8, When the above instructions are individually or collectively executed by the at least one processor (120; 410), the electronic device (101; 200; 300; 400) causes: Based on the scene information, calculating a gaze vector toward the geometric model of the reflector and a reflection vector reflected by the geometric model from the gaze vector, Rendering the geometric model by applying a color corresponding to the second intersection point of the reflection vector and the virtual object to the first intersection point of the gaze vector and the geometric model. To do Electronic devices (101; 200; 300; 400).

10. In any one of paragraphs 1 to 9, When the above instructions are individually or collectively executed by the at least one processor (120; 410), the electronic device (101; 200; 300; 400) causes: Based on the sensing data, it is determined whether to generate the reflected image according to the real-time view of the actual space. To do Electronic devices (101; 200; 300; 400).

11. In any one of paragraphs 1 to 10, When the above instructions are individually or collectively executed by the at least one processor (120; 410), the electronic device (101; 200; 300; 400) causes: When the reflector includes a plurality of reflectors, it is determined whether to generate the reflection image for each of the plurality of reflectors based on predetermined reflection information. To do, Electronic devices (101; 200; 300; 400).

12. In any one of paragraphs 1 to 11, When the above instructions are individually or collectively executed by the at least one processor (120; 410), the electronic device (101; 200; 300; 400) causes: When rendering the above geometric model, whether to hide the virtual object in the scene information is determined based on the attribute information of the virtual object. To do, Electronic devices (101; 200; 300; 400).

13. In any one of paragraphs 1 to 12, When the above instructions are individually or collectively executed by the at least one processor (120; 410), the electronic device (101; 200; 300; 400) causes: Tracking the gaze of a user wearing the above electronic device, When rendering the above geometric model, determine whether to hide the virtual object in the scene information based on the user's line of sight. To do, Electronic devices (101; 200; 300; 400).

14. A method for generating a content image, performed by an electronic device (101; 200; 300; 400), An operation of acquiring sensing data for an actual space from a sensor (176; 180; 250; 251; 253; 317; 311; 312; 313; 314; 315; 316; 325; 326; 430) of the electronic device (101; 200; 300; 400); An operation of identifying a reflector included in the actual space based on the sensing data; An action of adding a geometry model of the above reflector to scene information for generating a content image including a virtual object; An operation of generating a reflection image in which at least a portion of the virtual object is projected by rendering the geometric model based on the scene information; and An operation of generating a content image in which the reflected image is superimposed on the reflector in the actual space based on the sensing data. including, method.

15. In a non-transitory computer-readable recording medium, Store one or more programs containing instructions, When the above instructions are individually or collectively executed by at least one processor (120; 410) of the electronic device (101; 200; 300; 400), the electronic device (101; 200; 300; 400) causes: An operation of acquiring sensing data for an actual space from a sensor (176; 180; 250; 251; 253; 317; 311; 312; 313; 314; 315; 316; 325; 326; 430) of the electronic device (101; 200; 300; 400); An operation of identifying a reflector included in the actual space based on the sensing data; An action of adding a geometry model of the above reflector to scene information for generating a content image including a virtual object; An operation of generating a reflection image in which at least a portion of the virtual object is projected by rendering the geometric model based on the scene information; and An operation of generating a content image in which the reflected image is superimposed on the reflector in the actual space based on the sensing data. to perform, Non-transitory computer-readable recording medium.

Images