Electronic device, method, and non-transitory storage medium for displaying three-dimensional image in virtual environment

Abstract

The present document relates to an electronic device, a method, and a non-transitory storage medium for displaying a three-dimensional image in a virtual environment. An electronic device, according to one embodiment, comprises: a camera; a display; at least one processor including a processing circuit; and a memory, wherein the memory may store instructions that, when executed individually or collectively by the at least one processor, instruct the electronic device to: acquire at least one two-dimensional image; select a first image portion included in the at least one two-dimensional image; acquire a first three-dimensional virtual object for the first image portion generated by a generative AI model, on the basis of a command to generate the selected first image portion as a first three-dimensional virtual object; anchor the first virtual object to a first region identified through the camera in a real space; acquire a three-dimensional image generated by the generative AI model and in which the first virtual object moves in the first region; and control the display to display the three-dimensional image in the first region on a screen corresponding to the real space. Other embodiments are also possible.

Description

Electronic device, method and non-transitory storage medium for displaying three-dimensional images in a virtual environment

[0001] The present disclosure relates to an electronic device, method and non-transitory storage medium for displaying three-dimensional images in a virtual environment.

[0002] With the advancement of digital technology, electronic devices are available in various forms, such as smartphones, tablet personal computers (PCs), and personal digital assistants (PDAs). Electronic devices are also being developed into wearable forms to enhance portability and accessibility. As technology advances, technologies are being developed to provide virtual environments, allowing users to indirectly experience specific locations or situations through 3D computer graphics. In virtual environments, electronic devices can be configured in various forms to be worn on parts of the user's body and can display images in the virtual reality space of the virtual environment.

[0003] Recently, users have become increasingly interested in acquiring high-quality images, beyond simply capturing them using electronic devices. Electronic devices are actively developing technologies that generate images using artificial intelligence (generative AI models). Using image-generating applications, electronic devices can provide users with an environment where they can create images stored on the device.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0005] Conventional electronic devices display two-dimensional (2D) images in a virtual environment, and therefore cannot implement the image portion of an object, animal, or person included in the 2D image as a three-dimensional (3D) virtual object, and cannot reflect the movement of the virtual object from different viewpoints. Therefore, a technology is required to display or reproduce a three-dimensional image that implements a virtual object stereoscopically and dynamically in a virtual environment.

[0006] According to one embodiment of the present disclosure, an electronic device may include a camera, a display, at least one processor including a processing circuit, and a memory storing instructions.

[0007] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to acquire at least one two-dimensional (2D) image.

[0008] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to select a first image portion included in the at least one 2D image.

[0009] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain a first 3D virtual object for the first image portion generated by the generative AI model based on an instruction to generate the first image portion as a three-dimensional (3D) first virtual object.

[0010] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to anchor a first virtual object in a first area identified through the camera in real space.

[0011] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain a three-dimensional image generated by the generative AI model and showing a first virtual object moving in the first area.

[0012] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to display the three-dimensional image in the first area of ​​the screen corresponding to the real space.

[0013] According to one embodiment, a method of operating in an electronic device may include obtaining at least one two-dimensional (2D) image.

[0014] According to one embodiment, the method may include an operation of selecting a first image portion included in the at least one 2D image.

[0015] According to one embodiment, the method may include an operation of obtaining a three-dimensional (3D) first virtual object for the first image portion generated by a generative AI model based on a command to generate the selected first image portion as a three-dimensional (3D) first virtual object.

[0016] According to one embodiment, the method may include anchoring a first virtual object to a first area identified through a camera of the electronic device in real space.

[0017] According to one embodiment, the method may include an operation of obtaining a three-dimensional image generated by the generative AI model and in which a first virtual object moves in the first area.

[0018] According to one embodiment, the method may include an operation of controlling a display of the electronic device to display the three-dimensional image in the first area on a screen corresponding to the real space.

[0019] According to one embodiment, a non-transitory storage medium storing one or more programs may include instructions that, when executed by at least one processor of an electronic device, cause the electronic device to perform an operation of acquiring at least one two-dimensional (2D) image.

[0020] According to one embodiment, the one or more programs may include instructions that, when executed by at least one processor of the electronic device, cause the electronic device to perform an operation of selecting a first image portion included in the at least one two-dimensional image.

[0021] According to one embodiment, the one or more programs may include instructions that, when executed by at least one processor of the electronic device, cause the electronic device to perform an operation of obtaining a first 3D virtual object for the first image portion generated by the generative AI model based on an instruction to generate the first image portion as a three-dimensional (3D) first virtual object.

[0022] According to one embodiment, the one or more programs may include instructions that, when executed by at least one processor of the electronic device, cause the electronic device to perform an operation of anchoring a first virtual object to a first area identified through a camera of the electronic device in real space.

[0023] According to one embodiment, the one or more programs may include instructions that, when executed by at least one processor of the electronic device, cause the electronic device to perform an operation of obtaining a three-dimensional image generated by the generative AI model and in which a first virtual object moves in the first area.

[0024] According to one embodiment, the one or more programs may include instructions that, when executed by at least one processor of the electronic device, cause the electronic device to execute an operation of controlling a display of the electronic device to display the three-dimensional image in the first area on a screen corresponding to the real space.

[0025] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0026] FIG. 2 is a perspective view showing the structure of an electronic device according to one embodiment.

[0027] FIGS. 3A, 3B, and 3C are perspective views showing the structure of an electronic device according to one embodiment.

[0028] Fig. 4 is a block diagram showing an example configuration of an electronic device according to one embodiment.

[0029] FIG. 5 is a diagram illustrating an example for displaying a 3D image in a virtual environment in an electronic device according to one embodiment.

[0030] FIGS. 6A and 6B are drawings illustrating examples for displaying 3D images in a virtual environment in an electronic device according to one embodiment.

[0031] FIG. 7 is a diagram illustrating an example for displaying a 3D image in a virtual environment in an electronic device according to one embodiment.

[0032] FIGS. 8A and 8B are drawings illustrating examples for displaying 3D images in a virtual environment in an electronic device according to one embodiment.

[0033] FIG. 9 is a drawing showing an example of an operating method in an electronic device according to one embodiment.

[0034] FIG. 10 is a drawing showing an example of an operating method in an electronic device according to one embodiment.

[0035] FIG. 11 is a drawing showing an example of an operating method in an electronic device according to one embodiment.

[0036] FIG. 12 is a drawing showing an example of displaying a three-dimensional image in an electronic device according to one embodiment.

[0037] FIG. 13 is a drawing showing an example of displaying a three-dimensional image in an electronic device according to one embodiment.

[0038] FIG. 14 is a drawing showing an example of displaying a three-dimensional image in an electronic device according to one embodiment.

[0039] FIG. 15 is a drawing showing an example of displaying a three-dimensional image in an electronic device according to one embodiment.

[0040] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0041] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0042] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components. In addition, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness. The term "user" used in the embodiments of the present disclosure may refer to a person using an electronic device or a device (e.g., an artificial intelligence electronic device) using an electronic device.

[0043] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0044] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0045] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0046] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0047] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0048] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0049] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0050] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0051] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0052] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0053] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0054] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0055] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0056] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0057] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0058] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0059] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0060] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0061] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for realizing 1eMBB, a loss coverage (e.g., 164 dB or less) for realizing mMTC, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for realizing URLLC.

[0062] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0063] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0064] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0065] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0066] FIG. 2 is a perspective view showing the structure of an electronic device according to one embodiment.

[0067] Referring to FIGS. 1 and 2, an electronic device (200) according to one embodiment may be an electronic device (101) of FIG. 1, an electronic device (102 or 104) communicating with the electronic device (101) of FIG. 1, or a device capable of providing a service related to virtual reality technology that provides a virtual environment similar to the electronic device (101) of FIG. 1. Virtual reality (VR) technology, which is a technology for providing a virtual environment, may be developed into augmented reality (AR), mixed reality (MR), and / or extended reality (XR) encompassing these. According to one embodiment, virtual reality (VR) technology may be described to mean including augmented reality (AR), mixed reality (MR), and / or extended reality (XR).

[0068] The electronic device (200) may be a device configured to be worn on a user's body (e.g., a head-mounted display (HMD) or an AR glasses device in the form of glasses), as illustrated in FIG. 2. For example, the electronic device (200) may be configured to combine with an external electronic device, such as a mobile device, and may utilize components of the external electronic device (e.g., a display module, a camera module, an audio output module, or other components). Without being limited thereto, the electronic device (200) may be implemented in various forms that can be worn on a user's body.

[0069] According to one embodiment, the electronic device (200) configures a virtual reality space (e.g., an augmented reality space) that displays an augmented reality image corresponding to an actual environment captured in the surrounding environment where the user is located or an image provided virtually (e.g., a two-dimensional or three-dimensional image), and controls a display module (160) to display at least one virtual object corresponding to the user and / or at least one virtual object corresponding to an object for user interaction in the virtual reality space.

[0070] According to one embodiment, the electronic device (200) may include a processor (120), a memory (130), a display module (160), a sensor module (176), a camera module (180), a charging module (e.g., a battery (189) of FIG. 1), and a communication module (190) as illustrated in FIG. 1. The electronic device (200) may further include an audio output device (155) and an input module (150) as illustrated in FIG. 1, or other components as illustrated in FIG. 1. In addition, the electronic device (200) may further include other components necessary to provide an augmented reality function (e.g., a service or method).

[0071] According to one embodiment, the processor (120) may be electrically connected to other components and control the other components. The processor (120) may perform various data processing or calculations according to the execution of various functions (e.g., operations, services, or programs) provided by the electronic device (200). The processor (120) may perform various data processing or calculations to display at least one virtual object related to real objects included in an image captured in real space and / or a virtual object corresponding to a user (e.g., an avatar) in a virtual reality space. The processor (120) may perform various data processing or calculations to express user interaction or movement of a virtual object displayed in a virtual reality space.

[0072] Again, referring to FIG. 2, an electronic device (200) according to one embodiment will be described. As described above, the electronic device (200) is not limited to a glasses-type (e.g., AR glasses) augmented reality device, and may be implemented as various devices (e.g., AR head-mounted display type, 2D / 3D head-mounted display device, or VR head-mounted display device) capable of providing immersive content (e.g., content based on XR technology) to the user's eyes.

[0073] According to one embodiment, a camera module of the electronic device (200) (e.g., camera module (180) of FIG. 1) can capture still images and / or moving images. According to one embodiment, the camera module may be disposed within a lens frame and may be disposed around the first display (251) and the second display (252). According to one embodiment, the camera module may include one or more first cameras (211-1, 211-2), one or more second cameras (212-1, 212-2), and one or more third cameras (213). According to one embodiment, images acquired through the one or more first cameras (211-1, 211-2) may be used for detecting hand gestures by a user, tracking the user's head, and / or recognizing space. The one or more first cameras (211-1, 211-2) may be a global shutter (GS) camera or a rolling shutter (RS) camera. One or more first cameras (211-1, 211-2) can perform simultaneous localization and mapping (SLAM) operations through depth imaging. One or more first cameras (211-1, 211-2) can perform spatial recognition and / or movement recognition for 3DoF (depth of field) and / or 6DoF. According to one embodiment, the first cameras (211-1, 211-2) can periodically or aperiodically transmit information related to the trajectory of the user's eyes or gaze (e.g., trajectory information) to a processor (e.g., the processor (120) of FIG. 1).

[0074] According to one embodiment, the electronic device (200) may use another camera (e.g., a third camera (213)) for hand detection and tracking and user gesture recognition. According to one embodiment, at least one of the first camera (211-1, 211-2) to the third camera module (213) may be replaced with a sensor module (e.g., a LiDAR sensor). For example, the sensor module may include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0075] According to one embodiment, images acquired through one or more second cameras (212-1, 212-2) may be used to detect and track the user's pupils. One or more of the second cameras (212-1, 212-2) may be GS cameras. One or more of the second cameras (212-1, 212-2) may correspond to the left eye and the right eye, respectively, and the performance of one or more of the second cameras (212-1, 212-2) may be substantially the same. One or more of the third cameras (213) may be relatively high-resolution cameras. One or more of the third cameras (213) may perform an auto-focusing (AF) function and an optical image stabilization (OIS) function. One or more of the third cameras (213) may be a GS (global shutter) camera or an RS (rolling shutter) camera. One or more of the third cameras (213) may be a color camera.

[0076] According to one embodiment, the electronic device (200) may include one or more light-emitting elements (214-1, 214-2). The light-emitting elements (214-1, 214-2) are different from a light source, which will be described later, that radiates light to a screen output area of ​​a display. According to one embodiment, the light-emitting elements (214-1, 214-2) may radiate light to facilitate pupil detection when detecting and tracking the user's pupils through one or more second cameras (212-1, 212-2). According to one embodiment, the light-emitting elements (214-1, 214-2) may each include an LED. According to one embodiment, the light-emitting elements (214-1, 214-2) may radiate light in the infrared region. According to various embodiments, the light-emitting elements (214-1, 214-2) may be attached to the periphery of the frame of the augmented reality device (200). According to one embodiment, the light-emitting elements (214-1, 214-2) are positioned around one or more first cameras (211-1, 211-2) and can assist gesture detection, head tracking, and spatial recognition by one or more first cameras (211-1, 211-2) when the augmented reality device (200) is used in a dark environment. According to one embodiment, the light-emitting elements (214-1, 214-2) are positioned around one or more third cameras (213) and can assist image acquisition by one or more third cameras (213) when the augmented reality device (200) is used in a dark environment.

[0077] According to one embodiment, the electronic device (200) may include a battery (235-1, 235-2) (e.g., battery (189) of FIG. 1). The battery (235-1, 235-2) may store power to operate the remaining components of the augmented reality device (200).

[0078] According to one embodiment, a display module (e.g., display module (160) of FIG. 1) of an electronic device (200) may include a first display (251), a second display (252), one or more input optical members (253-1, 253-2), one or more transparent members (290-1, 290-2), and one or more screen display portions (254-1, 254-2). According to one embodiment, the first display (251) and the second display (252) may include, as light output modules, for example, a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED). According to one embodiment, when the first display (251) and the second display (252) are formed of one of a liquid crystal display (LCD), a digital mirror display, or a silicon liquid crystal display, the augmented reality device (200) may include a light source that irradiates light to the screen output area of ​​the display. According to one embodiment, when the first display (251) and the second display (252) can generate light on their own, for example, when they are formed of one of an organic light emitting diode (OLED) or a micro LED, the augmented reality device (200) may provide a good quality virtual image (e.g., an image of a virtual reality space) to the user even without including a separate light source.

[0079] According to one embodiment, one or more transparent members (290-1, 290-2) included in the electronic device (200) may be positioned to face the eyes of the user when the user wears the augmented reality device (200). The one or more transparent members (290-1, 290-2) may include at least one of a glass plate, a plastic plate, or a polymer. When the user wears the augmented reality device (200), the user may view the outside world through the one or more transparent members (290-1, 290-2).

[0080] According to one embodiment, one or more input optical elements (253-1, 253-2) included in the electronic device (200) can guide light generated from the first display (251) and the second display (252) to the user's eyes. An image based on the light generated from the first display (251) and the second display (252) is formed on one or more screen display portions (254-1, 254-2) on one or more transparent elements (290-1, 290-2), and the user can view the image formed on the one or more screen display portions (254-1, 254-2).

[0081] According to one embodiment, the electronic device (200) may include one or more optical waveguides (not shown). The optical waveguides may transmit light generated by the first display (251) and the second display (252) to the user's eyes. The augmented reality device (200) may include one optical waveguide corresponding to the left eye and one optical waveguide corresponding to the right eye, respectively. According to one embodiment, the optical waveguide may include at least one of glass, plastic, or polymer. The optical waveguide may include a nano-pattern formed on one surface of the inner or outer surface, for example, a grating structure having a polygonal or curved shape. The optical waveguide may include a free-form prism, in which case the optical waveguide may provide incident light to the user through a reflective mirror. According to one embodiment, the optical waveguide includes at least one diffractive element (e.g., a diffractive optical element (DOE) or a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror), and the at least one diffractive element or reflective element included in the optical waveguide can be used to guide display light emitted from a light source toward a user's eye. According to one embodiment, the diffractive element can include an input / output optical element. According to one embodiment, the reflective element can include an element that causes total internal reflection.

[0082] According to one embodiment, the electronic device (200) may include one or more voice input devices (262-1, 262-2, 262-3) and one or more voice output devices (263-1, 263-2).

[0083] According to one embodiment, the electronic device (200) may include a first PCB (270-1) and a second PCB (270-2). The first PCB (270-1) and the second PCB (270-2) may transmit electrical signals to components included in the electronic device (200), such as a first camera (211-1, 211-2), a second camera (212-1, 212-2), a third camera (213), a display (251, 252), an audio module (e.g., an audio module (170) of FIG. 1), and a sensor module (e.g., a sensor module (176) of FIG. 1) included in the camera module (180). According to one embodiment, the first PCB (270-1) and the second PCB (270-2) may be a flexible printed circuit board (FPCB). According to one embodiment, the first PCB (270-1) and the second PCB (270-2) may each include a first substrate, a second substrate, and an interposer disposed between the first substrate and the second substrate.

[0084] FIG. 3A is a perspective view showing the structure of an electronic device according to one embodiment.

[0085] Referring to FIG. 3A, an electronic device (300) according to one embodiment (e.g., the electronic device (101) of FIG. 1 or the electronic device (200) of FIG. 2) may be a wearable device such as a head-mounted device (HMD) that can be worn on a user's head to provide an image (e.g., a virtual reality space image) to the user's eyes. The configuration of the electronic device (300) of FIG. 3 may be all or part of the same as the configuration of the electronic device (200) of FIG. 2.

[0086] According to one embodiment, the electronic device (300) may include a housing (310, 320, 330) that may form an exterior and provide a space in which components of the electronic device (300) may be placed.

[0087] In one embodiment, the electronic device (300) may include a first housing (310) that can surround at least a portion of a user's head. In one embodiment, the first housing (310) may include a first side (300a) that faces the exterior of the electronic device (300) (e.g., in the +X direction).

[0088] In one embodiment, the first housing (310) can surround at least a portion of the internal space (I). For example, the first housing (310) can include a second surface (300b) facing the internal space (I) of the electronic device (300) and a third surface (300c) opposite the second surface (300b). In one embodiment, the first housing (310) can be combined with the third housing (330) to form a closed curve shape surrounding the internal space (I).

[0089] According to one embodiment, the first housing (310) can accommodate at least some of the components of the electronic device (300). For example, an optical output module, a circuit board, and a speaker module can be disposed within the first housing (310).

[0090] According to one embodiment, the electronic device (300) may include one display member (340) corresponding to the left and right eyes. The display member (340) may be disposed in the first housing (310). The configuration of the display member (340) of FIG. 3 may be all or part of the same as the configuration of the screen display portions (254-1, 254-2) of FIG. 2.

[0091] According to one embodiment, the electronic device (300) may include a second housing (320) that can be mounted on a user's face. According to one embodiment, the second housing (320) may include a fourth surface (300d) that can at least partially face the user's face. According to one embodiment, the fourth surface (300d) may be a surface facing the internal space (I) of the electronic device (300) (e.g., in the -X direction). According to one embodiment, the second housing (320) may be coupled with the first housing (310).

[0092] In one embodiment, the electronic device (300) may include a third housing (330) that may be mounted on the back of the user's head. In one embodiment, the third housing (330) may be coupled to the first housing (310). In one embodiment, the third housing (330) may accommodate at least some of the components of the electronic device (300). For example, a battery (e.g., batteries 235-1 and 235-2 of FIG. 2) may be placed within the third housing (330).

[0093] In order to improve the overall user experience, usage environment and usability of a head-mounted wearable type electronic device (300), it may be necessary for the sensations felt and experienced by the user in the VR (virtual reality), AR (augmented reality) and MR (mixed reality) space to be as similar as possible to the sensations of the real world.

[0094] FIGS. 3b and 3c are perspective views showing the structure of an electronic device according to one embodiment.

[0095] Referring to FIGS. 3b and 3c, in one embodiment, camera modules (311, 312, 313, 314, 315, 316) and / or depth sensors (317) may be arranged on a first surface (310) of the housing to obtain information related to the surrounding environment of an electronic device (300) (e.g., a wearable device).

[0096] In one embodiment, the camera modules (311, 312) can acquire images related to the environment surrounding the wearable electronic device.

[0097] In one embodiment, the camera modules (313, 314, 315, 316) can acquire images while the electronic device (300) is worn by the user. The camera modules (313, 314, 315, 316) can be used for hand detection, tracking, and recognition of user gestures (e.g., hand movements). The camera modules (313, 314, 315, 316) can be used for 3DoF, 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. In one embodiment, the camera modules (311, 312) can also be used for hand detection and tracking, and user gestures.

[0098] In one embodiment, a depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Instead of or in addition to the depth sensor (317), camera modules (313, 314, 315, 316) may determine the distance to an object.

[0099] According to one embodiment, a camera module (325, 326) for facial recognition and / or a display (321) (and / or a lens) may be disposed on the second side (320) of the housing.

[0100] In one embodiment, a face recognition camera module (325, 326) adjacent to the display may be used to recognize a user's face, or may recognize and / or track both eyes of the user.

[0101] In one embodiment, the display (321) (and / or lens) may be disposed on the second side (320) of the wearable electronic device (300). In one embodiment, the wearable electronic device (300) may not include camera modules (315, 316) among the plurality of camera modules (313, 314, 315, 316). Although not illustrated in FIGS. 3B and 3C , the electronic device (300) may further include at least one of the configurations illustrated in FIG. 2 .

[0102] As described above, according to one embodiment, the electronic device (300) may have a form factor for being worn on a user's head. The electronic device (300) may further include a strap and / or a wearable member for being secured to a body part of the user. The electronic device (300) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0103] Fig. 4 is a block diagram showing an example configuration of an electronic device according to one embodiment.

[0104] Referring to FIGS. 1 to 4, an electronic device (401) according to one embodiment may be a device capable of providing a service related to virtual reality technology that provides a virtual environment similar to the electronic device (101) of FIG. 1, or the electronic device (200) of FIG. 2 and the electronic device (300) of FIG. 3. Virtual reality (VR) technology, which is a technology for providing a virtual environment, may be developed into augmented reality (AR), mixed reality (MR), and / or extended reality (XR) that encompasses them. According to one embodiment, virtual reality (VR) technology may be described to mean including augmented reality (AR), mixed reality (MR), and / or extended reality (XR). The electronic device (401) may be a device configured to be wearable on a user's body (e.g., a head-mounted display (HMD) or an AR glass device in the form of glasses), as illustrated in FIGS. 2 and 3A to 3C. For example, the electronic device (401) may be configured to combine with an external electronic device, such as a mobile device, and may utilize components (e.g., a display module, a camera module, an audio output module, or other components) of the external electronic device (e.g., the electronic device (102 or 104) of FIG. 1). Without being limited thereto, the electronic device (401) may be implemented in various forms that can be worn on a user's body.

[0105] According to one embodiment, the electronic device (401) may configure a virtual reality space (e.g., an augmented reality space) that displays an augmented reality image corresponding to a real environment (e.g., a real space) captured in a surrounding environment in which a user is located or an image provided virtually (e.g., a two-dimensional or three-dimensional image), and may display at least one virtual object corresponding to the user and / or at least one virtual object corresponding to an object in the virtual reality space. According to one embodiment, when the electronic device (401) makes the real space of the real environment visible to the user through a transparent member, it may anchor and display at least one virtual object on a screen corresponding to the real space.

[0106] An electronic device (401) according to one embodiment may include at least one processor (410), a memory (420), a display (430), a camera (440), and a communication circuit (450). Without being limited thereto, the electronic device (401) may be implemented identically or similarly to the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, or the electronic device (300) of FIG. 3, and may further include other components of the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, or the electronic device (300) of FIG. 3. In addition, the electronic device (200) may further include other components necessary to provide an augmented reality function (e.g., a service or method).

[0107] According to one embodiment, the electronic device (401) may be an on-device device that may include a generative artificial intelligence (AI) model. According to one embodiment, the electronic device (401) may communicate with an external electronic device (e.g., electronic device (102 or 104) of FIG. 1) or server (e.g., server (108) of FIG. 1) that includes the generative AI model through a communication circuit (450). The generative AI model may generate 2D and / or 3D images through an image generation operation (e.g., an in-painting operation or an out-painting operation).

[0108] FIGS. 5, 6a, 6b, 7, 8a, and 8b are drawings showing examples for displaying a 3D image in a virtual environment in an electronic device according to one embodiment.

[0109] Referring to FIGS. 4, 5, 6a, 6b, 7, 8a, and 8b, according to one embodiment, a processor (410) of an electronic device (401) may be electrically connected to other components and control the other components. The processor (410) may perform various data processing or calculations according to the execution of various functions (e.g., operations, services, or programs) provided by the electronic device (401). The processor (410) may perform various data processing or calculations to display at least one virtual object related to an object or use included in an image captured in a real space (e.g., a real environment) or identified in a real space, in a virtual reality space. The processor (410) may perform various data processing or calculations to express a user interaction or movement of a virtual object displayed in a virtual reality space.

[0110] According to one embodiment, the processor (410) may obtain at least one two-dimensional (2D) image (510) from an external electronic device (403) (e.g., the electronic device (102 or 104) of FIG. 1) or a server (e.g., the server (108) of FIG. 1) via a communication circuit (450) using a wireless or wired communication method. According to one embodiment, the processor (410) may obtain at least one 2D image (510) captured via a camera (440). According to one embodiment, the processor (410) may display the obtained at least one 2D image (510) on a screen (540) corresponding to a real space through a display (430) in a virtual environment (e.g., a virtual reality space). According to one embodiment, the processor (410) can share images (e.g., photos) included in an image list (610) (e.g., a gallery) stored in a memory of an external electronic device (403) (e.g., an electronic device (102 or 104) of FIG. 1) or a server (e.g., a server (108) of FIG. 1) and display them in a virtual reality space through a display (430). According to one embodiment, the processor (410) can display a screen (e.g., an image) corresponding to the external electronic device (403) displaying the image list (610) on a screen (540), as illustrated in FIG. 6B, and, in response to a user interaction of selecting a 2D image (510) from the image list (610), display the selected 2D image (510) moving to a first area (541) on the screen (540).

[0111] According to one embodiment, the processor (410) can identify an object, such as an object, an animal, or a person, included in at least one 2D image (510), and obtain meta information (e.g., location information or identification information about the object) about the identified object. According to one embodiment, the processor (410) can obtain meta information about the object from an external electronic device when obtaining at least one 2D image (510), store the meta information in the memory (420), and obtain the meta information stored in the memory (420). According to one embodiment, the processor (410) can analyze the obtained at least one 2D image (510) to obtain meta information about the object identified in the at least one 2D image (510).

[0112] According to one embodiment, the processor (410) may select a first image portion (511) included in at least one 2D image (510) and transmit a command to the generative AI model (501) to generate the selected first image portion (511) into a three-dimensional (3D) virtual object (521) (e.g., a first virtual object). Here, the selected first image portion (511) may be a portion of an object representing one or more objects, animals, or people included in the at least one 2D image (510). According to one embodiment, the generative AI model (501) receives information about a selected image portion (511), and generates a three-dimensional virtual object (521) (e.g., a first virtual object) based on the information about the first image portion (511), for example, through a volumetric rendering and / or generative fill (e.g., NeRF, LRM model) operation as an image generation operation (520). The generative AI model (501) can generate the 3D virtual object based on data acquired through user- (e.g., subject-) customized 3D reinforcement learning. When the selected first image portion (511) includes two or more objects, the generative AI model (501) can generate three-dimensional virtual objects representing each of the two or more objects. The generative AI model (501) can perform post-processing operations on a 3D virtual object, for example, texturing, UV matching, material settings, or rendering, and can apply various effects (e.g., lighting effects, border adjustments, or animation effects) to the generated 3D virtual object.

[0113] According to one embodiment, the processor (410) can obtain a 3D virtual object (521) for the first image portion (511) from the generative AI model (501).

[0114] According to one embodiment, the processor (410) can identify a first region (541) (e.g., an anchoring region) for anchoring a virtual object (521) through a camera (440) in a real space (550). The processor (410) can identify the first region (541) by analyzing a screen (540) captured by the camera (440) within a field of view (FoV) (560) in the real space. The processor (410) can identify a real object, such as an object, an animal, or a person, located in the real space (550), and can identify (e.g., confirm, set, or designate) an area on the screen (540) that does not interfere with (e.g., overlap with) an area corresponding to the real object (521) so as to avoid the identified real object, as the first region (541) for anchoring the virtual object (521).

[0115] According to one embodiment, the processor (410) may display a graphic object (630) representing movement on a screen (540) of a virtual reality space in response to a user gesture of moving a first image portion (511) selected from a 2D image (510) to an identified first area (541). The graphic object (630) may be set to at least one of a dynamic movement of the first virtual object or a designated dynamic effect based on movement path information. According to one embodiment, the processor (410) may transmit a command to a generative AI model (501) to generate a 3D virtual object (521) for the first image portion (511) in response to a user gesture of moving the first image portion (511) to the identified first area (541). Without being limited thereto, the processor (410) may transmit a command to the generative AI model (501) to generate a 3D virtual object (521) by selecting a designated menu displayed on the screen (540) or by a designated user interaction.

[0116] According to one embodiment, the processor (410) may anchor a virtual object (521) to a first region (541) (e.g., anchoring region) identified through a camera (440) in a real space (550). The first region (541) is a region included in a screen (540) corresponding to the real space (550), and may be set based on structural information of the corresponding region in the real space (550) (e.g., region distance, size, or location information), and location information of at least one of an object, animal, or person within the real space (550).

[0117] According to one embodiment, the processor (410) may obtain movement path information for causing the first virtual object (521) to move along a designated path (711) in the first area (541), and obtain a three-dimensional image (530) generated by the generative AI model (501) based on the movement path information. The three-dimensional image (530) may be an image (e.g., multiple still images or a video) of the first virtual object (521) moving along the designated path (711) in the first area (541). The processor (410) may analyze an optimal movement path (e.g., a straight path) between the electronic device (401) and the first virtual object (521) or a real environment such as an object arrangement, texture, or spatial depth to set a movement path (e.g., a floor plane path with no complexity) to the designated path (711).

[0118] According to one embodiment, the processor (410) may control the display (430) to display (e.g., play) a three-dimensional image (530) so that a first virtual object (521) moves along a path (711) specified in a first area (541) on a screen (540) corresponding to a real space while the user is wearing the electronic device (401).

[0119] According to one embodiment, the generative AI model (501) may transmit a command to the generative AI model (501) to generate at least one other virtual object (810) (hereinafter referred to as a second virtual object) related to the first image portion (511) or the first virtual object (521) by a command from the processor (410), as illustrated in FIGS. 8A and 8B , and obtain the second virtual object (810) generated by the generative AI model (501). According to one embodiment, the generative AI model (501) may generate at least one second virtual object (810) related to the first image portion (511) through a volumetric rendering and / or generative fill (e.g., NeRF, LRM model) operation based on information about the first image portion (511). The generative AI model (501) may generate at least one second virtual object (810) in 2D and / or 3D related to each of the two or more objects when the selected first image portion (511) is two or more objects. According to one embodiment, the second virtual object may be newly generated in the generative AI model when the 2D image does not include an image portion (e.g., the second image portion) corresponding to the second virtual object (810), or may be generated by the generative AI model based on information about the second image portion selected from the 2D image. According to one embodiment, the electronic device may select the second image portion in relation to the first image portion in the 2D image, and may transmit information (e.g., relationship information and identification information) about the selected first image portion and the selected second image portion to the generative AI model through image analysis (e.g., a multi-object relationship mechanism of an AI learning model).For example, the relational information may be information about the relationship between a main subject (e.g., a first image portion) and surrounding objects (a second image portion) based on at least one of physical interaction-based, structural relationship-based, set relationship-based, spatial relationship-based, function / purpose relationship-based, or contextual relationship-based. For example, the physical interaction-based may represent a relationship in which the movement of the first object affects the physical phenomenon of the second object (e.g., a shooter - a target). The structural relationship-based may represent a relationship in which the first object and the second object are structurally connected, composed, and dependent on each other (e.g., a light bulb - a light bulb support bracket). The set relationship-based may represent a grouping relationship based on consistency and continuity information extracted from multiple objects (e.g., earbud left - right / case). The spatial relationship-based may represent a mutual arrangement relationship in which objects are related in terms of location, direction, distance, topology, and arrangement in physical space (e.g., a person - ground (grass)). A function / purpose relationship-based relationship can represent a relationship where a first object and a second object are related to each other to implement a single function and purpose (e.g., drummer - drum). A contextual relationship-based relationship can represent a set of objects with the same contextual properties (e.g., birthday person - cake), based on an understanding of the events, concepts, and immediate and subsequent situations within a scene.

[0120] According to one embodiment, the processor (410) may anchor a second virtual object (810) to be associated with a first virtual object (521) in a first area (541) (e.g., an anchoring area) identified through a camera (440) in a real space (550), as illustrated in FIG. 8A.

[0121] According to one embodiment, the processor (410) may obtain movement path information for moving the second virtual object (810) in the first area (541) along a specified path (811) (e.g., a movement path based on physical properties (weight, hardness, density, and / or force), gravity, and acceleration laws), and may obtain a three-dimensional image (830) including the second virtual object (810) from the generative AI model (501) based on the movement path information in association with the first virtual object (521) (or the three-dimensional image (530)). The three-dimensional image (830) including the second virtual object (810) may be an image (e.g., multiple still images or a video) of the second virtual object (810) moving along the specified path (811) in the first area (541). According to one embodiment, the processor (410) may apply (e.g., synthesize or display in a higher layer) a second virtual object (810) generated by the generative AI model to a three-dimensional image (530) including a first virtual object (521) without obtaining a newly generated three-dimensional image (830) from the generative AI model (501).

[0122] According to one embodiment, the processor (410) may control the display (430) to apply a second virtual object (810) moving along a designated path (811) in a first area (541) in association with a first virtual object (521) on a screen (540) corresponding to a real space while the user is wearing the electronic device (401) to a three-dimensional image (530) or to display (e.g., reproduce) the three-dimensional image (830).

[0123] According to one embodiment, the processor (410) may control the display (430) to further display a background image (820) on a three-dimensional image (830) (e.g., the three-dimensional image (530) of FIG. 5) to which a second virtual object (810) has been added, as illustrated in FIG. 8A. The background image (820) may be an image generated based on a background image included in a two-dimensional image (510) or an image newly generated in association with the first virtual object (521). The background image (820) may be generated as a two-dimensional or three-dimensional image by a generative AI model (501) according to a command of the processor (410).

[0124] According to one embodiment, the processor (410) may control the display (430) to display a three-dimensional image (530 or 830) or control the communication circuit (450) to transmit the three-dimensional image (530 or 830) to an external electronic device (403).

[0125] According to one embodiment, the processor (410) may be a hardware component (function) or a software component (program) including at least one component provided in the electronic device (401) as a hardware module or a software module (e.g., an application program). According to one embodiment, the processor (410) may include, for example, one or a combination of two or more of hardware, software, or firmware. The processor (410) may omit at least some of the above components, or may be configured to further include other components for performing image processing operations in addition to the above components.

[0126] According to one embodiment, the memory (420) (e.g., the memory (130) of FIG. 1) can store an application. For example, the memory (420) can store an application (function or program) related to an image (or image generation), an application related to image management, or an application related to a generative AI. The memory (420) can store a first image captured by an external electronic device or camera (440) (e.g., an original image), a second acquired image (e.g., an image for editing), a third acquired image (e.g., an edited result image), and information related to image editing.

[0127] According to one embodiment, the memory (420) may store a program used for functional operation (e.g., the program (140) of FIG. 1), as well as various data generated during execution of the program (140). For example, the memory (420) may include a program (140) area and a data area (not shown). The program (140) area may store related program information for driving the electronic device (401), such as an operating system (OS) (e.g., the operating system (142) of FIG. 1) that boots the electronic device (401). The data area (not shown) may store transmitted and / or received data and generated data according to various embodiments. In addition, the memory (420) may be configured to include at least one storage medium among flash memory, a hard disk, a multimedia card micro type memory (e.g., a secure digital (SD) or extreme digital (XD) memory), RAM, and ROM.

[0128] According to one embodiment, the display (430) (e.g., the display module (160) of FIG. 1, the displays (251, 252) of FIG. 2, or the display (321) of FIG. 3) may display an execution screen of an application related to an image (or image generation). The display (430) may display information related to image editing, a first image, or a third image under the control of the processor (410). When performing image editing, the display (430) may display a second image for editing under the control of the processor (410). According to one embodiment, the display (430) may be implemented in the form of a touch screen. When the display (430) is implemented together with an input module in the form of a touch screen, the display (430) may display various pieces of information generated according to a user's touch operation. According to one embodiment, the display (430) may be configured with at least one of a liquid crystal display (LCD), a thin film transistor LCD (TFT-LCD), organic light emitting diodes (OLED), a light emitting diode (LED), an active matrix organic LED (AMOLED), a flexible display, and a 3-dimensional display. In addition, some of these displays may be configured as transparent or light-transmitting so that the outside can be seen through them. This may be configured in the form of a transparent display including a transparent OLED (TOLED). According to one embodiment, in addition to the display (430), other display modules (e.g., an extended display or a flexible display) may be further installed.

[0129] According to one embodiment, a camera (440) (e.g., camera module (180) of FIG. 1, cameras (211-1, 211-2) of FIG. 2, or cameras (311, 312, 313, 314, 315, 316) of FIG. 3) can capture 2D images for generating 3D images. The camera (440) can capture a real environment (e.g., real space) and provide a screen corresponding to the real space.

[0130] According to one embodiment, the communication circuit (450) (e.g., the communication module (190) of FIG. 1) can communicate with an external electronic device (e.g., the electronic device (102, 104) of FIG. 1, the server (108) of FIG. 1, or another user's electronic device). For example, the communication circuit (450) can receive and transmit a first image, at least one 2D image, or a list of images from and to the external electronic device. The communication circuit (450) can transmit a generated 3D image (530 or 830) to the external electronic device. The communication circuit (450) can receive and / or transmit an application and / or information related to image generation from and / or to the external electronic device. According to one embodiment, the communication circuit (450) can include a cellular module, a wireless-fidelity (Wi-Fi) module, a Bluetooth module, or a near field communication (NFC) module.

[0131] An electronic device (101, 200, 300, 401) according to one embodiment may implement a software module (e.g., a program (140) of FIG. 1) for displaying a 3D image, which is created by converting at least a portion of a 2D image into a 3D virtual object, in a virtual reality space. A memory (420) of the electronic device (101, 200, 300, 401) may store commands (e.g., instructions) that, when executed by a processor (410), cause the electronic device (101, 200, 300, 401) to perform operations for displaying a 3D image, which is created by converting at least a portion of a 2D image into a 3D virtual object, in a virtual reality space, in order to implement the software module. At least one processor (120, 410) can execute instructions stored in a memory (130, 420) to implement a software module and control hardware (e.g., a sensor module (176), a power management module (188), or a communication module (190) of FIG. 1) associated with the function of the software module.

[0132] According to one embodiment, a software module of an electronic device (101, 200, 300, 401) may be configured to include a kernel (or HAL), a framework (e.g., middleware (144) of FIG. 1), and an application (e.g., application (146) of FIG. 1). At least a portion of the software module may be preloaded on the electronic device (101, 200, 300, 401) or may be downloadable from a server (e.g., server (108)).

[0133] According to one embodiment, the kernel may include, but is not limited to, a system resource manager or device driver, and may further include other modules. The system resource manager may perform control, allocation, or retrieval of system resources. The device driver may include, for example, a display driver, a camera driver, a Bluetooth driver, a shared memory driver, a USB driver, a keypad driver, a WIFI driver, an audio driver, or an inter-process communication (IPC) driver.

[0134] According to one embodiment, the framework may provide functions commonly required by applications or provide various functions to applications through an application programming interface (API) (not shown) so that the applications can efficiently use limited system resources within the electronic device (101, 200, 300, 401). The framework may include modules that form a combination of various functions of components. The framework may provide specialized modules for each type of operating system to provide differentiated functions. The framework may dynamically delete some existing components or add new components.

[0135] According to one embodiment, the application may be configured to include an application (e.g., a module, a manager, or a program) for displaying a 3D image in a virtual reality space. The application may include an application received from an external electronic device (e.g., a server (108) or an electronic device (102, 104)). According to one embodiment, the application may include a preloaded application or a third-party application downloadable from a server. The components and names of the components of the software module according to the illustrated embodiment may vary depending on the type of operating system. According to one embodiment, at least a portion of the software module may be implemented as software, firmware, hardware, or a combination of at least two or more thereof. At least a portion of the software module may be implemented (e.g., executed) by, for example, a processor (e.g., an AP). At least a portion of the software module may include, for example, a module, a program, a routine, a set of instructions, or a process for performing at least one function.

[0136] As such, in one embodiment, the main components of the electronic device (101, 200, 300, 401) of FIGS. 1, 2, 3A to 3C, and 4 are described. However, in various embodiments, not all of the components illustrated in FIGS. 1, 2, 3A to 3C, and 4 are essential components, and the electronic device (101, 200, 300, 401) may be implemented with more components than the illustrated components, or may be implemented with fewer components. In addition, the positions of the main components of the electronic device (101, 200, 300, 401) described above with FIGS. 1, 2, 3A to 3C, and 4 may be changed according to various embodiments.

[0137] According to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2, electronic device (300) of FIGS. 3A to 3C, and electronic device (401) of FIG. 4) may include a camera (180, 211-1, 211-2, 213, 311, 312, 313, 314, 315, 316, 440), a display (160, 251, 252, 321, 430), at least one processor (120, 410) including a processing circuit, and a memory (130, 420) for storing instructions.

[0138] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to acquire at least one two-dimensional (2D) image.

[0139] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to select a first image portion included in the at least one 2D image.

[0140] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain a first virtual object in three dimensions (3D) for the first image portion generated by the generative AI model (501) based on an instruction that causes the electronic device to generate the first image portion as a first virtual object in three dimensions (3D).

[0141] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to anchor a first virtual object to a first area identified through the camera in real space.

[0142] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain a three-dimensional image generated by the generative AI model and showing a first virtual object moving in the first area.

[0143] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to display the three-dimensional image in the first area on the screen corresponding to the real space.

[0144] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to: identify a second image portion related to the first image portion in the at least one 2D image; generate the second image portion as the second virtual object in the 3D based on the instructions; and control the display to display the second virtual object in the first area so as to be associated with the first virtual object.

[0145] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain movement path information for causing the first virtual object to move along a designated path in the first area, and to generate, based on the movement path information, the three-dimensional image of the first virtual object moving along the designated path in the first area by the generative AI model.

[0146] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to obtain a background image related to the first virtual object and to display the background image in the first area so as to be associated with the first virtual object.

[0147] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to display at least one of the first virtual object, the second virtual object, and the background image in the first area without overlapping with at least one of an object, an animal, or a person located within the real space.

[0148] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to identify an anchorable portion and a non-anchorable portion on the first screen based on position information of the first image portion, the anchorable portion being set as a margin area on the screen where the first virtual object can be displayed based on the position information or depth information of the first virtual object, and to display a first graphical object indicating anchorability on a first portion of the virtual space region corresponding to the anchorable portion, and to display a second graphical object indicating non-anchorability on a second portion of the virtual space region corresponding to the non-anchorable portion.

[0149] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to control the display circuit to display a third graphical object representing at least one of a dynamic movement of the first virtual object or a designated dynamic effect based on the movement path information, wherein the designated path can be set without overlapping at least one of the object, the animal or the person located in the real space.

[0150] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to display on the screen the at least one 2D image or a list of images including the at least one 2D image.

[0151] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to fix the three-dimensional image to a designated object corresponding to the object, animal or person located within the real space, and to adjust the size of the three-dimensional image based on a user input for size or size adjustment of the designated object and to display the three-dimensional image adjacent to the fixed designated object on the screen.

[0152] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to display a previous three-dimensional image stored in a memory in relation to the real space on the screen corresponding to the real space, to compare the generated three-dimensional image with another previously stored three-dimensional image to identify a virtual object for a portion having a low degree of similarity in the generated three-dimensional image, and to control the display to display the identified virtual object by replacing it with a virtual object included in the other three-dimensional image.

[0153] FIG. 9 is a diagram illustrating an example of an operating method in an electronic device according to one embodiment. FIG. 10 is a diagram illustrating an example of an operating method in an electronic device according to one embodiment. In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0154] Referring to FIGS. 9 and 10 , an electronic device (e.g., the electronic device 101 of FIG. 1 , the electronic device 200 of FIG. 2 , the electronic device 300 of FIGS. 3A to 3C , and the electronic device 401 of FIG. 4 ) according to one embodiment may obtain at least one two-dimensional (2D) image (1010) in operation 901. According to one embodiment, the electronic device may obtain at least one 2D image (1010) from a list of images stored in an external electronic device (e.g., the electronic device 102, 104 of FIG. 1 or the server (108)). According to one embodiment, the electronic device may obtain at least one 2D image (1010) from a memory (e.g., the memory 130 of FIG. 1 and the memory 420 of FIG. 4 ). According to one embodiment, the electronic device can display an image list (1001) in a part of a virtual reality space where a screen (540) corresponding to the real space is displayed. The electronic device can select a 2D image (1010) included in the image list (1001) based on a user interaction. Here, the 2D image can be captured by an external electronic device (e.g., the electronic device (102, 104) or the server (108) of FIG. 1) or a camera included in the electronic device, and can be acquired or stored in a memory together with meta information acquired through image analysis during or after capture. The meta information can include a capture location of the 2D image, identification information, layer information (e.g., depth information), and RGB information.

[0155] In operation 903, according to one embodiment, the electronic device may select a first image portion (1011) included in at least one acquired 2D image (1010). Here, the first image portion (1011) may be a portion including an object, such as an object, an animal, or a person, included in the at least one 2D image (1010). According to one embodiment, the electronic device may separate the selected first image portion (1011) from a background portion (e.g., the remaining portion).

[0156] In operation 905, according to one embodiment, the electronic device may obtain a three-dimensional (3D) first virtual object (1021) for the first image portion (1011) from the generative AI model. The electronic device may transmit (e.g., transfer) a command to generate the first image portion (1011) into the 3D first virtual object (1021) to the generative AI model (e.g., the generative AI model (501) of FIG. 5). Based on the received command, the generative AI model may generate the 3D first virtual object (1021) by stereoscopically implementing the first image portion (1011) through volumetric rendering and / or generative fill operations.

[0157] In operation 907, according to one embodiment, the electronic device may analyze a real space within a field of view (FoV) of a user wearing the electronic device through a camera, and identify (e.g., confirm, set, or designate) a first area (1031) for anchoring a first virtual object (1021) on a screen (540) corresponding to the analyzed real space. According to one embodiment, the electronic device may anchor the first virtual object (1021) to the identified first area (1031). According to one embodiment, the electronic device may identify (e.g., confirm, set, or designate) an area that does not interfere with (e.g., does not overlap with) an object (e.g., a subject) located within the real space, such as an object, animal, or person, as the first area (541).

[0158] In operation 909, according to one embodiment, the electronic device may obtain movement path information to cause the first virtual object (1021) in the first area to move along a specified path.

[0159] In operation 911, according to one embodiment, the electronic device may obtain a three-dimensional image (1030) generated by a generative AI model based on movement path information, and control the display to display the three-dimensional image (1030) in a first area (541) on a screen (540) corresponding to a real space. According to one embodiment, the electronic device may transmit (e.g., transfer) a command to generate the three-dimensional image (1030) by transferring the movement path information to the generative AI model. The generative AI model may generate the three-dimensional image (1030) moving along a specified path in the first area (541) based on the movement path information. Here, for example, the generative AI model may be stored in a memory of the electronic device and executed by a processor (e.g., the processor (120) of FIG. 1 or the processor (410) of FIG. 4). For another example, the generative AI model may be stored in an external electronic device (e.g., the server (108) of FIG. 1) and executed according to a command received from the electronic device. In one embodiment, the electronic device may generate an image as a background image (1040) in a background portion (1013) of a 2D image (1010) and display the generated background image (1040) on a screen (540) (e.g., an adjacent area or a first area (541) of a location where the 3D image (1030) is displayed) in relation to a 3D image (1030).

[0160] According to one embodiment, when the electronic device displays a background image (1040) in relation to a three-dimensional image (1030), if the background image (1040) interferes or overlaps with an object located in a real space, the electronic device may display the three-dimensional image (1030) and the background image (1040) on the screen (540) so as not to interfere with the object (e.g., without overlapping) the object located in the real space. According to one embodiment, when the three-dimensional image (1030) to be anchored to the screen (540) and the object located in the real space are within a specified distance, the electronic device may delete an area overlapping the object in the three-dimensional image (1030) or the background image (1040).

[0161] FIG. 11 is a diagram illustrating an example of an operating method in an electronic device according to one embodiment. FIG. 12 is a diagram illustrating an example of displaying a three-dimensional image in an electronic device according to one embodiment. In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

[0162] Referring to FIGS. 11 and 12 , an electronic device according to an embodiment (e.g., the electronic device 101 of FIG. 1 , the electronic device 200 of FIG. 2 , the electronic device 300 of FIGS. 3A to 3C , and the electronic device 401 of FIG. 4 ) may, in operation 1101 , obtain at least one two-dimensional (2D) image (1010). According to an embodiment, the electronic device may obtain at least one 2D image from a list of images stored in an external electronic device (e.g., the electronic device 102, 104 of FIG. 1 or the server 108). According to an embodiment, the electronic device may obtain at least one 2D image from a memory (e.g., the memory 130 of FIG. 1 and the memory 420 of FIG. 4 ). According to one embodiment, the electronic device may display a list of images (e.g., the list of images (1001) of FIG. 10) in a portion of a virtual reality space where a screen (540) corresponding to the real space is displayed. The electronic device may select at least one 2D image included in the list of images based on user interaction.

[0163] In operation 1103, according to one embodiment, the electronic device may select a first image portion included in at least one acquired 2D image. Here, the first image portion may be a portion including an object, such as an object, an animal, or a person included in the at least one 2D image. According to one embodiment, the electronic device may separate the selected first image portion from a background portion (e.g., the remaining portion).

[0164] In operation 1105, according to one embodiment, the electronic device may obtain a three-dimensional (3D) first virtual object (1221) for a first image portion from a generative AI model (e.g., the generative AI model (501) of FIG. 5 ). The electronic device may transmit (e.g., transfer) a command to generate the first image portion as the 3D first virtual object (1221) to the generative AI model. Based on the received command, the generative AI model may generate the 3D first virtual object (1221) by stereoscopically implementing the first image portion through a generative filling operation.

[0165] In operation 1107, an electronic device according to an embodiment may analyze a real space within a field of view (FoV) of a user wearing the electronic device through a camera, and identify (e.g., confirm, set, or designate) a first region for anchoring a first virtual object (1021) on a screen (540) corresponding to the analyzed real space. According to an embodiment, the electronic device may anchor the first virtual object (1021) to the identified first region (1031). According to an embodiment, the electronic device may identify (e.g., confirm, set, or designate) a region that does not interfere with (e.g., does not overlap with) an object (e.g., a subject) for an object, animal, or person located within the real space as the first region.

[0166] In operation 1109, an electronic device according to one embodiment may obtain movement path information to cause a first virtual object (1121) to move along a specified path in a first area.

[0167] In operation 1111, an electronic device according to an embodiment may obtain a three-dimensional image (1230) generated by a generative AI model based on movement path information, and control a display to display the three-dimensional image (1230) in a first area (541) on a screen (540) corresponding to a real space. According to an embodiment, the electronic device may transmit (e.g., transfer) a command to transfer the movement path information to the generative AI model to generate the three-dimensional image (1230). The generative AI model may generate a three-dimensional image (1230) moving along a specified path in the first area (541) based on the movement path information. Here, for example, the generative AI model may be stored in a memory of the electronic device and executed by a processor (e.g., the processor (120) of FIG. 1 or the processor (410) of FIG. 4). For another example, a generative AI model may be stored in an external electronic device (e.g., server (108) of FIG. 1) and executed in response to commands received from the electronic device.

[0168] In operation 1113, an electronic device according to an embodiment may determine whether to generate a second virtual object (1223) related to a three-dimensional image (1230). If the electronic device determines that generation of a second virtual object (1223) related to a three-dimensional image (1230) is required, the electronic device may perform operation 1115, and if the electronic device determines that generation of a second virtual object (1223) is not required, the electronic device may perform operation 1117.

[0169] In operation 1115, an electronic device according to an embodiment may acquire a second virtual object (1223) generated by a generative AI model in relation to a three-dimensional image (1230), and display the acquired second virtual object (1223) in a first area of ​​a screen (540) in relation to the three-dimensional image (1230). The generative AI model may generate the second virtual object (1223) in relation to the three-dimensional image (1130) based on identification information about the first virtual object (1221) in the three-dimensional image (1130). For example, when the generative AI model analyzes the identification information and identifies that the first virtual object (1221) is a person in a posture for playing golf, the generative AI model may determine that a golf ball is needed, and may generate a virtual object (1223) for a golf ball. According to one embodiment, the electronic device may obtain movement path information of the second virtual object (1223), and obtain the second virtual object (1223) moving along a specified path from the generative AI model based on the obtained movement path information. The electronic device may display the second virtual object (1223) moving along the specified path in relation to the three-dimensional image (1130), or display a new three-dimensional image (1231) in which the second virtual object (1223) moving along the specified path is applied to the three-dimensional image (1230). According to one embodiment, the second virtual object (1223) may be newly generated in the generative AI model or may be generated by the generative AI model based on information about the second image portion selected from the 2D image, without including an image portion (e.g., a second image portion) corresponding to the second virtual object (1223) in the 2D image. In one embodiment, the electronic device may select a second image portion in relation to a first image portion in a 2D image, and may transmit information (e.g., relationship information and identification information) about the selected first image portion and the selected second image portion to a generative AI model through image analysis (e.g., a multi-object relationship mechanism of an AI learning model).For example, the relational information may be information about the relationship between a main subject (e.g., a first image portion) and surrounding objects (a second image portion) based on at least one of physical interaction-based, structural relationship-based, set relationship-based, spatial relationship-based, function / purpose relationship-based, or contextual relationship-based. For example, the physical interaction-based may represent a relationship in which the movement of the first object affects the physical phenomenon of the second object (e.g., a shooter - a target). The structural relationship-based may represent a relationship in which the first object and the second object are structurally connected, composed, and dependent on each other (e.g., a light bulb - a light bulb support bracket). The set relationship-based may represent a grouping relationship based on consistency and continuity information extracted from multiple objects (e.g., earbud left - right / case). The spatial relationship-based may represent a mutual arrangement relationship in which objects are related in terms of location, direction, distance, topology, and arrangement in physical space (e.g., a person - ground (grass)). A function / purpose relationship-based relationship can represent a relationship where a first object and a second object are related to each other to implement a single function and purpose (e.g., drummer - drum). A contextual relationship-based relationship can represent a set of objects with the same contextual properties (e.g., birthday person - cake), based on an understanding of the events, concepts, and immediate and subsequent situations within a scene.

[0170] In operation 1117, an electronic device according to an embodiment may obtain a background image (1040) in relation to a three-dimensional image (1230) or a three-dimensional image to which a second virtual object (1223) is applied, and display the background image (1240) in relation to the three-dimensional image (1230) or the three-dimensional image (1231) to which the second virtual object (1223) is applied on a screen (540) (e.g., an adjacent area or a first area (541) to which the three-dimensional image (1230 or 1231) is displayed).

[0171] According to one embodiment, when the electronic device displays a background image (1240) in relation to a three-dimensional image (1230 or 1231), if the background image (1240) interferes or overlaps with an object located in a real space, the electronic device may display the three-dimensional image (1230 or 1231) and the background image (1240) on the screen (540) so as not to interfere with (e.g., without overlapping) the object (1241) of the object. According to one embodiment, when the three-dimensional image (1230 or 1231) to be anchored to the screen (540) and the object (1241) of the object located in a real space are within a specified distance, the electronic device may delete an area overlapping the object (1241) in the three-dimensional image (1230 or 1231) or the background image (1140).

[0172] FIG. 13 is a drawing showing an example of displaying a three-dimensional image in an electronic device according to one embodiment.

[0173] Referring to FIG. 13, when an electronic device according to an embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, the electronic device (300) of FIGS. 3A to 3C, and the electronic device (401) of FIG. 4) captures a 2D image or video, the electronic device classifies a main object included in the 2D image or video (e.g., an object designated through learning or an object, animal, or person set by the user), and stores a file including meta information about the classified main object and information (e.g., a 3D image) that is 3D modeled up to an invisible part through a generative AI model, together with the 2D image or video, in memory. The file can be acquired from an external electronic device and stored in memory. According to an embodiment, the electronic device can store the generated 3D image described in the drawings described above in memory.

[0174] In one embodiment, when a user selects a stored 2D image or video from a photo list (e.g., gallery) to recall memories in a space where the 2D image or video was taken over time, the electronic device may display (e.g., play) a screen (1301) corresponding to the current real environment (e.g., real space) together with a previously generated 3D image (1310) stored in memory to maximize realism.

[0175] FIG. 14 is a drawing showing an example of displaying a three-dimensional image in an electronic device according to one embodiment.

[0176] Referring to FIG. 14, an electronic device (401) according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, and the electronic device (300) of FIGS. 3A to 3C) can display a three-dimensional image (1420) (e.g., a currently generated three-dimensional image or a previously generated three-dimensional image stored in memory) by fixing (e.g., mapping) it to a designated object (1401) for an object, animal, or person on a screen (540) corresponding to a real space. When the electronic device (401) displays the three-dimensional image (1420) by fixing it to the designated object (14011), the electronic device (401) can adjust the size of the three-dimensional image (1420) based on the size of the designated object (1401) or a user input for size adjustment. According to one embodiment, the electronic device (401) can control the display to display a three-dimensional image (1420) whose size is adjusted on a screen (540) corresponding to a real space, adjacent to a position of a designated object (1401). According to one embodiment, the electronic device (401) can display a related graphic object or graphic effect (e.g., a message (1430) of “My cute daughter, working hard, looks good in three dimensions”) together with the three-dimensional image (1420).

[0177] FIG. 15 is a drawing showing an example of displaying a three-dimensional image in an electronic device according to one embodiment.

[0178] Referring to FIG. 15, an electronic device (401) according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (200) of FIG. 2, and the electronic device (300) of FIGS. 3A to 3C) can compare a currently generated three-dimensional image (1510) with another three-dimensional image at a specific point in time related to a real space to identify a portion having low similarity in the three-dimensional image (1510). The electronic device (401) can replace a virtual object (1511) for a portion having low similarity with another virtual object (1521) included in another three-dimensional image, and control the display to display the three-dimensional object replaced with the other virtual object (1521). According to one embodiment, the electronic device (401) may display a related graphic object or graphic effect (e.g., a message (1530) of "Teacher, was that building there in the past?") along with a three-dimensional image replaced with another virtual object (1521).

[0179] According to one embodiment, a method of operating an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (200) of FIG. 2, electronic device (300) of FIGS. 3A to 3C, and electronic device (401) of FIG. 4) may include an operation of acquiring at least one two-dimensional (2D) image.

[0180] According to one embodiment, the method may include an operation of selecting a first image portion included in the at least one 2D image.

[0181] According to one embodiment, the method may include an operation of obtaining a first 3D virtual object for the first image portion generated by the generative AI model (501) based on a command to generate the selected first image portion as a first 3D virtual object.

[0182] According to one embodiment, the method may include an operation of anchoring a first virtual object to a first area identified through a camera (180, 211-1, 211-2, 213, 311, 312, 313, 314, 315, 316, 440) of the electronic device in real space.

[0183] According to one embodiment, the method may include an operation of obtaining a three-dimensional image generated by the generative AI model and in which a first virtual object moves in the first area.

[0184] According to one embodiment, the method may include an operation of controlling a display (160, 251, 252, 321, 430) of the electronic device to display the three-dimensional image in the first area on a screen corresponding to the real space.

[0185] In one embodiment, the method may further include an operation of identifying a second image portion related to the first image portion in the at least one 2D image, an operation of obtaining the second virtual object generated by the generative AI model based on a command to generate the second image portion as the second virtual object in the 3D, and an operation of controlling the display to display the second virtual object in the first area so as to be associated with the first virtual object.

[0186] In one embodiment, the method may include an operation of obtaining movement path information set to cause the first virtual object to move along a specified path in the first area, and an operation of generating, by the generative AI model, the 3D image in which the first virtual object moves along the specified path in the first area based on the movement path information.

[0187] According to one embodiment, the method may further include an operation of obtaining a background image related to the first virtual object and an operation of controlling the display to display the background image in the first area so as to be associated with the first virtual object.

[0188] According to one embodiment, the method may further include an operation of controlling the display to display at least one of the first virtual object, the second virtual object, and the background image in the first area without overlapping with at least one object among objects, animals, or people located within the real space.

[0189] According to one embodiment, the method may further include an operation of identifying an anchorable portion and a non-anchorable portion in the first screen, and an operation of controlling the display to display a first graphic object indicating anchorability in a first portion of the virtual space region corresponding to the anchorable portion, and a second graphic object indicating non-anchorability in a second portion of the virtual space region corresponding to the non-anchorable portion.

[0190] According to one embodiment, the method includes an operation of controlling the display circuit to display a third graphic object representing at least one of dynamic movement of the first virtual object or a designated dynamic effect based on the movement path information, wherein the designated path can be set without overlapping at least one of the object, the animal, or the person located in the real space.

[0191] According to one embodiment, the method may further include an action of controlling the display to display on the screen at least one 2D image or a list of images including the at least one 2D image.

[0192] According to one embodiment, the method may further include an operation of controlling the display to fix the three-dimensional image to a designated object corresponding to the object, animal or person located in the real space, and to adjust the size of the three-dimensional image based on a user input for size or size adjustment of the designated object and to display the three-dimensional image adjacent to the fixed designated object on the screen.

[0193] According to one embodiment, the method may further include an operation of controlling the display to display a previous three-dimensional image stored in a memory in relation to the real space on the screen corresponding to the real space, an operation of comparing the generated three-dimensional image with another previously stored three-dimensional image to identify a virtual object for a portion having low similarity in the generated three-dimensional image, and an operation of controlling the display to display the identified virtual object by replacing it with a virtual object included in the other three-dimensional image.

[0194] According to one embodiment, in a non-transitory storage medium storing one or more programs, the program, when executed by at least one processor (120, 410) of an electronic device (101, 200, 300, 401), causes the electronic device to: obtain at least one two-dimensional (2D) image; select a first image portion included in the at least one 2D image; obtain a 3D first virtual object for the first image portion generated by a generative AI model (501) based on a command to generate the selected first image portion as a three-dimensional (3D) first virtual object; anchor the first virtual object in a first area identified through a camera (180, 211-1, 211-2, 213, 311, 312, 313, 314, 315, 316, 440) of the electronic device in a real space; It may include instructions for executing an operation of obtaining a three-dimensional image (530) generated by a generative AI model and in which a first virtual object moves in the first area, and an operation of controlling a display (160, 251, 252, 321, 430) of the electronic device to display the three-dimensional image in the first area on a screen corresponding to the real space.

[0195] Through this document, an electronic device according to one embodiment generates a part of a two-dimensional image as a three-dimensional virtual object, and displays a three-dimensional image that moves the generated three-dimensional virtual object along a designated path in a virtual environment (e.g., a virtual reality space), thereby reproducing a three-dimensional image more realistically in a virtual environment. In addition, various effects that can be directly or indirectly understood through this document can be provided. The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below.

[0196] The embodiments disclosed in this document are presented for the purpose of explaining and understanding the disclosed technical content, and do not limit the scope of the technology described in this document. Therefore, the scope of this document should be interpreted to include all modifications or various other embodiments based on the technical concepts of this document.

[0197] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0198] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0199] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0200] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0201] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0202] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices (101, 200, 300, 401), Camera (180, 211-1, 211-2, 213, 311, 312, 313, 314, 315, 316, 440); display(160, 251, 252, 321, 430); At least one processor (120, 410) comprising a processing circuit; and It includes memory (130, 420) for storing instructions, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Acquire at least one two-dimensional (2D) image, Selecting a first image portion included in at least one two-dimensional image, Based on a command to generate the first image portion selected above as a three-dimensional (3D) first virtual object, the first 3D virtual object for the first image portion generated by the generative AI model (501) is obtained, Anchoring of a first virtual object in a first area identified through the camera in a real space, Obtaining a three-dimensional image generated by the generative AI model and in which a first virtual object moves in the first area; An electronic device that causes the display to be controlled to display the three-dimensional image in the first area on the screen corresponding to the real space.

2. In the first paragraph, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Identifying a second image portion related to the first image portion in the at least one two-dimensional image, Obtaining the second virtual object generated by the generative AI model based on a command to generate the second image portion as the second virtual object of the 3D; An electronic device that causes the display to be controlled to display the second virtual object in the first area so as to be associated with the first virtual object.

3. In the first or second paragraph, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Obtaining movement path information set to make the first virtual object move along a specified path in the first area; An electronic device that causes the first virtual object to store instructions for generating a three-dimensional image of the first virtual object moving along the designated path in the first area by the generative AI model based on the movement path information.

4. In any one of paragraphs 1 to 3, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Obtain a background image related to the first virtual object, Controlling the display to display the background image in the first area so as to be associated with the first virtual object; An electronic device that causes the display to be controlled so as to display at least one of the first virtual object, the second virtual object, and the background image in the first area without overlapping with at least one object among objects, animals, or people located in the real space.

5. In any one of paragraphs 1 to 4, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Based on the location information of the first image portion, an anchorable portion and an unanchorable portion are identified on the first screen, and the anchorable portion is set as a margin area where the first virtual object can be displayed on the screen based on the location information or depth information of the first virtual object. An electronic device that causes the display to be controlled to display a first graphic object indicating anchorability in a first portion of the virtual space area corresponding to the anchorable portion, and to display a second graphic object indicating anchorability in a second portion of the virtual space area corresponding to the non-anchorable portion.

6. In any one of paragraphs 1 to 5, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Controlling the display to display a third graphic object representing at least one of the dynamic movement of the first virtual object or a specified dynamic effect based on the movement path information, Causing the display to be controlled to display on the screen at least one two-dimensional image or a list of images including the at least one two-dimensional image; An electronic device in which the above-mentioned designated path is set without overlapping at least one object among the object, the animal or the person located in the real space.

7. In any one of paragraphs 1 to 6, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Fixing the above 3D image to a designated object corresponding to the object, animal or person located within the above real space, An electronic device that causes the display to be controlled to adjust the size of the three-dimensional image based on a user input for adjusting the size or size of the specified object and to display the three-dimensional image adjacent to the specified object fixed on the screen.

8. In any one of paragraphs 1 to 7, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Controlling the display to display a previous three-dimensional image stored in memory in relation to the above real space on the screen corresponding to the above real space; By comparing the generated 3D image with another previously stored 3D image, a virtual object is identified for a part of the generated 3D image with low similarity, An electronic device that causes the display to be controlled to display the identified virtual object by replacing it with a virtual object included in the other three-dimensional image.

9. In the method of operation in an electronic device (101, 200, 300, 401), An act of acquiring at least one two-dimensional (2D) image; An operation of selecting a first image portion included in at least one two-dimensional image; An operation of obtaining a three-dimensional (3D) first virtual object for the first image portion generated by a generative AI model (501) based on a command to generate the selected first image portion as a three-dimensional (3D) first virtual object; An operation of anchoring a first virtual object in a first area identified through a camera (180, 211-1, 211-2, 213, 311, 312, 313, 314, 315, 316, 440) of the electronic device in a real space; An operation of obtaining a three-dimensional image generated by the generative AI model and in which a first virtual object moves in the first area; and A method comprising an operation of controlling a display (160, 251, 252, 321, 430) of the electronic device to display the three-dimensional image in the first area on a screen corresponding to the real space.

10. In the 9th paragraph, the method, An operation of identifying a second image portion related to the first image portion in the at least one two-dimensional image; An operation of obtaining the second virtual object generated by the generative AI model based on a command to generate the second image portion as the three-dimensional second virtual object; and Further comprising an action of controlling the display to display the second virtual object in the first area so as to be associated with the first virtual object; The operation of obtaining the above 3D image is as follows: An operation of obtaining movement path information set to cause the first virtual object to move along a specified path in the first area; and A method comprising an operation of generating, by the generative AI model, the 3D image in which the first virtual object moves along the designated path in the first area based on the movement path information.

11. In any one of paragraphs 9 and 10, the method, An operation of obtaining a background image related to the first virtual object; and Controlling the display to display at least one of the first virtual object, the second virtual object, and the background image in the first area without overlapping with at least one object among objects, animals, or people located in the real space; An operation for identifying anchorable and non-anchorable parts on the first screen; and A method further comprising an action of controlling the display to display a first graphic object indicating anchorability in a first portion of the virtual space area corresponding to the anchorable portion, and to display a second graphic object indicating anchorability in a second portion of the virtual space area corresponding to the non-anchorable portion.

12. In any one of the 9th to 11th clauses, the method, An operation for controlling the display circuit to display a third graphic object representing at least one of the dynamic movement of the first virtual object or a specified dynamic effect based on the movement path information; and Further comprising an action of controlling the display to display at least one 2D image or a list of images including at least one 2D image on the screen, A method in which the above-mentioned specified path is set without overlapping at least one object among the object, the animal or the person located in the real space.

13. In any one of paragraphs 9 to 12, the method, A method further comprising an action of controlling the display to fix the three-dimensional image to a designated object corresponding to the object, animal or person located in the real space, and to adjust the size of the three-dimensional image based on a user input for the size or size adjustment of the designated object and to display the three-dimensional image adjacent to the fixed designated object on the screen.

14. In any one of the 9th to 13th clauses, the method, An action of controlling the display to display a previous three-dimensional image stored in memory in relation to the real space on the screen corresponding to the real space; An operation of comparing the generated 3D image with another previously stored 3D image to identify a virtual object for a portion of the generated 3D image with low similarity; and A method further comprising an action of controlling the display to display the identified virtual object by replacing it with a virtual object included in the other three-dimensional image.

15. In a non-transitory storage medium storing one or more programs, the one or more programs, when executed by at least one processor (120, 410) of an electronic device (101, 200, 300, 401), cause the electronic device to: An act of acquiring at least one two-dimensional (2D) image; An operation of selecting a first image portion included in at least one two-dimensional image; An operation of obtaining a three-dimensional (3D) first virtual object for the first image portion generated by a generative AI model (501) based on a command to generate the selected first image portion as a three-dimensional (3D) first virtual object; An operation of anchoring a first virtual object in a first area identified through a camera (180, 211-1, 211-2, 213, 311, 312, 313, 314, 315, 316, 440) of the electronic device in a real space; An operation of obtaining a three-dimensional image (530) generated by the generative AI model and in which a first virtual object moves in the first area; and A non-transitory storage medium comprising instructions for executing an operation of controlling a display (160, 251, 252, 321, 430) of the electronic device to display the three-dimensional image in the first area on a screen corresponding to the real space.

Images