Wearable electronic device for displaying virtual objects and control method thereof

Abstract

An electronic device is provided. The electronic device includes: a display, a sensor, a memory for storing one or more computer programs, and a processor communicatively connected to the display, the sensor, and the memory. The computer program includes computer-executable instructions. When executed individually or jointly by one or more processors, the instructions can cause the electronic device to: acquire information related to a user's field of view via the sensor; acquire a two-dimensional image; identify a first portion of the two-dimensional image to be applied as a background image in a three-dimensional virtual space; identify a second portion of the two-dimensional image corresponding to an object of interest; generate an extended image corresponding to the first portion of the two-dimensional image; apply a first depth to the extended image; generate a three-dimensional virtual image by applying a second depth to the second portion of the two-dimensional image; and display a portion of the three-dimensional virtual image based on information related to the user's field of view.

Description

Technical Field

[0001] This disclosure relates to a wearable electronic device for displaying virtual objects and a method for controlling the same. Background Technology

[0002] Electronic devices (such as smartphones or other portable electronic devices) are offering an increasing number of services and additional functions. To meet the diverse needs of users and improve the efficiency of electronic device usage, communication service operators and device manufacturers are competing to develop electronic devices with differentiated and diversified functions. Therefore, the variety of functions offered through electronic devices is constantly evolving.

[0003] Wearable electronic devices, such as augmented reality glasses (AR glasses), virtual reality glasses (VR glasses), and head-mounted displays (HMDs), are offering an increasing number of services and additional features. To meet the diverse needs of users and improve the efficiency of electronic device use, communication service operators and device manufacturers are competing to develop electronic devices with differentiated and diversified functions. Therefore, the various functions offered through wearable electronic devices are increasingly evolving.

[0004] When worn on a user's head, AR or VR glasses can provide a realistic experience by displaying virtual images. AR or VR glasses can replace the availability of smartphones in various fields such as gaming, education, and social networking services. Users can receive content from their smartphones and / or realistic content through the AR or VR glasses they wear.

[0005] The above information is presented as background information only to aid in understanding this disclosure. No determination or assertion is made regarding whether any of the above content can be used as prior art with respect to this disclosure. Summary of the Invention

[0006] Solution to the problem

[0007] According to an embodiment, the electronic device may include a display, a sensor, a processor, and a memory that stores instructions that, when executed by the processor, enable the electronic device to perform the following operations.

[0008] According to an embodiment, the instructions, when executed by a processor, enable an electronic device to obtain information related to the user's field of view via a sensor.

[0009] According to an embodiment, the instructions can enable an electronic device to acquire a two-dimensional (2D) image when executed by a processor.

[0010] According to an embodiment, when executed by a processor, the instructions enable an electronic device to recognize a first portion of a 2D image that is to be applied as a background image for a three-dimensional (3D) virtual space.

[0011] According to an embodiment, when executed by a processor, the instructions enable an electronic device to identify a second portion of a 2D image corresponding to an object of interest.

[0012] According to an embodiment, when executed by a processor, the instructions enable an electronic device to generate an extended image corresponding to a first portion of a 2D image.

[0013] According to an embodiment, when executed by a processor, the instructions enable an electronic device to generate a 3D virtual image by applying a first depth to an extended image and a second depth to a second portion of a 2D image.

[0014] According to an embodiment, when the processor executes instructions, the instructions can cause the electronic device to display a portion of a 3D virtual image based on information related to the user's field of view.

[0015] According to an embodiment, a method for controlling an electronic device may include: obtaining information related to a user's field of view via sensors of the electronic device.

[0016] According to an embodiment, a method for controlling an electronic device may include obtaining a two-dimensional (2D) image.

[0017] According to an embodiment, a method for controlling an electronic device may include: identifying a first portion of a 2D image of a background image to be applied as a three-dimensional (3D) virtual space.

[0018] According to an embodiment, a method for controlling an electronic device may include: identifying a second portion of a 2D image corresponding to an object of interest.

[0019] According to an embodiment, a method for controlling an electronic device may include: generating an extended image corresponding to a first portion of a 2D image.

[0020] According to an embodiment, a method for controlling an electronic device may include generating a 3D virtual image by applying a first depth to an extended image and a second depth to a second portion of a 2D image.

[0021] According to an embodiment, a method for controlling an electronic device may include a portion that displays a 3D virtual image based on information related to the user's field of view.

[0022] According to an embodiment, in a non-transitory computer-readable recording medium storing one or more programs, the one or more programs may store instructions that enable an electronic device to obtain information related to a user's field of view via a sensor.

[0023] According to an embodiment, the one or more programs may store instructions that enable an electronic device to acquire two-dimensional (2D) images.

[0024] According to an embodiment, the one or more programs may store instructions to enable an electronic device to recognize a first portion of a 2D image that is to be applied as a background image for a three-dimensional (3D) virtual space.

[0025] According to an embodiment, the one or more programs may store instructions for enabling an electronic device to perform the following operation: identify a second portion of a 2D image corresponding to an object of interest.

[0026] According to an embodiment, the one or more programs may store instructions to enable an electronic device to generate an extended image corresponding to a first portion of a 2D image.

[0027] According to an embodiment, one or more programs may store instructions to enable an electronic device to generate a 3D virtual image by applying a first depth to an extended image and a second depth to a second portion of a 2D image.

[0028] According to an embodiment, one or more programs may store instructions to enable an electronic device to display a portion of a 3D virtual image based on information related to the user's field of view.

[0029] According to an embodiment, the electronic device may include a display, a sensor for sensing a user’s gaze as at least part of the field of view information, a memory for storing a virtual space including a three-dimensional (3D) background region, and a processor operatively connected to the sensor and the memory.

[0030] According to an embodiment, the memory can store instructions that, when executed by a processor, enable the electronic device to acquire a two-dimensional (2D) image.

[0031] According to an embodiment, the memory may store instructions that, when executed by a processor, enable an electronic device to generate an extended image based on the properties of a 2D image. The extended image further includes an image portion generated in an outward direction of the 2D image, such that the image is continuous at a boundary corresponding to at least one edge of the 2D image.

[0032] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to be configured to include an extended image in at least a portion of a 3D background area.

[0033] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to display at least a portion of a virtual space including a set 3D background area based on field-of-view information.

[0034] According to an embodiment, a head-mounted display (HMD) device may include: a display; a sensor configured to detect the user's gaze at the HMD as part of field of view information; a memory configured to store image information indicating a virtual space including a 3D background area; and a processor operatively connected to the display, the sensor, and the memory.

[0035] According to an embodiment, the memory can store instructions that, when executed by a processor, enable the head-mounted display device to acquire 2D images.

[0036] According to an embodiment, the memory may store instructions that, when executed by the processor, enable the head-mounted display device to display at least a portion of a virtual space via a display, such that an extended image corresponding to a 2D image is included in at least a portion of a 3D background area.

[0037] According to an embodiment, the electronic device may include a display, a sensor for sensing a user’s gaze as at least part of the field of view information, a memory for storing a virtual space including a 3D background area, and a processor.

[0038] According to an embodiment, the memory can store instructions that, when executed by a processor, enable the electronic device to acquire a 2D image.

[0039] According to an embodiment, the memory can store instructions that, when executed by a processor, enable the electronic device to identify background and object regions in a 2D image.

[0040] According to an embodiment, the memory can store instructions that, when executed by a processor, enable the electronic device to separate object regions from a 2D image.

[0041] According to an embodiment, the memory may store instructions that, when executed by a processor, enable an electronic device to generate an extended image based on attributes of a background region. The extended image also includes image portions generated outside the background region to continue at the boundaries of the background region.

[0042] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to be configured to include an extended image in at least a portion of the 3D background area.

[0043] According to an embodiment, the memory can store instructions that, when executed by the processor, enable the electronic device to arrange an object region between a set of points in virtual space and a 3D background region based on field-of-view information.

[0044] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to display at least a portion of a virtual space comprising a set 3D background area and arranged object areas based on field-of-view information.

[0045] The method for controlling an electronic device according to an embodiment may include obtaining a 2D image.

[0046] According to an embodiment, the method may further include generating an extended image based on the attributes of the 2D image, the extended image including an image portion generated in the outward direction of the 2D image such that the image is continuous at a boundary corresponding to at least one edge of the 2D image.

[0047] According to an embodiment, the method may further include setting the image to be expanded to be included in at least a portion of the 3D background area.

[0048] According to an embodiment, the method may further include displaying at least a portion of a virtual space including a set 3D background area based on field-of-view information.

[0049] According to an embodiment, a method for controlling a head-mounted display (HMD) device may include obtaining a 2D image.

[0050] According to an embodiment, the method may further include: displaying at least a portion of the virtual space on a display such that an extended image corresponding to the 2D image is included in at least a portion of the 3D background area.

[0051] According to an embodiment, a method for controlling an electronic device may include obtaining a two-dimensional (2D) image.

[0052] According to an embodiment, the method may further include identifying background regions and object regions in a 2D image.

[0053] According to an embodiment, the method may further include separating object regions from a 2D image.

[0054] According to an embodiment, the method may further include: generating an extended image based on the properties of a background region, the extended image including an image portion generated outside the background region to be continuous at the boundary of the background region.

[0055] According to an embodiment, the method may further include setting the image to be expanded to be included in at least a portion of the 3D background area.

[0056] According to an embodiment, the method may further include: arranging the object region between a set of points in virtual space and a 3D background region in relation to field of view information.

[0057] According to an embodiment, the method may further include: displaying at least a portion of a virtual space comprising a set 3D background area and arranged object areas based on field of view information.

[0058] According to an embodiment, in a non-transitory computer-readable recording medium storing one or more programs, the one or more programs may store instructions that enable an electronic device to acquire 2D images.

[0059] According to an embodiment, one or more programs may store instructions that enable an electronic device to generate an extended image based on the properties of a 2D image, the extended image further including an image portion generated in an outward direction of the 2D image such that the image is continuous at a boundary corresponding to at least one edge of the 2D image.

[0060] According to an embodiment, the one or more programs may store instructions that enable an electronic device to be configured to include an extended image in at least a portion of a 3D background area.

[0061] According to an embodiment, the one or more programs may store instructions that enable an electronic device to perform the following operation: display at least a portion of a virtual space including a set 3D background area based on field of view information.

[0062] According to an embodiment, in a non-transitory computer-readable recording medium storing one or more programs, the one or more programs may store instructions that enable a head-mounted display (HMD) device to acquire 2D images.

[0063] According to an embodiment, one or more programs may store instructions that enable a head-mounted display device to display at least a portion of a virtual space via a display, such that an extended image corresponding to a 2D image is included in at least a portion of a 3D background area.

[0064] According to an embodiment, in a non-transitory computer-readable recording medium storing one or more programs, the one or more programs may store instructions that enable an electronic device to acquire 2D images.

[0065] According to an embodiment, the one or more programs may store instructions that enable an electronic device to recognize background and object regions in a 2D image.

[0066] According to an embodiment, the one or more programs may store instructions that enable an electronic device to separate an object region from a 2D image.

[0067] According to an embodiment, the one or more programs may store instructions that enable an electronic device to generate an extended image based on attributes of a background region, the extended image including an image portion generated outside the background region to continue at the boundary of the background region.

[0068] According to an embodiment, the one or more programs may store instructions that enable an electronic device to be configured to include an extended image in at least a portion of a 3D background area.

[0069] According to an embodiment, the one or more programs may store instructions that enable an electronic device to arrange an object region between a set of points in virtual space and a 3D background region based on field-of-view information.

[0070] According to an embodiment, the one or more programs may store instructions that enable an electronic device to display at least a portion of a virtual space, including a set 3D background area and arranged object areas, based on field-of-view information. Attached Figure Description

[0071] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0072] Figure 1a This is a block diagram illustrating an electronic device in a network environment according to an embodiment of the present disclosure;

[0073] Figure 1b This is a block diagram illustrating the operation of obtaining results from generative artificial intelligence (AI) of an electronic device according to embodiments of the present disclosure;

[0074] Figure 2 This is a perspective view showing the internal configuration of a wearable electronic device according to an embodiment of the present disclosure;

[0075] Figure 3a This is a front perspective view showing a wearable electronic device according to an embodiment of the present disclosure;

[0076] Figure 3b This is a rear perspective view showing a wearable electronic device according to an embodiment of the present disclosure;

[0077] Figure 4 This is a perspective view illustrating a wearable electronic device according to an embodiment of the present disclosure;

[0078] Figure 5 This is a flowchart illustrating the operation of an electronic device displaying an extended image generated from a 2D image in a 3D background area according to an embodiment of the present disclosure;

[0079] Figure 6a This is a flowchart illustrating the operation of an electronic device generating an extended image based on a 2D image according to an embodiment of the present disclosure;

[0080] Figure 6b This is a view illustrating the operation of an electronic device generating an extended image based on a 2D image according to an embodiment of the present disclosure;

[0081] Figure 7a This is a flowchart illustrating the operation of an electronic device generating two extended images connected to each other at opposite ends based on a 2D image, according to an embodiment of the present disclosure;

[0082] Figure 7b This is a view illustrating the operation of an electronic device generating two extended images connected to each other at opposite ends based on a 2D image, according to an embodiment of the present disclosure;

[0083] Figure 8a This is a view illustrating the operation of connecting two opposite ends of an extended image by an electronic device according to an embodiment of the present disclosure;

[0084] Figure 8b This is a view illustrating the operation of connecting two opposite ends of an extended image by an electronic device according to an embodiment of the present disclosure;

[0085] Figure 9 This is a view illustrating the operation of an electronic device generating an extended image of a ceiling and floor according to an embodiment of the present disclosure;

[0086] Figure 10 This is a flowchart illustrating the operation of an electronic device arranging each object included in an extended image at different depths, according to embodiments of the present disclosure;

[0087] Figure 11 This is a flowchart illustrating the operation of an electronic device arranging each object included in an extended image at different depths, according to embodiments of the present disclosure;

[0088] Figure 12a This is a view illustrating the operation of an electronic device arranging each object included in an extended image at different depths, according to embodiments of the present disclosure;

[0089] Figure 12b This is a view showing the depth of a 3D space according to an embodiment of the present disclosure;

[0090] Figure 12c It is a view showing an extended image of objects arranged at different depths according to embodiments of the present disclosure;

[0091] Figure 12d It is a view showing an extended image of objects arranged at different depths according to embodiments of the present disclosure;

[0092] Figure 13a This is a view illustrating the operation of an electronic device providing interaction to an arranged object according to an embodiment of the present disclosure;

[0093] Figure 13b This is a view illustrating the operation of an electronic device providing interaction to an arranged object according to an embodiment of the present disclosure;

[0094] Figure 13c This is a view illustrating the operation of an electronic device providing interaction to an arranged object according to an embodiment of the present disclosure;

[0095] Figure 13d This is a view illustrating the operation of an electronic device providing interaction to an arranged object according to an embodiment of the present disclosure;

[0096] Figure 14 This is a view illustrating the operation of an electronic device providing interaction to an arranged object according to an embodiment of the present disclosure;

[0097] Figure 15 This is a view illustrating the operation of an electronic device providing interaction to an arranged object according to an embodiment of the present disclosure;

[0098] Figure 16 This is a flowchart illustrating the operation of an electronic device mapping audio to arranged objects according to an embodiment of the present disclosure;

[0099] Figure 17a This is a view illustrating the operation of an electronic device mapping audio to arranged objects according to an embodiment of the present disclosure;

[0100] Figure 17b This is a view illustrating the operation of an electronic device mapping audio to arranged objects according to an embodiment of the present disclosure;

[0101] Figure 17c This is a view illustrating the operation of an electronic device mapping audio to arranged objects according to an embodiment of the present disclosure;

[0102] Figure 17d This is a view illustrating the operation of an electronic device mapping audio to arranged objects according to an embodiment of the present disclosure;

[0103] Figure 18a This is a view illustrating the operation of changing the arrangement of objects by an electronic device according to an embodiment of the present disclosure;

[0104] Figure 18b This is a view illustrating the operation of changing the arrangement of objects by an electronic device according to an embodiment of the present disclosure;

[0105] Figure 18c This is a view illustrating the operation of changing the arrangement of objects by an electronic device according to an embodiment of the present disclosure;

[0106] Figure 19 This is a view illustrating the operation of changing the arrangement of objects by an electronic device according to an embodiment of the present disclosure;

[0107] Figure 20aThis is a view illustrating the operation of changing the arrangement of objects by an electronic device according to an embodiment of the present disclosure;

[0108] Figure 20b This is a view illustrating the operation of changing the arrangement of objects by an electronic device according to an embodiment of the present disclosure;

[0109] Figure 20c This is a view illustrating the operation of changing the arrangement of objects by an electronic device according to an embodiment of the present disclosure;

[0110] Figure 21a This is a view illustrating the operation of arranging objects on a lock screen or always-on display (AOD) screen by an electronic device according to embodiments of the present disclosure;

[0111] Figure 21b This is a view illustrating the operation of arranging objects on a lock screen or AOD screen by an electronic device according to an embodiment of the present disclosure;

[0112] Figure 22 This is a view of a virtual space displayed by an electronic device according to an embodiment of the present disclosure, based on rotation;

[0113] Figure 23a This is a view illustrating the operation of an electronic device displaying a keyword-based image as a 3D virtual image on an external electronic device, according to an embodiment of the present disclosure; and

[0114] Figure 23b This is a view illustrating the operation of an electronic device applying an image, which is changed based on keywords while displaying a 3D virtual image, to an external electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0115] The following description with reference to the accompanying drawings is intended to aid in a full understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. It includes various specific details to aid understanding, but these are considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Additionally, for clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0116] The terms and words used in the following description and claims are not limited to their literal meaning, but are used by the inventors only to enable a clear and consistent understanding of this disclosure. Therefore, those skilled in the art should understand that the following description of various embodiments of this disclosure is for illustrative purposes only and is not intended to limit the scope of this disclosure as defined by the appended claims and their equivalents.

[0117] It should be understood that, unless the context clearly specifies otherwise, the singular forms “a,” “an,” and “the” include plural indicators. Thus, for example, a reference to “component surface” includes a reference to one or more such surfaces.

[0118] It should be understood that the boxes in each flowchart and the combination of flowcharts can be executed by one or more computer programs that include computer-executable instructions. The entirety of one or more computer programs can be stored in a single memory device, or one or more computer programs can be divided into different parts stored in multiple different memory devices.

[0119] Any function or operation described herein can be processed by a processor or a combination of processors. A processor or combination of processors is circuitry that performs processing and includes, for example, an application processor (AP, such as a central processing unit (CPU)), a communication processor (CP, such as a modem), a graphics processing unit (GPU), a neural processing unit (NPU) (e.g., an artificial intelligence (AI) chip), a Wi-Fi chip, Bluetooth, etc. TM Circuits including chips, GPS chips, NFC chips, connectivity chips, sensor controllers, touch controllers, fingerprint sensor controllers, display driver integrated circuits (ICs), audio codec chips, Universal Serial Bus (USB) controllers, camera controllers, image processing ICs, microprocessor units (MPUs), system-on-a-chip (SoCs), and other ICs.

[0120] Figure 1a This is a block diagram illustrating an electronic device in a network environment according to an embodiment of the present disclosure.

[0121] Referring to FIG1, electronic device 101 in network environment 100 can communicate with electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or communicate with at least one of electronic device 104 or server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, electronic device 101 can communicate with electronic device 104 via server 108. According to an embodiment, electronic device 101 may include a processor 120, a memory 130, an input module 150, a sound output module 155, a display module 160, an audio module 170, a sensor module 176, an interface 177, a connection terminal 178, a haptic module 179, a camera module 180, a power management module 188, a battery 189, a communication module 190, a subscriber identification module (SIM) 196, or an antenna module 197. In some embodiments, at least one of the above components (e.g., connection terminal 178) may be omitted from electronic device 101, or one or more other components may be added to electronic device 101. In some embodiments, some of the components described above (e.g., sensor module 176, camera module 180, or antenna module 197) may be implemented as a single integrated component (e.g., display module 160).

[0122] Processor 120 may run software (e.g., program 140) to control at least one other component (e.g., hardware or software component) of electronic device 101 connected to processor 120, and may perform various data processing or calculations. According to one embodiment, as at least part of the data processing or calculation, processor 120 may store commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132, process the commands or data stored in volatile memory 132, and store the resulting data in non-volatile memory 134. According to embodiments, processor 120 may include a main processor 121 (e.g., central processing unit (CPU) or application processor (AP)) or an auxiliary processor 123 (e.g., graphics processing unit (GPU), neural processing unit (NPU), image signal processor (ISP), sensor central processor, or communication processor (CP)) that is operationally independent of or combined with the main processor 121. For example, when electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or to be dedicated to a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121, or may be implemented as part of the main processor 121.

[0123] When the main processor 121 is inactive (e.g., in sleep) state, the auxiliary processor 123 (rather than the main processor 121) can control at least some of the functions or states associated with at least one component of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190), or when the main processor 121 is active (e.g., running an application), the auxiliary processor 123 can work with the main processor 121 to control at least some of the functions or states associated with at least one component of the electronic device 101 (e.g., display module 160, sensor module 176, or communication module 190). According to embodiments, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 180 or communication module 190) functionally associated with the auxiliary processor 123. According to embodiments, the auxiliary processor 123 (e.g., a neural processing unit) may include hardware architecture dedicated to artificial intelligence model processing. Artificial intelligence models can be generated through machine learning. For example, such learning can be performed via electronic device 101 where artificial intelligence is performed or via a separate server (e.g., server 108). The learning algorithm may include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model may include multiple layers of artificial neural networks. The artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or a deep Q-network, or a combination of two or more thereof, but is not limited thereto. Additionally or optionally, the artificial intelligence model may include software structures in addition to hardware structures.

[0124] Memory 130 may store various data used by at least one component of electronic device 101 (e.g., processor 120 or sensor module 176). The various data may include, for example, software (e.g., program 140) and input or output data for commands associated with it. Memory 130 may include volatile memory 132 or non-volatile memory 134.

[0125] The program 140 may be stored as software in the memory 130, and the program 140 may include, for example, an operating system (OS) 142, middleware 144, or application 146.

[0126] The input module 150 can receive commands or data from outside the electronic device 101 (e.g., a user) that will be used by other components of the electronic device 101 (e.g., processor 120). The input module 150 may include, for example, a microphone, mouse, keyboard, keys (e.g., buttons), or digital pen (e.g., stylus).

[0127] The sound output module 155 can output sound signals to the outside of the electronic device 101. The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as playing multimedia or playing records. The receiver can be used to receive incoming calls. According to an embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0128] Display module 160 can visually provide information to the outside of electronic device 101 (e.g., to a user). Display device 160 may include, for example, a display, a holographic device, or a projector, and control circuitry for controlling a respective one of the display, holographic device, and projector. According to an embodiment, display module 160 may include a touch sensor adapted to detect touch or a pressure sensor adapted to measure the intensity of the force caused by touch.

[0129] The audio module 170 can convert sound into electrical signals and vice versa. According to an embodiment, the audio module 170 can obtain sound via the input module 150, or output sound via the sound output module 155 or headphones of an external electronic device (e.g., electronic device 102) that is directly (e.g., wired) or wirelessly connected to the electronic device 101.

[0130] Sensor module 176 can detect the operating state of electronic device 101 (e.g., power or temperature) or the environmental state outside electronic device 101 (e.g., user state), and then generate an electrical signal or data value corresponding to the detected state. According to embodiments, sensor module 176 may include, for example, a gesture sensor, gyroscope sensor, atmospheric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biometric sensor, temperature sensor, humidity sensor, or illuminance sensor.

[0131] Interface 177 may support one or more specific protocols used to enable electronic device 101 to connect directly (e.g., wired) or wirelessly to external electronic devices (e.g., electronic device 102). According to embodiments, interface 177 may include, for example, a High Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital Card (SD) interface, or an audio interface.

[0132] Connection 178 may include a connector, through which electronic device 101 may be physically connected to an external electronic device (e.g., electronic device 102). According to embodiments, connection 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0133] The haptic module 179 can convert electrical signals into mechanical stimuli (e.g., vibration or motion) or electrical stimuli that can be recognized by a user through his touch or kinesthesia. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0134] Camera module 180 can capture still or moving images. According to an embodiment, camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0135] The power management module 188 manages the power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0136] Battery 189 can power at least one component of electronic device 101. According to an embodiment, battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable rechargeable battery, or a fuel cell.

[0137] Communication module 190 can support the establishment of a direct (e.g., wired) or wireless communication channel between electronic device 101 and external electronic devices (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. Communication module 190 may include one or more communication processors capable of operating independently of processor 120 (e.g., application processor (AP)) and support direct (e.g., wired) or wireless communication. According to embodiments, communication module 190 may include wireless communication module 192 (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module) or wired communication module 194 (e.g., local area network (LAN) communication module or power line communication (PLC) module). One of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth, Wi-Fi Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, 5G network, next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip) or as multiple components separate from each other (e.g., multiple chips). The wireless communication module 192 can identify and verify the electronic device 101 in the communication network (such as the first network 198 or the second network 199) using user information (e.g., the International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196.

[0138] Wireless communication module 192 can support 5G networks following 4G networks and next-generation communication technologies (such as new radio (NR) access technologies). NR access technologies can support enhanced mobile broadband (eMBB), massive machine-type communication (mMTC), or ultra-reliable low-latency communication (URLLC). Wireless communication module 192 can support high-frequency bands (e.g., millimeter-wave bands) to achieve, for example, high data transmission rates. Wireless communication module 192 can support various technologies used to ensure performance in high-frequency bands, such as, for example, beamforming, massive MIMO, full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. Wireless communication module 192 can support various requirements specified in electronic device 101, external electronic devices (e.g., electronic device 104), or network systems (e.g., second network 199). According to an embodiment, the wireless communication module 192 may support peak data rates (e.g., 20 Gbps or greater) for implementing eMBB, lost coverage (e.g., 164 dB or less) for implementing mMTC, or U-plane latency (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less round trip) for implementing URLLC.

[0139] Antenna module 197 can transmit or receive signals or power to or from the exterior of electronic device 101 (e.g., external electronic device). According to an embodiment, antenna module 197 may include an antenna comprising a radiating element formed of a conductive material or conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, antenna module 197 may include multiple antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as a first network 198 or a second network 199) can be selected from the multiple antennas by, for example, communication module 190 (e.g., wireless communication module 192). Signals or power can then be transmitted or received between communication module 190 and the external electronic device via the selected at least one antenna. According to an embodiment, additional components besides the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may be additionally incorporated into antenna module 197.

[0140] According to various embodiments, antenna module 197 may form a millimeter-wave antenna module. According to embodiments, the millimeter-wave antenna module may include a printed circuit board, a radio frequency integrated circuit (RFIC), and multiple antennas (e.g., an array antenna), wherein the RFIC is disposed on or adjacent to a first surface (e.g., a bottom surface) of the printed circuit board and is capable of supporting a specified high-frequency band (e.g., a millimeter-wave band), and the multiple antennas are disposed on or adjacent to a second surface (e.g., a top surface or a side surface) of the printed circuit board and are capable of transmitting or receiving signals in the specified high-frequency band.

[0141] At least some of the aforementioned components can be interconnected and communicate signals (e.g., commands or data) between them via an inter-peripheral communication scheme (e.g., bus, general purpose input / output (GPIO), serial peripheral interface (SPI), or mobile industrial processor interface (MIPI)).

[0142] According to an embodiment, commands or data can be sent or received between electronic device 101 and external electronic device 104 via server 108 connected to a second network 199. Each of electronic device 102 or electronic device 104 can be a device of the same type as electronic device 101, or a device of a different type. According to an embodiment, all or some operations that would be performed on electronic device 101 can be performed on one or more of external electronic devices 102, external electronic devices 104, or server 108. For example, if electronic device 101 is required to automatically perform a function or service, or is required to perform a function or service in response to a request from a user or another device, electronic device 101 may request the one or more external electronic devices to perform at least a portion of the function or service, instead of running the function or service, or electronic device 101 may request the one or more external electronic devices to perform at least a portion of the function or service in addition to running the function or service. Upon receiving the request, one or more external electronic devices may perform at least a portion of the requested function or service, or perform additional functions or services related to the request, and transmit the result of the execution to electronic device 101. Electronic device 101 may provide the result as at least a partial response to the request, with or without further processing of the result. For this purpose, technologies such as cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing may be used. Electronic device 101 may use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, external electronic device 104 may include an Internet of Things (IoT) device. Server 108 may be an intelligent server using machine learning and / or neural networks. According to embodiments, external electronic device 104 or server 108 may be included in a second network 199. Electronic device 101 may be applied to intelligent services based on 5G communication technology or IoT-related technologies (e.g., smart homes, smart cities, smart cars, or healthcare).

[0143] Figure 1b This is a block diagram illustrating the operation of obtaining results from the generating AI of an electronic device according to an embodiment of the present disclosure.

[0144] Reference Figure 1b Electronic device 101 may include processor 120 (e.g., Figure 1a and Figure 1b The processor 120) and memory 130 (e.g., Figure 1a and Figure 1b (Memory 130).

[0145] According to an embodiment, processor 120 may use generated AI 131 stored in memory 130 or generate AI server 108 as an external electronic device (e.g., Figure 1a and Figure 1b The server 108 is used to obtain the generated results 16 and 18. According to an embodiment, the generated AI server 108 may be a generated AI stored in an external electronic device that serves as a terminal device.

[0146] According to an embodiment, the processor 120 can transmit a prompt 15 for generating an extended image to a generation AI 131 stored in memory 130, and receive a generation result 16 corresponding to the prompt 15 from the generation AI 131. For example, the generation result 16 may be an extended image obtained based on a 2D image through external drawing and / or internal drawing techniques.

[0147] According to an embodiment, the processor 120 can transmit a prompt 17 for generating an extended image to a generation AI server 108, which is an external electronic device, and can receive a generation result 18 corresponding to the prompt 17 from the generation AI server 108. For example, the generation result 18 may be an extended image obtained based on a 2D image through external drawing and / or internal drawing techniques.

[0148] According to the embodiments, refer to Figure 5 , Figure 6a , Figure 6b , Figure 7a , Figure 7b , Figure 8a , Figure 8b and Figure 9 Describe the operation for generating an extended image.

[0149] Figure 2 This is a perspective view showing the internal configuration of a wearable electronic device according to an embodiment of the present disclosure.

[0150] refer to Figure 2 The wearable electronic device 200 according to the embodiment may include at least one of a light output module 211, a display component 201, and a camera module 250.

[0151] According to an embodiment, the light output module 211 may include a light source capable of outputting an image and a lens for guiding the image to the display member 201. According to an embodiment, the light output module 211 may include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), liquid crystal on silicon (LCoS), an organic light-emitting diode (OLED), or a micro light-emitting diode (micro LED).

[0152] According to an embodiment, the display component 201 may include an optical waveguide (e.g., a waveguide). According to an embodiment, an image output from the light output module 211, incident on one end of the optical waveguide, can propagate within the optical waveguide and be provided to the user. According to an embodiment, the optical waveguide may include at least one of a diffractive element (e.g., a diffractive optical element (DOE) or a holographic optical element (HOE)) or a reflective element (e.g., a mirror). For example, the optical waveguide may use at least one diffractive element or a reflective element to guide the image output from the light output module 211 to the user's eye.

[0153] According to an embodiment, camera module 250 can capture still images and / or video. According to an embodiment, camera module 250 can be disposed in a lens frame and can be disposed around display member 201.

[0154] According to an embodiment, the first camera module 251 can capture and / or identify the trajectory of a user's eyes (e.g., pupil or iris) or gaze. According to an embodiment, the first camera module 251 can periodically or non-periodically transmit data to a processor (e.g., ...). Figure 1a and Figure 1b The processor 120 sends information (e.g., trajectory information) related to the trajectory of the user's eye or gaze.

[0155] According to an embodiment, the second camera module 253 can capture external images.

[0156] According to an embodiment, the third camera module 255 can be used for hand detection and tracking, as well as recognizing user gestures (e.g., hand movements). According to an embodiment, the third camera module 255 can be used for 3DoF or 6DoF head tracking, position (space, environment) recognition, and / or motion recognition. According to an embodiment, the second camera module 253 can also be used for hand detection, tracking, and recognizing user gestures. According to an embodiment, at least one of the first camera modules 251 to the third camera module 255 can be replaced by a sensor module (e.g., a LiDAR sensor). For example, the sensor module may include at least one of a vertical-cavity surface-emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0157] Figure 3a This is a front perspective view showing a wearable electronic device according to an embodiment of the present disclosure.

[0158] Figure 3b This is a rear perspective view showing a wearable electronic device according to an embodiment of the present disclosure.

[0159] Reference Figure 3a and Figure 3bIn an embodiment, camera modules 311, 312, 313, 314, 315 and 316 and / or depth sensor 317 for obtaining information related to the surrounding environment of the wearable electronic device 300 may be disposed on the first surface 310 of the housing.

[0160] In an embodiment, camera modules 311 and 312 can acquire images related to the surrounding environment of the wearable electronic device.

[0161] In embodiments, camera modules 313, 314, 315, and 316 can acquire images when a user wears a wearable electronic device. Camera modules 313, 314, 315, and 316 can be used for hand detection, tracking, and recognition of user gestures (e.g., hand movements). Camera modules 313, 314, 315, and 316 can be used for 3DoF or 6DoF head tracking, position (spatial or environmental) recognition, and / or motion recognition. In embodiments, camera modules 311 and 312 can be used for hand detection and for tracking and recognizing user gestures.

[0162] In an embodiment, depth sensor 317 may be configured to send and receive signals reflected from an object and to identify the distance to the object, such as time-of-flight (TOF). As an alternative to or supplement to depth sensor 217, camera modules 213, 214, 215, and 216 may identify the distance to the object.

[0163] According to an embodiment, camera modules 325 and 326 and / or display 321 (and / or lens) for facial recognition may be arranged on the second surface 320 of the housing.

[0164] In an embodiment, facial recognition camera modules 325 and 326 adjacent to the display can be used to recognize the user's face, or to recognize and / or track the user's eyes.

[0165] In one embodiment, the display 321 (and / or lens) may be disposed on the second surface 320 of the wearable electronic device 300. In another embodiment, the wearable electronic device 300 may not include camera modules 315 and 316 among the plurality of camera modules 313, 314, 315, and 316. Although not explicitly stated... Figure 3a and Figure 3b As shown, however, the wearable electronic device 300 may also include Figure 2 At least one of the components shown.

[0166] As described above, according to the embodiments, the wearable electronic device 300 may have a shape factor intended to be worn on a user's head. The wearable electronic device 300 may also include straps and / or wearable components to be secured to a part of the user's body. The wearable electronic device 300 may provide a user experience based on augmented reality, virtual reality, and / or mixed reality when worn on a user's head.

[0167] Figure 4 This is another perspective view illustrating an electronic device according to an embodiment of the present disclosure.

[0168] refer to Figure 4 The electronic device 400 may be a head-mounted device (HMD) capable of providing an image in front of the user's eyes. Figure 4 The configuration of the electronic device 400 can be wholly or partially integrated with... Figure 2 The configuration of the electronic device 200 is the same.

[0169] According to an embodiment, the electronic device 400 may form the exterior of the electronic device 400 and may include housings 410, 420 and 430, which may provide space in which components of the electronic device 400 may be arranged.

[0170] According to an embodiment, the electronic device 400 may include a first housing 410 that may surround at least a portion of a user's head. According to an embodiment, the first housing 410 may include a first surface 400a facing the exterior (e.g., in the +Z direction) of the electronic device 400.

[0171] According to an embodiment, the first housing 410 may surround at least a portion of the internal space I. For example, the first housing 410 may include a second surface 400b facing the internal space I of the electronic device 400 and a third surface 400c opposite to the second surface 400b. According to an embodiment, the first housing 410 may be coupled to a third housing 430 and may be formed in a closed-loop shape surrounding the internal space I.

[0172] According to an embodiment, the first housing 410 may surround at least some components of the electronic device 400. For example, an optical output module and a circuit board may be arranged within the first housing 410.

[0173] According to an embodiment, the electronic device 400 may include a display member 440 corresponding to the left and right eyes. The display member 440 may be disposed in the first housing 410. Figure 5 The configuration of the display component 440 can be wholly or partially integrated with... Figure 2 The configuration of the display component 201 is the same.

[0174] According to an embodiment, the electronic device 400 may include a second housing 420 that can be placed on a user's face. According to an embodiment, the second housing 420 may include a fourth surface 400d that can at least partially face the user's face. According to an embodiment, the fourth surface 400d may be a surface in a direction toward the internal space I of the electronic device 400 (e.g., the -Z direction). According to an embodiment, the second housing 420 may be coupled to a first housing 410.

[0175] According to an embodiment, the electronic device 400 may include a third housing 430, which may be positioned behind the user's head. According to an embodiment, the third housing 430 may be coupled to the first housing 410. According to an embodiment, the third housing 430 may surround at least some components of the electronic device 400. For example, a battery (e.g., Figure 1a and Figure 1b The battery 189 can be arranged in the third housing 430.

[0176] Figure 5 This is a flowchart illustrating the operation of an electronic device according to an embodiment of the present disclosure displaying an extended image generated based on a 2D image in a 3D background area.

[0177] In the following embodiments, operations can be performed sequentially, but they can also be performed out of sequence. For example, the order of operations can be changed, and at least two operations can be performed in parallel.

[0178] refer to Figure 5 In operation 510, electronic devices (e.g., Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can obtain 2D images.

[0179] According to embodiments, the electronic device may include a display. For example, the electronic device may include a smartphone, tablet PC, smart glasses, smart contact lenses, or a head-mounted display device.

[0180] According to an embodiment, the electronic device can receive 2D images from an external electronic device. For example, the electronic device can receive 2D images from an external electronic device associated with a user account of the electronic device.

[0181] According to embodiments, the electronic device can access the memory stored in the electronic device (e.g., Figure 1a and Figure 1b 2D images stored in memory 130. For example, 2D images can be captured by a camera (e.g., included in an electronic device) in the memory 130. Figure 1a and Figure 1b The camera module 180 captures the image, either by screen capture or by downloading it to storage in memory.

[0182] According to an embodiment, in operation 520, the electronic device can generate an extended image based on the properties of a 2D image, the extended image further including an image portion generated in the outward direction of the 2D image such that the image is continuous at a boundary corresponding to at least one edge of the 2D image.

[0183] According to an embodiment, an electronic device can use an artificial intelligence model to obtain an image expanded into output data using a 2D image as input data. For example, the AI ​​model could be a generative AI model. According to an embodiment, the artificial intelligence model can be stored in the memory of the electronic device (e.g., ...). Figure 1a and Figure 1b The memory 130 may be stored in the external electronic device (e.g., server). Figure 1a and Figure 1b In server 108).

[0184] According to an embodiment, an electronic device can obtain an extended image based on a 2D image using out-of-image rendering technology. Out-of-image rendering technology is a technique that uses an artificial intelligence model to generate an external region that extends beyond the edges of an image.

[0185] According to an embodiment, the electronic device can obtain an extended image using 2D images based on field-of-view information.

[0186] According to an embodiment, field of view information may include the user's gaze direction and / or field of view (FOV). According to an embodiment, this can be achieved through sensors in an electronic device (e.g., Figure 1a and Figure 1b The sensor module 176 obtains field-of-view information. For example, the sensor may include a camera for detecting the movement of the user's eyes and / or a motion sensor for detecting the movement of the user's head.

[0187] For example, the electronic device can set the size of the drawing area (e.g., an image portion) to be larger than the user's field of view (FOV). According to an embodiment, the electronic device can generate an image portion corresponding to a determined size based on at least one of the size or orientation of the field of view corresponding to the field of view information.

[0188] According to an embodiment, the electronic device can extend in at least two directions along the edges of the 2D image that serves as the original image.

[0189] According to an embodiment, the electronic device can generate an extended image such that the generated image portion includes a region extending from a first boundary of the 2D image along a first direction and a region extending from a second boundary different from the first boundary along a second direction different from the first direction. For example, the electronic device can generate an extended image to include an image portion extending to the left from the left boundary of the 2D image and an image portion extending to the right from the right boundary of the 2D image.

[0190] According to an embodiment, as part of generating the extended image, the electronic device can generate a region corresponding to a first range belonging to the main movable range of the field of view by performing rendering based on a 2D image, at least based on field of view information. The main movable range of the field of view may include a range that can change based on movement of the user's eyes or head. For example, the main movable range of the field of view may be a range of 240 degrees in total, including 120 degrees to the left and 120 degrees to the right relative to the user's forward field of view. This is merely an example, and the present disclosure is not limited thereto.

[0191] According to an embodiment, the electronic device can generate an image corresponding to a second range, different from the first range, by performing in-drawing based on an image generated in the region corresponding to the first range, at least based on field-of-view information. In-drawing technology is a technique used to fill in damaged or blank portions of an image or to remove objects included in an image using an artificial intelligence model. According to an embodiment, the electronic device can generate an image of the region corresponding to the second range based on images of two opposite ends of the region corresponding to the first range. According to an embodiment, the extended image may include a 360-degree image. According to an embodiment, the 360-degree image may exclude ceiling and floor images.

[0192] According to the embodiments, refer to Figure 6a , 6b 7a and 7b describe operations for obtaining an extended image by performing outward and inward drawing based on field of view information.

[0193] According to an embodiment, the electronic device can generate an image portion by connecting two opposite boundaries of an image generated from outward drawing without performing inward drawing. According to an embodiment, the extended image may include a 360-degree image. According to an embodiment, the 360-degree image may exclude ceiling and floor images.

[0194] For example, an electronic device can connect the left and right boundaries of an image portion generated by outlining using bilateral symmetry. For example, as part of generating an extended image, the electronic device can generate an image portion that is symmetrical with respect to at least one of virtual lines or boundary lines at one or more points of the image portion.

[0195] According to the embodiments, refer to Figure 7a and Figure 7bDescribes the operation of connecting the boundaries of an image generated by bilateral symmetry.

[0196] According to an embodiment, the electronic device can adjust the left and right boundaries of an image portion generated by outlining using fade-in / fade-out processing. For example, as part of generating an extended image, the electronic device can generate the image portion by performing fade-in / fade-out processing on at least one of virtual lines or boundary lines at one or more points of the image portion. The fade-in / fade-out processing can be performed by adjusting their positions and overlapping them, or by repeatedly adjusting their positions and overlapping them, to process the images of two opposing boundary regions into a natural connection.

[0197] According to an embodiment, the electronic device can adjust the left and right boundaries of the image portion generated by the outlining on both sides symmetrically, and then fade in / fade out of the connecting area.

[0198] According to the embodiments, refer to Figure 8a and Figure 8b Describes the operation of connecting the boundaries of an image generated through fade-in / fade-out processing.

[0199] According to an embodiment, the electronic device can store a virtual space including a 3D background area. According to an embodiment, the virtual space may include a sphere or a hemisphere. According to an embodiment, the 3D background area may be included in a boundary surface indicating the maximum size of the virtual space. For example, the 3D background area may be included in the boundary surface of a spherical or hemispherical virtual space.

[0200] According to an embodiment, at least a portion of the 3D background region can be configured as a curved surface shape, which has a substantially equal distance from points set in the virtual space. For example, the set of points in the virtual space can be the center point of the virtual space. The center point of the virtual space can be the user's position in the virtual space.

[0201] According to an embodiment, the electronic device can correct an expanded image to correspond to a curved surface shape. For example, the electronic device can correct an expanded image to correspond to a 3D background area. According to an embodiment, the electronic device can correct an expanded image to correspond to a shape in a virtual space.

[0202] According to the embodiments, although the generation of extended images by electronic devices has been described above, electronic devices can generate extended images or can receive extended images generated by external electronic devices.

[0203] For example, electronic devices can communicate via a communication module (e.g., Figure 1a and Figure 1bThe communication module 190 sends the 2D image to a second external electronic device (e.g., a server). According to an embodiment, the 2D image can be received from a first external electronic device (e.g., an electronic device), acquired through the camera of the electronic device, or stored in a memory.

[0204] According to an embodiment, the electronic device can receive an extended image from a second external electronic device via a communication module in response to transmission. The extended image may be an extended image derived from a 2D image.

[0205] According to an embodiment, the extended image received from the external electronic device can be generated by the second external electronic device through at least some of the extended image generation operations of the aforementioned electronic device.

[0206] According to an embodiment, as part of generating the extended image, the electronic device can transmit a generation command prompt for generating the extended image to an external electronic device (e.g., a server) via a communication module. According to an embodiment, the electronic device can receive the extended image generated based on the generation command prompt from the external electronic device.

[0207] According to an embodiment, the command prompt can store instructions for generating an extended image, which further includes an image portion generated in the outward direction of the 2D image based on the attributes of the 2D image, such that the image is continuous at a boundary corresponding to at least one edge of the 2D image.

[0208] According to an embodiment, the command prompt can store instructions for generating an image portion having a determined size based on at least one of the size or orientation of the field of view corresponding to the field of view information.

[0209] According to an embodiment, the command prompt can store instructions for generating an extended image, such that the generated image portion includes a region extending from a first boundary of the 2D image along a first direction and a region extending from a second boundary different from the first boundary along a second direction different from the first direction.

[0210] According to an embodiment, the generated command prompt can store instructions for correcting the expanded image to correspond to a curved surface shape. According to an embodiment, the curved surface shape can be in the form of a 3D background area in virtual space.

[0211] According to an embodiment, the command prompt can store instructions as part of generating an extended image, generating a region corresponding to a first range belonging to the main movable range of the field of view by performing outdrawing based on a 2D image, and generating a region corresponding to a second range different from the first range by performing indrawing based at least on information related to the field of view, based on an image generated in the region corresponding to the first range.

[0212] According to an embodiment, the command prompt can store instructions for generating an image portion symmetrical to at least one of virtual lines or boundary lines at one or more points relative to the image portion, as part of generating an extended image.

[0213] According to an embodiment, the command prompt can store instructions as part of generating an extended image, which is generated by fading in / out at least one of virtual lines or boundary lines at one or more points of the image portion.

[0214] According to an embodiment, an electronic device can generate an extended image by loading images of a similar category to the original 2D image from memory. For example, the electronic device can generate an extended image by arranging the 2D image and similar category images in several layouts.

[0215] According to an embodiment, in operation 530, the electronic device may configure the extended image to be included in at least a portion of the 3D background area.

[0216] According to an embodiment, when the extended image does not include the ceiling and floor images, the electronic device can set the extended image to be included in an area of ​​the 3D background area other than the ceiling and floor areas. According to an embodiment, when the extended image includes the ceiling and floor images, the extended image can be set to be included in the 3D background area.

[0217] According to an embodiment, in operation 540, the electronic device can display at least a portion of a virtual space including a set 3D background area based on field-of-view information.

[0218] According to an embodiment, the electronic device can display an extended image at least within the field of view (FOV) based on the user's field of view information. According to an embodiment, the electronic device can display the extended image such that a 2D image, serving as the original image, is positioned in front of the user's field of view based on the user's FOV information.

[0219] According to an embodiment, the electronic device can separate objects from a 2D image, display an expanded image in a 3D background area, and display the separated objects between the user and the 3D background area in virtual space. (See also...) Figure 10 , Figure 11 , Figure 12a , Figure 12b , Figure 12c , Figure 12d , Figure 13a , Figure 13b , Figure 13c , Figure 13d , Figure 14 and Figure 15 Describes the operation of displaying separated objects according to an embodiment.

[0220] Figure 6a This is a flowchart illustrating the operation of an electronic device generating an extended image based on a 2D image according to an embodiment of the present disclosure.

[0221] Figure 6b This is a view illustrating the operation of an electronic device generating an extended image based on a 2D image according to an embodiment of the present disclosure.

[0222] In the following embodiments, operations can be performed sequentially, but they can also be performed out of sequence. For example, the order of operations can be changed, and at least two operations can be performed in parallel.

[0223] Reference Figure 6a In operation 610, electronic devices (e.g., Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can collect original background (wallpaper) images from the database.

[0224] According to an embodiment, the electronic device can receive a 2D image as an original image (or a background original image) from an external electronic device. For example, the electronic device can receive the 2D image through an external electronic device (e.g., a database) associated with the user account of the electronic device.

[0225] According to an embodiment, the electronic device can access the memory stored in the electronic device (e.g., Figure 1a and Figure 1b The original image is stored in the memory 130. For example, a 2D image, which is the original image, can be obtained from a camera (e.g., included in the electronic device). Figure 1a and Figure 1b The camera module 180 captures the image, either by screen capture or by downloading it to storage in memory.

[0226] According to an embodiment, in operation 620, the electronic device can identify whether the resolution of the original image is greater than or equal to a first value. For example, the electronic device can identify whether the resolution of the original image is 4k or higher. When obtaining an extended image through out-drawing and / or in-drawing, if the resolution of the original image is low, an incomplete result is obtained; therefore, the electronic device can identify the resolution of the original image.

[0227] According to an embodiment, when the resolution is less than a first value (No in operation 620), in operation 625, the electronic device can adjust to a high-resolution image (or frame).

[0228] According to an embodiment, an electronic device can obtain a high-resolution image by means of an artificial intelligence model trained to output a high-resolution image from a low-resolution image.

[0229] According to an embodiment, the electronic device can use the Single Frame Super-Resolution (SFSR) method to optimize the storage capacity of virtual space. The SFSR method is used to obtain a single high-resolution image by matching multiple low-resolution images, and machine learning and similarity estimation techniques can be repeatedly performed.

[0230] According to an embodiment, when the original image is video, the electronic device can use the multi-frame super-resolution (MFSR) method. Although motion estimation is accurate and performs well, it is difficult to implement in real time due to motion estimation and repeated estimation operations for multiple images. Therefore, the MFSR method can operate by storing images sequentially and listing frames in time order in a cylindrical space.

[0231] According to an embodiment, when the resolution is equal to or greater than a first value ("Yes" in operation 620), in operation 630, the electronic device can identify whether the horizontal length of the original image is greater than the vertical length.

[0232] According to an embodiment, when the vertical length of the original image is greater than the horizontal length (No in operation 630), in operation 635, the electronic device can generate an image by drawing at a minimum 1:1 ratio.

[0233] According to an embodiment, the electronic device can generate images in two opposite lateral directions from two opposite boundary regions of an original image. According to an embodiment, the electronic device can generate an image in one direction from one lateral boundary region of the original image.

[0234] According to an embodiment, when the horizontal length of the original image is greater than the vertical length (Yes in operation 630), in operation 640, the electronic device can identify whether the drawn portion exceeds 240 degrees.

[0235] According to an embodiment, the electronic device can generate an image by drawing an original image. According to an embodiment, the electronic device can generate an image corresponding to a curved surface shape in virtual space. For example, the curved surface shape may have substantially the same distance between the user and any point on the curved surface shape.

[0236] According to an embodiment, the electronic device can identify whether the angle between the user and a straight line connecting two opposite boundaries of an image generated by drawing exceeds 240 degrees. According to the embodiment, 240 degrees is merely an example of a set range, and this disclosure is not limited thereto.

[0237] According to an embodiment, when the drawn portion does not exceed 240 degrees (No in operation 640), in operation 645, the electronic device can automatically generate an image by inferring from the external drawing scheme.

[0238] According to an embodiment, an electronic device can generate an image by drawing an outer region of a defined area and drawing an inner region of the remaining area based on two relative boundary regions of the image generated by the outer drawing.

[0239] According to the embodiments, such as Figure 6b As shown, the electronic device can perform outdrawing to additionally generate images in the directions on both sides of the original image 611 arranged in the forward gaze direction 11 of the user 10.

[0240] According to an embodiment, the electronic device can maintain outlining when the angle between the user 10 and the straight lines of the two opposite boundaries of the generated image 641 is within a set range. According to an embodiment, the electronic device can perform outlining within a range at least wider than the user 10's field of view (FoV) 12.

[0241] According to an embodiment, the electronic device can perform outlining until the angle between the straight lines connecting the two opposite boundaries of the generated image 641 reaches 240 degrees. For example, the electronic device can perform outlining until the angle between the straight line connecting the left boundary of the generated image 641 and the user 10 and the user 10's forward viewing direction 11 is 120 degrees, and the angle between the straight line connecting the right boundary of the generated image 641 and the user 10 and the user 10's forward viewing direction 11 is 120 degrees.

[0242] return Figure 6a According to an embodiment, when the drawn portion exceeds 240 degrees ("Yes" in operation 640), in operation 650, the electronic device can generate an image of the remaining area by in-drawing. For example, as Figure 6b As shown, after outlining is performed, blank portion 651 can be inferred to fill the image. According to an embodiment, blank portion 651 can be within a range of 120 degrees and can be arranged behind user 10 away from the FOV 12 of user 10.

[0243] According to an embodiment, the electronic device can fill the blank portion 651 by in-line drawing based on the image 641 generated by out-drawing. For example, the electronic device can generate an image of the region corresponding to the blank portion 651 based on images of two opposite ends of the image 641 generated by out-drawing.

[0244] According to an embodiment, the electronic device can generate a 360-degree image by performing external drawing and internal drawing.

[0245] According to an embodiment, the electronic device can generate a spherical 360-degree image. According to an embodiment, the electronic device can generate a spherical 360-degree image including the ceiling and floor by performing external and internal drawing.

[0246] According to one embodiment, the electronic device can generate a hemispherical image. According to another embodiment, the electronic device can generate a 180-degree image including the ceiling and floor by performing outlining.

[0247] return Figure 6a According to an embodiment, in operation 660, the electronic device can render a 360-degree background.

[0248] According to an embodiment, the electronic device can display the generated image in at least a portion of a 3D background area in a virtual space. For example, the electronic device can display the generated image in an area of ​​the 3D background area that includes the user's field of view (FOV).

[0249] According to an embodiment, the electronic device can separate objects from a 2D image, display an expanded image in a 3D background area, and display the separated objects between the user and the 3D background area in virtual space. (See also...) Figure 10 , Figure 11 , Figure 12a , Figure 12b , Figure 12c , Figure 12d , Figure 13a , Figure 13b , Figure 13c , Figure 13d , Figure 14 and Figure 15 Describes the operation of displaying separated objects according to an embodiment.

[0250] Figure 7a This is a flowchart illustrating the operation of an electronic device generating two extended images connected to each other at opposite ends based on a 2D image, according to an embodiment of the present disclosure.

[0251] In the following embodiments, operations can be performed sequentially, but they can also be performed out of sequence. For example, the order of operations can be changed, and at least two operations can be performed in parallel.

[0252] refer to Figure 7a In operation 710, electronic devices (e.g., Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4The electronic device 400 can separate the original image into individual objects.

[0253] According to an embodiment, the electronic device can identify at least one object region included in an original image through image analysis. According to an embodiment, the electronic device can separate the identified at least one object region from the original image. According to an embodiment, the electronic device can divide the original image into a background region and at least one object region.

[0254] According to an embodiment, when the original image does not include objects different from the background area, the electronic device may omit operation 710.

[0255] According to an embodiment, in operation 720, the electronic device can perform in-line drawing / out-line drawing by applying generative AI to the original image.

[0256] According to an embodiment, the electronic device can perform outlining based on an original image to additionally generate an image in at least one direction of the original image. For example, the electronic device can perform outlining based on one side boundary of the original image to additionally generate an image in a first direction. According to an embodiment, the electronic device can additionally generate an image in a left direction based on the left boundary of the original image, and can additionally generate an image in a right direction based on the right boundary.

[0257] According to an embodiment, when the object region is separated from the original image, the electronic device can generate an image by drawing within the blank space of the original image generated based on the separation of the object region.

[0258] According to an embodiment, in operation 730, the electronic device can connect images symmetrically to each other.

[0259] According to an embodiment, the electronic device can symmetrically connect two opposite boundaries of an image generated by outlining. For example, when another image is generated from one side boundary of the original image in a first direction, the other side boundary of the original image and the boundary of the newly generated image can be symmetrically connected to each other.

[0260] According to the embodiments, such as Figure 7b As shown, when an image is generated in the left direction based on the left boundary of the original image and an image is generated in the right direction based on the right boundary, the boundaries of the image generated in the left direction and the boundaries of the image generated in the right direction can be connected symmetrically to each other.

[0261] Figure 7b This is a view illustrating the operation of an electronic device generating two extended images connected to each other at opposite ends based on a 2D image, according to an embodiment of the present disclosure.

[0262] refer to Figure 7bThe electronic device can separate the object 712 from the original image 711. According to an embodiment, the electronic device can generate an external drawing image 721 by generating an additional image in the left direction based on the left boundary of the original image 711 and an additional image in the right direction based on the right boundary.

[0263] According to an embodiment, the electronic device can generate an image 722 (e.g., a horizontally reversed image A) obtained by reversing an external drawing image 721 (e.g., image A). According to an embodiment, the electronic device can connect the external drawing image 721 and the horizontally reversed image 722.

[0264] For example, the electronic device can connect one boundary of the external drawing image 721 and one boundary of the horizontally flipped image 722. The electronic device can also connect a second boundary of the external drawing image 721 and a second boundary of the horizontally flipped image 722.

[0265] According to an embodiment, the first region 731 connecting the second boundary of the outer image 721 and the second boundary of the horizontally reversed image 722 can be symmetrical about each other with respect to the connecting boundary 732. According to an embodiment, the second region 741 connecting the first boundary of the outer image 721 and the first boundary of the horizontally reversed image 722 can be symmetrical about each other with respect to the connecting boundary 742.

[0266] According to an embodiment, Figure 7b The outer drawing image 721 and the left-right reversed image 722 are shown to be linearly connected, but the two opposite boundaries of the outer drawing image 721 and the two opposite boundaries of the left-right reversed image 722 can be connected to each other in the form of a 360-degree image.

[0267] According to an embodiment, the first region 731 and the second region 741 may be approximately symmetrical with respect to the connecting boundaries 732 and 742, and the images of the connecting boundaries 732 and 742 may be unnatural. According to an embodiment, the electronic device may process the connecting boundaries 732 and 742 to make them appear natural.

[0268] return Figure 7a According to an embodiment, in operation 740, the electronic device can adjust the loop image at the point where the left and right ends intersect, such as... Figure 8a Or as shown in 8b.

[0269] Figure 8a This is a view illustrating the operation of an extended image connected by an electronic device according to an embodiment of the present disclosure.

[0270] refer to Figure 8a Electronic devices (e.g.) Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can identify the connection area 810 between the external drawing image and the horizontally reversed image.

[0271] According to an embodiment, the electronic device can identify a portion 811 that is part of the connecting area 810 of the external drawing image and a second portion 812 that is part of the connecting area 810 of the left-right reversed image.

[0272] According to an embodiment, the electronic device can perform a fade-in / fade-out operation on the generated image by overlapping the positions of the first portion 811 and the second portion 812. According to an embodiment, the electronic device can generate an image including a connected region 813 that is naturally processed by the fade-in / fade-out operation.

[0273] According to an embodiment, the electronic device can input images in which side-drawn images and horizontally reversed images are connected to an artificial intelligence model trained to naturally process connection regions 810. According to an embodiment, the electronic device can obtain images including naturally processed connection regions 813 as output data from the artificial intelligence model.

[0274] Figure 8b This is a view illustrating the operation of an extended image connected by an electronic device according to an embodiment of the present disclosure.

[0275] refer to Figure 8b The electronic device can identify the connection area 820 between the external drawing image and the horizontally reversed image.

[0276] According to an embodiment, the electronic device can identify multiple connection portions 821, 822, and 823 of the connection boundary in the connection area 820.

[0277] According to an embodiment, the electronic device can generate an image of each of the plurality of connecting portions 821, 822, and 823 by recognizing the surrounding environment of each portion. For example, the electronic device can generate an image of the first connecting portion 821 by recognizing the peripheral portion of the first connecting portion 821, generate an image of the second connecting portion 822 by recognizing the peripheral portion of the second connecting portion 822, and generate an image of the third connecting portion 823 by recognizing the peripheral portion of the third portion 823.

[0278] According to an embodiment, the electronic device can generate a blank space of a predetermined size between a plurality of connecting portions 821, 822, and 823, and can generate an image in the blank space by drawing inwards based on the outer portions of the plurality of connecting portions in an external drawing image and the outer portions of the plurality of connecting portions in a left-right reversed image. Therefore, the electronic device can generate an image including a connecting region 824 that has been processed into a natural shape.

[0279] According to an embodiment, the electronic device can connect an externally drawn image and a horizontally reversed image to an image input of an artificial intelligence model trained to naturally process the connection region 820. According to an embodiment, the electronic device can obtain an image including the naturally processed connection region 824 as output data from the artificial intelligence model.

[0280] Return to reference Figure 7a According to an embodiment, in operation 750, the electronic device can generate an image after dividing the space into a ceiling and a floor. According to an embodiment, the electronic device can generate an expanded image including both a ceiling image and a floor image.

[0281] According to the embodiments, such as Figure 9 As shown, the electronic device can generate ceiling and floor images based on the extended image.

[0282] Figure 9 This is a view illustrating the operation of an electronic device generating an extended image of a ceiling and floor according to an embodiment of the present disclosure.

[0283] Reference Figure 9 Electronic devices (e.g.) Figure 1a and Figure 1b Electronic device 101 Figure 1a And the processor 120 in Figure 1B Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can identify the ceiling region 911 and the floor region 912 in the image 910 obtained by outlining the original image. According to an embodiment, the image 910 obtained by outlining can be a 360-degree panoramic image in the form of the side surface of a cylinder.

[0284] According to an embodiment, the electronic device can identify the ceiling region 911 and the floor region 912, and can identify the peripheral regions of the ceiling region 911 and the peripheral regions of the floor region 912.

[0285] According to an embodiment, the electronic device can generate an image 930 up to the ceiling vanishing point 920 based on at least a portion of the ceiling region 911. According to an embodiment, the electronic device can generate the image 930 up to the ceiling vanishing point 920 based on the ceiling region 911 and its peripheral region.

[0286] According to an embodiment, the electronic device can generate an image 931 up to the vanishing point 921 of the floor region 912 based on at least a portion of the floor region 912. According to an embodiment, the electronic device can generate the image 931 up to the vanishing point 921 of the floor region 912 based on the floor region 912 and the peripheral region of the floor region 912.

[0287] According to an embodiment, the electronic device can use generative AI to generate an image 930 up to the ceiling vanishing point 920 and / or an image 931 up to the floor vanishing point 921.

[0288] return Figure 7a According to an embodiment, in operation 760, the electronic device can render processed images, videos, and / or audio.

[0289] According to an embodiment, when an extended image is generated, the electronic device can display the extended image in a virtual space. For example, the electronic device can display the extended image in a 3D background area of ​​the virtual space.

[0290] According to an embodiment, when an extended video is generated (e.g., an extended frame is generated for each frame), the electronic device can display the extended video in a 3D background area of ​​virtual space.

[0291] According to an embodiment, the electronic device can output audio corresponding to the extended image or extended video. For example, the audio can be included in the image or video, or it can be assigned to the image or video by the electronic device.

[0292] Figure 10 This is a flowchart illustrating the operation of an electronic device arranging each object included in an extended image at different depths, according to embodiments of the present disclosure.

[0293] In the following embodiments, operations can be performed sequentially, but they can also be performed out of sequence. For example, the order of operations can be changed, and at least two operations can be performed in parallel.

[0294] refer to Figure 10 In operation 1010, electronic devices (e.g., Figure 1a and 1b Electronic device 101 Figure 1a and 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3aWearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can acquire information related to the user's field of view. For example, the field-of-view related information may include positional and dimensional information about the area displayed on the display in the 3D virtual space. Since the area displayed on the display in the 3D virtual space changes according to the movement (e.g., movement and / or rotation) of the electronic device, the field-of-view related information can be modified.

[0295] According to an embodiment, sensors in electronic devices (e.g., Figure 1a and Figure 1b The sensor module 176 obtains field-of-view information. For example, the sensor may include a sensor for detecting movement of the electronic device and / or movement of the user's head. For example, the sensor may include a motion sensor for detecting movement of the electronic device and / or movement of the head of a user wearing the electronic device, and may include a camera for capturing the external environment. According to an embodiment, the movement of the electronic device can be identified by the movement of feature points in an image captured via the camera.

[0296] According to an embodiment, in operation 1020, the electronic device can acquire a 2D image. According to an embodiment, the electronic device can acquire the 2D image based on user input. For example, the electronic device can receive user input for selecting a background image. According to an embodiment, the electronic device can receive user input for selecting a background image for an external electronic device linked to a user account. According to an embodiment, the electronic device can receive user input for selecting an image to use as a background from images displayed on the execution screen of a gallery application (e.g., a gallery).

[0297] According to an embodiment, the electronic device can collect original background (wallpaper) images from a database. According to an embodiment, the electronic device can receive an original background image corresponding to a selected background image from the database.

[0298] According to an embodiment, the electronic device can receive a 2D image as an original image (or a background original image) from an external electronic device. For example, the electronic device can receive the 2D image through an external electronic device (e.g., a database) associated with the user account of the electronic device.

[0299] According to an embodiment, the electronic device can access the memory stored in the electronic device (e.g., Figure 1a and Figure 1b The original image is stored in the memory 130. For example, a 2D image, which is the original image, can be obtained from a camera (e.g., included in the electronic device). Figure 1a and Figure 1bThe camera module 180 captures the image, either by screen capture or by downloading it to storage in memory.

[0300] According to an embodiment, in operation 1030, the electronic device can identify a first portion of a 2D image to be applied as a background image to the 3D virtual space. For example, applying the background image to the 3D virtual space can display it in a background area of ​​the 3D virtual space.

[0301] According to an embodiment, in operation 1040, the electronic device can identify a second portion of the 2D image corresponding to the object of interest. According to the embodiment, operations 1030 and 1040 are shown to be performed sequentially, but the order can be changed, and at least some of operations 1030 and 1040 can be performed simultaneously.

[0302] According to an embodiment, the electronic device can identify whether a depth-appropriate object of interest exists in a 2D image. For example, the electronic device can identify whether objects and / or text, other than the background, exist in a 2D image.

[0303] According to an embodiment, when a depth-appropriate object of interest exists in a 2D image, the electronic device can identify the entire 2D image as the first part to be applied as a background image.

[0304] According to an embodiment, when a depth-appropriate object of interest exists in a 2D image, the electronic device can separate the 2D image. For example, the electronic device can identify a first portion of the 2D image to be applied as a background image in a 3D virtual space and a second portion of the 2D image corresponding to the object of interest.

[0305] According to an embodiment, an electronic device can generate an image by drawing blank space within an image or a 360-degree background, the blank space being generated by separating a second portion corresponding to an object of interest included in a 2D image.

[0306] According to an embodiment, in operation 1050, the electronic device can generate an extended image corresponding to a first portion of the 2D image.

[0307] According to an embodiment, an electronic device can generate an extended image based on the properties of a first portion of a 2D image to be applied as a background image, the extended image further including an image portion generated outside the first portion to be continuous at the boundary of the first portion.

[0308] According to an embodiment, the electronic device can be used as Figure 5 , Figure 6a , Figure 6b , Figure 7a , Figure 7b , Figure 8a , Figure 8b and Figure 9At least one of the methods shown is used to generate an extended image.

[0309] According to an embodiment, the electronic device can prioritize generating a background that requires a large area.

[0310] According to an embodiment, in operation 1060, the electronic device can generate a 3D virtual image by applying a first depth to an extended image and a second depth to a second portion of a 2D image. For example, the electronic device can arrange objects in a virtual space by reflecting depth values. According to an embodiment, the virtual space may include a VR home screen capable of executing VR applications.

[0311] According to an embodiment, the electronic device can arrange an object region at a second depth, i.e., a first depth, between a point set in virtual space relative to the field of view information and a 3D background region. For example, the set of points in the virtual space may include the user's location, and the electronic device can arrange the object between the user and the 3D background region.

[0312] According to an embodiment, as part of the arrangement, the electronic device can determine the attributes of objects included in the object area. According to an embodiment, the electronic device can arrange objects capable of interacting with the user at a first point designated for user interaction, based at least partially on the object's attributes. According to an embodiment, the electronic device can arrange objects that only provide information and do not interact with the user at a second point designated for non-user interaction, based at least partially on the object's attributes.

[0313] According to an embodiment, the electronic device can assign user interactions to images based on the characteristics of the virtual space.

[0314] According to an embodiment, the electronic device can interact with each of the 360-degree background and separate objects.

[0315] According to an embodiment, the electronic device can add motion graphics effects to an object based at least in part on the object's attributes. For example, when the object is a moving natural object (e.g., a person, animal, cloud, or fire) or a moving object (e.g., a ball or a car), the electronic device can add motion graphics effects to the object.

[0316] According to an embodiment, the electronic device can obtain environmental information. For example, the environmental information may include at least one of time information, weather information, seasonal information, and the location of the electronic device.

[0317] According to an embodiment, the electronic device can determine the attributes of a graphic effect based at least on environmental information. For example, when the environmental information includes weather information, the electronic device can determine the attributes of the graphic effect differently depending on the weather information. For example, when the weather is currently windy, the electronic device can determine an attribute that causes the clouds, as the object, to move rapidly. According to an embodiment, when the weather is currently sunny, the electronic device can determine that the background brightness is bright.

[0318] According to an embodiment, the electronic device can identify contact information associated with an object area (e.g., a person). According to an embodiment, the electronic device can provide contact information in response to receiving user input about the object area.

[0319] According to an embodiment, the electronic device can receive events related to contact information. For example, when an event such as "call received" is received, the electronic device can add or change event-related graphical effects to the target area.

[0320] According to an embodiment, when no object exists that can be interacted with by the user, the interaction assignment operation can be omitted. According to an embodiment, in operation 1070, the electronic device can display a portion of a 3D virtual image based on information related to the user's field of view. According to an embodiment, the electronic device can render processed images (e.g., extended images, object images), video, and / or audio.

[0321] According to embodiments, an electronic device can display a portion of a 3D virtual image obtained by applying a first depth to an extended image and a second depth to a second portion of a 2D image based on information related to the field of view. For example, the electronic device can display at least a portion of a virtual space comprising a set 3D background area and arranged object areas. For example, the electronic device can display a background image in the 3D background area of ​​a virtual space within the user's field of view. According to embodiments, the electronic device can display object images in the space between the user and the 3D background area within the virtual space of the user's field of view.

[0322] According to an embodiment, the electronic device can display an image in the area viewed by the user (or camera) by adjusting the user's (or camera's) gaze, position, movement, and lens value.

[0323] According to an embodiment, when an extended video is generated (e.g., an extended frame is generated for each frame), the electronic device can display the extended video in a 3D background area of ​​a virtual space, and can display an object image in the space between the user and the 3D background area.

[0324] According to an embodiment, the electronic device can output audio corresponding to the extended image or extended video. For example, the audio can be included in the image or video, or it can be assigned to the image or video by the electronic device.

[0325] Figure 11 This is a flowchart illustrating the operation of an electronic device arranging each object included in an extended image at different depths, according to embodiments of the present disclosure.

[0326] In the following embodiments, operations can be performed sequentially, but they can also be performed out of sequence. For example, the order of operations can be changed, and at least two operations can be performed in parallel.

[0327] refer to Figure 11 In operation 1101, electronic devices (e.g., Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can identify whether there are depth-applicable objects in the original image.

[0328] According to an embodiment, the electronic device can identify whether a depth-adapted object exists in the original image. For example, the electronic device can identify whether there are objects and / or text in the original image other than the background.

[0329] According to an embodiment, when there is no depth-applicable object in the original image (No in operation 1101), the electronic device can generate a 360-degree background in operation 1102.

[0330] According to an embodiment, the electronic device can generate an extended image based on attributes of a background region. This extended image also includes image portions generated outside the background region to ensure continuity at the boundaries of the background region. For example, the extended image may include a 360-degree panoramic image, a 360-degree spherical image, or a hemispherical image.

[0331] According to an embodiment, the electronic device can be used as Figure 5 , Figure 6a , Figure 6b , Figure 7a , Figure 7b , Figure 8a , Figure 8b and Figure 9 At least one of the methods shown is used to generate an extended image.

[0332] According to an embodiment, in operation 1103, the electronic device can render a background image. According to an embodiment, the electronic device can render a 360-degree background, generated independently of the region of the virtual space, onto the background region of the virtual space.

[0333] According to an embodiment, when a depth-applicable object exists in the original image (Yes in operation 1101), in operation 1104, the electronic device can identify the background and the object as separate objects. According to an embodiment, the electronic device can separate the background region and the object region of the original image.

[0334] According to an embodiment, the electronic device can generate an image by performing in-line drawing on an image generated based on an image separated from an object image or on blank space in a 360-degree background.

[0335] According to an embodiment, in operation 1105, the electronic device can detect a virtual space region. For example, the electronic device can detect the virtual space region between a set point (e.g., the user's location) and a background region.

[0336] According to an embodiment, in operation 1106, the electronic device can identify whether two or more objects exist.

[0337] According to an embodiment, when there are fewer than two objects (no in operation 1106), in operation 1107, the electronic device can divide the virtual space region. For example, when there is one object, the electronic device can divide the virtual space region into region A and region A'.

[0338] According to an embodiment, region A in the virtual space may be a region where no objects are placed. According to an embodiment, region A' in the virtual space may be a region where objects are placed. According to an embodiment, region A in the virtual space is a space related to the user's minimum movement and may be a space within a set distance from the user. According to an embodiment, region A in the virtual space may be the remaining region in the virtual space other than region A, and may be a space at a set distance from the user to the background region. According to an embodiment, refer to... Figure 12b Describe the operation of dividing virtual space.

[0339] According to an embodiment, in operation 1108, the electronic device can render an object image in region A'. According to an embodiment, the electronic device can arrange an object in region A', which is at a set distance or further from the user and closer to the background region.

[0340] According to an embodiment, when two or more objects exist (Yes in operation 1106), in operation 1109, the electronic device can divide the virtual space region into region A and region A', and can further divide region A'. For example, the electronic device can divide region A' into region A1 and region A2.

[0341] According to an embodiment, region A1 may be a region in region A' where objects capable of interacting with the user are arranged. According to an embodiment, region A2 may be a region in region A' where objects that cannot be interacted with the user are arranged. According to an embodiment, region A1 and region A2 may be divided based on their distance from the user. For example, region A1 may be a region close to the user in region A', and region A2 may be a region far from the user in region A'. According to an embodiment, refer to... Figure 12b Describe the operation of dividing virtual space.

[0342] According to an embodiment, in operation 1110, the electronic device can render objects by reflecting depth and / or height values ​​based on the object's category. According to an embodiment, the electronic device can determine depth based on the object's position in the content and the object's category, and arrange objects in virtual space based on the determined depth. According to an embodiment, when multiple objects exist, the electronic device can arrange the multiple objects based on an order of depth that can be identified in the original image.

[0343] According to an embodiment, the electronic device can render an object in region A1 or region A2 based on the object's attributes. According to an embodiment, when the object is a floating object (e.g., a sphere), the electronic device can render the object in a region near the ceiling area of ​​region A1 based on the object's attributes. According to an embodiment, when the object is a floating object, the electronic device can render the object in a ceiling area based on the object's attributes.

[0344] Figure 12a This is a view illustrating the operation of an electronic device arranging each object included in an extended image at different depths, according to an embodiment of the present disclosure.

[0345] refer to Figure 12a Electronic devices (e.g.) Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can generate an extended image 1220 based on the 2D image 1210. According to an embodiment, the 2D image 1210 may be an image displayed on a background screen.

[0346] According to an embodiment, the electronic device can use generative AI to generate an extended image 1220 based on a 2D image 1210. According to an embodiment, the electronic device can separate at least one object 1221 or 1222 included in the 2D image 1210.

[0347] According to an embodiment, the electronic device can separate a 2D image 1210 into a background image and one or more objects 1221 and 1222, and then perform out-drawing and in-drawing on the background image to generate an extended image 1220.

[0348] According to an embodiment, the electronic device can generate an extended image 1220 by performing outward and inward drawing on a 2D image 1210, and then can separate at least one object 1221 or 1222 included in the extended image 1220. According to an embodiment, the electronic device can generate an image by performing inward drawing on the blank space generated by separating at least one object 1221 or 1222.

[0349] According to an embodiment, the electronic device can divide a virtual space into multiple areas. For example, the electronic device can be divided into multiple areas based on the distance to the user 10 located at the center of the virtual space.

[0350] For example, the area within a first distance from user 10 can be a first area 1230 (e.g., a closed-range view). According to an embodiment, the electronic device can have a second area 1231 (e.g., an intermediate-range view) that is greater than or equal to the first distance from user 10 and less than the second distance from user 10. According to an embodiment, the second distance can be the boundary of a virtual space. According to an embodiment, the area corresponding to the boundary of the virtual space can be a third area 1232 (e.g., a distant view (skybox)). According to an embodiment, the range of at least one of the first area 1230, the second area 1231, and the third area 1232 can be adjusted by user input. According to an embodiment, the third area 1232 can have a curved surface shape corresponding to the shape of the virtual space. For example, the third area can be spherical or hemispherical.

[0351] According to an embodiment, the first area 1230 is the main usage space for the user 10, and the electronic device can display objects related to the VR application (e.g., a home screen or execution screen). According to an embodiment, the electronic device can arrange floating objects and interactive objects 1222 in the first area 1230 based on the attributes of objects in at least one object 1221 or 1222 obtained from the original image 1210.

[0352] According to an embodiment, the electronic device can display objects 1221, which are included in categories such as people, animals, or plants, in a second area 1231 based on the attributes of objects in at least one of the objects 1221 or 1222.

[0353] According to an embodiment, the electronic device can display the extended image 1220 in at least a portion of the third region 1232. According to an embodiment, when the extended image 1220 is a 360-degree panoramic image, the electronic device can display the extended image 1220 in an area other than the ceiling and floor areas of the third region 1232. According to an embodiment, when the extended image 1220 is a spherical image, the electronic device can display the extended image 1220 over the entire area of ​​the third region 1232.

[0354] According to the embodiments, such as Figure 12b or Figure 12c As shown, the electronic device can divide the second area 1231 into multiple areas to arrange objects.

[0355] Figure 12b This is a view showing the depth of a 3D space according to an embodiment of the present disclosure.

[0356] See Figure 12b The virtual space can be divided into a first region 1230 (e.g., region A), a second region 1231 (e.g., region A') and a third region 1232 (e.g., background region) based on the distance from the user 10.

[0357] According to an embodiment, the electronic device can further divide the second region 1231 into a second-1 region 1240 (e.g., region A1) and a second-2 region 1241 (e.g., region A2) based on the distance from the user 10. Figure 12b In the middle, the second region 1231 is divided into two regions, but it can be divided into three or more regions.

[0358] According to an embodiment, the electronic device can also divide an area with a height greater than or equal to the height set based on the user 10 into a ceiling area 1233 (e.g., area B). According to an embodiment, the ceiling area 1233 can be the space where the user 10 must raise his head.

[0359] According to an embodiment, the electronic device can be arranged in a first region 1230, a second-1st region 1240, a second-2nd region 1241, or a ceiling region 1233 based on the attributes of at least one object separated from the original image.

[0360] According to an embodiment, an electronic device can additionally generate and arrange virtual objects based on the attributes of at least one object. According to an embodiment, reference is made below. Figure 13b Describes the operations for additionally generating and arranging virtual objects.

[0361] According to an embodiment, floating objects (e.g., petals, balls, or airplanes) can be arranged in a first area 1230, a second-first area 1240, a second-second area 1241, or a ceiling area 1233 based on object attributes by applying height values.

[0362] Figure 12c It is a view showing an extended image of objects arranged at different depths according to embodiments of the present disclosure.

[0363] refer to Figure 12c Based on the distance between the virtual space and the user 10, the electronic device may include a first area 1230 (e.g., area A), a second-first area 1240 (e.g., area A1), a second-second area 1241 (e.g., area A2), a third area 1232 (e.g., background area) and a ceiling area 1233 (e.g., area B).

[0364] According to an embodiment, the electronic device can arrange an extended image 1250 generated from the original image and at least one object 1251, 1252 or 1253 separated from the original image in corresponding areas of a virtual space.

[0365] For example, the electronic device can display an extended image 1250 in a third region 1232 corresponding to the boundary of the virtual space. According to an embodiment, the electronic device can display a first object 1251 as a character object in a second-second region 1241 based on the object's attributes. According to an embodiment, the electronic device can display a second object 1252 as an interactive object in a second-first region 1240 based on the object's attributes. According to an embodiment, the electronic device can display a third object 1253 as a floating object in a ceiling region 1233 based on the object's attributes.

[0366] Figure 12d It is a view showing an extended image of objects arranged at different depths according to embodiments of the present disclosure.

[0367] refer to Figure 12d The electronic device can overlay and provide extended image 1250, first object 1251, second object 1252, and third object 1253. According to an embodiment, when viewed from above, extended image 1250, first object 1251, second object 1252, and third object 1253 can be arranged relative to each other. Therefore, when viewed from the front, the electronic device can provide a 3D effect by displaying extended image 1250, first object 1251, second object 1252, and third object 1253.

[0368] Figure 13a This is a view illustrating the operation of an electronic device providing interaction to an arranged object according to an embodiment of the present disclosure.

[0369] refer to Figure 13a Electronic devices (e.g.) Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can display an extended image 1310, a first object 1320, and a second object 1330. According to an embodiment, the electronic device can interact with a user by applying motion to at least one of the extended image 1310, the first object 1320, and the second object 1330 based on attributes.

[0370] For example, the electronic device can assign flight motion to a second object 1330, which includes a floating attribute. According to an embodiment, the electronic device can display a second object 1331 arranged at different locations by applying motion to change the position of the second object 1330. According to an embodiment, the second object 1331 arranged at different locations can have different heights and left-right positions. According to an embodiment, the size of the second object 1331 arranged at different locations can vary based on its distance from the user.

[0371] Figure 13b This is a view illustrating the operation of an electronic device providing interaction to an arranged object according to an embodiment of the present disclosure.

[0372] refer to Figure 13b The electronic device can display a movable second object 1330 based on attributes. According to an embodiment, when a user receives first user input (e.g., touch) for selecting the second object 1330, the electronic device can display a second object 1332 with a changed color. According to an embodiment, the color can change randomly. This embodiment is an example of changing the color of an object based on user input, but it is not limited to this; different colors, types, and / or sizes of highlighting effects can be applied to objects based on user input.

[0373] According to an embodiment, when a second user input (e.g., a long touch) for selecting a second object 1330 is received, the electronic device can display a plurality of additionally generated objects 1333 based on the second object 1330. According to an embodiment, the electronic device can assign movement to each of the plurality of additionally generated objects 1333.

[0374] According to an embodiment, in Figure 13bThe image shows multiple objects 1333 shaped like spheres, but according to an embodiment, when an object is selected as a petal, multiple petals can be displayed as if they were scattered.

[0375] According to an embodiment, when a user input (e.g., touch) is received for selecting one of a plurality of additionally generated objects 1333, the electronic device can display a plurality of recolored objects 1334. According to the embodiment, this is an example of changing the color of an object based on user input, but it is not limited thereto; different colors, types, and / or sizes of highlighting effects can be applied to objects based on user input.

[0376] In this way, electronic devices can add or change objects through interaction with the user.

[0377] Figure 13c This is a view illustrating the operation of an electronic device providing interaction to an arranged object according to an embodiment of the present disclosure.

[0378] refer to Figure 13c The electronic device can perform interactions with the user by applying actions to at least one of the first object 1320 and the second object 1330 based on attributes.

[0379] For example, the electronic device can assign flight motion to a second object 1330, which includes a floating attribute. According to an embodiment, the electronic device can assign actions to change the posture of a first object 1320, including attributes of movement while changing posture.

[0380] According to an embodiment, the duration of movement can be determined based on the category of the object. For example, since the first object 1320, being a person, does not move in the air for an extended period, the electronic device can assign movement over approximately 5 seconds. According to an embodiment, the electronic device can assign the movement of the first object 1320 as a Graphics Exchange Format (GIF).

[0381] According to an embodiment, when a slight wagging of the tail is assigned to an object as an animal, the electronic device may not impose a limit on the duration of the movement.

[0382] According to an embodiment, the electronic device can allocate movements to change the position and orientation of a first object 1320 to display the first object 1321 arranged in different positions. According to an embodiment, the electronic device can allocate movements to change the position of a second object 1330 to display the second object 1335 arranged in different positions. According to an embodiment, the second object 1335 arranged in different positions may have different heights and left-right positions. According to an embodiment, the size of the second object 1335 arranged in different positions may vary based on its distance from the user.

[0383] As mentioned above, a sense of life can be provided by assigning motion to objects, and this allows them to be used as screensavers.

[0384] Figure 13d This is a view illustrating the operation of an electronic device providing interaction to an arranged object according to an embodiment of the present disclosure.

[0385] refer to Figure 13d The electronic device can assign motion to the background. For example, the electronic device can hold a first object 1320 and a second object 1330, and can assign movement to the background 1340. For example, the electronic device can assign motion to an image included in the background 1340, but can set a small range of motion and repeatedly reproduce the image.

[0386] According to an embodiment, the electronic device may additionally arrange an image included in the background 1340. For example, the electronic device may additionally generate and arrange viewer objects 1341 included in the background 1340. According to an embodiment, the electronic device may also assign motion to additionally arranged objects.

[0387] According to an embodiment, the electronic device can add images not included in the background 1340 based on the content (e.g., category) of the original image. For example, when the original image is a game scene, the electronic device can add images of spectators, another player, electronic displays, and / or signage that are not included in the background 1340.

[0388] According to an embodiment, the image additionally arranged in the background 1340 can also be interactive with the user. For example, when user input (e.g., pinch) is received for selecting an additionally arranged electronic display, the electronic device can add movement so that the player's name moves across the electronic display image.

[0389] According to an embodiment, the electronic device can control the brightness of the entire virtual space based on the content (e.g., category) of the original image or background 1340. As described above, an immersive experience can be provided by controlling the brightness of the virtual space based on the content of the original image or background.

[0390] It can also be made available for viewing by assigning movement to the background. For example, when the background is a natural object such as fire, mountains, or water, it can be made available for viewing by giving movement to the background, and it can be used as a screensaver.

[0391] Figure 14 This is a view illustrating the operation of an electronic device providing interaction to an arranged object according to an embodiment of the present disclosure.

[0392] refer to Figure 14 Electronic devices (e.g.) Figure 1a and Figure 1bElectronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can assign movement to the background image 1410 based on environmental information. For example, based on time information, a background image 1410 corresponding to daytime can be provided during daytime, and a background image 1420 corresponding to nighttime can be provided during nighttime. For example, the electronic device can gradually change the background image over time.

[0393] According to an embodiment, the background image 1410 corresponding to daytime can be changed to a background image 1420 corresponding to nighttime by user input. For example, when the device receives a user's drag-and-drop input of selecting and pulling down the ceiling area of ​​a virtual space, it can change the background image 1410 corresponding to daytime to the background image 1420 corresponding to nighttime.

[0394] According to an embodiment, when the background image 1420 is changed to correspond to nighttime, the electronic device may additionally arrange objects that are not included in the background image 1410 corresponding to daytime but are suitable for the background image 1420 corresponding to nighttime. For example, the electronic device may additionally arrange objects such as the moon, stars, comets, and auroras in the background image 1420 corresponding to nighttime.

[0395] Figure 15 This is a view illustrating the operation of an electronic device providing interaction to an arranged object according to an embodiment of the present disclosure.

[0396] refer to Figure 15 When a user input 1511 (e.g., touch) is received for selecting a first object 1510, the electronic device (e.g., Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can display information related to the first object 1510.

[0397] For example, when a user input 1511 is received for selecting a first object 1510 to provide current time information, the electronic device can display a screen 1520 in virtual space that includes at least one of relevant widgets, weather information, schedule information, and reminder information. According to an embodiment, screen 1520 may be a pop-up screen.

[0398] According to an embodiment, when user input is received for selecting an object related to a person, plant or animal, building, or thing, the electronic device can provide information related to the selected object. For example, when user input is received for selecting an object related to a person, the electronic device can provide contact information corresponding to that person. According to an embodiment, the contact information may include a user interface for making a call or sending a message to the contact person corresponding to that person.

[0399] According to an embodiment, when user input for selecting a background image is received, the electronic device can provide information related to the background image. For example, when user input for selecting a background image is received, the electronic device can provide information related to a location included in the background image (e.g., location name or location).

[0400] Figure 16 This is a flowchart illustrating the operation of an electronic device mapping audio to arranged objects according to an embodiment of the present disclosure.

[0401] In the following embodiments, operations can be performed sequentially, but they can also be performed out of sequence. For example, the order of operations can be changed, and at least two operations can be performed in parallel.

[0402] Reference Figure 16 In operation 1610, electronic devices (e.g., Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can identify whether the original document is a video. According to an embodiment, the electronic device can identify whether the original image is a video or a video.

[0403] According to an embodiment, when the original object is not video (No in operation 1610), in operation 1611, the electronic device can automatically generate audio after identifying the category of the object. According to an embodiment, when the original image is an image, since the original image does not include audio, the electronic device can identify the category of the image and then generate audio corresponding to that category. For example, depending on whether the image is a natural environment, an urban environment, or a specific situation (e.g., a sporting event or a festival), audio corresponding to the image can be generated.

[0404] According to an embodiment, the electronic device can generate a background image of an image and audio corresponding to at least one object included in the image.

[0405] According to an embodiment, in operation 1612, the electronic device can encode the background volume based on spatial depth. For example, the electronic device can encode the volume based on the attributes of an object and the area to be arranged. For example, when an object is positioned close to the user, the volume can be encoded as increasing. For example, since the background image is arranged furthest from the user, the size of the audio associated with the background image can be encoded as smaller than the size of other objects.

[0406] According to an embodiment, if the original image is a video (in operation 1610), then in operation 1620, the electronic device can identify whether sound exists in the original image.

[0407] According to an embodiment, when sound is present in the original image (yes in operation 1620), in operation 1630, the electronic device can identify whether the object is a person, animal, or plant. According to an embodiment, the electronic device can identify whether an object's category, such as a person, animal, or plant, can emit sound through object recognition.

[0408] According to an embodiment, when the object is neither a person, animal, nor plant (no in operation 1630), in operation 1640, the electronic device can encode the background volume based on the depth of the space. For example, as Figure 17a As shown, the electronic device can encode volume based on the attributes of the objects and the area to which they are to be placed. For example, when an object is positioned close to the user, the volume can be encoded as increasing. Similarly, since a background image is positioned furthest from the user, the size of the audio associated with the background image can be encoded as smaller than the size of other objects.

[0409] Figure 17a This is a view illustrating the operation of an electronic device mapping audio to arranged objects according to an embodiment of the present disclosure.

[0410] refer to Figure 17a Electronic devices (e.g.) Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can map different sounds to different intensities based on the distance between the object and the user (e.g., nearsightedness, intermediate vision, or farsightedness). For example, the sound of an object 1711 (e.g., a ball) that is close to the user can be mapped to a louder sound than the sound of an object 1710 (e.g., an audience member) that is far from the user.

[0411] According to an embodiment, when the background image is natural, sounds corresponding to nature can be applied. For example, based on the background image, sounds such as desert wind, snowfall, forest, burning firewood, or muffled sounds can be applied. According to an embodiment, the electronic device can apply the sound corresponding to the background image as a low volume.

[0412] According to an embodiment, when displaying video, sounds corresponding to nearby objects and sounds corresponding to the background image can be applied at different decibels or frequencies. For example, the sound corresponding to a person near the user can be applied loudly, while the sound of wind in the desert corresponding to the background image can be applied softly.

[0413] According to an embodiment, when the background image is an unnatural road or stadium, noise can be identified and removed, or the volume can be reduced and applied based on the large amount of noise included in the sound.

[0414] According to an embodiment, the electronic device can apply different sounds corresponding to the background image and the object depending on whether it is day or night. For example, at night, the sounds corresponding to the background image and the object can be applied to resonate more than during the day.

[0415] return Figure 16 According to an embodiment, when the object is a person, animal, or plant ("Yes" in operation 1630), in operation 1621, the electronic device can automatically separate the audio after identifying the category of the object.

[0416] According to an embodiment, when the object is a person or animal, the electronic device can separate the sound corresponding to the object from the sound included in the original image. According to an embodiment, when the sound is too faint to extract the object's sound, artificial intelligence resources can be used to map the sound corresponding to the object.

[0417] return Figure 16According to an embodiment, in operation 1622, the electronic device can encode the background volume based on the depth of space, and as... Figure 17b As shown, electronic devices can encode objects with sound (such as people, animals, or plants) only when the user interacts with the object.

[0418] Figure 17b This is a view illustrating the operation of an electronic device mapping audio to arranged objects according to an embodiment of the present disclosure.

[0419] refer to Figure 17b When the object is a person 1720, a sound-producing object 1721, or an animal 1722, the electronic device can separate the sound corresponding to the object. According to an embodiment, when the objects are a person 1720 and an object 1721, the electronic device can provide interaction between the objects (e.g., an image showing a person 1720 playing with an object 1721).

[0420] According to an embodiment, when the electronic device is a sound-producing object (e.g., a crying baby or a barking dog), the electronic device can output a sound corresponding to the object only when there is interaction between the object and the user. For example, when the electronic device does not output a sound corresponding to the object, it can output a sound corresponding to the object when the user looks at the object or receives user input for selecting the object.

[0421] As mentioned above, in cases where sound might disrupt the immersion level of the virtual space, the sound can be output only when there is user interaction.

[0422] According to an embodiment, the electronic device can encode sound by taking into account sounds in real space. For example, when several sounds are present, since the sounds can be provided overlappingly, only the sounds corresponding to a set number (e.g., two) of objects can be generated, and sounds corresponding to the background image are preferentially generated.

[0423] According to an embodiment, the sound can be the result of providing an image as input and outputting an image, and the electronic device can interpret the input image, generate interpretive information as a prompt, and reproduce the sound generated based on the prompt.

[0424] According to an embodiment, when there is no sound in the original image (No in operation 1620), the electronic device can proceed to operation 1611 to automatically generate audio after identifying the category of the object. According to an embodiment, in operation 1612, as... Figure 17c As shown, the electronic device can encode background volume based on the depth of the space.

[0425] Figure 17c This is a view illustrating the operation of an electronic device mapping audio to arranged objects according to an embodiment of the present disclosure.

[0426] refer to Figure 17c The electronic device can map sounds associated with the background image 1730, or provide a related music playback interface.

[0427] According to an embodiment, the electronic device can analyze locations, regions, and cities related to the background image 1730, and can provide relevant sounds based on the analysis results. For example, the electronic device can search for songs related to the region of the background image 1730 and provide the searched songs.

[0428] According to an embodiment, when user input 1731 for selecting the background image 1730 is received while outputting sound related to the background image 1730, the electronic device can display an interface 1732 for the sound related to the background image 1730. According to an embodiment, the interface 1732 may include information related to the sound and / or buttons for controlling the sound.

[0429] According to an embodiment, the electronic device can repeatedly reproduce a sound source for a set time (e.g., 2-3 minutes).

[0430] According to an embodiment, the electronic device can change the music output based on user input.

[0431] Figure 17d This is a view illustrating the operation of an electronic device mapping audio to arranged objects according to an embodiment of the present disclosure.

[0432] refer to Figure 17d The electronic device can map audio onto each of the background image 1740 and at least one object 1741, 1742, and 1743. According to an embodiment, each of the background image 1740 and at least one object 1741, 1742, and 1743 can be generated or extracted based on the original image.

[0433] According to an embodiment, the electronic device can output relevant audio based on the original image or the generated background image 1740 and at least one object 1741, 1742, and 1743. For example, when the original image or the generated background image 1740 and at least one object 1741, 1742, or 1743 are in an urban environment, the electronic device can automatically output urban noise.

[0434] According to embodiments, the electronic device can map audio to objects that do not emit sound (e.g., buildings, plants, or objects) based on the attributes of the objects. According to embodiments, the electronic device can map audio to objects based on user input. According to embodiments, the electronic device can output the audio mapped to the object, either during user interaction (e.g., touching the object) or by displaying an audio-related player. For example, when user input 1750 is received for selecting a first object 1741, the electronic device can display a media player screen 1751 associated with the audio mapped to the first object 1741. For example, the media player screen 1751 may include audio-related information (e.g., song title or artist) and / or buttons for playback control.

[0435] According to an embodiment, when user interaction is performed on an object that emits a sound (e.g., a person or an animal), audio corresponding to the object can be output. For example, when user input is received for selecting a second object 1742 as a person, the electronic device can output a movement sound and a ball bouncing sound as audio corresponding to the second object 1742. According to an embodiment, when user input is received for selecting a third object 1743 as an animal, the electronic device can output a barking sound as audio corresponding to the third object 1743.

[0436] According to an embodiment, when an object is selected, the electronic device can also provide movement of the object and audio output. For example, when user input is received for selecting a second object 1742 as a person, the electronic device can output the movement of a bouncing ball on the second object 1742. According to an embodiment, when user input is received for selecting a third object 1743 as an animal, the electronic device can output a tail wagging motion to the third object 1743.

[0437] In this way, the electronic device can provide a sense of immersion by outputting audio related to the background image 1740 and at least one object 1741, 1742, and 1743. More specifically, for at least one object 1741, 1742, and 1743, the level of immersion in the virtual space can be prevented from deteriorating by outputting audio and / or movement based on user interaction.

[0438] Figure 18a This is a view illustrating the operation of an object whose arrangement is changed by an electronic device according to an embodiment of the present disclosure. Figure 18a This demonstrates partial removal and alteration of the close-up view (e.g., Figure 12a , Figure 12b , Figure 12c and Figure 12d First area 1230), intermediate view (e.g., Figure 12a , Figure 12b , Figure 12c and Figure 12d The second area 1231) and the distant view (e.g., Figure 12a , Figure 12b , Figure 12c and Figure 12d An embodiment of the distant view in the third region 1232).

[0439] refer to Figure 18a Electronic devices (e.g.) Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can change the background image 1810. According to an embodiment, the electronic device can change the background image 1810 based on user input or the passage of time. According to an embodiment, the electronic device can change the background image 1810 based on a change in an object.

[0440] According to an embodiment, the electronic device can change to a monochrome or gradient background image 1811 based on the color extracted from the background image 1810. For example, the color extracted from the background image 1810 could be the color in the largest area of ​​the background image 1810.

[0441] According to an embodiment, the electronic device can transform a background image 1810 into an illustration-type image 1812. According to an embodiment, the electronic device can input the background image 1810 into an artificial intelligence model trained to transform an image into an illustration-type image, and display the illustration-type image 1812 as output data from the artificial intelligence model. According to an embodiment, the electronic device can use an algorithm for transforming an image into an illustration-type image to transform the background image 1810 into the illustration-type image 1812.

[0442] According to an embodiment, the electronic device can transform a background image 1810 into a line image 1813. According to an embodiment, the electronic device can input the background image 1810 into an artificial intelligence model trained to transform an image into a line image, and display the line image 1813 obtained from the artificial intelligence model as output data. According to an embodiment, the electronic device can use an algorithm for transforming an image into a line image to transform the background image 1810 into the line image 1813.

[0443] According to an embodiment, Figure 18a The illustration shows the background image being changed to one of three types, but this disclosure is not limited to this, and the background image can be changed to such a type using an artificial intelligence model or algorithm. Figure 19Another type of image is shown.

[0444] Figure 18b This is a view illustrating the operation of changing the arrangement of objects by an electronic device according to embodiments of the present disclosure. For example, Figure 18b An example is shown where the intermediate view is partially removed and altered from the close-up view, intermediate view, and far-view.

[0445] refer to Figure 18b The electronic device can transform objects arranged in a close-up view, objects 1820 arranged in an intermediate view, and objects 1820 arranged in a background image arranged in a distant view into objects 1821 and 1822 of another category. For example, the electronic device can transform an object 1820 of a person exercising into a flashing illustration object 1821 and an object 1822 in the shape of a seated person. According to an embodiment, the electronic device can provide a virtual space with an exotic atmosphere (e.g., a popular or lighthearted atmosphere) by arranging different categories of objects in the intermediate view.

[0446] Figure 18c This is a view illustrating the operation of changing the arrangement of objects by an electronic device according to embodiments of the present disclosure. For example, Figure 18c An example is shown where the intermediate view is partially removed and altered from the close-up view, intermediate view, and far-view.

[0447] refer to Figure 18c The electronic device can transform an object 1830 in a close-up view, an object in an intermediate view, and an object in a background image in a distant view into an object of another category. For example, the electronic device can transform a flying spherical object 1830 into an object 1831 that is related to providing information (e.g., a business card, an electronic display, or a pop-up screen).

[0448] According to an embodiment, an electronic device can provide a virtual space with an exotic atmosphere (e.g., a popular or entertaining atmosphere) by arranging different types of objects in the foreground.

[0449] exist Figures 18a to 18c The document discloses changing the displayed object to another object. However, according to embodiments, the electronic device can arrange images of a similar category to the object in several layouts.

[0450] Figure 19 This is a view illustrating the operation of an object whose arrangement is changed by an electronic device according to an embodiment of the present disclosure.

[0451] refer to Figure 19 Electronic devices (e.g.) Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1bProcessor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can divide the virtual space into multiple areas and can arrange background images 1910 or objects for each of the multiple areas.

[0452] According to an embodiment, the electronic device can change the style of the background image 1910 or at least some objects. For example, the electronic device can change the background image 1910 into a line image 1920, a watercolor image 1921, or an oil painting image 1922.

[0453] Figure 20a This is a view illustrating the operation of an object whose arrangement is changed by an electronic device according to an embodiment of the present disclosure.

[0454] refer to Figure 20a Electronic devices (e.g.) Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can display an image 2010 generated by connecting (or merging) multiple images 2011 and 2012 as a background image.

[0455] For example, an electronic device can generate a 360-degree image 2010 by connecting a daytime image 2011 and a nighttime image 2012 of a location. According to an embodiment, the electronic device can display the 360-degree image 2010 such that the daytime image 2011 is positioned in front of the user and the nighttime image 2012 is positioned behind the user.

[0456] According to an embodiment, when multiple contents (e.g., images or videos) are applied to a virtual space, the electronic device can change the applied contents at set time intervals, such as... Figure 20b As shown, or attributes that can change over time, such as... Figure 20c As shown. According to an embodiment, when user input (e.g., swiping) is received, the electronic device can change the content applied to the virtual space.

[0457] Figure 20b This is a view illustrating the operation of an object whose arrangement is changed by an electronic device according to an embodiment of the present disclosure.

[0458] refer to Figure 20b The electronic device can change the content applied to the virtual space in the background image 2020 and multiple contents 2021, 2022, and 2023 at predetermined time intervals (e.g., n minutes). According to an embodiment, the electronic device can arrange the background image 2020 and at least one object in the virtual space. According to an embodiment, the electronic device can change the background image 2020 at predetermined time intervals while maintaining at least one object.

[0459] According to an embodiment, the electronic device can slide the displayed content at predetermined time intervals, or replace the displayed content with another content by fading in / out. According to an embodiment, the electronic device can sequentially display a background image 2020 and multiple contents 2021, 2022, and 2023 at predetermined time intervals based on a set order.

[0460] Figure 20c This is a view illustrating the operation of an object whose arrangement is changed by an electronic device according to an embodiment of the present disclosure.

[0461] refer to Figure 20c Electronic devices can change the same content based on time and / or season. For example, electronic devices can change the attributes of the same content based on time and / or season.

[0462] According to an embodiment, when the first content 2030 is applied to a virtual space, the electronic device can display the second content 2031, wherein attributes such as the color of the background image change based on the passage of time and / or seasonal changes. For example, the first content 2030 may correspond to daytime, and the second content 2031 may correspond to sunset. According to an embodiment, the electronic device can sequentially move the brightest part to the right from the first content 2030 and sequentially change the background color to purple over time, wherein in the first content 2030, the brightest part corresponding to the sun's position is located in the center, and the sky as the background is blue during the day. According to an embodiment, the electronic device can display the second content 2031 at a time corresponding to sunset, wherein the brightest part is on the right and the sky is purple.

[0463] According to embodiments, the electronic device can not only change the properties of the background image, but also change the object images (e.g., trees, plants) included in the background image and the objects arranged spaced apart from the background image based on the passage of time and / or seasonal changes. For example, the electronic device can change the size of the tree images included in the background image or the trees arranged spaced apart from the background image over time, and can change according to seasonal changes, such as causing leaves to sprout or fall.

[0464] Figure 21aThis is a view illustrating the operation of an electronic device according to an embodiment of the present disclosure arranging objects on a lock screen or an always-on display (AOD) screen. According to the embodiment, Figure 21a The lock screen or AOD screen 2110, 2120, 2130 or 2140 shown can be a 360-degree panoramic, spherical or hemispherical image arranged in virtual space.

[0465] refer to Figure 21a Electronic devices (e.g.) Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can retain only the clock content 2111 included in the lock screen or AOD screen 2110, while dimming the rest.

[0466] According to an embodiment, the electronic device can arrange clock content 2122 within a field of view (FOV) 2121, and can display a dimmed lock screen or AOD screen 2120 in addition to the clock content 2122. According to an embodiment, a floating object 2123 can display a lock screen or AOD screen 2110 arranged in an area other than the FOV. According to an embodiment, the floating object 2123 can be arranged in virtual space, spaced apart from a background image. According to an embodiment, the floating object 2123 can interact with the user.

[0467] According to an embodiment, the electronic device may display a lock screen or AOD screen 2130, wherein the portion of the screen except for the clock content is blurred to enhance security. According to an embodiment, the electronic device may display a deblurred lock screen or AOD screen based on performing authentication for release security (such as in biometric identification).

[0468] According to an embodiment, the electronic device may display a lock screen or AOD screen 2140, which includes images of a similar category to the background image, excluding clock content, to enhance security. According to an embodiment, the electronic device may display an image or object related to the background image on top of similar category images, while blurring it.

[0469] Figure 21b This is a view illustrating the operation of arranging objects on a lock screen or AOD screen by an electronic device according to an embodiment of the present disclosure. According to the embodiment, Figure 21bThe lock screens or AOD screens 2150, 2160 and 2170 shown can be 360-degree panoramic, spherical or hemispherical images arranged in virtual space.

[0470] refer to Figure 21b The electronic device can display a lock screen or AOD screen 2150, wherein a representative color extracted from a background image is displayed as a single color. According to an embodiment, the electronic device can display a lock screen or AOD screen 2160 that is displayed by gradients of multiple colors extracted from a background image.

[0471] According to an embodiment, the electronic device may display a lock screen or AOD screen 2170 that displays at least one object 2171 (e.g., a floating object) on a monochrome or gradient background.

[0472] According to an embodiment, at least one object 2171 may be arranged in a virtual space, spaced apart from a background image. According to an embodiment, at least one object 2171 may be interactive with a user.

[0473] Figure 22 This is a view of a virtual space displayed by an electronic device according to an embodiment of the present disclosure.

[0474] refer to Figure 22 Electronic devices (e.g.) Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 The electronic device 400 can generate an extended image 2220-1 or 2220-2 based on the 2D image 2210. According to an embodiment, the extended image 2220-1 or 2220-2 can be a 360-degree panoramic image, a spherical image, or a hemispherical image.

[0475] According to an embodiment, the electronic device can display extended images 2220-1 or 2220-2 in a 3D background area. According to an embodiment, the electronic device can apply workspace-related objects (i.e., board 2221 or virtual monitor screen 2222) to a virtual space displaying extended images 2220-1 or 2220-2 in the 3D background area. For example, the electronic device may include board 2221 or virtual monitor screen 2222, which includes functions related to performing tasks. According to an embodiment, board 2221 may include icons for task-related applications and widgets for functions related to performing tasks (e.g., memos or keyboard). According to an embodiment, virtual monitor screen 2222 can display an execution screen for task-related applications. According to an embodiment, the electronic device can display operations based on user input performed on virtual monitor screen 2222 or user input performed via board 2221.

[0476] According to an embodiment, when a user gazes or moves their head, the electronic device can change the field of view (FOV) in the virtual space based on the gaze or head movement. According to an embodiment, the electronic device can display a screen based on the changed FOV.

[0477] For example, when a user looks upwards while displaying a first background image 2220-1 and a virtual monitor screen 2222 displayed on and above board 2221, the electronic device can change the field of view (FOV) in the virtual space upwards and display the screen based on the changed FOV. According to an embodiment, the electronic device can display a second background image 2220-2, which is a higher FOV than the first background image 2220-1. According to an embodiment, the electronic device can move the position of the virtual monitor screen 2222 downwards within the FOV and can display time content 2223 arranged above the virtual monitor screen 2222 within the FOV. According to an embodiment, the electronic device can also display function icons arranged around the time content 2223.

[0478] In this way, electronic devices can apply the images desired by the user to a virtual workspace.

[0479] Figure 23a This is a view illustrating the operation of an electronic device according to an embodiment of the present disclosure, displaying a keyword-generated image as a 3D virtual image in an external electronic device.

[0480] refer to Figure 23a External electronic devices (e.g.) Figure 1a and Figure 1bThe external electronic device 104 may display a screen 2310 for generating a background (wallpaper) image. The screen 2310 for generating the background image may include at least one keyword 2311 to be applied to the background image to be generated and a button 2312 for commanding the operation of generating the background image.

[0481] According to an embodiment, at least one keyword 2311 to be applied to the background image to be generated can be selected from a plurality of keywords through user input or text input performed by the user. According to an embodiment, a plurality of optional keywords can be pre-stored for each category related to image style. According to an embodiment, an external electronic device can display an example image together for each of the plurality of keywords.

[0482] According to an embodiment, an external electronic device can apply a background image 2320 generated based on at least one keyword 2311 as a background screen. For example, when flowers, blue, navy, and castle are included in at least one keyword 2311, the generated background image 2320 can include a blue-roofed castle surrounded by blue flowers and a blue sky.

[0483] According to an embodiment, the background image 2320 applied to an external electronic device can be applied to an interoperable electronic device (e.g., ...). Figure 1a and Figure 1b Electronic device 101 Figure 1a and Figure 1b Processor 120 Figure 2 Wearable electronic devices 200 Figure 3a Wearable electronic devices 300 Figure 3b Wearable electronic devices 300 or Figure 4 (electronic device 400). For example, external electronic devices and electronic devices can interact with each other through an account.

[0484] According to an embodiment, the electronic device can generate a 3D virtual image 2330 based on a 2D background image 2320 received from an external electronic device. For example, the 3D virtual image 2330 may include a background image 2331 applied to a background region at a first depth in the 3D virtual space, an object image 2332 applied to a second depth in the 3D virtual space, and at least one newly generated object 2333. For example, at least one newly generated object 2333 may be generated based on attributes of the background image 2331 and / or the object image 2332, and may be arranged at a third depth different from the second depth or in a ceiling area (or a public area).

[0485] Figure 23bThis is a view illustrating the operation of an electronic device applying an image, which is changed based on keywords while displaying a 3D virtual image, to an external electronic device according to an embodiment of the present disclosure.

[0486] refer to Figure 23b When a user input for changing a 3D virtual image is received while the 3D virtual image is being displayed, the electronic device can display a screen 2340 for changing the 3D virtual image. According to an embodiment, the electronic device can change the displayed 3D virtual image to an image generated based on the changed keyword by changing a keyword.

[0487] According to an embodiment, the screen 2340 for changing the 3D virtual image may include at least one keyword 2341 to be applied to the 3D virtual image to be generated and a button 2342 for commanding the operation of generating the 3D virtual image. According to an embodiment, the screen 2340 for changing the 3D virtual image may also include a preview 2343 of the image to be generated based on at least one keyword. For example, the preview 2343 may be a 2D image.

[0488] According to an embodiment, at least one keyword 2341 to be applied to the 3D virtual image to be generated can be one selected from multiple keywords by user input, or it can be text input by the user. According to an embodiment, multiple optional keywords can be pre-stored for each category related to image style. According to an embodiment, an external electronic device can display sample images together for each of the multiple keywords.

[0489] According to an embodiment, an external electronic device can display a 3D virtual image 2350 generated based on at least one keyword 2341 in a 3D virtual space. For example, when at least one keyword 2341 includes flowers, pink, purple, and a castle, the generated 3D virtual image 2350 may include a sky image 2351 with a changed color applied to a background area of ​​a first depth in the 3D virtual space, a castle image 2352 with a pink roof surrounded by pink flowers applied to a second depth in the 3D virtual space, and at least one pink heart image 2353 arranged in the ceiling area of ​​the 3D virtual space.

[0490] According to an embodiment, the 3D virtual image 2350 applied to the electronic device can be used as a background image 2360 on the background screen of an interactive external electronic device. For example, the electronic device and the external electronic device can interact with each other through an account.

[0491] According to an embodiment, an external electronic device can generate a 2D background image 2360 based on a 3D virtual image 2350 received from the electronic device. According to an embodiment, the 2D background image 2360 can be applied to the main screen and / or lock screen of the external electronic device.

[0492] According to this disclosure, an electronic device can generate images suitable for virtual spaces by obtaining extended images based on 2D images through external and internal drawing. According to this disclosure, immersion can be provided to users by separating the background and objects in the image and arranging them at different depths in the virtual space. According to this disclosure, immersion can be provided to users in virtual spaces by mapping audio according to the properties of the background and objects and outputting them.

[0493] According to an embodiment, electronic devices (e.g., Figure 1a and Figure 1b Electronic device 101) may include a display (e.g., Figure 1a and Figure 1b The display module 160), and sensors (e.g., Figure 1a and Figure 1b Sensor module 176), processor (e.g., Figure 1a and Figure 1b The processor 120) and memory (e.g., Figure 1a and Figure 1b The memory 130 includes instructions that enable the electronic device to perform operations when executed by a processor.

[0494] According to an embodiment, the instructions, when executed by a processor, enable an electronic device to obtain information related to the user's field of view via a sensor.

[0495] According to an embodiment, the instructions can enable an electronic device to acquire a two-dimensional (2D) image when executed by a processor.

[0496] According to an embodiment, when executed by a processor, the instructions enable an electronic device to recognize a first portion of a 2D image that is to be applied as a background image for a three-dimensional (3D) virtual space.

[0497] According to an embodiment, when executed by a processor, the instructions enable an electronic device to recognize a second portion of a 2D image corresponding to an object of interest.

[0498] According to an embodiment, when executed by a processor, the instructions enable an electronic device to generate an extended image corresponding to a first portion of a 2D image.

[0499] According to an embodiment, when executed by a processor, the instructions enable an electronic device to generate a 3D virtual image by applying a first depth to an extended image and a second depth to a second portion of a 2D image.

[0500] According to an embodiment, when the processor executes instructions, the instructions can cause the electronic device to display a portion of a 3D virtual image based on information related to the user's field of view.

[0501] According to an embodiment, when the processor executes instructions, the instructions can cause the electronic device to display a portion of a 3D virtual image, the size of which is determined based on at least one of the size or orientation of the field of view, corresponding to information related to the field of view.

[0502] According to an embodiment, when the processor executes instructions, the instructions can cause the electronic device to generate an extended image, such that the generated image portion of the extended image includes a region extending from a first boundary of the 2D image along a first direction and a region extending from a second boundary different from the first boundary along a second direction different from the first direction.

[0503] According to an embodiment, the background area of ​​the 3D virtual space can be configured as a curved surface, the distance between the curved surface and a point set in the 3D virtual space being a first depth.

[0504] According to an embodiment, the instructions can enable the electronic device to correct the expanded image to correspond to the shape of the curved surface when executed by the processor.

[0505] According to an embodiment, when the processor executes instructions, the instructions may cause the electronic device, as part of generating an extended image, to generate a region corresponding to a first range belonging to the main movable range of the field of view by performing outward drawing based on a 2D image, and to generate a region corresponding to a second range different from the first range by performing inward drawing based on an image generated in the region corresponding to the first range, at least based on information related to the field of view.

[0506] According to an embodiment, when executed by a processor, the instructions enable an electronic device, as part of generating an extended image, to generate at least one symmetrical image portion at one or more points of a boundary line or imaginary line of an image portion of an extended image generated from a 2D image.

[0507] According to an embodiment, when executed by a processor, the instructions can enable an electronic device to generate an image portion as part of generating an extended image by fading in based on at least one of a boundary line or an imaginary line at one or more points in an image generated from a 2D image.

[0508] According to an embodiment, when executed by a processor, the instructions enable an electronic device to receive 2D images from an external electronic device associated with the user account of the electronic device.

[0509] According to an embodiment, the electronic device may be a head-mounted display (HMD) device.

[0510] According to an embodiment, the electronic device may also include communication circuitry.

[0511] According to an embodiment, when the instructions are executed by the processor, they enable the electronic device to obtain a 2D image from a first external electronic device via a communication circuit.

[0512] According to an embodiment, when the processor executes instructions, the instructions can cause the electronic device to send a 2D image obtained from a first external electronic device to a second external electronic device via a communication circuit.

[0513] According to an embodiment, when the processor executes instructions, the instructions can cause the electronic device to respond to receiving an extended image from a second external electronic device via a communication circuit.

[0514] According to an embodiment, when executed by a processor, the instructions enable an electronic device to determine the attributes of an object of interest included in the second part.

[0515] According to an embodiment, when the processor executes instructions, the instructions can cause the electronic device to add motion graphics effects to the object of interest, at least in part, based on the attributes of the object of interest.

[0516] According to an embodiment, the instructions can enable an electronic device to obtain environmental information when executed by a processor.

[0517] According to an embodiment, the instructions, when executed by a processor, enable an electronic device to determine the attributes of a graphics effect based at least on environmental information.

[0518] According to an embodiment, the instructions can enable an electronic device to identify contact information related to an object of interest when executed by a processor.

[0519] According to an embodiment, the instructions, when executed by a processor, enable an electronic device to provide contact information in response to receiving user input about an object of interest.

[0520] According to an embodiment, the instructions can enable an electronic device to receive events related to contact information when executed by a processor.

[0521] According to an embodiment, instructions can enable an electronic device to add or change graphical effects related to an object of interest based on an event when executed by a processor.

[0522] According to an embodiment, when the processor executes instructions, the instructions can cause the electronic device, as part of generating a 3D virtual image, to determine the attributes of an object of interest included in the second part, and to selectively arrange the second part at one of a first point that is configured to allow interaction with the user or a second point that is configured not to allow interaction with the user, based at least in part on the attributes of the object of interest.

[0523] According to an embodiment, a method for controlling an electronic device may include: obtaining information related to a user's field of view via sensors of the electronic device.

[0524] According to an embodiment, a method for controlling an electronic device may include obtaining a two-dimensional (2D) image.

[0525] According to an embodiment, a method for controlling an electronic device may include: identifying a first portion of a 2D image to be applied as a background image in a three-dimensional (3D) virtual space.

[0526] According to an embodiment, a method for controlling an electronic device may include: identifying a second portion of a 2D image corresponding to an object of interest.

[0527] According to an embodiment, a method for controlling an electronic device may include generating an extended image corresponding to a first portion of a 2D image.

[0528] According to an embodiment, a method for controlling an electronic device may include generating a 3D virtual image by applying a first depth to an extended image and a second depth to a second portion of a 2D image.

[0529] According to an embodiment, a method of controlling an electronic device may include displaying a portion of a 3D virtual image based on information related to the user's field of view.

[0530] According to an embodiment, generating a 3D virtual image may include determining the attributes of an object of interest contained in the second part, and selectively arranging the second part at one of a first point that is configured to allow user interaction or a second point that is configured not to allow user interaction, based at least in part on the attributes of the object of interest.

[0531] According to an embodiment, generating a 3D virtual image may include determining the attributes of an object of interest contained in the second part, and adding motion graphics effects to the object of interest based at least in part on the attributes of the object of interest.

[0532] According to an embodiment, in a non-transitory computer-readable recording medium storing one or more programs, the one or more programs may store instructions that enable an electronic device to obtain information related to a user's field of view via a sensor.

[0533] According to an embodiment, the one or more programs may store instructions that enable an electronic device to acquire two-dimensional (2D) images.

[0534] According to an embodiment, the one or more programs may store instructions to enable an electronic device to recognize a first portion of a 2D image to be applied as a background image for a three-dimensional (3D) virtual space.

[0535] According to an embodiment, the one or more programs may store instructions for enabling an electronic device to perform the following operation: identify a second portion of a 2D image corresponding to an object of interest.

[0536] According to an embodiment, the one or more programs may store instructions to enable an electronic device to generate an extended image corresponding to a first portion of a 2D image.

[0537] According to an embodiment, the one or more programs may store instructions to enable an electronic device to generate a 3D virtual image by applying a first depth to an extended image and a second depth to a second portion of a 2D image.

[0538] According to an embodiment, the one or more programs may store instructions to enable an electronic device to display a portion of a 3D virtual image based on information related to the user's field of view.

[0539] According to an embodiment, the electronic device may include a display, a sensor for sensing a user’s gaze as at least part of the field of view information, a memory for storing a virtual space including a three-dimensional (3D) background region, and a processor operatively connected to the sensor and the memory.

[0540] According to an embodiment, the memory can store instructions that, when executed by a processor, enable the electronic device to acquire a two-dimensional (2D) image.

[0541] According to an embodiment, the memory may store instructions that, when executed by a processor, enable an electronic device to generate an extended image based on the properties of a 2D image. The extended image further includes an image portion generated in an outward direction of the 2D image, such that the image is continuous at a boundary corresponding to at least one edge of the 2D image.

[0542] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to be configured to include an extended image in at least a portion of the 3D background area.

[0543] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to display at least a portion of a virtual space, including a set 3D background area, based on field-of-view information.

[0544] According to an embodiment, the memory may store instructions that, when executed by a processor, enable an electronic device to generate an image portion having a determined size based on at least one of field of view size or field of view orientation, the image portion corresponding to field of view information.

[0545] According to an embodiment, the memory may store instructions that, when executed by a processor, enable an electronic device to generate an extended image, such that the generated image portion of the extended image includes a region extending from a first boundary of a 2D image along a first direction and a region extending from a second boundary different from the first boundary along a second direction different from the first direction.

[0546] According to an embodiment, at least a portion of the 3D background area can be configured as a curved surface shape, which has a substantially the same distance from a point set in the virtual space.

[0547] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to correct an expanded image to correspond to the shape of a curved surface.

[0548] According to an embodiment, the memory may store instructions that, when executed by the processor, enable the electronic device, as part of generating an extended image, to generate an area corresponding to a first range belonging to the main movable range of the field of view by performing out-drawing based on a 2D image, and to generate an area corresponding to a second range different from the first range by performing in-drawing based on an image generated in the area corresponding to the first range based at least on field of view information.

[0549] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device, as part of generating an extended image, to generate at least one symmetrical image portion with respect to one or more points of the image portion, either a boundary line or an imaginary line.

[0550] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to generate an extended image portion by fading in at least one of a boundary line or an imaginary line at one or more points of the image portion.

[0551] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to receive 2D images from an external electronic device associated with the user account of the electronic device.

[0552] According to an embodiment, the electronic device may also include a communication module.

[0553] According to an embodiment, the memory can store instructions that, when executed by a processor, enable the electronic device, as part of generating the extended image, to transmit generation command prompts for generating the extended image to an external device via a communication device.

[0554] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to receive an extended image generated based on a command prompt from an external device.

[0555] According to an embodiment, the command prompt can store instructions for generating an image portion of the 2D image in an outward direction based on the attributes of the 2D image, such that the image is continuous at a boundary corresponding to at least one edge of the 2D image.

[0556] According to an embodiment, the command prompt can store instructions for generating an image portion, the size of which is determined based on at least one of the field size or field direction corresponding to the field of view information.

[0557] According to an embodiment, the command prompt can store instructions for generating an extended image, such that the image portion includes a region extending from a first boundary of the 2D image along a first direction and a region extending from a second boundary different from the first boundary along a second direction different from the first direction.

[0558] According to an embodiment, at least a portion of the 3D background area can be configured as a curved surface shape, which has a substantially the same distance from a point set in the virtual space.

[0559] According to an embodiment, the generated command prompt can store instructions for correcting the extended image to correspond to the shape of a curved surface.

[0560] According to an embodiment, as part of the operation of generating an extended image, a command prompt can store instructions to generate, based on a 2D image and at least based on field of view information, an area corresponding to a first range belonging to the main movable range of the field of view by performing outdrawing, and an area corresponding to a second range different from the first range by performing indrawing based on an image generated in the area corresponding to the first range.

[0561] According to an embodiment, as part of generating an extended image, a generation command prompt may store instructions for generating an image portion that is symmetrical with respect to at least one of virtual lines or boundary lines at one or more points of the image portion.

[0562] According to an embodiment, as part of generating an extended image, the generation command prompt may store instructions for generating the image portion by fading in / out based on at least one of virtual lines or boundary lines at one or more points of the image portion.

[0563] According to an embodiment, a head-mounted display (HMD) device may include: a display; a sensor configured to detect a user's gaze at the HMD as part of field-of-view information; a memory configured to store image information indicating a virtual space including a 3D background area; and a processor operatively connected to the display, the sensor, and the memory.

[0564] According to an embodiment, the memory can store instructions that, when executed by a processor, enable the head-mounted display device to acquire 2D images.

[0565] According to an embodiment, the memory may store instructions that, when executed by the processor, enable the head-mounted display device to display at least a portion of a virtual space via a display, such that an extended image corresponding to a 2D image is included in at least a portion of a 3D background area.

[0566] According to an embodiment, the extended image may include a first image portion corresponding to a 2D image, a second image portion corresponding to an image continuously extending outward from a first boundary portion corresponding to an edge of the 2D image, and a third image portion corresponding to an image continuously extending in a second direction opposite to the first direction from a second boundary portion corresponding to an edge different from the first boundary of the 2D image.

[0567] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the head-mounted display device to generate an extended image based on the properties of a 2D image.

[0568] According to an embodiment, at least a portion of the 3D background area can be configured as a curved surface shape, which has a substantially the same distance from a point set in the virtual space.

[0569] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the head-mounted display to correct an expanded image to correspond to the shape of a curved surface.

[0570] According to an embodiment, the head-mounted display device may also include a communication module.

[0571] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the head-mounted display to acquire 2D images from a first external electronic device via a communication module.

[0572] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the head-mounted display device to transmit 2D images obtained from a first external electronic device to a second external electronic device via a communication module.

[0573] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the head-mounted display device to receive extended images from a second external electronic device via a communication module in response to transmission.

[0574] According to an embodiment, the command prompt can store instructions for generating an image portion of the 2D image in an outward direction based on the attributes of the 2D image, such that the image is continuous at a boundary corresponding to at least one edge of the 2D image.

[0575] According to an embodiment, the extended image received from an external device may be a portion of a generated image, the size of which is determined based on at least one of the field of view size or field of view direction corresponding to the field of view information.

[0576] According to an embodiment, an extended image received from an external device can be generated as including a region extending from a first boundary of a 2D image along a first direction and a region extending from a second boundary different from the first boundary along a second direction different from the first direction.

[0577] According to an embodiment, at least a portion of the 3D background area can be configured as a curved surface shape, which has a substantially the same distance from a point set in the virtual space.

[0578] According to an embodiment, the extended image received from an external device may be an image obtained by correcting the extended image to correspond to the shape of a curved surface.

[0579] According to an embodiment, in an extended image received from an external device, based at least on field of view information, an area corresponding to a first range belonging to the main movable range of the field of view can be generated by performing out-drawing based on a 2D image, and an area corresponding to a second range different from the first range can be generated by performing in-drawing based on an image generated in the area corresponding to the first range.

[0580] According to an embodiment, the extended image received from an external device may be a generated image portion that is symmetrical with respect to at least one of virtual lines or boundary lines at one or more points of the image portion.

[0581] According to an embodiment, the extended image received from an external device may be an image portion generated by fading in / out based on at least one of virtual lines or boundary lines at one or more points of an image portion.

[0582] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the head-mounted display to receive 2D images via an external electronic device associated with a user account of the electronic device.

[0583] According to an embodiment, the electronic device may include a display, a sensor for sensing a user’s gaze as at least a portion of the field of view information, a memory for storing a virtual space including a 3D background area, and a processor.

[0584] According to an embodiment, the memory can store instructions that, when executed by a processor, enable the electronic device to acquire a 2D image.

[0585] According to an embodiment, the memory can store instructions that, when executed by a processor, enable the electronic device to identify background and object regions in a 2D image.

[0586] According to an embodiment, the memory can store instructions that, when executed by a processor, enable the electronic device to separate object regions from a 2D image.

[0587] According to an embodiment, the memory may store instructions that, when executed by a processor, enable an electronic device to generate an extended image based on attributes of a background region. The extended image also includes image portions generated outside the background region to continue at the boundaries of the background region.

[0588] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to be configured to include an extended image in at least a portion of the 3D background area.

[0589] According to an embodiment, the memory can store instructions that, when executed by the processor, enable the electronic device to arrange an object region between a set of points in virtual space and a 3D background region based on field-of-view information.

[0590] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to display at least a portion of a virtual space comprising a set 3D background area and arranged object areas based on field-of-view information.

[0591] According to an embodiment, the memory may store instructions that, when executed by a processor, enable an electronic device to generate an image in an object region by performing in-drawing using at least a portion of a background region adjacent to the object region, as at least part of generating an extended image.

[0592] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to determine the attributes of an object included in an object region.

[0593] According to an embodiment, the memory may store instructions that, when executed by a processor, enable an electronic device to add motion graphics effects to an object, at least in part, based on the object's attributes.

[0594] According to an embodiment, the memory can store instructions that, when executed by a processor, enable the electronic device to obtain environmental information.

[0595] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to determine the attributes of a graphics effect based at least on environmental information.

[0596] According to an embodiment, the memory can store instructions that, when executed by a processor, enable the electronic device to identify contact information associated with an object area.

[0597] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to provide contact information in response to receiving user input on an object area.

[0598] According to an embodiment, the memory may store instructions that, when executed by a processor, enable the electronic device to receive events related to contact information.

[0599] According to an embodiment, the memory can store instructions that, when executed by a processor, enable the electronic device to add or change graphical effects related to an object area based on an event.

[0600] According to an embodiment, the memory may store instructions that, when executed by the processor, enable the electronic device, as part of the arrangement, to determine the attributes of objects included in the object region, and to selectively arrange the object region at one of a first point that allows interaction with the user or a second point that does not allow interaction with the user, based at least in part on the attributes of the objects.

[0601] The method for controlling an electronic device according to an embodiment may include obtaining a 2D image.

[0602] According to an embodiment, the method may further include generating an extended image based on the attributes of the 2D image, the extended image including an image portion generated in the outward direction of the 2D image such that the image is continuous at a boundary corresponding to at least one edge of the 2D image.

[0603] According to an embodiment, the method may further include setting the image to be expanded to be included in at least a portion of the 3D background area.

[0604] According to an embodiment, the method may further include displaying at least a portion of a virtual space including a set 3D background area based on field-of-view information.

[0605] According to an embodiment, generating an extended image can generate an image portion having a determined size based on at least one of the field size or field direction corresponding to the field information.

[0606] According to an embodiment, generating an extended image can generate an extended image such that the image portion includes a region extending from a first boundary of the 2D image along a first direction and a region extending from a second boundary different from the first boundary along a second direction different from the first direction.

[0607] According to an embodiment, at least a portion of the 3D background area can be configured with a curved surface shape to have substantially the same distance from a point set in the virtual space.

[0608] According to an embodiment, generating an extended image may further include correcting the extended image to correspond to the shape of a curved surface.

[0609] According to an embodiment, generating an extended image can be based at least on field of view information, by performing out-drawing based on a 2D image to generate a region corresponding to a first range belonging to the main movable range of the field of view, and by performing in-drawing based on an image generated in the region corresponding to the first range to generate a region corresponding to a second range different from the first range.

[0610] According to an embodiment, generating an extended image can generate at least one symmetrical image portion relative to one or more points of the image portion, either a virtual line or a boundary line.

[0611] According to an embodiment, the extended image can be generated by fading in at least one of the virtual lines or boundary lines at one or more points of the image portion.

[0612] According to an embodiment, obtaining a 2D image can be achieved by receiving the 2D image through an external electronic device associated with a user account of the electronic device.

[0613] According to an embodiment, a method for controlling a head-mounted display (HMD) device may include obtaining a 2D image.

[0614] According to an embodiment, the method may further include: displaying at least a portion of a virtual space on a display such that an extended image corresponding to a 2D image is included in at least a portion of a 3D background area.

[0615] According to an embodiment, the extended image may include a first image portion corresponding to a 2D image, a second image portion corresponding to an image continuously extending outward from a first boundary portion corresponding to an edge of the 2D image, and a third image portion corresponding to an image continuously extending in a second direction opposite to the first direction from a second boundary portion corresponding to an edge different from the first boundary of the 2D image.

[0616] According to an embodiment, a method for controlling a head-mounted display device may include generating an extended image corresponding to the attributes of a 2D image.

[0617] According to an embodiment, at least a portion of the 3D background area can be configured as a curved surface shape, which has a substantially the same distance from a point set in the virtual space.

[0618] According to an embodiment, the method for controlling a head-mounted display device may further include correcting an expanded image to correspond to a curved surface shape.

[0619] According to an embodiment, obtaining a 2D image may include obtaining a 2D image from a first external electronic device via a communication module.

[0620] According to an embodiment, the method for controlling a head-mounted display device may further include: transmitting a 2D image obtained from a first external electronic device to a second external electronic device via a communication module.

[0621] According to an embodiment, the method for controlling a head-mounted display device may further include: receiving an extended image from a second external electronic device via a communication module in response to transmission.

[0622] According to an embodiment, a method for controlling an electronic device may include obtaining a two-dimensional (2D) image.

[0623] According to an embodiment, the method may further include identifying background regions and object regions in a 2D image.

[0624] According to an embodiment, the method may further include separating object regions from a 2D image.

[0625] According to an embodiment, the method may further include: generating an extended image based on the properties of a background region, the extended image including an image portion generated outside the background region to be continuous at the boundary of the background region.

[0626] According to an embodiment, the method may further include setting the image to be expanded to be included in at least a portion of the 3D background area.

[0627] According to an embodiment, the method may further include: arranging the object region between a set of points in virtual space and a 3D background region in relation to field of view information.

[0628] According to an embodiment, the method may further include: displaying at least a portion of a virtual space comprising a set 3D background area and arranged object areas based on field of view information.

[0629] According to an embodiment, generating an extended image can be achieved by performing inpainting in the object region using at least a portion of the background region adjacent to the object region.

[0630] According to an embodiment, the method for controlling an electronic device may further include: determining the attributes of objects included in the object region.

[0631] According to an embodiment, the method for controlling an electronic device may further include adding motion graphics effects to an object, at least in part, based on the object's attributes.

[0632] According to an embodiment, the method for controlling an electronic device may further include obtaining environmental information.

[0633] According to an embodiment, the method for controlling an electronic device may further include determining attributes of a graphic effect based at least on environmental information.

[0634] According to an embodiment, the method for controlling an electronic device may further include identifying contact information associated with a target area.

[0635] According to an embodiment, the method for controlling an electronic device may further include providing contact information in response to receiving user input on an object area.

[0636] According to an embodiment, the method for controlling an electronic device may further include receiving events related to contact information.

[0637] According to an embodiment, the method for controlling an electronic device may further include adding or changing graphical effects related to an object area based on an event.

[0638] According to an embodiment, the arrangement can determine the attributes of the objects included in the object region, and at least in part based on the attributes of the objects, selectively arrange the object region at one of a first point that is set to allow interaction with the user or a second point that is set to not allow interaction with the user.

[0639] According to an embodiment, in a non-transitory computer-readable recording medium storing one or more programs, the one or more programs may store instructions that enable an electronic device to acquire 2D images.

[0640] According to an embodiment, one or more programs may store instructions that enable an electronic device to generate an extended image based on the properties of a 2D image, the extended image further including an image portion generated in an outward direction of the 2D image such that the image is continuous at a boundary corresponding to at least one edge of the 2D image.

[0641] According to an embodiment, one or more programs may store instructions that enable an electronic device to be configured to include an extended image in at least a portion of a 3D background area.

[0642] According to an embodiment, one or more programs may store instructions that enable an electronic device to perform the following operation: display at least a portion of a virtual space including a set 3D background area based on field-of-view information.

[0643] According to an embodiment, one or more programs may store instructions that enable an electronic device to generate an image portion having a determined size based on at least one of field of view size or field of view orientation, the image portion corresponding to field of view information.

[0644] According to an embodiment, one or more programs may store instructions that enable an electronic device to generate an extended image, such that the generated image portion of the extended image includes a region extending from a first boundary of a 2D image along a first direction and a region extending from a second boundary different from the first boundary along a second direction different from the first direction.

[0645] According to an embodiment, at least a portion of the 3D background area can be configured as a curved surface shape, which has a substantially the same distance from a point set in the virtual space.

[0646] According to an embodiment, one or more programs may store instructions that enable an electronic device to correct an expanded image to correspond to a curved surface shape.

[0647] According to an embodiment, one or more programs may store instructions that enable an electronic device, as part of generating an extended image, to generate an area corresponding to a first range belonging to the main movable range of the field of view by performing outdrawing based on a 2D image, and to generate an area corresponding to a second range different from the first range by performing indrawing based on an image generated in the area corresponding to the first range based at least on field of view information.

[0648] According to an embodiment, one or more programs may store instructions that enable an electronic device to: as part of generating an extended image, generate at least one symmetrical image portion at one or more points of the image portion, either a boundary line or an imaginary line.

[0649] According to an embodiment, one or more programs may store instructions that enable an electronic device to generate an extended image by fading in at least one of the boundary lines or imaginary lines at one or more points of the image portion.

[0650] According to an embodiment, one or more programs may store instructions that enable an electronic device to receive 2D images from an external electronic device associated with a user account of the electronic device.

[0651] According to an embodiment, one or more programs may store instructions that enable an electronic device, as part of generating an extended image, to transmit generation command prompts for generating an extended image to an external device via a communication device.

[0652] According to an embodiment, one or more programs may store instructions that enable an electronic device to receive an extended image generated based on a command prompt from an external device.

[0653] According to an embodiment, the command prompt can store instructions for generating an image portion of the 2D image in an outward direction based on the attributes of the 2D image, such that the image is continuous at a boundary corresponding to at least one edge of the 2D image.

[0654] According to an embodiment, the command prompt can store instructions for generating an image portion, the size of which is determined based on at least one of the field size or field direction corresponding to the field of view information.

[0655] According to an embodiment, the command prompt can store instructions for generating an extended image, such that the image portion includes a region extending from a first boundary of the 2D image along a first direction and a region extending from a second boundary different from the first boundary along a second direction different from the first direction.

[0656] According to an embodiment, at least a portion of the 3D background area can be configured as a curved surface shape, which has a substantially the same distance from a point set in the virtual space.

[0657] According to an embodiment, the generated command prompt can store instructions for correcting the extended image to correspond to the shape of a curved surface.

[0658] According to an embodiment, as part of the operation of generating an extended image, a command prompt can store instructions to generate a region corresponding to a first range belonging to the main movable range of the field of view by performing outdrawing based on a 2D image, at least based on field of view information, and to generate a region corresponding to a second range different from the first range by performing inlaying based on an image generated in the region corresponding to the first range.

[0659] According to an embodiment, as part of generating an extended image, a generation command prompt may store instructions for generating an image portion that is symmetrical with respect to at least one of virtual lines or boundary lines at one or more points of the image portion.

[0660] According to an embodiment, as part of generating an extended image, the generation command prompt may store instructions for generating the image portion by fading in / out based on at least one of virtual lines or boundary lines at one or more points of the image portion.

[0661] According to an embodiment, in a non-transitory computer-readable recording medium storing one or more programs, the one or more programs may store instructions that enable a head-mounted display (HMD) device to acquire 2D images.

[0662] According to an embodiment, one or more programs may store instructions that enable a head-mounted display device to display at least a portion of a virtual space via a display, such that an extended image corresponding to a 2D image is included in at least a portion of a 3D background area.

[0663] According to an embodiment, the extended image may include a first image portion corresponding to a 2D image, a second image portion corresponding to an image continuously extending outward from a first boundary portion corresponding to an edge of the 2D image, and a third image portion corresponding to an image continuously extending in a second direction opposite to the first direction from a second boundary portion corresponding to an edge different from the first boundary of the 2D image.

[0664] According to an embodiment, one or more programs may store instructions that enable a head-mounted display device to generate an extended image based on the properties of a 2D image.

[0665] According to an embodiment, at least a portion of the 3D background area can be configured as a curved surface shape, which has a substantially the same distance from a point set in the virtual space.

[0666] According to an embodiment, one or more programs may store instructions that enable the head-mounted display device to correct an expanded image to correspond to the shape of a curved surface.

[0667] According to an embodiment, one or more programs may store instructions that enable the head-mounted display device to obtain 2D images from a first external electronic device via a communication module.

[0668] According to an embodiment, one or more programs may store instructions that enable the head-mounted display device to transmit 2D images obtained from a first external electronic device to a second external electronic device via a communication module.

[0669] According to an embodiment, one or more programs may store instructions that enable the head-mounted display device to receive extended images from a second external electronic device in response to transmission via a communication module.

[0670] According to an embodiment, the command prompt can store instructions for generating an image portion of the 2D image in an outward direction based on the attributes of the 2D image, such that the image is continuous at a boundary corresponding to at least one edge of the 2D image.

[0671] According to an embodiment, the extended image received from an external device may be a portion of a generated image, the size of which is determined based on at least one of the field of view size or field of view direction corresponding to the field of view information.

[0672] According to an embodiment, an extended image received from an external device can be generated as including a region extending from a first boundary of a 2D image along a first direction and a region extending from a second boundary different from the first boundary along a second direction different from the first direction.

[0673] According to an embodiment, at least a portion of the 3D background area can be configured as a curved surface shape, which has a substantially the same distance from a point set in the virtual space.

[0674] According to an embodiment, the extended image received from an external device may be an image obtained by correcting the extended image to correspond to the shape of a curved surface.

[0675] According to an embodiment, in an extended image received from an external device, based at least on field of view information, an area corresponding to a first range belonging to the main movable range of the field of view can be generated by performing out-drawing based on a 2D image, and an area corresponding to a second range different from the first range can be generated by performing in-drawing based on an image generated in the area corresponding to the first range.

[0676] According to an embodiment, the extended image received from an external device may be a generated image portion that is symmetrical with respect to at least one of virtual lines or boundary lines at one or more points of the image portion.

[0677] According to an embodiment, the extended image received from an external device may be an image portion generated by fading in / out based on at least one of virtual lines or boundary lines at one or more points of an image portion.

[0678] According to an embodiment, one or more programs may store instructions that enable a head-mounted display device to receive 2D images via an external electronic device associated with a user account of the electronic device.

[0679] According to an embodiment, in a non-transitory computer-readable recording medium storing one or more programs, the one or more programs may store instructions that enable an electronic device to acquire 2D images.

[0680] According to an embodiment, one or more programs may store instructions that enable an electronic device to identify background and object regions in a 2D image.

[0681] According to an embodiment, one or more programs may store instructions that enable an electronic device to separate an object region from a 2D image.

[0682] According to an embodiment, one or more programs may store instructions that enable an electronic device to generate an extended image based on attributes of a background region, the extended image including image portions generated outside the background region to continue at the boundaries of the background region.

[0683] According to an embodiment, one or more programs may store instructions that enable an electronic device to be configured to include an extended image in at least a portion of a 3D background area.

[0684] According to an embodiment, one or more programs may store instructions that enable an electronic device to arrange an object region between a set of points in virtual space and a 3D background region based on field-of-view information.

[0685] According to an embodiment, one or more programs may store instructions that enable an electronic device to display at least a portion of a virtual space, including a set 3D background area and arranged object areas, based on field-of-view information.

[0686] According to an embodiment, one or more programs may store instructions that enable an electronic device to generate an image in an object region by performing in-line drawing using at least a portion of a background region adjacent to the object region, as at least part of generating an extended image.

[0687] According to an embodiment, one or more programs may store instructions that enable an electronic device to determine the attributes of an object included in an object area.

[0688] According to an embodiment, one or more programs may store instructions that enable an electronic device to add motion graphics effects to an object, at least in part, based on the object's attributes.

[0689] According to an embodiment, one or more programs may store instructions that enable an electronic device to obtain environmental information.

[0690] According to an embodiment, one or more programs may store instructions that enable an electronic device to determine the attributes of a graphic effect based at least on environmental information.

[0691] According to an embodiment, one or more programs may store instructions that enable an electronic device to identify contact information associated with a target area.

[0692] According to an embodiment, one or more programs may store instructions that enable an electronic device to provide contact information in response to receiving user input on an object area.

[0693] According to an embodiment, one or more programs may store instructions that enable an electronic device to receive events related to contact information.

[0694] According to an embodiment, one or more programs may store instructions that enable an electronic device to add or change graphical effects related to an object area based on an event.

[0695] According to an embodiment, one or more programs may store instructions that enable an electronic device, as part of an arrangement, to determine the attributes of objects included in an object region, and to selectively arrange the object region at one of a first point that allows interaction with a user or a second point that does not allow interaction with a user, based at least in part on the attributes of the objects.

[0696] The electronic device according to various embodiments can be one of a variety of types of electronic devices. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. According to embodiments of this disclosure, the electronic device is not limited to those described above.

[0697] It should be understood that the various embodiments of this disclosure and the terminology used therein are not intended to limit the technical features set forth herein to the specific embodiments, but rather to include various changes, equivalents, or substitutions to the respective embodiments. In the description of the drawings, similar reference numerals may be used to refer to similar or related elements. It will be understood that nouns in the singular form corresponding to terms may include one or more things unless the relevant context clearly indicates otherwise. As used herein, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” may include any one or all possible combinations of the items enumerated together with the corresponding phrase among the plurality of phrases. As used herein, terms such as “first” and “second” or “first” and “second” may be used to simply distinguish the respective component from another component and do not limit the component in other respects (e.g., importance or order). It will be understood that, whether the terms “operably” or “communically” are used or not, if an element (e.g., a first element) is referred to as “combined with another element (e.g., a second element),” “combined to another element (e.g., a second element),” “connected to another element (e.g., a second element),” or “connected to another element (e.g., a second element)”, it means that the element can be directly (e.g., wiredly) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0698] As used in connection with various embodiments of this disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms such as "logic," "logic block," "part," or "circuit." A module may be a single integrated component adapted to perform one or more functions, or the smallest unit or part of such a single integrated component. For example, according to embodiments, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0699] The various embodiments set forth herein can be implemented as software (e.g., program 140) containing one or more instructions readable by a machine (e.g., electronic device 101) stored in a storage medium (e.g., internal memory 136 or external memory 138). For example, under the control of a processor, the processor (e.g., processor 120) of the machine (e.g., electronic device 101) can invoke and execute at least one of the one or more instructions stored in the storage medium, with or without the use of one or more other components. This enables the machine to operate to perform at least one function according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. Machine-readable storage media may be provided in the form of non-transitory storage media. The term "non-transitory" simply means that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between data being stored semi-permanently in the storage medium and data being temporarily stored in the storage medium.

[0700] According to embodiments, methods according to various embodiments of this disclosure may be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disk read-only memory (CD-ROM)) or via an app store (e.g., the Play Store). TM The computer program product may be published online (e.g., downloaded or uploaded), or may be distributed directly between two user devices (e.g., smartphones) (e.g., downloaded or uploaded). If published online, at least a portion of the computer program product may be temporarily generated, or at least a portion of the computer program product may be temporarily stored in a machine-readable storage medium (such as the memory of a manufacturer's server, an app store's server, or a forwarding server).

[0701] According to various embodiments, each of the above-described components (e.g., a module or program) may include a single entity or multiple entities, and some of the multiple entities may be separately disposed in different components. According to various embodiments, one or more of the above-described components may be omitted, or one or more other components may be added. Optionally or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform the one or more functions of each of the multiple components in the same or similar manner as the corresponding component of the multiple components performed one or more functions before integration. According to various embodiments, the operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be run in a different order or omitted, or one or more other operations may be added.

[0702] It should be understood that the various embodiments of this disclosure described in the claims and specification can be implemented in hardware, software, or a combination of hardware and software.

[0703] Any such software may be stored in a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores one or more computer programs (software modules) that include computer-executable instructions, which, when executed by one or more processors of an electronic device, cause the electronic device to perform the methods of this disclosure.

[0704] Any such software may be stored in the form of volatile or non-volatile memory, such as a storage device like read-only memory (ROM), whether erasable or rewritable, or in the form of memory such as random access memory (RAM), memory chips, devices, or integrated circuits, or stored on an optical or magnetically readable medium, such as an optical disc (CD), a digital versatile disc (DVD), a magnetic disk, or magnetic tape. It should be understood that storage devices and storage media are various embodiments of non-transitory machine-readable storage suitable for storing one or more computer programs including instructions that, when executed, implement various embodiments of this disclosure. Therefore, various embodiments provide a program and a non-transitory machine-readable storage medium for storing such a program, the program including code for implementing the means or methods claimed as any one of the claims of this specification.

[0705] While this disclosure has been shown and described with reference to various embodiments thereof, those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope of this disclosure as defined by the appended claims and their equivalents.

Claims

1. An electronic device (101), comprising: Display (160); Sensor (176); Memory (130) stores one or more computer programs; as well as One or more processors (120) are communicatively coupled to the display, the sensor, and the memory; as well as The one or more computer programs include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: Information related to the user's field of view is obtained through the sensor. Obtain a 2D image. Identify the first part of the 2D image that will be used as the background image for a 3D virtual space. Identify the second part of the 2D image corresponding to the object of interest. Generate an expanded image corresponding to the first part of the 2D image. A 3D virtual image is generated by applying a first depth to an extended image and a second depth to a second portion of a 2D image. Display portions of the 3D virtual image based on information related to the user's field of view.

2. The electronic device according to claim 1, in, The one or more computer programs further include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: A portion of a 3D virtual image is displayed with a determined size based on at least one of the size or orientation of the field of view, which corresponds to information related to the field of view.

3. The electronic device according to claim 1, in, The one or more computer programs further include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: Generate an expanded image, such that the generated image portion of the expanded image includes: The region extending along a first direction from the first boundary of the 2D image; and The region extending from a second boundary that is different from the first boundary along a second direction that is different from the first direction.

4. The electronic device according to claim 1, in, The background area of ​​the 3D virtual space is configured as a curved surface, and the distance between the curved surface and a point set in the 3D virtual space is a first depth. The one or more computer programs further include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: Correct the expanded image to correspond to the shape of the curved surface.

5. The electronic device according to claim 1, in, The one or more computer programs further include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: As part of generating the extended image, an area corresponding to a first range belonging to the main movable range of the field of view is generated by performing outlining based on the 2D image, and Based at least on information related to the field of view, and based on an image generated in a region corresponding to the first range, an inset drawing is performed to generate a region corresponding to a second range that is different from the first range.

6. The electronic device according to claim 1, in, The one or more computer programs further include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: As part of generating the extended image, at least one symmetrical image portion is generated at one or more points of the image portion of the extended image, which is generated based on a 2D image.

7. The electronic device according to claim 1, in, The one or more computer programs further include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: As part of generating the extended image, the image portion is generated by fading in at least one of the boundary lines or imaginary lines at one or more points of the image portion of the extended image, which is generated based on a 2D image.

8. The electronic device according to claim 1, in, The one or more computer programs further include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: 2D images are received via an external electronic device associated with the user account of the electronic device.

9. The electronic device according to claim 1, wherein, The electronic device is a head-mounted display (HMD) device.

10. The electronic device according to claim 1, further comprising: Communication circuit (190); The one or more computer programs further include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: 2D images are obtained from a first external electronic device via a communication circuit. The 2D image obtained from the first external electronic device is transmitted to the second external electronic device via a communication circuit, and In response to transmission, an extended image is received from a second external electronic device via a communication circuit.

11. The electronic device according to claim 1, in, The one or more computer programs further include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: Determine the attributes of the object of interest included in Part Two, and Add motion graphics effects to the object of interest, at least in part, based on the object's attributes. The one or more computer programs further include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: Obtaining environmental information; and The attributes of the graphic effect should be determined based on at least environmental information.

12. The electronic device according to claim 11, wherein, The one or more computer programs further include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: Identify contact information related to the object of interest, and In response to receiving user input about an object of interest, contact information is provided. The one or more computer programs further include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: Receive events related to contact information, and Add or change graphical effects related to objects of interest based on events.

13. The electronic device according to claim 1, wherein, The one or more computer programs further include computer-executable instructions that, when executed individually or jointly by the one or more processors, cause the electronic device to: As part of generating 3D virtual images, the attributes of the objects of interest included in the second part are determined, and Based at least in part on the attributes of the object of interest, the second part may be selectively placed at either the first point, which is set to allow user interaction, or the second point, which is set to not allow user interaction.

14. A method for controlling an electronic device (101), the method comprising: Information related to the user's field of view is obtained (1010) via the sensor (176) of the electronic device; Obtain a (1020) two-dimensional 2D image; Identify (1030) the first part of the 2D image to be applied as the background image of the three-dimensional 3D virtual space; Identify (1040) the second part of the 2D image corresponding to the object of interest; Generate (1050) an extended image corresponding to the first part of the 2D image; A (1060) 3D virtual image is generated by applying a first depth to an extended image and a second depth to a second portion of a 2D image; as well as Display a portion of the (1070) 3D virtual image based on information related to the user's field of view.

15. One or more non-transitory computer-readable storage media store computer-executable instructions that, when executed individually or jointly by one or more processors of the electronic device (101), cause the electronic device (101) to perform operations, said operations including: Information related to the user's field of view is obtained via sensors (1010); Obtain a (1020) two-dimensional 2D image; Identify (1030) the first part of the 2D image to be applied as the background image of the three-dimensional 3D virtual space; Identify (1040) the second part of the 2D image corresponding to the object of interest; Generate (1050) an extended image corresponding to the first part of the 2D image; A (1060) 3D virtual image is generated by applying a first depth to an extended image and a second depth to a second portion of a 2D image; as well as Display a portion of the (1070) 3D virtual image based on information related to the user's field of view.

Images