Wearable electronic device for displaying virtual object and control method therefor

Abstract

Provided is an electronic device. The electronic device comprises: a display, a sensor, a memory for storing one or more computer programs, and a processor communicatively connected to the display, the sensor, and the memory. The computer programs include computer-executable instructions. The instructions, when executed individually or collectively by the one or more processors, may cause the electronic device to: acquire information related to a field of view of a user via the sensor; acquire a two-dimensional image; identify a first portion, to be applied as a background image to a three-dimensional virtual space, of the two-dimensional image; identify a second portion, corresponding to an object of interest, of the two-dimensional image; generate an extended image corresponding to the first portion of the two-dimensional image; apply a first depth to the extended image; generate a three-dimensional virtual image by applying a second depth to the second portion of the two-dimensional image; and display a portion of the three-dimensional virtual image on the basis of the information related to the field of view of the user.

Description

Wearable electronic device displaying virtual objects and method for controlling the same

[0001] The present disclosure relates to a wearable electronic device displaying a virtual object and a method for controlling the same.

[0002] The variety of services and additional features offered through electronic devices, such as smartphones, is steadily increasing. To enhance the utility of these devices and satisfy the diverse needs of users, telecommunications service providers and electronic device manufacturers are competitively developing electronic devices to offer a variety of features and differentiate themselves from competitors. Consequently, the various functions offered through electronic devices are also becoming increasingly sophisticated.

[0003] The variety of services and additional features offered through wearable electronic devices, such as augmented reality glasses (AR glasses), virtual reality glasses (VR glasses), and head-mounted displays (HMDs), is steadily increasing. To enhance the utility of these devices and satisfy the needs of diverse users, telecommunications service providers and electronic device manufacturers are competitively developing electronic devices that offer diverse features and differentiate themselves from competitors. Consequently, the various functions offered through wearable electronic devices are also becoming increasingly sophisticated.

[0004] AR glasses or VR glasses, worn on the user's head, can provide a realistic experience by displaying virtual images. AR glasses or VR glasses can replace smartphones in various fields, such as gaming, entertainment, education, and social networking services (SNS). Users can access smartphone content and / or content similar to reality through the AR glasses or VR glasses worn on their head.

[0005] The above information is provided solely as background information to aid in understanding the present disclosure. No determination has been made, and no claim is made, as to whether any of the above constitutes prior art relating to the present disclosure.

[0006] According to one embodiment, an electronic device may include a display, a sensor, a processor, and a memory including instructions that, when executed by the processor, cause the electronic device to perform the following operations.

[0007] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to obtain, through the sensor, information related to a user's field of view.

[0008] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to obtain a two-dimensional image.

[0009] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to identify a first portion of the two-dimensional image to be applied as a background image in a three-dimensional virtual space.

[0010] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to identify a second portion of the two-dimensional image corresponding to an object of interest.

[0011] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to generate an extended image corresponding to the first portion of the two-dimensional image.

[0012] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to generate a three-dimensional virtual image by applying a first depth to the extended image and applying a second depth to the second portion of the two-dimensional image.

[0013] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to display a portion of the three-dimensional virtual image based on information related to the user's field of view.

[0014] According to one embodiment, a method of controlling an electronic device may include an operation of obtaining information related to a user's field of view through a sensor of the electronic device.

[0015] According to one embodiment, a method of controlling an electronic device may include an operation of obtaining a two-dimensional image.

[0016] According to one embodiment, a method of controlling an electronic device may include an operation of identifying a first portion of the two-dimensional image to be applied as a background image in a three-dimensional virtual space.

[0017] According to one embodiment, a method of controlling an electronic device may include an operation of identifying a second portion of the two-dimensional image corresponding to an object of interest.

[0018] According to one embodiment, a method of controlling an electronic device may include generating an extended image corresponding to the first portion of the two-dimensional image.

[0019] According to one embodiment, a method of controlling an electronic device may include applying a first depth to the extended image and applying a second depth to the second portion of the two-dimensional image to generate a three-dimensional virtual image.

[0020] According to one embodiment, a method of controlling an electronic device may include an action of displaying a portion of the three-dimensional virtual image based on information related to the field of view of the user.

[0021] According to one embodiment, a non-transitory computer-readable recording medium storing one or more programs may store instructions that cause an electronic device to obtain information related to a user's field of view through the sensor.

[0022] According to one embodiment, the one or more programs may store instructions for causing the electronic device to obtain a two-dimensional image.

[0023] According to one embodiment, the one or more programs may store instructions for the electronic device to identify a first portion of the two-dimensional image to be applied as a background image in a three-dimensional virtual space.

[0024] According to one embodiment, the one or more programs may store instructions for causing the electronic device to identify a second portion of the two-dimensional image corresponding to an object of interest.

[0025] According to one embodiment, the one or more programs cause the electronic device to generate an extended image corresponding to the first portion of the two-dimensional image,

[0026] In one embodiment, the one or more programs may store instructions causing the electronic device to apply a first depth to the extended image and apply a second depth to the second portion of the two-dimensional image to generate a three-dimensional virtual image.

[0027] According to one embodiment, the one or more programs may store instructions that cause the electronic device to display a portion of the three-dimensional virtual image based on information related to the user's field of view.

[0028] According to one embodiment, an electronic device may include a display, a sensor for detecting a user's gaze as at least a portion of the field of view information, a memory for storing a virtual space including a three-dimensional background area, and a processor operatively connected to the sensor and the memory.

[0029] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to obtain a two-dimensional image.

[0030] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to generate an extended image that further includes an image portion generated in an outer direction of the two-dimensional image such that the image is continuous at a boundary corresponding to at least one of the corners of the two-dimensional image, based on an attribute of the two-dimensional image.

[0031] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to set at least a portion of the three-dimensional background area to include the expanded image.

[0032] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to display at least a portion of the virtual space including the set three-dimensional background area based on the field of view information.

[0033] According to one embodiment, a head-mounted display device (HMD) may include a display, a sensor configured to detect a gaze of a user of the HMD as part of field of view information, a memory configured to store image information representing a virtual space including a three-dimensional background area, and a processor operatively connected to the display, the sensor, and the memory.

[0034] In one embodiment, the memory may store instructions that, when executed by the processor, cause the head-mountable display device to acquire a two-dimensional image.

[0035] In one embodiment, the memory may store instructions that, when executed by the processor, cause the head-mounted display device to display at least a portion of the virtual space through the display such that an extended image corresponding to the two-dimensional image is included in at least a portion of the three-dimensional background area.

[0036] According to one embodiment, an electronic device may include a display, a sensor for detecting a user's gaze as at least part of the field of view information, a memory for storing a virtual space including a three-dimensional background area, and a processor.

[0037] According to one embodiment, the memory, when executed by the processor, causes the electronic device to obtain a two-dimensional image.

[0038] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to recognize a background area and an object area within the two-dimensional image.

[0039] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to separate the object region from the two-dimensional image.

[0040] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to generate an extended image that further includes a portion of the image generated outside the background area so as to extend beyond the boundary of the background area, based on an attribute of the background area.

[0041] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to set at least a portion of the three-dimensional background area to include the expanded image.

[0042] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to position the object area between a point set within the virtual space and the three-dimensional background area in relation to the field of view information.

[0043] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to display at least a portion of the virtual space including the set three-dimensional background area and the placed object area based on the field of view information.

[0044] According to one embodiment, a method of controlling an electronic device may include an operation of obtaining a two-dimensional image.

[0045] According to one embodiment, a method of controlling an electronic device may include generating an extended image further including an image portion generated in an outer direction of the two-dimensional image such that the image is continuous at a boundary corresponding to at least one of the corners of the two-dimensional image, based on an attribute of the two-dimensional image.

[0046] According to one embodiment, a method of controlling an electronic device may include an operation of setting the expanded image to be included in at least a portion of the three-dimensional background area.

[0047] According to one embodiment, a control method of an electronic device may include an operation of displaying at least a portion of the virtual space including the set three-dimensional background area based on the field of view information.

[0048] According to one embodiment, a method of controlling a head-mounted display device (HMD) may include an operation of acquiring a two-dimensional image.

[0049] According to one embodiment, the control method of the head-mounted display device may include an operation of displaying at least a portion of the virtual space through the display such that an extended image corresponding to the two-dimensional image is included in at least a portion of the three-dimensional background area.

[0050] According to one embodiment, a method of controlling an electronic device may include an operation of obtaining a two-dimensional image.

[0051] According to one embodiment, the control method of the electronic device may include an operation of recognizing a background area and an object area within the two-dimensional image.

[0052] According to one embodiment, the control method of the electronic device may include an operation of separating the object area from the two-dimensional image.

[0053] According to one embodiment, the control method of the electronic device may include generating an extended image that further includes a portion of the image generated outside the background area so as to continue from a boundary of the background area, based on an attribute of the background area.

[0054] According to one embodiment, the control method of the electronic device may include an operation of setting the expanded image to be included in at least a portion of the three-dimensional background area.

[0055] According to one embodiment, the control method of the electronic device may include an operation of positioning the object area between a point set within the virtual space and the three-dimensional background area in relation to the field of view information.

[0056] According to one embodiment, the control method of the electronic device may include an operation of displaying at least a portion of the virtual space including the set three-dimensional background area and the placed object area based on the field of view information.

[0057] According to one embodiment, a non-transitory computer-readable recording medium storing one or more programs may store instructions that cause an electronic device to obtain a two-dimensional image.

[0058] In one embodiment, the one or more programs may store instructions that cause the electronic device to generate an extended image that further includes an image portion generated in an outer direction of the two-dimensional image such that the image continues at a boundary corresponding to at least one of the corners of the two-dimensional image, based on properties of the two-dimensional image.

[0059] According to one embodiment, the one or more programs may store instructions that cause the electronic device to set at least a portion of the three-dimensional background area to include the extended image.

[0060] According to one embodiment, the one or more programs may store instructions that cause the electronic device to display at least a portion of the virtual space including the set three-dimensional background area based on the field of view information.

[0061] According to one embodiment, a non-transitory computer-readable recording medium storing one or more programs, wherein the one or more programs can store instructions that cause a head-mounted display device (HMD) to acquire a two-dimensional image.

[0062] According to one embodiment, the one or more programs may store instructions that cause the head-mounted display device to display at least a portion of the virtual space through the display such that an extended image corresponding to the two-dimensional image is included in at least a portion of the three-dimensional background area.

[0063] In one embodiment, a non-transitory computer-readable recording medium storing one or more programs, wherein the one or more programs cause the electronic device to obtain a two-dimensional image.

[0064] According to one embodiment, the one or more programs may store instructions that cause the electronic device to recognize a background area and an object area within the two-dimensional image.

[0065] According to one embodiment, the one or more programs may store instructions that cause the electronic device to separate the object region from the two-dimensional image.

[0066] In one embodiment, the one or more programs may store instructions that cause the electronic device to generate an extended image that further includes a portion of the image generated outside the background area so as to extend beyond the boundary of the background area, based on properties of the background area.

[0067] In one embodiment, the one or more programs may store instructions that cause the electronic device to set the expanded image to be included in at least a portion of the three-dimensional background area.

[0068] According to one embodiment, the one or more programs may store instructions that cause the electronic device to place the object area between a point set within the virtual space and the three-dimensional background area in relation to the field of view information.

[0069] According to one embodiment, the one or more programs may store instructions that cause the electronic device to display at least a portion of the virtual space including the set three-dimensional background area and the placed object area based on the field of view information.

[0070] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0071] FIG. 1A is a block diagram of an electronic device within a network environment according to one embodiment of the present disclosure.

[0072] FIG. 1b is a block diagram illustrating an operation of obtaining a result from a generative artificial intelligence (AI) of an electronic device according to one embodiment of the present disclosure.

[0073] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0074] FIG. 3A is a perspective view showing the front of a wearable electronic device according to one embodiment of the present disclosure.

[0075] FIG. 3b is a perspective view showing the rear side of a wearable electronic device according to one embodiment of the present disclosure.

[0076] FIG. 4 is a perspective view of a wearable electronic device according to one embodiment of the present disclosure.

[0077] FIG. 5 is a flowchart illustrating an operation of displaying an extended image generated based on a two-dimensional image in a three-dimensional background area of ​​an electronic device according to one embodiment of the present disclosure.

[0078] FIG. 6A is a flowchart illustrating an operation of generating an extended image based on a two-dimensional image of an electronic device according to one embodiment of the present disclosure.

[0079] FIG. 6b is a diagram for explaining an operation of generating an extended image based on a two-dimensional image of an electronic device according to one embodiment of the present disclosure.

[0080] FIG. 7A is a flowchart illustrating an operation of generating an extended image with both ends connected based on a two-dimensional image of an electronic device according to one embodiment of the present disclosure.

[0081] FIG. 7b is a diagram for explaining an operation of generating an extended image with both ends connected based on a two-dimensional image of an electronic device according to one embodiment of the present disclosure.

[0082] FIG. 8A is a drawing for explaining an operation of connecting both ends of an extended image of an electronic device according to one embodiment of the present disclosure.

[0083] FIG. 8b is a drawing for explaining an operation of connecting both ends of an extended image of an electronic device according to one embodiment of the present disclosure.

[0084] FIG. 9 is a diagram illustrating an operation of generating a ceiling and floor of an extended image of an electronic device according to one embodiment of the present disclosure.

[0085] FIG. 10 is a flowchart illustrating an operation of arranging objects included in an extended image at different depths in an electronic device according to an embodiment of the present disclosure.

[0086] FIG. 11 is a flowchart illustrating an operation of arranging objects included in an extended image at different depths in an electronic device according to an embodiment of the present disclosure.

[0087] FIG. 12A is a diagram for explaining an operation of arranging objects included in an extended image at different depths according to an embodiment of the present disclosure in an electronic device.

[0088] FIG. 12b is a drawing for explaining the depth of a three-dimensional space according to one embodiment of the present disclosure.

[0089] FIG. 12c is a drawing for explaining objects of an extended image arranged at different depths according to one embodiment of the present disclosure.

[0090] FIG. 12d is a drawing illustrating objects of an extended image arranged at different depths according to one embodiment of the present disclosure.

[0091] FIG. 13A is a drawing for explaining an operation of providing interaction to a placed object in an electronic device according to one embodiment of the present disclosure.

[0092] FIG. 13b is a drawing for explaining an operation of providing interaction to a placed object in an electronic device according to one embodiment of the present disclosure.

[0093] FIG. 13c is a drawing for explaining an operation of providing interaction to a placed object in an electronic device according to one embodiment of the present disclosure.

[0094] FIG. 13d is a drawing for explaining an operation of providing interaction to a placed object in an electronic device according to one embodiment of the present disclosure.

[0095] FIG. 14 is a drawing for explaining an operation of providing interaction to a placed object in an electronic device according to one embodiment of the present disclosure.

[0096] FIG. 15 is a drawing for explaining an operation of providing interaction to a placed object in an electronic device according to one embodiment of the present disclosure.

[0097] FIG. 16 is a flowchart illustrating an operation of mapping a placed object and audio in an electronic device according to one embodiment of the present disclosure.

[0098] FIG. 17A is a diagram illustrating an operation of mapping a placed object and audio in an electronic device according to one embodiment of the present disclosure.

[0099] FIG. 17b is a diagram illustrating an operation of mapping a placed object and audio in an electronic device according to one embodiment of the present disclosure.

[0100] FIG. 17c is a diagram illustrating an operation of mapping a placed object and audio in an electronic device according to one embodiment of the present disclosure.

[0101] FIG. 17d is a diagram for explaining an operation of mapping a placed object and audio in an electronic device according to one embodiment of the present disclosure.

[0102] FIG. 18A is a drawing for explaining an operation of changing a placed object of an electronic device according to one embodiment of the present disclosure.

[0103] FIG. 18b is a drawing for explaining an operation of changing a placed object of an electronic device according to one embodiment of the present disclosure.

[0104] FIG. 18c is a drawing for explaining an operation of changing a placed object of an electronic device according to one embodiment of the present disclosure.

[0105] FIG. 19 is a drawing for explaining an operation of changing a placed object of an electronic device according to one embodiment of the present disclosure.

[0106] FIG. 20A is a drawing for explaining an operation of changing a placed object of an electronic device according to one embodiment of the present disclosure.

[0107] FIG. 20b is a drawing for explaining an operation of changing a placed object of an electronic device according to one embodiment of the present disclosure.

[0108] FIG. 20c is a drawing for explaining an operation of changing a placed object of an electronic device according to one embodiment of the present disclosure.

[0109] FIG. 21A is a drawing for explaining an operation of placing an object on a lock screen or an always-on display (AOD) screen of an electronic device according to one embodiment of the present disclosure.

[0110] FIG. 21b is a drawing for explaining an operation of placing an object on a lock screen or AOD screen of an electronic device according to an embodiment of the present disclosure.

[0111] FIG. 22 is a drawing for explaining a virtual space displayed according to rotation of an electronic device according to one embodiment of the present disclosure.

[0112] FIG. 23A is a diagram for explaining an operation of displaying an image generated based on a keyword in an external electronic device as a three-dimensional virtual image in an electronic device according to one embodiment of the present disclosure.

[0113] FIG. 23b is a diagram for explaining an operation of applying a changed image based on a keyword to an external electronic device while a three-dimensional virtual image is displayed on an electronic device according to one embodiment of the present disclosure.

[0114] The following description, with reference to the accompanying drawings, is provided to facilitate a comprehensive understanding of various embodiments of the present invention as defined by the claims and their equivalents. While it includes numerous specific details to facilitate understanding, these should be considered illustrative only. Accordingly, those skilled in the art will recognize that various changes and modifications to the various embodiments described herein can be made without departing from the scope and spirit of the present invention. Furthermore, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0115] The terms and words used in the following description and claims are not limited to their bibliographic meanings, but are merely used to ensure a clear and consistent understanding of the disclosure by the inventor. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present invention is provided for illustrative purposes only and is not intended to limit the disclosure as defined by the appended claims and their equivalents.

[0116] It should be understood that the singular forms "a" and "an" include plural references unless the context clearly dictates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more of those surfaces.

[0117] It should be recognized that the blocks and combinations of flowcharts in each flowchart can be executed by one or more computer programs containing instructions. The entirety of one or more computer programs may be stored in a single memory device, or the one or more computer programs may be divided into different parts stored in different memory devices.

[0118] Any of the functions or operations described herein may be performed by a single processor or a combination of processors. A single processor or a combination of processors is a circuit that performs processing and includes an application processor (AP, e.g., a central processing unit (CPU)), a communication processor (CP, e.g., a modem), a graphics processing unit (GPU), a neural processing unit (NPU) (e.g., an artificial intelligence (AI) chip), a Wi-Fi chip, a Bluetooth® chip, a global positioning system (GPS) chip, a near-field communication (NFC) chip, a connection chip, a sensor controller, a touch controller, a fingerprint sensor controller, a display driver integrated circuit (IC), an audio codec chip, a universal serial bus (USB) controller, a camera controller, an image processing IC, a microprocessor unit (MPU), a system on a chip (SoC), an IC, or similar circuits.

[0119] FIG. 1A is a block diagram of an electronic device within a network environment according to one embodiment of the present disclosure.

[0120] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an external electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an external electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the external electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0121] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0122] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0123] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0124] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0125] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0126] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0127] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0128] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., an external electronic device (102)) (e.g., a speaker or headphones) directly or wirelessly connected to the electronic device (101).

[0129] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0130] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) to an external electronic device (e.g., the external electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0131] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., an external electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0132] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0133] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0134] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0135] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0136] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., external electronic device (102), external electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a fifth generation (5G) network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network or a wide area network)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0137] The wireless communication module (192) can support the 5G network and next-generation communication technologies after the 4th generation (4G) network, for example, new radio access technology (NR). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency communications (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., millimeter wave (mmWave) band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0138] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0139] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0140] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0141] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed by one or more of the external electronic devices (102, 104) or the server (108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0142] FIG. 1b is a block diagram illustrating an operation of obtaining a result from a generative AI of an electronic device according to one embodiment of the present disclosure.

[0143] Referring to FIG. 1B, the electronic device (101) may include a processor (120) (e.g., the processor (120) of FIGS. 1A and 1B) and a memory (130) (e.g., the memory (130) of FIGS. 1A and 1B).

[0144] According to one embodiment, the processor (120) may obtain the generation result (16, 18) by using a generation AI (131) stored in a memory (130) or by using a generation AI server (108) that is an external electronic device (e.g., the server (108) of FIGS. 1A and 1B). According to one embodiment, the generation AI server (108) may be a generation AI stored in an external electronic device that is a terminal device.

[0145] According to one embodiment, the processor (120) may transmit a prompt (15) for generating an extended image to a generative AI (131) stored in a memory (130), and receive a generated result (16) corresponding to the prompt (15) from the generative AI (131). For example, the generated result (16) may be an extended image obtained based on a two-dimensional image through an out-painting and / or in-painting technique.

[0146] According to one embodiment, the processor (120) may transmit a prompt (17) for generating an extended image to a generative AI server (108), which is an external electronic device, and receive a generated result (18) corresponding to the prompt (17) from the generative AI server (108). For example, the generated result (18) may be an extended image obtained based on a two-dimensional image through out-painting and / or in-painting techniques.

[0147] According to one embodiment, the operation of generating an extended image is described in more detail with reference to FIGS. 5, 6a, 6b, 7a, 8b, 8a, 8b and 9 below.

[0148] FIG. 2 is a perspective view illustrating the internal configuration of a wearable electronic device according to one embodiment of the present disclosure.

[0149] Referring to FIG. 2, a wearable electronic device (200) according to one embodiment may include at least one of a light output module (211), a display member (201), and a camera module (250).

[0150] According to one embodiment, the light output module (211) may include a light source capable of outputting an image and a lens for guiding the image to the display member (201). According to one embodiment, the light output module (211) may include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0151] According to one embodiment, the display member (201) may include an optical waveguide (e.g., a waveguide). According to one embodiment, an output image of the optical output module (211) incident on one end of the optical waveguide may be propagated inside the optical waveguide and provided to a user. According to one embodiment, the optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). For example, the optical waveguide may guide the output image of the optical output module (211) to the user's eyes by using at least one diffractive element or reflective element.

[0152] In one embodiment, the camera module (250) can capture still images and / or moving images. In one embodiment, the camera module (250) is disposed within a lens frame and can be disposed around the display member (201).

[0153] According to one embodiment, the first camera module (251) can capture and / or recognize the trajectory of the user's eye (e.g., pupil, iris) or gaze. According to one embodiment, the first camera module (251) can periodically or aperiodically transmit information related to the trajectory of the user's eye or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIGS. 1A and 1B ).

[0154] According to one embodiment, the second camera module (253) can capture an external image.

[0155] According to one embodiment, the third camera module (255) can be used for hand detection and tracking, and user gesture (e.g., hand movement) recognition. According to one embodiment, the third camera module (255) can be used for 3 degrees of freedom (3DoF), 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. According to one embodiment, the second camera module (253) can also be used for hand detection and tracking, and user gesture recognition. According to one embodiment, at least one of the first camera module (251) to the third camera module (255) can be replaced with a sensor module (e.g., a LiDAR sensor). For example, the sensor module can include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0156] FIG. 3A is a perspective view showing the front of a wearable electronic device according to one embodiment of the present disclosure.

[0157] FIG. 3b is a perspective view showing the rear side of a wearable electronic device according to one embodiment of the present disclosure.

[0158] Referring to FIGS. 3A and 3B, in one embodiment, camera modules (311, 312, 313, 314, 315, 316) and / or depth sensors (317) for obtaining information related to the surrounding environment of the wearable electronic device (300) may be arranged on the first surface (310) of the housing.

[0159] In one embodiment, the camera modules (311, 312) can acquire images related to the environment surrounding the wearable electronic device.

[0160] In one embodiment, the camera modules (313, 314, 315, 316) can acquire images while the wearable electronic device is worn by a user. The camera modules (313, 314, 315, 316) can be used for hand detection and tracking, and user gesture (e.g., hand movement) recognition. The camera modules (313, 314, 315, 316) can be used for 3DoF, 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. In one embodiment, the camera modules (311, 312) can also be used for hand detection and tracking, and user gesture.

[0161] In one embodiment, a depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Instead of or in addition to the depth sensor (217), the camera modules (213, 214, 215, 216) may determine the distance to an object.

[0162] According to one embodiment, a camera module (325, 326) for facial recognition and / or a display (321) (and / or a lens) may be disposed on the second side (320) of the housing.

[0163] In one embodiment, a face recognition camera module (325, 326) adjacent to the display may be used to recognize a user's face, or may recognize and / or track both eyes of the user.

[0164] In one embodiment, the display (321) (and / or lens) may be disposed on the second side (320) of the wearable electronic device (300). In one embodiment, the wearable electronic device (300) may not include camera modules (315, 316) among the plurality of camera modules (313, 314, 315, 316). Although not illustrated in FIGS. 3A and 3B , the wearable electronic device (300) may further include at least one of the configurations illustrated in FIG. 2 .

[0165] As described above, according to one embodiment, the wearable electronic device (300) may have a form factor for being worn on a user's head. The wearable electronic device (300) may further include a strap and / or a wearing member for being secured to a body part of the user. The wearable electronic device (300) may provide a user experience based on augmented reality, virtual reality, and / or mixed reality while being worn on the user's head.

[0166] FIG. 4 is another perspective view of an electronic device according to one embodiment of the present disclosure.

[0167] Referring to FIG. 4, the electronic device (400) may be a head-mounted device (HMD) capable of providing images in front of the user's eyes. The configuration of the electronic device (400) of FIG. 4 may be entirely or partially identical to the configuration of the electronic device (200) of FIG. 2.

[0168] According to one embodiment, the electronic device (400) may include a housing (410, 420, 430) that may form the exterior of the electronic device (400) and provide a space in which components of the electronic device (400) may be placed.

[0169] In one embodiment, the electronic device (400) may include a first housing (410) that can surround at least a portion of a user's head. In one embodiment, the first housing (410) may include a first side (400a) that faces the exterior of the electronic device (400) (e.g., in the +Z direction).

[0170] In one embodiment, the first housing (410) can surround at least a portion of the internal space (I). For example, the first housing (410) can include a second surface (400b) facing the internal space (I) of the electronic device (400) and a third surface (400c) opposite the second surface (400b). In one embodiment, the first housing (410) can be combined with the third housing (430) to form a closed curve shape surrounding the internal space (I).

[0171] According to one embodiment, the first housing (410) can accommodate at least some of the components of the electronic device (400). For example, an optical output module and a circuit board can be disposed within the first housing (410).

[0172] According to one embodiment, the electronic device (400) may include one display member (440) corresponding to the left and right eyes. The display member (440) may be disposed in the first housing (410). The configuration of the display member (440) of FIG. 5 may be all or part of the same as the configuration of the display member (201) of FIG. 2.

[0173] According to one embodiment, the electronic device (400) may include a second housing (420) that can be mounted on a user's face. According to one embodiment, the second housing (420) may include a fourth surface (400d) that can at least partially face the user's face. According to one embodiment, the fourth surface (400d) may be a surface facing the internal space (I) of the electronic device (400) (e.g., the -Z direction). According to one embodiment, the second housing (420) may be coupled with the first housing (410).

[0174] In one embodiment, the electronic device (400) may include a third housing (430) that may be mounted on the back of the user's head. In one embodiment, the third housing (430) may be coupled to the first housing (410). In one embodiment, the third housing (430) may accommodate at least some of the components of the electronic device (400). For example, a battery (e.g., battery (189) of FIGS. 1A and 1B) may be placed within the third housing (430).

[0175] FIG. 5 is a flowchart illustrating an operation of displaying an extended image generated based on a two-dimensional image in a three-dimensional background area of ​​an electronic device according to one embodiment of the present disclosure.

[0176] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0177] Referring to FIG. 5, in operation 510, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) can obtain a two-dimensional image.

[0178] In one embodiment, the electronic device may include a display. For example, the electronic device may include a smartphone, a tablet PC, smart glasses, smart contact lenses, or a head-mounted display device.

[0179] In one embodiment, the electronic device may receive a two-dimensional image from an external electronic device. For example, the electronic device may receive the two-dimensional image through an external electronic device associated with a user account of the electronic device.

[0180] In one embodiment, the electronic device may acquire a two-dimensional image stored in a memory of the electronic device (e.g., memory (130) of FIGS. 1A and 1B). For example, the two-dimensional image may be captured by a camera included in the electronic device (e.g., camera module (180) of FIGS. 1A and 1B), acquired through a screen capture, or downloaded and stored in the memory.

[0181] In one embodiment, in operation 520, the electronic device may generate an extended image that further includes a portion of the image generated in an outer direction of the two-dimensional image such that the image is continuous at a boundary corresponding to at least one of the edges of the two-dimensional image, based on properties of the two-dimensional image.

[0182] In one embodiment, an electronic device may use a two-dimensional image as input data through an artificial intelligence model to obtain an expanded image as output data. For example, the artificial intelligence model may be a generative artificial intelligence (AI) model. In one embodiment, the AI ​​model may be stored in a memory of the electronic device (e.g., memory (130) of FIGS. 1A and 1B ) or may be stored in an external electronic device (e.g., server) (e.g., server (108) of FIGS. 1A and 1B ).

[0183] In one embodiment, an electronic device can acquire an expanded image based on a two-dimensional image using out-painting technology. Out-painting technology is a technology that generates an outer region beyond the image's border using an artificial intelligence model.

[0184] According to one embodiment, the electronic device can obtain an extended image using a two-dimensional image based on field of view information.

[0185] In one embodiment, the field of view information may include the user's gaze direction and / or field of view (FOV). In one embodiment, the field of view information may be acquired through a sensor of the electronic device (e.g., the sensor module (176) of FIGS. 1A and 1B ). For example, the sensor may include a camera for detecting the movement of the user's eyes and / or a motion sensor for detecting the movement of the user's head.

[0186] For example, the electronic device may set the size of the out-of-painting area (e.g., a portion of the image) to be larger than the user's field of view (FOV). In one embodiment, the electronic device may generate an image portion corresponding to a size determined based on at least one of the size of the field of view corresponding to the field of view information or the direction of the field of view.

[0187] In one embodiment, the electronic device can extend in at least two directions the border of the two-dimensional image, which is the original image.

[0188] In one embodiment, the electronic device may generate an extended image such that the image portion includes a region extending in a first direction from a first boundary of the two-dimensional image and a region extending in a second direction different from the first direction from a second boundary different from the first boundary. For example, the electronic device may generate the extended image such that the image portion extends leftward from a left boundary of the two-dimensional image and an image portion extends rightward from a right boundary of the two-dimensional image.

[0189] In one embodiment, as part of the operation of generating an extended image, the electronic device may perform outpainting on the two-dimensional image to generate an area corresponding to a first range within a primary motion range of the field of view based at least on field of view information. The primary motion range of the field of view may include a range of the field of view that can be changed based on movement of the user's eyes or head. For example, the primary motion range of the field of view may be a range of 240 degrees in total, including 120 degrees to the left and 120 degrees to the right based on the user's frontal field of view. This is merely an example and is not limited thereto.

[0190] According to one embodiment, the electronic device may generate an area corresponding to a second range, which is different from the first range, by performing in-painting based on an image generated in the area corresponding to the first range, based on at least the field of view information. The in-painting technique is a technique for filling in damaged or empty portions in an image or deleting objects included in the image using an artificial intelligence model. According to one embodiment, the electronic device may generate an image of an area corresponding to the second range based on images of both ends of the area corresponding to the first range. According to one embodiment, the extended image may include a 360-degree image. According to one embodiment, the 360-degree image may not include a ceiling image or a floor image.

[0191] According to one embodiment, the operation of obtaining an extended image by performing out-painting and in-painting based on field of view information is described in more detail below with reference to FIGS. 6a, 6b, 7a, and 7b.

[0192] In one embodiment, the electronic device may generate an image portion by connecting both edges of an image generated by out-painting without performing in-painting. In one embodiment, the expanded image may include a 360-degree image. In one embodiment, the 360-degree image may not include a ceiling image or a floor image.

[0193] For example, an electronic device may connect the left and right boundaries of an image portion generated by outpainting to achieve bilateral symmetry. For example, as part of the operation of generating an extended image, the electronic device may generate an image portion so that it is symmetrical with respect to at least one imaginary line or boundary line at one or more points in the image portion.

[0194] According to one embodiment, the operation of connecting the boundaries of an image generated through left-right symmetry will be described below with reference to FIGS. 7a and 7b.

[0195] According to one embodiment, the electronic device may adjust the left and right boundaries of an image portion generated by out-painting by fading in and out. For example, as part of an operation of generating an extended image, the electronic device may generate an image portion by fading in and out based on at least one of a virtual line or boundary line at one or more points of the image portion. The fading in and out processing may be performed by adjusting the position of the images in the two boundary areas to overlap them, or by repeating the position adjustment and overlapping so that the images are naturally connected.

[0196] According to one embodiment, the electronic device may adjust the left and right boundaries of an image portion generated by out painting by connecting them symmetrically left and right and then fading in / out the connected area.

[0197] According to one embodiment, the operation of connecting the boundaries of an image generated through fade in / out processing will be described below with reference to FIGS. 8a and 8b.

[0198] In one embodiment, an electronic device may store a virtual space including a three-dimensional background area. In one embodiment, the virtual space may include a sphere or a hemisphere. In one embodiment, the three-dimensional background area may be included in a boundary representing the maximum size of the virtual space. For example, the three-dimensional background area may be included in a boundary of a sphere or a hemisphere of a virtual space.

[0199] In one embodiment, at least a portion of the 3D background area may be configured as a curved surface with a substantially equal distance from a point set within the virtual space. For example, the point set within the virtual space may be the center point of the virtual space. The center point of the virtual space may be the user's location within the virtual space.

[0200] In one embodiment, the electronic device may correct the expanded image to correspond to a curved shape. For example, the electronic device may correct the expanded image to correspond to a three-dimensional background area. In one embodiment, the electronic device may correct the expanded image to correspond to the shape of the virtual space.

[0201] In one embodiment, while the electronic device is described above as generating an extended image, the electronic device may also generate an extended image or receive an extended image generated by an external electronic device.

[0202] For example, an electronic device may transmit a two-dimensional image to a second external electronic device (e.g., a server) via a communication module (e.g., the communication module (190) of FIGS. 1A and 1B). In one embodiment, the two-dimensional image may be received from the first external electronic device (e.g., the electronic device), acquired via a camera of the electronic device, or stored in memory.

[0203] In one embodiment, the electronic device may, in response to the transmission, receive an expanded image from a second external electronic device via the communication module. The expanded image may be an image expanded from a two-dimensional image.

[0204] According to one embodiment, the extended image received from the external electronic device may be generated by the second external electronic device through operations identical to at least some of the extended image generating operations of the electronic device described above.

[0205] In one embodiment, as part of the operation of generating an extended image, the electronic device may transmit a generation command prompt for generating the extended image to an external electronic device (e.g., a server) via a communication module. In one embodiment, the electronic device may receive the extended image generated based on the generation command prompt from the external electronic device.

[0206] According to one embodiment, the generation command prompt may store instructions for generating an extended image that further includes a portion of the image generated in an outer direction of the two-dimensional image such that the image is continuous at a boundary corresponding to at least one of the edges of the two-dimensional image, based on properties of the two-dimensional image.

[0207] According to one embodiment, the generation command prompt may store instructions for generating an image portion with a size determined based on at least one of a size of the field of view corresponding to the field of view information or a direction of the field of view.

[0208] According to one embodiment, the generation command prompt may store instructions that generate an extended image such that the image portion includes a region extending in a first direction from a first boundary of the two-dimensional image and a region extending in a second direction different from the first direction from a second boundary different from the first boundary.

[0209] In one embodiment, the generation command prompt may store instructions for correcting an extended image to correspond to a curved shape. In one embodiment, the curved shape may be the shape of a three-dimensional background area in a virtual space.

[0210] According to one embodiment, the generation command prompt may store instructions that, as part of the operation of generating an extended image, generate an area corresponding to a first range within a primary operating range of the field of view based at least on field of view information by performing out-painting based on a two-dimensional image, and generate an area corresponding to a second range different from the first range by performing in-painting based on an image generated in the area corresponding to the first range.

[0211] According to one embodiment, the generation command prompt may store instructions for generating an image portion so as to be symmetrical about at least one of a virtual line or boundary line at one or more points in the image portion, as part of the operation of generating the extended image.

[0212] According to one embodiment, the generation command prompt may store instructions for generating an image portion by fading in / out at least one of a virtual line or boundary line at one or more points in the image portion as part of the operation of generating the extended image.

[0213] In one embodiment, the electronic device may generate an expanded image by loading images of a similar category to the original two-dimensional image from memory. For example, the electronic device may generate an expanded image by arranging the two-dimensional image and images of a similar category in multiple layouts.

[0214] In one embodiment, in operation 530, the electronic device may be configured to include an extended image in at least a portion of the three-dimensional background area.

[0215] According to one embodiment, the electronic device may set the extended image to be included in an area of ​​the 3D background region excluding the ceiling region and the floor region, if the extended image does not include a ceiling image and a floor image. According to one embodiment, if the extended image includes a ceiling image and a floor image, the electronic device may set the extended image to be included in the 3D background region.

[0216] In one embodiment, in operation 540, the electronic device may display at least a portion of a virtual space including a set three-dimensional background area based on the field of view information.

[0217] In one embodiment, the electronic device may display an expanded image at least for the FOV based on the user's field of view information. In one embodiment, the electronic device may display an expanded image so that the original image, which is a two-dimensional image, is positioned in front of the user's field of view based on the user's field of view information.

[0218] According to one embodiment, the electronic device may separate an object from a two-dimensional image, display the expanded image in a three-dimensional background area, and display the separated object between the user and the three-dimensional background area in a virtual space. An operation of displaying the separated object according to one embodiment will be described in more detail with reference to FIGS. 10, 11, 12a, 12b, 12c, 12d, 13a, 13b, 13c, 13d, 14, and 15.

[0219] FIG. 6A is a flowchart illustrating an operation of generating an extended image based on a two-dimensional image of an electronic device according to one embodiment of the present disclosure.

[0220] FIG. 6b is a diagram for explaining an operation of generating an extended image based on a two-dimensional image of an electronic device according to one embodiment of the present disclosure.

[0221] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0222] Referring to FIG. 6A, in operation 610, an electronic device (e.g., the electronic device (101) of FIGS. 1A and 1B, the processor (120) of FIGS. 1A and 1B, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3A, the wearable electronic device (300) of FIG. 3B, or the electronic device (400) of FIG. 4) may collect a wallpaper original image from a database.

[0223] In one embodiment, the electronic device may receive a two-dimensional image, which is a source image (or a background source image), from an external electronic device. For example, the electronic device may receive the two-dimensional image through an external electronic device (e.g., a database) associated with a user account of the electronic device.

[0224] According to one embodiment, the electronic device may obtain an original image stored in a memory of the electronic device (e.g., memory (130) of FIGS. 1A and 1B). For example, the original image, which is a two-dimensional image, may be captured by a camera included in the electronic device (e.g., camera module (180) of FIGS. 1A and 1B), obtained by screen capture, or downloaded and stored in the memory.

[0225] In one embodiment, in operation 620, the electronic device may determine whether the resolution of the original image is greater than or equal to a first value. For example, the electronic device may determine whether the resolution of the original image is greater than or equal to 4k. When acquiring an expanded image through outpainting and / or inpainting, a low resolution of the original image may result in incomplete results, so the electronic device may determine the resolution of the original image.

[0226] In one embodiment, if the resolution is less than the first value (action 620 - No), in operation 625, the electronic device may adjust to a higher resolution image (or frame).

[0227] In one embodiment, the electronic device can obtain a high-resolution image through an artificial intelligence model trained to output a high-resolution image from a low-resolution image.

[0228] In one embodiment, the electronic device may utilize a single-frame super-resolution (SFSR) method to increase the memory capacity of the virtual space. SFSR involves aligning multiple low-resolution images to obtain a single high-resolution image, and may repeatedly perform machine learning and similarity estimation techniques.

[0229] In one embodiment, the electronic device may utilize the multi-frame super-resolution (MFSR) method when the original image is a video. While the MFSR method offers accurate motion estimation and good performance, real-time implementation is difficult due to the motion estimation of multiple images and repetitive estimation operations. Therefore, the method may operate by storing images in a sequence and arranging the frames in chronological order in a cylindrical space.

[0230] In one embodiment, if the resolution is greater than or equal to the first value (operation 620 - yes), in operation 630, the electronic device may determine whether the width of the original image is longer than the height.

[0231] In one embodiment, if the vertical length of the original image is longer than the horizontal length (operation 630 - No), then in operation 635, the electronic device can generate the image by outpainting up to a ratio of at least 1:1.

[0232] In one embodiment, the electronic device may generate an image in a lateral direction from both boundary regions of the original image. In one embodiment, the electronic device may also generate an image in a unilateral direction from one boundary region of the original image.

[0233] In one embodiment, if the width of the original image is longer than the height (operation 630 - Yes), then in operation 640, the electronic device can determine whether the painted portion exceeds 240 degrees.

[0234] In one embodiment, the electronic device can generate an image by outpainting an original image. In one embodiment, the electronic device can generate the image so that it corresponds to a curved shape in the virtual space. For example, the curved shape may include a shape in which the distance between the user and any point on the curved shape is substantially the same.

[0235] In one embodiment, the electronic device may determine whether the angle between a straight line connecting a point on each edge of an image generated by the user and the outpainting exceeds 240 degrees. In one embodiment, 240 degrees is merely an example of the set range and is not limited thereto.

[0236] In one embodiment, if the painted portion does not exceed 240 degrees (operation 640-No), then in operation 645, the electronic device can automatically generate an image by inferring an out-painting technique.

[0237] According to one embodiment, the electronic device can generate an image by performing out-painting up to a set range and then performing in-painting on the remaining area based on both boundary areas of the image generated through out-painting.

[0238] According to one embodiment, as illustrated in FIG. 6b, the electronic device may perform out-painting to additionally generate images in the lateral directions of an original image (611) positioned in the frontal gaze direction (11) of the user (10).

[0239] In one embodiment, the electronic device may continue performing out-painting if the angle between the straight lines connecting the user (10) and the two boundaries of the generated image (641) is within a set range. In one embodiment, the electronic device may perform out-painting over a range at least wider than the field of view (or FOV) (12) of the user (10).

[0240] In one embodiment, the electronic device may perform outpainting until the angle between the straight lines connecting the two boundaries of the generated image (641) becomes 240 degrees. For example, the electronic device may perform outpainting until the angle between the straight line connecting the left boundary of the generated image (641) and the user (10) and the frontal gaze direction (11) of the user (10) becomes 120 degrees, and the angle between the straight line connecting the right boundary of the generated image (641) and the user (10) and the frontal gaze direction (11) of the user (10) becomes 120 degrees.

[0241] Returning to FIG. 6A, in one embodiment, if the painted portion exceeds 240 degrees (operation 640 - yes), in operation 650, the electronic device may generate an image for the remaining area using an in-painting technique. For example, as illustrated in FIG. 6B, the image may be filled by inferring the empty portion (651) after performing out-painting. In one embodiment, the empty portion (651) may be within a 120-degree range and may be positioned behind the user (10), outside the FOV (12) of the user (10).

[0242] In one embodiment, the electronic device may fill in an empty portion (651) through in-painting based on an image (641) generated through out-painting. For example, the electronic device may generate an image of an area corresponding to the empty portion (651) based on images of both ends of the image (641) generated through out-painting.

[0243] In one embodiment, the electronic device can perform out-painting and in-painting to generate a 360-degree image.

[0244] In one embodiment, the electronic device can generate a 360-degree image in the shape of a sphere. In one embodiment, the electronic device can perform out-painting and in-painting to generate a 360-degree image in the shape of a sphere including the ceiling and floor.

[0245] In one embodiment, the electronic device can generate a hemispherical image. In one embodiment, the electronic device can perform outpainting to generate a 180-degree image including the ceiling and floor.

[0246] Returning to FIG. 6A, in one embodiment, at operation 660, the electronic device may render a 360 degree background.

[0247] In one embodiment, the electronic device may display the generated image in at least a portion of a three-dimensional background area of ​​a virtual space. For example, the electronic device may display the generated image in an area of ​​the three-dimensional background area that includes the user's field of view.

[0248] According to one embodiment, the electronic device may separate an object from a two-dimensional image, display the expanded image in a three-dimensional background area, and display the separated object between the user and the three-dimensional background area in a virtual space. An operation of displaying the separated object according to one embodiment will be described in more detail with reference to FIGS. 10, 11, 12a, 12b, 12c, 12d, 13a, 13b, 13c, 13d, 14, and 15.

[0249] FIG. 7A is a flowchart illustrating an operation of generating an extended image with both ends connected based on a two-dimensional image of an electronic device according to one embodiment of the present disclosure.

[0250] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0251] Referring to FIG. 7A, in operation 710, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) can separate an original image into independent objects.

[0252] According to one embodiment, the electronic device can identify at least one object region included in an original image through image analysis. According to one embodiment, the electronic device can separate the identified at least one object region from the original image. According to one embodiment, the electronic device can separate the original image into a background region and at least one object region.

[0253] In one embodiment, if the original image does not contain an object that is distinct from the background area, the electronic device may omit operation 710.

[0254] In one embodiment, in operation 720, the electronic device can perform in / out painting by applying generative AI to the original image.

[0255] According to one embodiment, the electronic device may perform outpainting to additionally generate an image in at least one direction relative to the original image based on the original image. For example, the electronic device may perform outpainting to additionally generate an image in a first direction relative to one boundary of the original image. According to one embodiment, the electronic device may additionally generate an image in the left direction relative to the left boundary of the original image, and additionally generate an image in the right direction relative to the right boundary of the original image.

[0256] According to one embodiment, when an object area is separated from an original image, the electronic device can generate an image by painting an empty space in the original image generated according to the separation of the object area.

[0257] In one embodiment, in operation 730, the electronic device may connect the images symmetrically left and right.

[0258] In one embodiment, the electronic device can symmetrically connect the two sides of an image generated through outpainting. For example, if an additional image is generated in a first direction from one side of the original image, the other side of the original image and the boundary of the additionally generated image can be symmetrically connected.

[0259] According to one embodiment, as illustrated in FIG. 7b, when an image is additionally generated in the left direction based on the left boundary of the original image and an image is additionally generated in the right direction based on the right boundary, the boundary of the image additionally generated in the left direction and the boundary of the image additionally generated in the right direction can be connected symmetrically left and right.

[0260] FIG. 7b is a diagram for explaining an operation of generating an extended image with both ends connected based on a two-dimensional image of an electronic device according to one embodiment of the present disclosure.

[0261] Referring to FIG. 7B, the electronic device can separate an object (712) from an original image (711). According to one embodiment, the electronic device can generate an out-painted image (721) by additionally generating an image in a left direction based on a left boundary of the original image (711) and additionally generating an image in a right direction based on a right boundary.

[0262] In one embodiment, the electronic device may generate an image (722) (e.g., image A) that is a horizontally inverted image (721) of an out-painted image (721). In one embodiment, the electronic device may connect the out-painted image (721) and the horizontally inverted image (722).

[0263] For example, the electronic device can connect one boundary of an out-painted image (721) and one boundary of a left-right inverted image (722). The electronic device can connect two boundaries of the out-painted image (721) and two boundaries of a left-right inverted image (722).

[0264] According to one embodiment, a first region (731) in which two boundaries of an out-painted image (721) and two boundaries of a left-right inverted image (722) are connected may be symmetrical left-right with respect to the connection boundary (732). According to one embodiment, a second region (741) in which one boundary of an out-painted image (721) and one boundary of a left-right inverted image (722) are connected may be symmetrical left-right with respect to the connection boundary (742).

[0265] In one embodiment, in FIG. 7b, the out-painted image (721) and the left-right inverted image (722) are shown as being linearly connected, but the boundaries of both sides of the out-painted image (721) and the left-right inverted image (722) may be interconnected to form a 360-degree image.

[0266] In one embodiment, the first region (731) and the second region (741) may be symmetrical left and right with respect to the connection boundary (732, 742), and the image of the connection boundary (732, 742) may be unnatural. In one embodiment, the electronic device may process the connection boundary (732, 742) to make it look natural.

[0267] Returning to FIG. 7a, in one embodiment, at operation 740, the electronic device may adjust the looping image to the point where the left and right ends meet, as illustrated in FIG. 8a or FIG. 8b.

[0268] FIG. 8A is a drawing for explaining an operation of connecting both ends of an extended image of an electronic device according to one embodiment of the present disclosure.

[0269] Referring to FIG. 8A, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) can identify a connection area (810) of an out-painted image and a left-right inverted image.

[0270] According to one embodiment, the electronic device can identify a portion (811) that is a portion of a connected area (810) of an out-painted image and a second portion (812) that is a portion of a connected area (810) of a left-right inverted image.

[0271] In one embodiment, the electronic device can perform a fade in / out operation to create an image by moving and overlapping the positions of the first portion (811) and the second portion (812). In one embodiment, the electronic device can create an image including a naturally processed connection area (813) through the fade in / out operation.

[0272] In one embodiment, the electronic device may input an image that connects an out-painted image and a left-right inverted image to an artificial intelligence model trained to naturally process a connection area (810). In one embodiment, the electronic device may obtain an image including a naturally processed connection area (813) from the artificial intelligence model as output data.

[0273] FIG. 8b is a drawing for explaining an operation of connecting both ends of an extended image of an electronic device according to one embodiment of the present disclosure.

[0274] Referring to FIG. 8b, the electronic device can check the connection area (820) of the out-painted image and the left-right inverted image.

[0275] In one embodiment, the electronic device can identify a plurality of connection portions (821, 822, 823) of a connection boundary within a connection area (820).

[0276] According to one embodiment, the electronic device may recognize the periphery of each of the plurality of connecting portions (821, 822, 823) and generate an image of each portion. For example, the electronic device may recognize the periphery of the first connecting portion (821) to generate an image of the first connecting portion (821), recognize the periphery of the second connecting portion (822) to generate an image of the second connecting portion (822), and recognize the periphery of the third portion (823) to generate an image of the third connecting portion (823).

[0277] According to one embodiment, the electronic device may create an empty space of a predetermined size between a plurality of connected portions (821, 822, 823), and perform in-painting based on the surrounding portions of the plurality of connected portions within the out-painted image and the surrounding portions of the plurality of connected portions within the left-right inverted image to create an image in the empty space. This allows the electronic device to create an image including a naturally processed connected area (824).

[0278] In one embodiment, the electronic device may input an image that connects an out-painted image and a horizontally inverted image to an artificial intelligence model trained to naturally process a connection area (820). In one embodiment, the electronic device may obtain an image including the naturally processed connection area (824) from the artificial intelligence model as output data.

[0279] Returning to FIG. 7A, in one embodiment, at operation 750, the electronic device may generate an image after segmenting each ceiling and floor. In one embodiment, the electronic device may generate an extended image including the ceiling image and the floor image.

[0280] According to one embodiment, as illustrated in FIG. 9, the electronic device can generate a ceiling image and a floor image based on the extended image.

[0281] FIG. 9 is a diagram illustrating an operation of generating a ceiling and floor of an extended image of an electronic device according to one embodiment of the present disclosure.

[0282] Referring to FIG. 9, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) can identify a ceiling area (911) and a floor area (912) within an image (910) obtained by out-painting an original image. According to one embodiment, the image (910) obtained by out-painting may be a 360-degree panoramic image in the shape of a side surface of a cylinder.

[0283] In one embodiment, the electronic device can identify a ceiling area (911) and a floor area (912) and recognize a surrounding area of ​​the ceiling area (911) and a surrounding area of ​​the floor area (912).

[0284] In one embodiment, the electronic device can generate an image (930) based on at least a portion of the ceiling area (911) to a ceiling vanishing point (920). In one embodiment, the electronic device can generate an image (930) based on the ceiling area (911) and a surrounding area of ​​the ceiling area (911) to a ceiling vanishing point (920).

[0285] In one embodiment, the electronic device can generate an image (931) based on at least a portion of the floor area (912) up to the floor vanishing point (921). In one embodiment, the electronic device can generate an image (931) based on the floor area (912) and a surrounding area of ​​the floor area (912) up to the floor vanishing point (921).

[0286] In one embodiment, the electronic device may use generative AI to generate an image (930) up to a ceiling vanishing point (920) and / or an image (931) up to a floor vanishing point (921).

[0287] Returning to FIG. 7A, in one embodiment, at operation 760, the electronic device may render processed images, video, and / or audio.

[0288] In one embodiment, if an extended image is generated, the electronic device may display the extended image in a virtual space. For example, the electronic device may display the extended image in a three-dimensional background area of ​​the virtual space.

[0289] In one embodiment, when an extended video is generated (e.g., extended frame by frame), the electronic device may display the extended video in a three-dimensional background area of ​​the virtual space.

[0290] In one embodiment, the electronic device may output audio corresponding to the expanded image or expanded video. For example, the audio may be included in the image or video, or may be assigned to the image or video by the electronic device.

[0291] FIG. 10 is a flowchart illustrating an operation of arranging objects included in an extended image at different depths in an electronic device according to an embodiment of the present disclosure.

[0292] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0293] Referring to FIG. 10, in operation 1010, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) may obtain information related to a user's field of view. For example, the information related to the field of view may include position information and size information of an area displayed on a display within a three-dimensional virtual space. Since an area displayed on a display within the three-dimensional virtual space changes according to a movement (e.g., movement and / or rotation) of the electronic device, information related to the field of view may change.

[0294] In one embodiment, the field of view information may be acquired through a sensor of the electronic device (e.g., the sensor module (176) of FIGS. 1A and 1B). For example, the sensor may include a sensor for detecting movement of the electronic device and / or movement of the user's head. For example, the sensor may include a motion sensor for detecting movement of the electronic device and / or movement of the user's head wearing the electronic device, and may include a camera for capturing an external environment. In one embodiment, the electronic device may determine movement of the electronic device through movement of feature points within an image captured by the camera.

[0295] According to one embodiment, in operation 1020, the electronic device may obtain a two-dimensional image. According to one embodiment, the electronic device may obtain the two-dimensional image based on a user input. For example, the electronic device may receive a user input for selecting a background image. According to one embodiment, the electronic device may receive a user input for selecting a background image of an external electronic device linked to a user account. According to one embodiment, the electronic device may receive a user input for selecting an image to be used as a background among images displayed through the execution screen of a photo album application (e.g., a gallery).

[0296] In one embodiment, the electronic device may collect a wallpaper original image from a database. In one embodiment, the electronic device may receive a background original image corresponding to a selected background image from the database.

[0297] In one embodiment, the electronic device may receive a two-dimensional image, which is a source image (or a background source image), from an external electronic device. For example, the electronic device may receive the two-dimensional image through an external electronic device (e.g., a database) associated with a user account of the electronic device.

[0298] According to one embodiment, the electronic device may obtain an original image stored in a memory of the electronic device (e.g., memory (130) of FIGS. 1A and 1B). For example, the original image, which is a two-dimensional image, may be captured by a camera included in the electronic device (e.g., camera module (180) of FIGS. 1A and 1B), obtained by screen capture, or downloaded and stored in the memory.

[0299] In one embodiment, in operation 1030, the electronic device may identify a first portion of a two-dimensional image to be applied as a background image in a three-dimensional virtual space. For example, applying a background image to a three-dimensional virtual space may mean displaying the image in a background area of ​​the three-dimensional virtual space.

[0300] In one embodiment, in operation 1040, the electronic device may identify a second portion of the two-dimensional image corresponding to the object of interest. In one embodiment, operations 1030 and 1040 are illustrated as being performed sequentially, but the order may be changed, and at least some of operations 1030 and 1040 may be performed simultaneously.

[0301] In one embodiment, an electronic device may determine whether an object of interest capable of applying depth exists in a two-dimensional image. For example, the electronic device may determine whether objects and / or text other than the background exist within the two-dimensional image.

[0302] In one embodiment, if there is an object of interest within the two-dimensional image to which depth can be applied, the electronic device can identify the entire two-dimensional image as the first portion to be applied as a background image.

[0303] In one embodiment, if a depth-applicable object of interest exists within a two-dimensional image, the electronic device may separate the two-dimensional image. For example, the electronic device may identify a first portion of the two-dimensional image to be applied as a background image in a three-dimensional virtual space and a second portion of the two-dimensional image corresponding to the object of interest.

[0304] In one embodiment, the electronic device can generate an image by performing in-painting on an empty space within the image or 360-degree background, which is generated by separating a second portion corresponding to an object of interest included in the two-dimensional image.

[0305] In one embodiment, in operation 1050, the electronic device may generate an extended image corresponding to a first portion of the two-dimensional image.

[0306] According to one embodiment, the electronic device may generate an extended image that further includes an image portion generated outside the first portion so as to extend from the boundary of the first portion, based on properties of a first portion of a two-dimensional image to be applied as a background image.

[0307] According to one embodiment, the electronic device can generate an extended image in at least one of the manners illustrated in FIGS. 5, 6a, 6b, 7a, 7b, 8a, 8b, and 9.

[0308] In one embodiment, the electronic device may generate backgrounds that require a large area first.

[0309] In one embodiment, in operation 1060, the electronic device may apply a first depth to the expanded image and a second depth to a second portion of the two-dimensional image to generate a three-dimensional virtual image. For example, the electronic device may place an object in the virtual space by reflecting the depth value. In one embodiment, the virtual space may include a VR home screen capable of executing a VR application.

[0310] In one embodiment, the electronic device may position an object area at a second depth, which is between a point set within the virtual space and a three-dimensional background area, which is a first depth, in relation to field of view information. For example, the point set within the virtual space may include the user's location, and the electronic device may position the object between the user and the three-dimensional background area.

[0311] According to one embodiment, the electronic device may determine the properties of an object included in an object area as part of a placement operation. According to one embodiment, the electronic device may place an object capable of interacting with a user at a first location set to be user-interactive, at least in part based on the properties of the object. According to one embodiment, the electronic device may place an object that only provides information without interacting with a user at a second location set to be user-interactive, at least in part based on the properties of the object.

[0312] In one embodiment, the electronic device may impart user interaction to an image based on characteristics of the virtual space.

[0313] In one embodiment, the electronic device can provide interaction for each of the 360-degree background and isolated objects.

[0314] In one embodiment, the electronic device may add a moving graphic effect to an object based at least in part on the object's properties. For example, the electronic device may add a moving graphic effect to an object if the object is a moving natural object (e.g., a person, an animal, a cloud, a fire) or a moving object (e.g., a ball, a car).

[0315] In one embodiment, an electronic device may obtain environmental information. For example, the environmental information may include at least one of time information, weather information, seasonal information, and the location of the electronic device.

[0316] In one embodiment, the electronic device may determine the properties of a graphic effect based at least on environmental information. For example, if the environmental information includes weather information, the electronic device may determine different properties of the graphic effect based on the weather information. For example, if the weather is currently windy, the electronic device may determine properties such that the object, a cloud, moves quickly. In one embodiment, if the weather is currently clear, the electronic device may determine the background brightness to be bright.

[0317] In one embodiment, the electronic device may identify contact information associated with an object region (e.g., a person object). In one embodiment, the electronic device may provide contact information in response to receiving user input regarding the object region.

[0318] In one embodiment, the electronic device may receive an event related to contact information. For example, when an event such as "incoming call" is received, the electronic device may add or modify a graphic effect related to the event to an object area.

[0319] According to one embodiment, if there is no object with which the user can interact, the interaction granting operation may be omitted. According to one embodiment, in operation 1070, the electronic device may display a portion of a three-dimensional virtual image based on information related to the user's field of view. According to one embodiment, the electronic device may render a processed image (e.g., an expanded image, an object image), a video, and / or audio.

[0320] According to one embodiment, the electronic device may display a portion of a three-dimensional virtual image by applying a first depth to an extended image and applying a second depth to a second portion of a two-dimensional image based on information related to a field of view. For example, the electronic device may display at least a portion of a virtual space including a set three-dimensional background region and a placed object region. For example, the electronic device may display a background image in a three-dimensional background region of a virtual space within a user's field of view. According to one embodiment, the electronic device may display an object image in a space between the user and the three-dimensional background region of a virtual space within the user's field of view.

[0321] In one embodiment, the electronic device can display an image in an area viewed by the user by adjusting the user's (or camera's) gaze, position, movement, and lens values.

[0322] In one embodiment, when an extended video is generated (e.g., extended frame by frame), the electronic device may display the extended video in a three-dimensional background area of ​​the virtual space and display an object image in the space between the user and the three-dimensional background area.

[0323] In one embodiment, the electronic device may output audio corresponding to the expanded image or expanded video. For example, the audio may be included in the image or video, or may be assigned to the image or video by the electronic device.

[0324] FIG. 11 is a flowchart illustrating an operation of arranging objects included in an extended image at different depths in an electronic device according to an embodiment of the present disclosure.

[0325] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0326] Referring to FIG. 11, in operation 1101, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) can check whether an object to which depth can be applied exists in an original image.

[0327] In one embodiment, the electronic device can determine whether an object capable of applying depth exists in the original image. For example, the electronic device can determine whether objects and / or text other than the background exist in the original image.

[0328] In one embodiment, if no depth-applicable object exists in the original image (operation 1101 - No), in operation 1102, the electronic device may generate a 360-degree background.

[0329] In one embodiment, the electronic device may generate an extended image that further includes portions of the image generated outside the background area, extending beyond the boundaries of the background area, based on properties of the background area. For example, the extended image may include a 360-degree panoramic image, a 360-degree spherical image, or a hemispherical image.

[0330] According to one embodiment, the electronic device can generate an extended image in at least one of the manners illustrated in FIGS. 5, 6a, 6b, 7a, 7b, 8a, 8b, and 9.

[0331] In one embodiment, in operation 1103, the electronic device may render a background image. In one embodiment, the electronic device may render a 360-degree background generated without region division of the virtual space in a background area of ​​the virtual space.

[0332] In one embodiment, if an object capable of applying depth exists in the original image (operation 1101 - Yes), in operation 1104, the electronic device may recognize the background and the object as independent objects. In one embodiment, the electronic device may separate the background area and the object area of ​​the original image.

[0333] According to one embodiment, the electronic device can generate an image by performing in-painting on an empty space within an image or 360-degree background generated according to object image separation.

[0334] In one embodiment, in operation 1105, the electronic device may detect a virtual space region. For example, the electronic device may detect a virtual space region between a set point (e.g., a user location) within the virtual space and a background region.

[0335] In one embodiment, in operation 1106, the electronic device may determine whether there are two or more objects.

[0336] In one embodiment, if there are less than two objects (operation 1106 - No), in operation 1107, the electronic device may divide the virtual space area. For example, if there is one object, the electronic device may divide the virtual space area into area A and area A'.

[0337] According to one embodiment, Area A in the virtual space may be an area where objects cannot be placed. According to one embodiment, Area A' in the virtual space may be an area where objects can be placed. According to one embodiment, Area A in the virtual space may be a space associated with the minimum movement of the user, and may be a space within a set distance from the user. According to one embodiment, Area A' in the virtual space may be a space excluding Area A in the virtual space from after the set distance from the user to before the background area. According to one embodiment, an operation of dividing the virtual space will be described below with reference to FIG. 12b.

[0338] In one embodiment, in operation 1108, the electronic device may render an object image in area A'. In one embodiment, the electronic device may place one object in area A' at a set distance from the user and closer than the background area.

[0339] In one embodiment, if there are two or more objects (operation 1106 - Yes), in operation 1109, the electronic device may divide the virtual space area into area A and area A', and further divide area A'. For example, the electronic device may divide area A' into area A1 and area A2.

[0340] According to one embodiment, the A1 region may be a region in the A' region where an object with which a user can interact is placed. According to one embodiment, the A2 region may be a region in the A' region where an object with which a user cannot interact is placed. According to one embodiment, the A1 region and the A2 region may be distinguished based on a distance from the user. For example, the A1 region may be a region in the A' region that is close to the user, and the A2 region may be a region in the A' region that is far from the user. According to one embodiment, an operation of distinguishing a virtual space will be described below with reference to FIG. 12b.

[0341] According to one embodiment, in operation 1110, the electronic device may render an object by reflecting depth and / or height values ​​according to the object's category. According to one embodiment, the electronic device may determine the depth not only based on the object's category but also based on the object's location within the content, and may place the object in the virtual space based on the determined depth. According to one embodiment, when there are multiple objects, the electronic device may place the multiple objects based on the order of depths that can be confirmed in the original image.

[0342] According to one embodiment, the electronic device may render an object in area A1 or area A2 based on the properties of the object. According to one embodiment, if the object is a floating object (e.g., a ball), the electronic device may render the object in an area closer to the ceiling area in area A1 based on the properties of the object. According to one embodiment, if the object is a floating object, the electronic device may also render the object in the ceiling area based on the properties of the object.

[0343] FIG. 12A is a diagram for explaining an operation of arranging objects included in an extended image at different depths according to an embodiment of the present disclosure in an electronic device.

[0344] Referring to FIG. 12A, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) may generate an extended image (1220) based on a two-dimensional image (1210). In one embodiment, the two-dimensional image (1210) may be an image displayed on a background screen.

[0345] In one embodiment, the electronic device may generate an extended image (1220) based on a two-dimensional image (1210) using generative AI. In one embodiment, the electronic device may separate at least one object (1221, 1222) contained within the two-dimensional image (1210).

[0346] According to one embodiment, the electronic device may separate a two-dimensional image (1210) into a background image and at least one object (1221, 1222), and then perform out-painting and in-painting on the background image to generate an extended image (1220).

[0347] In one embodiment, the electronic device may perform out-painting and in-painting on a two-dimensional image (1210) to generate an extended image (1220) and then separate at least one object (1221, 1222) included in the extended image (1220). In one embodiment, the electronic device may perform in-painting on an empty space generated by separating at least one object (1221, 1222) to generate an image.

[0348] In one embodiment, the electronic device may divide the virtual space into multiple regions. For example, the electronic device may divide the virtual space into multiple regions based on the distance from the user (10) positioned at the center of the virtual space.

[0349] For example, an area within a first distance from a user (10) of an electronic device may be a first area (1230) (e.g., a closed range view). According to one embodiment, an area within a first distance from a user (10) of an electronic device may be a second area (1231) (e.g., a middle range view). According to one embodiment, the second distance may be a boundary of a virtual space. According to one embodiment, an area corresponding to the boundary of the virtual space may be a third area (1232) (e.g., a distant view (sky box)). According to one embodiment, the range of at least one of the first area (1230), the second area (1231), and the third area (1232) may be adjustable by a user input. According to one embodiment, the third area (1232) may have a curved shape corresponding to the shape of the virtual space. For example, the third area may have a spherical shape or a hemispherical shape.

[0350] According to one embodiment, the first area (1230) is the main use space of the user (10), and the electronic device may display objects (e.g., a home screen, a running screen) related to a VR application. According to one embodiment, the electronic device may place a floating object or an object (1222) capable of interaction with the user, among at least one object (1221, 1222) obtained from the original image (1210), based on the properties of the object, in the first area (1230).

[0351] According to one embodiment, the electronic device may display an object (1221) included in a category such as a person, an animal, or a plant in a second area (1231) based on an attribute of the object among at least one object (1221, 1222).

[0352] According to one embodiment, the electronic device may display the extended image (1220) in at least a portion of the third region (1232). According to one embodiment, if the extended image (1220) is a 360-degree panoramic image, the electronic device may display the extended image (1220) in an area of ​​the third region (1232) excluding a ceiling area and a floor area. According to one embodiment, if the extended image (1220) is a spherical image, the electronic device may display the extended image (1220) in the entire area of ​​the third region (1232).

[0353] According to one embodiment, the electronic device may place objects by dividing the second area (1231) into a plurality of areas, as illustrated in FIG. 12b or FIG. 12c.

[0354] FIG. 12b is a drawing for explaining the depth of a three-dimensional space according to one embodiment of the present disclosure.

[0355] Referring to FIG. 12b, the virtual space may include a first area (1230) (e.g., area A), a second area (1231) (e.g., area A'), and a third area (1232) (e.g., background area) based on the distance from the user (10).

[0356] According to one embodiment, the electronic device may further divide the second area (1231) into a second-first area (1240) (e.g., area A1) and a second-second area (1241) (e.g., area A2) based on the distance from the user (10). In FIG. 12b, the second area (1231) is divided into two areas, but it may also be divided into three or more areas.

[0357] According to one embodiment, the electronic device may further divide an area above a height set based on the position of the user (10) into a ceiling area (1233) (e.g., area B). According to one embodiment, the ceiling area (1233) may be a space where the user (10) must look up.

[0358] According to one embodiment, the electronic device may be positioned in the first area (1230), the second-first area (1240), the second-second area (1241), or the ceiling area (1233) based on properties of at least one object separated from the original image.

[0359] According to one embodiment, the electronic device may additionally create and place a virtual object based on the properties of at least one object. According to one embodiment, the operation of additionally creating and placing a virtual object will be described below with reference to FIG. 13b.

[0360] In one embodiment, based on object properties, a floatable object (e.g., a flower petal, a ball, an airplane) can be placed in a first area (1230), a second-first area (1240), a second-second area (1241), or a ceiling area (1233) by applying a height value.

[0361] FIG. 12c is a drawing for explaining objects of an extended image arranged at different depths according to one embodiment of the present disclosure.

[0362] Referring to FIG. 12c, the electronic device may include a first area (1230) (e.g., area A), a second-1 area (1240) (e.g., area A1), a second-2 area (1241) (e.g., area A2), a third area (1232) (e.g., background area), and a ceiling area (1233) (e.g., area B) based on the distance from the user (10) in the virtual space.

[0363] According to one embodiment, the electronic device may place an extended image (1250) generated from an original image and at least one object (1251, 1252, 1253) separated from the original image in a corresponding area of ​​the virtual space.

[0364] For example, the electronic device may display the extended image (1250) in a third area (1232) corresponding to the boundary of the virtual space. According to one embodiment, the electronic device may display a first object (1251), which is a person object, in a second-second area (1241), based on the properties of the object. According to one embodiment, the electronic device may display a second object (1252), which is an object that can interact with a user, in a second-first area (1240), based on the properties of the object. According to one embodiment, the electronic device may display a third object (1253), which is an object that can float, in a ceiling area (1233), based on the properties of the object.

[0365] FIG. 12d is a drawing illustrating objects of an extended image arranged at different depths according to one embodiment of the present disclosure.

[0366] Referring to FIG. 12D, the electronic device may provide an extended image (1250), a first object (1251), a second object (1252), and a third object (1253) by overlapping them. According to one embodiment, the extended image (1250), the first object (1251), the second object (1252), and the third object (1253) may be arranged relative to each other when viewed from above. Accordingly, the electronic device may provide a three-dimensional effect through the display of the extended image (1250), the first object (1251), the second object (1252), and the third object (1253) when viewed from the front by a user (10).

[0367] FIG. 13A is a drawing for explaining an operation of providing interaction to a placed object in an electronic device according to one embodiment of the present disclosure.

[0368] Referring to FIG. 13A, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) may display an extended image (1310), a first object (1320), and a second object (1330). According to one embodiment, the electronic device may perform an interaction with a user by applying a motion to at least one of the extended image (1310), the first object (1320), and the second object (1330) based on an attribute.

[0369] For example, the electronic device may give a flying motion to a second object (1330) having a floatable property. According to one embodiment, the electronic device may give a motion to change the position of the second object (1330) to display a second object (1331) placed at a different position. According to one embodiment, the second object (1331) placed at a different position may have different heights as well as left and right positions. According to one embodiment, the second object (1331) placed at a different position may have different sizes based on the distance from the user.

[0370] FIG. 13b is a drawing for explaining an operation of providing interaction to a placed object in an electronic device according to one embodiment of the present disclosure.

[0371] Referring to FIG. 13B, the electronic device may display a second object (1330) capable of movement based on an attribute. According to one embodiment, when a first user input (e.g., touch) is received by the user to select the second object (1330), the electronic device may display a second object (1332) with a changed color. According to one embodiment, the color may be changed randomly. According to one embodiment, the color of the object being changed based on the user input is not limited to this example, and a highlight effect of a different color, type, and / or size may be applied to the object based on the user input.

[0372] According to one embodiment, when the electronic device receives a second user input (e.g., a long touch) for selecting a second object (1330), the electronic device may display a plurality of additionally generated objects (1333) based on the second object (1330). According to one embodiment, the electronic device may assign motion to each of the additionally generated plurality of objects (1333).

[0373] According to one embodiment, in FIG. 13b, a plurality of objects (1333) in the shape of a ball are displayed, but according to another embodiment, when an object in the shape of a petal is selected, a plurality of petals may be displayed in the form of fluttering petals.

[0374] According to one embodiment, when a user input (e.g., a touch) for selecting one of the plurality of additionally generated objects (1333) is received, the electronic device may display a plurality of objects (1334) each having a different color. According to one embodiment, the color of the object is changed based on the user input, but is not limited thereto, and a highlight effect of a different color, type, and / or size may be applied to the object based on the user input.

[0375] In this way, the electronic device can add or change objects through interaction with the user.

[0376] FIG. 13c is a drawing for explaining an operation of providing interaction to a placed object in an electronic device according to one embodiment of the present disclosure.

[0377] Referring to FIG. 13c, the electronic device can interact with the user by applying motion to at least one of the first object (1320) and the second object (1330) based on the properties.

[0378] For example, an electronic device may impart a flying motion to a second object (1330) that includes a floatable property. In one embodiment, the electronic device may impart a motion that changes the pose of a first object (1320) that includes a moving property.

[0379] In one embodiment, the duration of movement can be determined based on the object's category. For example, in the case of the first object (1320), which is a person, the time it can move in the air is limited, so the electronic device can assign the movement within approximately 5 seconds. In one embodiment, the electronic device can assign the movement of the first object (1320) as a GIF (graphics interchange format).

[0380] In one embodiment, the electronic device may not impose a limitation on the time of movement when imparting a tail-wagging motion to an object that is an animal.

[0381] According to one embodiment, the electronic device may display a first object (1321) positioned at a different location by applying a motion that changes the location and posture of the first object (1320). According to one embodiment, the electronic device may display a second object (1335) positioned at a different location by applying a motion that changes the location of the second object (1330). According to one embodiment, the second object (1335) positioned at a different location may have different heights as well as different left and right positions. According to one embodiment, the second object (1335) positioned at a different location may have different sizes based on the distance from the user.

[0382] In this way, by giving movement to objects, you can provide a sense of life and can also be used as a screen saver.

[0383] FIG. 13d is a drawing for explaining an operation of providing interaction to a placed object in an electronic device according to one embodiment of the present disclosure.

[0384] Referring to FIG. 13D, the electronic device can impart motion to the background. For example, the electronic device can maintain the first object (1320) and the second object (1330) and impart motion to the background (1340). For example, the electronic device can impart motion to images included in the background (1340), but set the range of motion to a small value, and repeatedly play them.

[0385] In one embodiment, the electronic device may additionally place an image included in the background (1340). For example, the electronic device may additionally create and place a spectator object (1341) included in the background (1340). In one embodiment, the electronic device may also impart motion to the additionally placed objects.

[0386] In one embodiment, the electronic device may add images not included in the background (1340) based on the content (e.g., category) of the original image. For example, if the original image is a game scene, the electronic device may add images of spectators, other players, scoreboard images, and / or placard images that were not included in the background (1340).

[0387] In one embodiment, images additionally placed on the background (1340) may also be interactive with the user. For example, when the electronic device receives a user input (e.g., pinching) to select an additionally placed scoreboard, the electronic device may add a motion to the scoreboard image so that a player's name passes by.

[0388] In one embodiment, the electronic device can control the brightness of the entire virtual space based on the content (e.g., category) of the original image or background (1340). In this way, controlling the brightness of the virtual space based on the content of the original image or background can provide an immersive experience.

[0389] Additionally, by adding movement to the background, it can be used for viewing purposes. For example, if the background is a natural element such as fire, mountains, or water, adding movement to the background can be used for viewing purposes and can also function as a screensaver.

[0390] FIG. 14 is a drawing for explaining an operation of providing interaction to a placed object in an electronic device according to one embodiment of the present disclosure.

[0391] Referring to FIG. 14, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) can provide movement to a background image (1410) based on environmental information. For example, based on time information, a background image (1410) corresponding to daytime can be provided during the day, and a background image (1420) corresponding to nighttime can be provided during the night. For example, the electronic device can gradually change the background image over time.

[0392] In one embodiment, a background image (1410) corresponding to daytime may be changed to a background image (1420) corresponding to nighttime based on user input. For example, when a drag-and-drop input is received in which the user selects and drags a ceiling area of ​​a virtual space, the electronic device may change the background image (1410) corresponding to daytime to a background image (1420) corresponding to nighttime.

[0393] In one embodiment, when the background image (1420) corresponding to the night is changed, the electronic device may additionally place objects that are not included in the background image (1410) corresponding to the day but are suitable for the background image (1420) corresponding to the night. For example, the electronic device may additionally place objects such as the moon, stars, comets, and aurora in the background image (1420) corresponding to the night.

[0394] FIG. 15 is a drawing for explaining an operation of providing interaction to a placed object in an electronic device according to one embodiment of the present disclosure.

[0395] Referring to FIG. 15, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) may display information related to a first object (1510) when a user input (1511) (e.g., a touch) for selecting a first object (1510) is received.

[0396] For example, when a user input (1511) is received for selecting a first object (1510) providing current time information, the electronic device may display a screen (1520) containing at least one of a related widget, weather information, schedule information, and reminder information in a virtual space. In one embodiment, the screen (1520) may be a pop-up screen.

[0397] In one embodiment, when a user input is received to select an object related to a person, plant, animal, building, or object, the electronic device may provide information related to the selected object. For example, when a user input is received to select an object related to a person, the electronic device may provide contact information corresponding to the person. In one embodiment, the contact information may include a user interface for making a call or sending a message to the contact corresponding to the person.

[0398] In one embodiment, when a user input for selecting a background image is received, the electronic device may provide information related to the background image. For example, when a user input for selecting a background image is received, the electronic device may provide information related to a location included in the background image (e.g., location name, location).

[0399] FIG. 16 is a flowchart illustrating an operation of mapping a placed object and audio in an electronic device according to one embodiment of the present disclosure.

[0400] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0401] Referring to FIG. 16, in operation 1610, an electronic device (e.g., the electronic device (101) of FIGS. 1A and 1B, the processor (120) of FIGS. 1A and 1B, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIG. 3A, the wearable electronic device (300) of FIG. 3B, or the electronic device (400) of FIG. 4) can determine whether the original is a video. According to one embodiment, the electronic device can determine whether the original is an image or a video.

[0402] In one embodiment, if the original is not a video (operation 1610 - No), in operation 1611, the electronic device may automatically generate audio after recognizing the category of the object. In one embodiment, if the original is an image, since the original does not include audio, the electronic device may generate audio corresponding to the category after identifying the category of the image. For example, audio corresponding to the image may be generated depending on whether the image is in a natural environment, an urban environment, or a specific situation (e.g., a sporting event or a festival).

[0403] According to one embodiment, the electronic device can generate audio corresponding to a background image of an image and at least one object included in the image, respectively.

[0404] In one embodiment, in operation 1612, the electronic device may encode the background sound level based on the depth of the space. For example, the electronic device may encode the sound level based on the area in which the object is to be placed, based on the properties of the object. For example, the closer the object is placed to the user, the louder the sound may be encoded. For example, since the background image is placed farthest from the user, the audio level associated with the background image may be encoded to be smaller than that of other objects.

[0405] In one embodiment, if the original is a video (operation 1610 - Yes), then in operation 1620, the electronic device can determine whether sound is present in the original.

[0406] In one embodiment, if the source contains sound (operation 1620 - Yes), in operation 1630, the electronic device may determine whether the object is a person, an animal, or a plant. In one embodiment, the electronic device may determine, through object recognition, whether the object is a category capable of making sounds, such as a person, an animal, or a plant.

[0407] In one embodiment, if the object is not a person, animal or plant (operation 1630 - No), then in operation 1640, the electronic device may encode the background sound level based on the depth of the space. For example, as illustrated in FIG. 17A, the electronic device may encode the sound level based on the area in which the object is to be placed based on the properties of the object. For example, the closer the object is placed to the user, the louder the sound may be encoded. For example, since the background image is placed farthest from the user, the audio level associated with the background image may be encoded to be smaller than that of other objects.

[0408] FIG. 17A is a diagram illustrating an operation of mapping a placed object and audio in an electronic device according to one embodiment of the present disclosure.

[0409] Referring to FIG. 17A, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) may map different sounds to different intensities depending on the distance between an object and a user (e.g., near, middle, far). For example, a sound of an object (1711) corresponding to a close distance from a user (e.g., a ball) may be mapped to a louder sound than a sound of an object (1710) corresponding to a far distance from the user (e.g., a spectator).

[0410] In one embodiment, if the background image is a natural scene, sounds corresponding to nature may be applied. For example, based on the background image, sounds such as desert wind, falling snow, forest sounds, crackling firewood, or melancholic sounds may be applied. In one embodiment, the electronic device may apply sounds corresponding to the background image at a low volume.

[0411] In one embodiment, when a video is displayed, sounds corresponding to nearby objects and sounds corresponding to background images can be applied at different decibels or frequencies. For example, the sound of a voice corresponding to a person object close to the user can be louder, while the sound of a desert wind corresponding to a background image can be softer.

[0412] In one embodiment, if the background image is a road or a stadium rather than a natural one, the noise can be identified and removed or the sound can be lowered and applied based on the noise contained in the sound.

[0413] In one embodiment, the electronic device may apply different sounds corresponding to background images and objects depending on whether it is day or night. For example, during nighttime, the sounds corresponding to background images and objects may be applied more loudly than during the day.

[0414] Returning to FIG. 16, according to one embodiment, if the object is a person or an animal or plant (operation 1630 - yes), then in operation 1621, the electronic device can automatically separate the audio after recognizing the category of the object.

[0415] In one embodiment, if the object is a person or an animal, the electronic device can isolate the sound corresponding to the object from the original sound. In another embodiment, if the sound is too low to extract the object's sound, the electronic device can use artificial intelligence resources to map the sound corresponding to the object.

[0416] Returning to FIG. 16, in one embodiment, at operation 1622, the electronic device may encode the background sound level according to the depth of the space, and may encode only when the user interacts with an object having sound, such as a person, plant or animal, as illustrated in FIG. 17b.

[0417] FIG. 17b is a diagram illustrating an operation of mapping a placed object and audio in an electronic device according to one embodiment of the present disclosure.

[0418] Referring to FIG. 17B, the electronic device can isolate a sound corresponding to an object when the object is a person (1720), a sound-producing object (1721), or an animal (1722). In one embodiment, when the objects are a person (1720) and an object (1721), the electronic device can also provide a mutual interaction between the objects (e.g., a video of a person (1720) playing with an object (1721).

[0419] In one embodiment, if the electronic device is an object that produces sound (e.g., a crying baby or a barking dog), the electronic device may output a sound corresponding to the object only when there is user interaction with the object. For example, if the electronic device does not output a sound corresponding to the object, but receives user input such as looking at the object or selecting the object, the electronic device may output a sound corresponding to the object.

[0420] Likewise, sounds that may harm the immersion of a virtual space can only be output when there is user interaction.

[0421] In one embodiment, the electronic device can encode sounds by considering the acoustics of the real space. For example, if multiple sounds are present, they may overlap, so sounds corresponding to the background image may be given priority, and only sounds corresponding to a set number of objects (e.g., two) may be generated.

[0422] In one embodiment, the sound may be a result obtained by providing an image as input and obtaining it as output, and the electronic device may interpret the input image, generate the interpreted information as a prompt, and play the generated sound based on the prompt.

[0423] In one embodiment, if no sound is present in the source (operation 1620 - No), the electronic device may proceed to operation 1611, where the device may automatically generate audio after recognizing the category of the object. In one embodiment, in operation 1612, the electronic device may encode the background sound level according to the depth of the space, as illustrated in FIG. 17C.

[0424] FIG. 17c is a diagram illustrating an operation of mapping a placed object and audio in an electronic device according to one embodiment of the present disclosure.

[0425] Referring to FIG. 17c, the electronic device may map a sound associated with a background image (1730) or provide a related music playback interface.

[0426] In one embodiment, the electronic device may analyze a location, region, or city associated with the background image (1730) and provide a related sound based on the analysis results. For example, the electronic device may search for a song associated with the region of the background image (1730) and provide the searched song.

[0427] In one embodiment, when a user input (1731) for selecting a background image (1730) is received while a sound associated with the background image (1730) is being output, the electronic device may display an interface (1732) for the sound associated with the background image (1730). In one embodiment, the interface (1732) may include information associated with the sound and / or buttons for controlling the sound.

[0428] In one embodiment, the electronic device may repeatedly play a sound source for a set period of time (e.g., 2-3 minutes).

[0429] In one embodiment, the electronic device may change the music it outputs based on user input.

[0430] FIG. 17d is a diagram for explaining an operation of mapping a placed object and audio in an electronic device according to one embodiment of the present disclosure.

[0431] Referring to FIG. 17D, the electronic device can map audio to each of a background image (1740) and at least one object (1741, 1742, 1743). In one embodiment, each of the background image (1740) and at least one object (1741, 1742, 1743) may be generated or extracted based on an original image.

[0432] In one embodiment, the electronic device may output related audio based on an original image or a generated background image (1740) and at least one object (1741, 1742, 1743). For example, if the original image or the generated background image (1740) and at least one object (1741, 1742, 1743) are urban environments, the electronic device may automatically output urban noise.

[0433] In one embodiment, the electronic device may map audio to an object that does not emit sound (e.g., a building, a plant, or an object) based on the properties of the object. In one embodiment, the electronic device may map audio to an object based on a user input. In one embodiment, the electronic device may output audio mapped to the object, output the mapped audio when a user interaction (e.g., touching the object) is performed, or display a player related to the audio. For example, when a user input (1750) selecting a first object (1741) is received, the electronic device may display a media player screen (1751) related to the audio mapped to the first object (1741). For example, the media player screen (1751) may include audio-related information (e.g., song title, singer) and / or buttons for playback control.

[0434] In one embodiment, when a user interaction is performed on an object that produces sound (e.g., a person or an animal), audio corresponding to the object may be output. For example, when a user input is received for selecting a second object (1742) that is a person, the electronic device may output audio corresponding to the second object (1742), such as a moving sound or a bouncing ball sound. In one embodiment, when a user input is received for selecting a third object (1743) that is an animal, the electronic device may output audio corresponding to the third object (1743), such as a barking sound.

[0435] In one embodiment, the electronic device may provide not only audio output but also movement of the object when selecting the object. For example, if a user input is received to select a second object (1742) that is a person, the electronic device may output a motion of bouncing a ball to the second object (1742). In one embodiment, if a user input is received to select a third object (1743) that is an animal, the electronic device may output a motion of wagging its tail to the third object (1743).

[0436] In this way, the electronic device can provide a sense of presence by outputting audio related to the background image (1740) and at least one object (1741, 1742, 1743). More specifically, the immersion of the virtual space can be prevented by outputting audio and / or movement based on the user's interaction with at least one object (1741, 1742, 1743).

[0437] FIG. 18A is a diagram illustrating an operation of changing a placed object in an electronic device according to an embodiment of the present disclosure. FIG. 18A illustrates an embodiment of partially removing and changing a distant view among a near view (e.g., a first region (1230) of FIGS. 12A, 12B, 12C, and 12D), a middle view (e.g., a second region (1231) of FIGS. 12A, 12B, 12C, and 12D), and a distant view (e.g., a third region (1232) of FIGS. 12A, 12B, 12C, and 12D).

[0438] Referring to FIG. 18A, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) can change a background image (1810). According to one embodiment, the electronic device can change the background image (1810) based on a user input or the passage of time. According to one embodiment, the electronic device can also change the background image (1810) based on a change in an object.

[0439] In one embodiment, the electronic device may change to a monochrome or gradient background image (1811) based on a color extracted from the background image (1810). For example, the color extracted from the background image (1810) may be the color of the largest area in the background image (1810).

[0440] According to one embodiment, the electronic device can change the background image (1810) into an illustration type image (1812). According to one embodiment, the electronic device can input the background image (1810) into an artificial intelligence model trained to change an image into an illustration type image, and display the illustration type image (1812) obtained as output data from the artificial intelligence model. According to one embodiment, the electronic device can change the background image (1810) into an illustration type image (1812) using an algorithm for changing an image into an illustration type image.

[0441] According to one embodiment, the electronic device can change the background image (1810) into a line type image (1813). According to one embodiment, the electronic device can input the background image (1810) into an artificial intelligence model trained to change an image into a line type image, and display the line type image (1813) obtained as output data from the artificial intelligence model. According to one embodiment, the electronic device can change the background image (1810) into a line type image (1813) using an algorithm for changing an image into a line type image.

[0442] In one embodiment, FIG. 18a illustrates that the background image is changed to one of three types, but is not limited thereto, and may be changed to other types of images as illustrated in FIG. 19 through an artificial intelligence model or algorithm.

[0443] FIG. 18B is a diagram illustrating an operation of changing a placed object in an electronic device according to one embodiment of the present disclosure. For example, FIG. 18B illustrates an embodiment in which the middle ground among the foreground, middle ground, and far ground is partially removed and changed.

[0444] Referring to FIG. 18B, the electronic device may change the object (1820) placed in the foreground, the object (1820) placed in the middle ground, and the background image placed in the distance, among the objects placed in the middle ground, into objects (1821, 1822) of different categories. For example, the electronic device may change the object (1820) of an exercising person into an illustration object (1821) of a shiny shape and an object (1822) of a sitting person shape. According to one embodiment, the electronic device may provide a virtual space with a heterogeneous atmosphere (e.g., a kitschy atmosphere) by placing objects of different categories in the middle ground.

[0445] FIG. 18c is a diagram illustrating an operation of changing a placed object in an electronic device according to one embodiment of the present disclosure. For example, FIG. 18c illustrates an embodiment in which the middle ground among the foreground, middle ground, and far ground is partially removed and changed.

[0446] Referring to FIG. 18c, the electronic device can change the object (1830) positioned in the foreground among the objects positioned in the foreground, objects positioned in the middle ground, and background images positioned in the distance to an object of a different category. For example, the electronic device can change the flying ball-shaped object (1830) to an object (1831) related to providing information (e.g., a business card, an electronic billboard, a pop-up screen).

[0447] In one embodiment, the electronic device may provide a virtual space with a heterogeneous atmosphere (e.g., a kitschy atmosphere) by placing objects of different categories in the foreground.

[0448] Although FIGS. 18A to 18C disclose changing the displayed object to another object, according to one embodiment, the electronic device may also arrange images of similar categories to the object in multiple layouts.

[0449] FIG. 19 is a drawing for explaining an operation of changing a placed object of an electronic device according to one embodiment of the present disclosure.

[0450] Referring to FIG. 19, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) can divide a virtual space into a plurality of areas and place a background image (1910) or an object in each of the plurality of areas.

[0451] In one embodiment, the electronic device may change the style of a background image (1910) or at least a portion of an object. For example, the electronic device may change the background image (1910) to a line-type image (1920), a watercolor-type image (1921), or an oil-type image (1922).

[0452] FIG. 20A is a drawing for explaining an operation of changing a placed object of an electronic device according to one embodiment of the present disclosure.

[0453] Referring to FIG. 20A, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) may display a single image (2010) generated by connecting (or merging) multiple images (2011, 2012) as a background image.

[0454] For example, an electronic device may generate a single 360-degree image (2010) by linking a daytime image (2011) and a nighttime image (2012) of a location. In one embodiment, the electronic device may display the 360-degree image (2010) such that the daytime image (2011) is positioned in front of the user and the nighttime image (2012) is positioned behind the user.

[0455] According to one embodiment, when applying a plurality of contents (e.g., images or videos) to a virtual space, the electronic device may change the contents to be applied at set intervals, as illustrated in FIG. 20b, or change the properties of the contents over time, as illustrated in FIG. 20c. According to one embodiment, the electronic device may also change the contents to be applied to the virtual space when a user input (e.g., a swipe) is received.

[0456] FIG. 20b is a drawing for explaining an operation of changing a placed object of an electronic device according to one embodiment of the present disclosure.

[0457] Referring to FIG. 20B, the electronic device may change a background image (2020) and a content applied to a virtual space among a plurality of contents (2021, 2022, 2023) at regular intervals (e.g., n minutes). According to one embodiment, the electronic device may place the background image (2020) and at least one object in the virtual space. According to one embodiment, the electronic device may maintain at least one object and change the background image (2020) at regular intervals.

[0458] In one embodiment, the electronic device may slide the displayed content or fade it in / out to replace it with other content at regular intervals. In one embodiment, the electronic device may sequentially display multiple background images (2020) and contents (2021, 2022, 2023) at regular intervals based on a set order.

[0459] FIG. 20c is a drawing for explaining an operation of changing a placed object of an electronic device according to one embodiment of the present disclosure.

[0460] Referring to FIG. 20c, the electronic device may change the same content based on time and / or season. For example, the electronic device may change the properties of the same content based on time and / or season.

[0461] According to one embodiment, the electronic device may display second content (2031) in which properties such as the color of the background image are changed based on the passage of time and / or change of season, while applying the first content (2030) to a virtual space. For example, the first content (2030) may correspond to daytime, and the second content (2031) may correspond to sunset. According to one embodiment, the electronic device may sequentially move the brightest part to the right and sequentially change the background color to purple from the first content (2030) in which the brightest part corresponding to the position of the sun is located in the center and the background sky is blue during daytime. According to one embodiment, the electronic device may display second content (2031) in which the brightest part is located on the right and the sky is purple during sunset.

[0462] In one embodiment, the electronic device may change not only the properties of the background image, but also the object images (e.g., trees, plants) included in the background image and the objects positioned apart from the background image based on the passage of time and / or changes in seasons. For example, the electronic device may change the size of a tree image included in the background image or a tree positioned apart from the background image based on the passage of time and change the leaves to sprout or fall based on the change of seasons.

[0463] FIG. 21A is a diagram illustrating an operation of placing an object on a lock screen or an always-on display (AOD) screen of an electronic device according to one embodiment of the present disclosure. According to one embodiment, the lock screen or AOD screen (2110, 2120, 2130, 2140) illustrated in FIG. 21A may be a 360-degree panoramic, spherical, or hemispherical image placed in a virtual space.

[0464] Referring to FIG. 21A, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) may maintain only the clock content (2111) included in a lock screen or AOD screen (2110), and dim the rest.

[0465] According to one embodiment, the electronic device may place the clock content (2122) in the FOV (2121) and display a dimmed lock screen or AOD screen (2120) other than the clock content (2122). According to one embodiment, the electronic device may display the lock screen or AOD screen (2110) in which floating objects (2123) are placed in an area other than the FOV. According to one embodiment, the floating objects (2123) may be placed in the virtual space in a state separated from the background image. According to one embodiment, the floating objects (2123) may interact with the user.

[0466] In one embodiment, the electronic device may display a lock screen or AOD screen (2130) with the clock content blurred out for enhanced security. In one embodiment, the electronic device may display the lock screen or AOD screen with the blurring removed based on authentication performed to unlock security, such as biometric authentication.

[0467] In one embodiment, the electronic device may display a lock screen or AOD screen (2140) that includes images of a similar category to the background image, except for the clock content, for enhanced security. In one embodiment, the electronic device may display images or objects related to the background image in a blurred state over images of a similar category.

[0468] FIG. 21B is a diagram illustrating an operation of placing an object on a lock screen or an AOD screen of an electronic device according to one embodiment of the present disclosure. According to one embodiment, the lock screen or AOD screen (2150, 2160, 2170) illustrated in FIG. 21B may be a 360-degree panoramic, spherical, or hemispherical image placed in a virtual space.

[0469] Referring to FIG. 21b, the electronic device may display a lock screen or AOD screen (2150) that displays a representative color extracted from a background image in a single color. According to one embodiment, the electronic device may display a lock screen or AOD screen (2160) that displays a plurality of colors extracted from a background image in a gradient form.

[0470] In one embodiment, the electronic device may display a lock screen or AOD screen (2170) displaying at least one object (2171) (e.g., a floating object) on a solid color or gradient background.

[0471] In one embodiment, at least one object (2171) may be placed in the virtual space, separated from the background image. In one embodiment, at least one object (2171) may interact with a user.

[0472] FIG. 22 is a drawing for explaining a virtual space displayed according to rotation of an electronic device according to one embodiment of the present disclosure.

[0473] Referring to FIG. 22, an electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B, a processor (120) of FIGS. 1A and 1B, a wearable electronic device (200) of FIG. 2, a wearable electronic device (300) of FIG. 3A, a wearable electronic device (300) of FIG. 3B, or an electronic device (400) of FIG. 4) may generate an extended image (2220-1, 2220-2) based on a two-dimensional image (2210). According to one embodiment, the extended image (2220-1, 2220-2) may be a 360-degree panoramic image, a spherical image, or a hemispherical image.

[0474] According to one embodiment, the electronic device may display the extended images (2220-1, 2220-2) in a three-dimensional background area. According to one embodiment, the electronic device may apply an object (e.g., a board (2221) or a virtual monitor screen (2222)) related to a workspace to the virtual space where the extended images (2220-1, 2220-2) are displayed in the three-dimensional background area. For example, the electronic device may include a board (2221) or a virtual monitor screen (2222) that includes functions related to performing tasks. According to one embodiment, the board (2221) may include icons of tasks-related applications and widgets for functions related to performing tasks (e.g., memos, keyboards). According to one embodiment, the virtual monitor screen (2222) may display an execution screen of a tasks-related application. In one embodiment, the electronic device may display an action based on user input performed on a virtual monitor screen (2222) or user input performed through a board (2221).

[0475] In one embodiment, the electronic device may change the field of view (FOV) within the virtual space based on the user's gaze or head movement, if the user's gaze or head movement is changed. In one embodiment, the electronic device may display a screen based on the changed field of view (FOV).

[0476] For example, when a user looks up while a first background image (2220-1) is displayed and a virtual monitor screen (2222) is displayed above a board (2221) and the board, the electronic device may change the FOV within the virtual space upward and display a screen based on the changed FOV. According to one embodiment, the electronic device may display a second background image (2220-2) that is an FOV higher than the first background image (2220-1). According to one embodiment, the electronic device may move the position of the virtual monitor screen (2222) downward within the FOV and display time content (2223) positioned higher than the virtual monitor screen (2222) within the FOV. According to one embodiment, the electronic device may also display function icons positioned around the time content (2223).

[0477] In this way, electronic devices can apply the image desired by the user to the virtual workspace.

[0478] FIG. 23A is a diagram for explaining an operation of displaying an image generated based on a keyword in an external electronic device as a three-dimensional virtual image in an electronic device according to one embodiment of the present disclosure.

[0479] Referring to FIG. 23A, an external electronic device (e.g., the external electronic device (104) of FIGS. 1A and 1B) may display a screen (2310) for generating a wallpaper image. The screen (2310) for generating a background image may include at least one keyword (2311) to be applied to the background image to be generated and a button (2312) for commanding a background image generation operation.

[0480] According to one embodiment, at least one keyword (2311) to be applied to the background image to be generated may be at least one keyword selected from a plurality of keywords by user input, or may be text entered by user input. According to one embodiment, the plurality of selectable keywords may be pre-stored by category related to the image style. According to one embodiment, the external electronic device may display example images for each of the plurality of keywords.

[0481] In one embodiment, the external electronic device may apply a background image (2320) generated based on at least one keyword (2311) as a wallpaper. For example, if at least one keyword (2311) includes flowers, blue and navy, and a castle, the generated background image (2320) may include a blue-roofed castle surrounded by blue-colored flowers and a blue sky.

[0482] According to one embodiment, a background image (2320) applied to an external electronic device may be applied to a linked electronic device (e.g., an electronic device (101) of FIGS. 1A and 1B , a processor (120) of FIGS. 1A and 1B , a wearable electronic device (200) of FIG. 2 , a wearable electronic device (300) of FIG. 3A , a wearable electronic device (300) of FIG. 3B , or an electronic device (400) of FIG. 4 ). For example, the external electronic device and the electronic device may be linked through an account.

[0483] According to one embodiment, the electronic device may generate a three-dimensional virtual image (2330) based on a two-dimensional background image (2320) received from an external electronic device. For example, the three-dimensional virtual image (2330) may include a background image (2331) applied to a background area of ​​a first depth in a three-dimensional virtual space, an object image (2332) applied to a second depth in the three-dimensional virtual space, and at least one newly generated object (2333). For example, the at least one newly generated object (2333) may be generated based on properties of the background image (2331) and / or properties of the object image (2332), and may be placed in a third depth different from the second depth or a ceiling area (or an air area).

[0484] FIG. 23b is a diagram for explaining an operation of applying a changed image based on a keyword to an external electronic device while a three-dimensional virtual image is displayed on an electronic device according to an embodiment of the present disclosure.

[0485] Referring to FIG. 23b, when a user input for changing a 3D virtual image is received while the electronic device is displaying a 3D virtual image, the electronic device may display a screen (2340) for changing the 3D virtual image. According to one embodiment, by changing a keyword, the electronic device may change the displayed 3D virtual image into an image generated based on the changed keyword.

[0486] According to one embodiment, a screen (2340) for changing a three-dimensional virtual image may include at least one keyword (2341) to be applied to a three-dimensional virtual image to be generated and a button (2342) for commanding a three-dimensional virtual image generation operation. According to one embodiment, the screen (2340) for changing a three-dimensional virtual image may further include a preview (2343) of an image to be generated based on the at least one keyword. For example, the preview (2343) of the image may be a two-dimensional image.

[0487] According to one embodiment, at least one keyword (2341) to be applied to the 3D virtual image to be generated may be at least one keyword selected from a plurality of keywords by user input, or may be text entered by user input. According to one embodiment, the plurality of selectable keywords may be pre-stored by category related to the image style. According to one embodiment, the external electronic device may display example images for each of the plurality of keywords.

[0488] According to one embodiment, the external electronic device may display a three-dimensional virtual image (2350) generated based on at least one keyword (2341) in a three-dimensional virtual space. For example, if the at least one keyword (2341) includes flowers, pink and purple, and castle, the generated three-dimensional virtual image (2350) may include a sky image (2351) of a changed color applied to a background area of ​​a first depth of the three-dimensional virtual space, an image of a castle with a pink roof surrounded by pink flowers applied to a second depth of the three-dimensional virtual space (2352), and at least one pink heart image (2353) placed in a ceiling area of ​​the three-dimensional virtual space.

[0489] In one embodiment, a three-dimensional virtual image (2350) applied to an electronic device may be applied as a background image (2360) to the background screen of a linked external electronic device. For example, the electronic device and the external electronic device may be linked through an account.

[0490] In one embodiment, the external electronic device may generate a two-dimensional background image (2360) based on a three-dimensional virtual image (2350) received from the electronic device. In one embodiment, the two-dimensional background image (2360) may be applied to the home screen and / or lock screen of the external electronic device.

[0491] According to the present disclosure, an electronic device can generate an image suitable for a virtual space by acquiring an expanded image through out-painting and in-painting based on a two-dimensional image. Furthermore, according to the present disclosure, the background and objects within the image can be separated and placed within the virtual space at different depths, thereby providing a sense of immersion to the user. Furthermore, according to the present disclosure, audio can be mapped and output based on the properties of the background and objects, thereby providing a sense of immersion to the user within the virtual space.

[0492] According to one embodiment, an electronic device (e.g., electronic device (101) of FIGS. 1A and 1B) may include a display (e.g., display module (160) of FIGS. 1A and 1B), a sensor (e.g., sensor module (176) of FIGS. 1A and 1B), a processor (e.g., processor (120) of FIGS. 1A and 1B), and a memory (e.g., memory (130) of FIGS. 1A and 1B) that includes instructions that, when executed by the processor, cause the electronic device to perform the following operations.

[0493] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to obtain, through the sensor, information related to a user's field of view.

[0494] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to obtain a two-dimensional image.

[0495] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to identify a first portion of the two-dimensional image to be applied as a background image in a three-dimensional virtual space.

[0496] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to identify a second portion of the two-dimensional image corresponding to an object of interest.

[0497] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to generate an extended image corresponding to the first portion of the two-dimensional image.

[0498] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to generate a three-dimensional virtual image by applying a first depth to the extended image and applying a second depth to the second portion of the two-dimensional image.

[0499] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to display a portion of the three-dimensional virtual image based on information related to the user's field of view.

[0500] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to display the portion of the three-dimensional virtual image at a size determined based on at least one of a size of the field of view or a direction of the field of view corresponding to the information associated with the field of view.

[0501] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to generate the extended image such that the image portion includes a region extending in a first direction from a first boundary of the two-dimensional image and a region extending in a second direction different from the first direction from a second boundary different from the first boundary.

[0502] According to one embodiment, the background area of ​​the three-dimensional virtual space may be configured as a curved surface having the same distance from a point set within the three-dimensional virtual space as the first depth.

[0503] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to correct the extended image to correspond to the curved shape.

[0504] In one embodiment, the instructions, when executed by the processor, may cause the electronic device, as part of the operation of generating the extended image, to generate an area corresponding to a first range within a primary movable range of the field of view by performing out-painting based on the two-dimensional image, based on at least information related to the field of view, and to generate an area corresponding to a second range different from the first range by performing in-painting based on an image generated in the area corresponding to the first range.

[0505] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to, as part of the operation of generating the extended image, generate the image portion based on the two-dimensional image so as to be symmetrical about at least one of a virtual line or boundary line at one or more points of the image portion.

[0506] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to generate the image portion by fading in at least one of a virtual line or boundary line at one or more points of the image portion generated based on the two-dimensional image, as part of the operation of generating the extended image.

[0507] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to receive the two-dimensional image through an external electronic device associated with a user account of the electronic device.

[0508] In one embodiment, the electronic device may be a head-mountable display device (HMD).

[0509] According to one embodiment, the electronic device may further include communication circuitry.

[0510] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to obtain the two-dimensional image from a first external electronic device via the communication circuit.

[0511] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to transmit the two-dimensional image acquired from the first external electronic device to a second external electronic device via the communication circuit.

[0512] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to receive, via the communication circuitry, the expanded image from the second external electronic device in response to the transmission.

[0513] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to determine properties of the object of interest included in the second portion.

[0514] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to add a moving graphic effect to the object of interest based at least in part on properties of the object of interest.

[0515] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to obtain environmental information.

[0516] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to determine properties of the graphic effect based at least on the environmental information.

[0517] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to identify contact information associated with the object of interest.

[0518] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to provide the contact information in response to receiving user input regarding the object of interest.

[0519] According to one embodiment, the instructions, when executed by the processor, may cause the electronic device to receive an event related to the contact information.

[0520] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to add or change a graphical effect associated with the object of interest based on the event.

[0521] In one embodiment, the instructions, when executed by the processor, may cause the electronic device, as part of generating the three-dimensional virtual image, to determine a property of an object of interest included in the second portion, and, based at least in part on the property of the object of interest, selectively place the second portion at one of a first point configured to be user-interactive or a second point configured not to be user-interactive.

[0522] According to one embodiment, a method of controlling an electronic device may include an operation of obtaining information related to a user's field of view through a sensor of the electronic device.

[0523] According to one embodiment, a method of controlling an electronic device may include an operation of obtaining a two-dimensional image.

[0524] According to one embodiment, a method of controlling an electronic device may include an operation of identifying a first portion of the two-dimensional image to be applied as a background image in a three-dimensional virtual space.

[0525] According to one embodiment, a method of controlling an electronic device may include an operation of identifying a second portion of the two-dimensional image corresponding to an object of interest.

[0526] According to one embodiment, a method of controlling an electronic device may include generating an extended image corresponding to the first portion of the two-dimensional image.

[0527] According to one embodiment, a method of controlling an electronic device may include applying a first depth to the extended image and applying a second depth to the second portion of the two-dimensional image to generate a three-dimensional virtual image.

[0528] According to one embodiment, a method of controlling an electronic device may include an action of displaying a portion of the three-dimensional virtual image based on information related to the field of view of the user.

[0529] In one embodiment, the operation of generating the three-dimensional virtual image may include determining an attribute of an object of interest included in the second portion, and selectively placing the second portion at one of a first point set to be interactive with the user or a second point set not to be interactive with the user, based at least in part on the attribute of the object of interest.

[0530] In one embodiment, the operation of generating the three-dimensional virtual image may include determining properties of the object of interest included in the second portion and adding a moving graphic effect to the object of interest based on at least a portion of the properties of the object of interest.

[0531] According to one embodiment, a non-transitory computer-readable recording medium storing one or more programs may store instructions that cause an electronic device to obtain information related to a user's field of view through the sensor.

[0532] According to one embodiment, the one or more programs may store instructions for causing the electronic device to obtain a two-dimensional image.

[0533] According to one embodiment, the one or more programs may store instructions for the electronic device to identify a first portion of the two-dimensional image to be applied as a background image in a three-dimensional virtual space.

[0534] According to one embodiment, the one or more programs may store instructions for causing the electronic device to identify a second portion of the two-dimensional image corresponding to an object of interest.

[0535] According to one embodiment, the one or more programs may store instructions that cause the electronic device to generate an extended image corresponding to the first portion of the two-dimensional image.

[0536] In one embodiment, the one or more programs may store instructions causing the electronic device to apply a first depth to the extended image and apply a second depth to the second portion of the two-dimensional image to generate a three-dimensional virtual image.

[0537] According to one embodiment, the one or more programs may store instructions that cause the electronic device to display a portion of the three-dimensional virtual image based on information related to the user's field of view.

[0538] According to one embodiment, an electronic device may include a display, a sensor for detecting a user's gaze as at least a portion of the field of view information, a memory for storing a virtual space including a three-dimensional background area, and a processor operatively connected to the sensor and the memory.

[0539] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to obtain a two-dimensional image.

[0540] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to generate an extended image that further includes an image portion generated in an outer direction of the two-dimensional image such that the image is continuous at a boundary corresponding to at least one of the corners of the two-dimensional image, based on an attribute of the two-dimensional image.

[0541] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to set at least a portion of the three-dimensional background area to include the expanded image.

[0542] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to display at least a portion of the virtual space including the set three-dimensional background area based on the field of view information.

[0543] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to generate the image portion at a size determined based on at least one of a size of the field of view or a direction of the field of view corresponding to the field of view information.

[0544] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to generate the extended image such that the image portion includes a region extending in a first direction from a first boundary of the two-dimensional image and a region extending in a second direction different from the first direction from a second boundary different from the first boundary.

[0545] According to one embodiment, at least a portion of the three-dimensional background area may be configured as a curved surface having a substantially equal distance from a point set within the virtual space.

[0546] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to correct the extended image to correspond to the curved shape.

[0547] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to, as part of the operation of generating the extended image, generate an area corresponding to a first range within a primary movable range of the field of view based on at least the field of view information by performing out-painting based on the two-dimensional image, and generate an area corresponding to a second range different from the first range by performing in-painting based on an image generated in the area corresponding to the first range.

[0548] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to, as part of the operation of generating the extended image, generate the image portion so as to be symmetrical about at least one of a virtual line or boundary line at one or more points in the image portion.

[0549] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to generate the image portion by fading in at least one of a virtual line or boundary line at one or more points of the image portion as part of the operation of generating the extended image.

[0550] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to receive the two-dimensional image through an external electronic device associated with a user account of the electronic device.

[0551] According to one embodiment, the electronic device may further include a communication module.

[0552] In one embodiment, the memory, when executed by the processor, causes the electronic device to, as part of an operation of generating the extended image, transmit a generation command prompt for generating the extended image to an external device using the communication device.

[0553] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to receive, from the external device, the expanded image generated based on the generation command prompt.

[0554] In one embodiment, the generation command prompt may store instructions for generating an extended image that further includes a portion of the image generated in an outer direction of the two-dimensional image such that the image is continuous at a boundary corresponding to at least one of the corners of the two-dimensional image, based on properties of the two-dimensional image.

[0555] In one embodiment, the generation command prompt may store instructions for generating the image portion at a size determined based on at least one of a size of the field of view or a direction of the field of view corresponding to the field of view information.

[0556] In one embodiment, the generation command prompt may store instructions for generating the extended image such that the image portion includes a region extending in a first direction from a first boundary of the two-dimensional image and a region extending in a second direction different from the first direction from a second boundary different from the first boundary.

[0557] According to one embodiment, at least a portion of the three-dimensional background area may be configured as a curved surface having a substantially equal distance from a point set within the virtual space.

[0558] In one embodiment, the generation command prompt may store instructions for correcting the extended image to correspond to the curved shape.

[0559] In one embodiment, the generation command prompt may store instructions for generating, as part of the operation of generating the extended image, an area corresponding to a first range belonging to a main movable range of the field of view based at least on the field of view information by performing out-painting based on the two-dimensional image, and generating an area corresponding to a second range different from the first range by performing in-painting based on an image generated in the area corresponding to the first range.

[0560] In one embodiment, the generation command prompt may store instructions for generating the image portion so as to be symmetrical about at least one of a virtual line or boundary line at one or more points of the image portion, as part of the operation of generating the extended image.

[0561] In one embodiment, the generation command prompt may store instructions for generating the image portion by fading in at least one of a virtual line or boundary line at one or more points of the image portion as part of the operation of generating the extended image.

[0562] According to one embodiment, a head-mounted display device (HMD) may include a display, a sensor configured to detect a gaze of a user of the HMD as part of field of view information, a memory configured to store image information representing a virtual space including a three-dimensional background area, and a processor operatively connected to the display, the sensor, and the memory.

[0563] In one embodiment, the memory may store instructions that, when executed by the processor, cause the head-mountable display device to acquire a two-dimensional image.

[0564] In one embodiment, the memory may store instructions that, when executed by the processor, cause the head-mounted display device to display at least a portion of the virtual space through the display such that an extended image corresponding to the two-dimensional image is included in at least a portion of the three-dimensional background area.

[0565] In one embodiment, the extended image may include a first image portion corresponding to the two-dimensional image, a second image portion corresponding to an image extended continuously in a first direction outward from an image of a first boundary portion corresponding to an edge of the two-dimensional image, and a third image portion corresponding to an image extended continuously in a second direction opposite to the first direction from an image of a second boundary portion corresponding to an edge of the two-dimensional image different from the first boundary.

[0566] In one embodiment, the memory may store instructions that, when executed by the processor, cause the head-mountable display device to generate the extended image based on properties of the two-dimensional image.

[0567] According to one embodiment, at least a portion of the three-dimensional background area may be configured as a curved surface having a substantially equal distance from a point set within the virtual space.

[0568] In one embodiment, the memory may store instructions that, when executed by the processor, cause the head-mounted display device to correct the extended image to correspond to the curved shape.

[0569] According to one embodiment, the head-mountable display device may further include a communication module.

[0570] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the head-mountable display device to obtain the two-dimensional image from a first external electronic device via the communication module.

[0571] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the head-mountable display device to transmit the two-dimensional image acquired from the first external electronic device to a second external electronic device via the communication module.

[0572] In one embodiment, the memory may store instructions that, when executed by the processor, cause the head-mountable display device to receive, in response to the transmission, the expanded image from the second external electronic device via the communication module.

[0573] In one embodiment, the extended image received from the external device may further include an image portion generated in an outer direction of the two-dimensional image such that the shape continues at a boundary corresponding to at least one of the corners of the two-dimensional image, based on properties of the two-dimensional image.

[0574] In one embodiment, the extended image received from the external device may be generated such that the image portion corresponds to a size determined based on at least one of a size of the field of view corresponding to the field of view information or a direction of the field of view.

[0575] In one embodiment, the extended image received from the external device may be generated such that the image portion includes a region extending in a first direction from a first boundary of the two-dimensional image and a region extending in a second direction different from the first direction from a second boundary different from the first boundary.

[0576] According to one embodiment, at least a portion of the three-dimensional background area may be configured as a curved surface having a substantially equal distance from a point set within the virtual space.

[0577] In one embodiment, the extended image received from the external device may be corrected to correspond to the curved shape of the extended image.

[0578] According to one embodiment, the extended image received from the external device may be generated by performing out-painting based on the two-dimensional image for an area corresponding to a first range belonging to the main movable range of the field of view based at least on the field of view information, and by performing in-painting based on an image generated for an area corresponding to a second range different from the first range.

[0579] In one embodiment, the extended image received from the external device may be generated such that the image portion is symmetrical with respect to at least one of a virtual line or boundary line at one or more points of the image portion.

[0580] According to one embodiment, the extended image received from the external device may be generated by fading in at least one of a virtual line or a boundary line at one or more points of the image portion.

[0581] In one embodiment, the memory, when executed by the processor, causes the head-mountable display device to receive the two-dimensional image through an external electronic device associated with a user account of the electronic device.

[0582] According to one embodiment, an electronic device may include a display, a sensor for detecting a user's gaze as at least part of the field of view information, a memory for storing a virtual space including a three-dimensional background area, and a processor.

[0583] According to one embodiment, the memory, when executed by the processor, causes the electronic device to obtain a two-dimensional image.

[0584] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to recognize a background area and an object area within the two-dimensional image.

[0585] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to separate the object region from the two-dimensional image.

[0586] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to generate an extended image that further includes a portion of the image generated outside the background area so as to extend beyond the boundary of the background area, based on an attribute of the background area.

[0587] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to set at least a portion of the three-dimensional background area to include the expanded image.

[0588] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to position the object area between a point set within the virtual space and the three-dimensional background area in relation to the field of view information.

[0589] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to display at least a portion of the virtual space including the set three-dimensional background area and the placed object area based on the field of view information.

[0590] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to generate an image in the object area by performing in-painting using at least a portion of the background area adjacent to the object area, as at least part of an operation of generating the extended image.

[0591] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to determine attributes of an object included in the object area.

[0592] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to add a moving graphic effect to the object based at least in part on properties of the object.

[0593] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to obtain environmental information.

[0594] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to determine properties of the graphic effect based at least on the environmental information.

[0595] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to identify contact information associated with the object area.

[0596] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to provide the contact information in response to receiving user input for the object area.

[0597] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to receive an event related to the contact information.

[0598] According to one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device to add or change a graphic effect associated with the object area based on the event.

[0599] In one embodiment, the memory may store instructions that, when executed by the processor, cause the electronic device, as part of the placing operation, to determine an attribute of an object included in the object area and, based at least in part on the attribute of the object, selectively place the object area at one of a first point set to be user-interactive or a second point set not to be user-interactive.

[0600] According to one embodiment, a method of controlling an electronic device may include an operation of obtaining a two-dimensional image.

[0601] According to one embodiment, a method of controlling an electronic device may include generating an extended image further including an image portion generated in an outer direction of the two-dimensional image such that the image is continuous at a boundary corresponding to at least one of the corners of the two-dimensional image, based on an attribute of the two-dimensional image.

[0602] According to one embodiment, a method of controlling an electronic device may include an operation of setting the expanded image to be included in at least a portion of the three-dimensional background area.

[0603] According to one embodiment, a control method of an electronic device may include an operation of displaying at least a portion of the virtual space including the set three-dimensional background area based on the field of view information.

[0604] According to one embodiment, the operation of generating the extended image may generate the image portion with a size determined based on at least one of a size of the field of view or a direction of the field of view corresponding to the field of view information.

[0605] In one embodiment, the operation of generating the extended image may generate the extended image such that the image portion includes a region extending in a first direction from a first boundary of the two-dimensional image and a region extending in a second direction different from the first direction from a second boundary different from the first boundary.

[0606] According to one embodiment, at least a portion of the three-dimensional background area may be configured as a curved surface so that the distance from a point set within the virtual space is substantially the same.

[0607] According to one embodiment, the operation of generating the extended image may further include an operation of correcting the extended image to correspond to the curved shape.

[0608] According to one embodiment, the operation of generating the extended image may include generating an area corresponding to a first range belonging to the main movable range of the field of view based at least on the field of view information by performing out-painting based on the two-dimensional image, and generating an area corresponding to a second range different from the first range by performing in-painting based on an image generated in the area corresponding to the first range.

[0609] According to one embodiment, the operation of generating the extended image may generate the image portion so as to be symmetrical with respect to at least one of a virtual line or boundary line at one or more points of the image portion.

[0610] According to one embodiment, the operation of generating the extended image may generate the image portion by performing a fade-in process based on at least one of a virtual line or a boundary line at one or more points of the image portion.

[0611] According to one embodiment, the operation of obtaining the two-dimensional image may include receiving the two-dimensional image through an external electronic device associated with a user account of the electronic device.

[0612] According to one embodiment, a method of controlling a head-mounted display device (HMD) may include an operation of acquiring a two-dimensional image.

[0613] According to one embodiment, the control method of the head-mounted display device may include an operation of displaying at least a portion of the virtual space through the display such that an extended image corresponding to the two-dimensional image is included in at least a portion of the three-dimensional background area.

[0614] In one embodiment, the extended image may include a first image portion corresponding to the two-dimensional image, a second image portion corresponding to an image extended continuously in a first direction outward from an image of a first boundary portion corresponding to an edge of the two-dimensional image, and a third image portion corresponding to an image extended continuously in a second direction opposite to the first direction from an image of a second boundary portion corresponding to an edge of the two-dimensional image different from the first boundary.

[0615] According to one embodiment, the control method of the head-mountable display device may include an operation of generating the extended image based on properties of the two-dimensional image.

[0616] According to one embodiment, at least a portion of the three-dimensional background area may be configured as a curved surface having a substantially equal distance from a point set within the virtual space.

[0617] According to one embodiment, the control method of the head-mountable display device may further include an operation of correcting the extended image to correspond to the curved shape.

[0618] According to one embodiment, the operation of obtaining the two-dimensional image may include an operation of obtaining the two-dimensional image from a first external electronic device via a communication module.

[0619] According to one embodiment, the control method of the head-mountable display device may further include an operation of transmitting the two-dimensional image acquired from the first external electronic device to a second external electronic device via the communication module.

[0620] According to one embodiment, the control method of the head-mountable display device may further include, in response to the transmission, receiving the expanded image from the second external electronic device via the communication module.

[0621] According to one embodiment, a method of controlling an electronic device may include an operation of obtaining a two-dimensional image.

[0622] According to one embodiment, the control method of the electronic device may include an operation of recognizing a background area and an object area within the two-dimensional image.

[0623] According to one embodiment, the control method of the electronic device may include an operation of separating the object area from the two-dimensional image.

[0624] According to one embodiment, the control method of the electronic device may include generating an extended image that further includes a portion of the image generated outside the background area so as to continue from a boundary of the background area, based on an attribute of the background area.

[0625] According to one embodiment, the control method of the electronic device may include an operation of setting the expanded image to be included in at least a portion of the three-dimensional background area.

[0626] According to one embodiment, the control method of the electronic device may include an operation of positioning the object area between a point set within the virtual space and the three-dimensional background area in relation to the field of view information.

[0627] According to one embodiment, the control method of the electronic device may include an operation of displaying at least a portion of the virtual space including the set three-dimensional background area and the placed object area based on the field of view information.

[0628] In one embodiment, the operation of generating the extended image may generate an image in the object area by performing in-painting using at least a portion of the background area adjacent to the object area.

[0629] According to one embodiment, the control method of the electronic device may further include an operation of determining an attribute of an object included in the object area.

[0630] According to one embodiment, the control method of the electronic device may further include adding a moving graphic effect to the object based on at least some of the properties of the object.

[0631] According to one embodiment, the control method of the electronic device may further include an operation of obtaining environmental information.

[0632] According to one embodiment, the control method of the electronic device may further include an operation of determining a property of the graphic effect based at least on the environmental information.

[0633] According to one embodiment, the control method of the electronic device may further include an operation of identifying contact information associated with the object area.

[0634] According to one embodiment, the control method of the electronic device may further include an action of providing the contact information in response to receiving a user input for the object area.

[0635] According to one embodiment, the control method of the electronic device may further include an operation of receiving an event related to the contact information.

[0636] According to one embodiment, the control method of the electronic device may further include an operation of adding or changing a graphic effect related to the object area based on the event.

[0637] In one embodiment, the placing action may determine an attribute of an object included in the object area, and selectively place the object area at one of a first point set to be interactive with the user or a second point set not to be interactive with the user, based at least in part on the attribute of the object.

[0638] According to one embodiment, a non-transitory computer-readable recording medium storing one or more programs may store instructions that cause an electronic device to obtain a two-dimensional image.

[0639] In one embodiment, the one or more programs may store instructions that cause the electronic device to generate an extended image that further includes an image portion generated in an outer direction of the two-dimensional image such that the image continues at a boundary corresponding to at least one of the corners of the two-dimensional image, based on properties of the two-dimensional image.

[0640] According to one embodiment, the one or more programs may store instructions that cause the electronic device to set at least a portion of the three-dimensional background area to include the extended image.

[0641] According to one embodiment, the one or more programs may store instructions that cause the electronic device to display at least a portion of the virtual space including the set three-dimensional background area based on the field of view information.

[0642] According to one embodiment, the one or more programs may store instructions that cause the electronic device to generate the image portion with a size determined based on at least one of a size of the field of view or a direction of the field of view corresponding to the field of view information.

[0643] In one embodiment, the one or more programs may store instructions that cause the electronic device to generate the extended image such that the image portion includes a region extending in a first direction from a first boundary of the two-dimensional image and a region extending in a second direction different from the first direction from a second boundary different from the first boundary.

[0644] According to one embodiment, at least a portion of the three-dimensional background area may be configured as a curved surface having a substantially equal distance from a point set within the virtual space.

[0645] According to one embodiment, the one or more programs may store instructions that cause the extended image to be corrected to correspond to the curved shape.

[0646] According to one embodiment, the one or more programs may store instructions that cause the electronic device, as part of the operation of generating the extended image, to generate an area corresponding to a first range within a main movable range of the field of view based at least on the field of view information by performing out-painting based on the two-dimensional image, and to generate an area corresponding to a second range different from the first range by performing in-painting based on an image generated in the area corresponding to the first range.

[0647] In one embodiment, the one or more programs may store instructions that cause the electronic device, as part of the operation of generating the extended image, to generate the image portion so as to be symmetrical about at least one of a virtual line or boundary line at one or more points of the image portion.

[0648] According to one embodiment, the one or more programs may store instructions that cause the electronic device to generate the image portion by fading in at least one of a virtual line or boundary line at one or more points of the image portion as part of the operation of generating the extended image.

[0649] According to one embodiment, the one or more programs may store instructions that cause the electronic device to receive the two-dimensional image through an external electronic device associated with a user account of the electronic device.

[0650] According to one embodiment, the one or more programs may store instructions that cause the electronic device to, as part of the operation of generating the extended image, transmit a generation command prompt for generation of the extended image to an external device using a communication device.

[0651] According to one embodiment, the one or more programs may store instructions that cause the electronic device to receive, from the external device, the expanded image generated based on the generation command prompt.

[0652] In one embodiment, the generation command prompt may store instructions for generating an extended image that further includes a portion of the image generated in an outer direction of the two-dimensional image such that the image is continuous at a boundary corresponding to at least one of the corners of the two-dimensional image, based on properties of the two-dimensional image.

[0653] In one embodiment, the generation command prompt may store instructions for generating the image portion at a size determined based on at least one of a size of the field of view or a direction of the field of view corresponding to the field of view information.

[0654] In one embodiment, the generation command prompt may store instructions for generating the extended image such that the image portion includes a region extending in a first direction from a first boundary of the two-dimensional image and a region extending in a second direction different from the first direction from a second boundary different from the first boundary.

[0655] According to one embodiment, at least a portion of the three-dimensional background area may be configured as a curved surface having a substantially equal distance from a point set within the virtual space.

[0656] In one embodiment, the generation command prompt may store instructions for correcting the extended image to correspond to the curved shape.

[0657] In one embodiment, the generation command prompt may store instructions for generating, as part of the operation of generating the extended image, an area corresponding to a first range belonging to a main movable range of the field of view based at least on the field of view information by performing out-painting based on the two-dimensional image, and generating an area corresponding to a second range different from the first range by performing in-painting based on an image generated in the area corresponding to the first range.

[0658] In one embodiment, the generation command prompt may store instructions for generating the image portion so as to be symmetrical about at least one of a virtual line or boundary line at one or more points of the image portion, as part of the operation of generating the extended image.

[0659] In one embodiment, the generation command prompt may store instructions for generating the image portion by fading in at least one of a virtual line or boundary line at one or more points of the image portion as part of the operation of generating the extended image.

[0660] According to one embodiment, a non-transitory computer-readable recording medium storing one or more programs, wherein the one or more programs can store instructions that cause a head-mounted display device (HMD) to acquire a two-dimensional image.

[0661] According to one embodiment, the one or more programs may store instructions that cause the head-mounted display device to display at least a portion of the virtual space through the display such that an extended image corresponding to the two-dimensional image is included in at least a portion of the three-dimensional background area.

[0662] In one embodiment, the extended image may include a first image portion corresponding to the two-dimensional image, a second image portion corresponding to an image extended continuously in a first direction outward from an image of a first boundary portion corresponding to an edge of the two-dimensional image, and a third image portion corresponding to an image extended continuously in a second direction opposite to the first direction from an image of a second boundary portion corresponding to an edge of the two-dimensional image different from the first boundary.

[0663] According to one embodiment, the one or more programs may store instructions that cause the head-mountable display device to generate the extended image based on properties of the two-dimensional image.

[0664] According to one embodiment, at least a portion of the three-dimensional background area may be configured as a curved surface having a substantially equal distance from a point set within the virtual space.

[0665] In one embodiment, the one or more programs may store instructions that cause the head-mounted display device to correct the extended image to correspond to the curved shape.

[0666] According to one embodiment, the one or more programs may store instructions that cause the head-mountable display device to obtain the two-dimensional image from a first external electronic device via a communication module.

[0667] According to one embodiment, the one or more programs may store instructions that cause the head-mountable display device to transmit the two-dimensional image acquired from the first external electronic device to a second external electronic device via the communication module.

[0668] In one embodiment, the one or more programs may store instructions that cause the head-mountable display device to receive, in response to the transmission, the expanded image from the second external electronic device via the communication module.

[0669] In one embodiment, the extended image received from the external device may further include an image portion generated in an outer direction of the two-dimensional image such that the shape continues at a boundary corresponding to at least one of the corners of the two-dimensional image, based on properties of the two-dimensional image.

[0670] In one embodiment, the extended image received from the external device may be generated such that the image portion corresponds to a size determined based on at least one of a size of the field of view corresponding to the field of view information or a direction of the field of view.

[0671] In one embodiment, the extended image received from the external device may be generated such that the image portion includes a region extending in a first direction from a first boundary of the two-dimensional image and a region extending in a second direction different from the first direction from a second boundary different from the first boundary.

[0672] According to one embodiment, at least a portion of the three-dimensional background area may be configured as a curved surface having a substantially equal distance from a point set within the virtual space.

[0673] In one embodiment, the extended image received from the external device may be corrected to correspond to the curved shape of the extended image.

[0674] According to one embodiment, the extended image received from the external device may be generated by performing out-painting based on the two-dimensional image for an area corresponding to a first range belonging to the main movable range of the field of view based at least on the field of view information, and by performing in-painting based on an image generated for an area corresponding to a second range different from the first range.

[0675] In one embodiment, the extended image received from the external device may be generated such that the image portion is symmetrical with respect to at least one of a virtual line or boundary line at one or more points of the image portion.

[0676] According to one embodiment, the extended image received from the external device may be generated by fading in at least one of a virtual line or a boundary line at one or more points of the image portion.

[0677] In one embodiment, the one or more programs may cause the head-mountable display device to receive the two-dimensional image via an external electronic device associated with a user account of the electronic device.

[0678] In one embodiment, a non-transitory computer-readable recording medium storing one or more programs, wherein the one or more programs cause the electronic device to obtain a two-dimensional image.

[0679] According to one embodiment, the one or more programs may store instructions that cause the electronic device to recognize a background area and an object area within the two-dimensional image.

[0680] According to one embodiment, the one or more programs may store instructions that cause the electronic device to separate the object region from the two-dimensional image.

[0681] In one embodiment, the one or more programs may store instructions that cause the electronic device to generate an extended image that further includes a portion of the image generated outside the background area so as to extend beyond the boundary of the background area, based on properties of the background area.

[0682] In one embodiment, the one or more programs may store instructions that cause the electronic device to set the expanded image to be included in at least a portion of the three-dimensional background area.

[0683] According to one embodiment, the one or more programs may store instructions that cause the electronic device to place the object area between a point set within the virtual space and the three-dimensional background area in relation to the field of view information.

[0684] According to one embodiment, the one or more programs may store instructions that cause the electronic device to display at least a portion of the virtual space including the set three-dimensional background area and the placed object area based on the field of view information.

[0685] In one embodiment, the one or more programs may store instructions that cause the electronic device to, as at least part of the operation of generating the extended image, perform in-painting using at least a portion of the background area adjacent to the object area to generate an image in the object area.

[0686] According to one embodiment, the one or more programs may store instructions that cause the electronic device to determine properties of an object included in the object area.

[0687] According to one embodiment, the one or more programs may store instructions that cause the electronic device to add a moving graphic effect to the object based at least in part on properties of the object.

[0688] According to one embodiment, the one or more programs may store instructions that cause the electronic device to obtain environmental information.

[0689] According to one embodiment, the one or more programs may store instructions that cause the electronic device to determine properties of the graphic effect based at least on the environmental information.

[0690] According to one embodiment, the one or more programs may store instructions that cause the electronic device to identify contact information associated with the object area.

[0691] According to one embodiment, the one or more programs may store instructions that cause the electronic device to, in response to receiving user input for the object area, provide the contact information.

[0692] According to one embodiment, the one or more programs may store instructions that cause the electronic device to receive an event related to the contact information.

[0693] According to one embodiment, the one or more programs may store instructions that cause the electronic device to add or change a graphic effect associated with the object area based on the event.

[0694] In one embodiment, the one or more programs may store instructions that cause the electronic device, as part of the placing operation, to determine an attribute of an object included in the object area and, based at least in part on the attribute of the object, selectively place the object area at one of a first point set to be user-interactive or a second point set not to be user-interactive.

[0695] Electronic devices according to the embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments disclosed in this document are not limited to the aforementioned devices.

[0696] The embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together with the corresponding phrase among the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish the corresponding component from other corresponding components and do not limit the corresponding components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0697] The term "module" used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0698] One embodiment of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0699] According to one embodiment, the method according to one embodiment disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0700] According to one embodiment, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to one embodiment, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to one embodiment, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0701] It will be appreciated that the various embodiments of the disclosure according to the claims and detailed description of the specification may be realized in the form of hardware, software, or a combination of hardware and software.

[0702] Such software may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores one or more computer programs (software modules), and the one or more computer programs include computer-executable instructions that, when individually or collectively executed by one or more processors of an electronic device, cause the electronic device to perform the method of disclosure.

[0703] Such software may be stored in the form of volatile or non-volatile storage, such as, for example, a storage device such as read-only memory (ROM), whether erasable or rewritable, or in the form of memory such as, for example, random access memory (RAM), a memory chip, device or integrated circuit, or on, for example, a compact disc (CD), a digital versatile disc (DVD), a magnetic disk or magnetic tape or similar optically or magnetically readable medium. It will be appreciated that the storage device and the storage medium are various embodiments of non-transitory machine-readable storage suitable for storing a computer program or computer programs comprising instructions that, when executed, implement various embodiments of the disclosure. Accordingly, various embodiments provide a program comprising code for implementing an apparatus or method as claimed in one of the claims of the present specification and a non-transitory machine-readable storage storing such a program.

[0704] While the present disclosure has been shown and described with reference to various embodiments, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present disclosure as defined by the appended claims and their equivalents.

Claims

1. In an electronic device (101), display (160); sensor (176); a memory (130) storing one or more computer programs; and comprising one or more processors (120) communicatively connected to the display, the sensor and the memory; The one or more computer programs comprise computer-executable instructions, which, when individually or collectively executed by the one or more processors, cause the electronic device to: Through the above sensor, information related to the user's field of view is obtained, Obtain a two-dimensional image, Check the first part of the above two-dimensional image to be applied as a background image in the three-dimensional virtual space, Identify the second part of the two-dimensional image corresponding to the object of interest, Generating an extended image corresponding to the first portion of the above two-dimensional image, Applying a first depth to the extended image and applying a second depth to the second part of the two-dimensional image to generate a three-dimensional virtual image, and An electronic device that displays a portion of the three-dimensional virtual image based on information related to the user's field of view.

2. In paragraph 1, The one or more computer programs, when individually or collectively executed by the one or more processors, cause the electronic device to: An electronic device further comprising instructions for displaying said portion of said three-dimensional virtual image at a size determined based on at least one of a size of the field of view or a direction of the field of view corresponding to said information related to said field of view.

3. In paragraph 1, The one or more computer programs, when individually or collectively executed by the one or more processors, cause the electronic device to: An electronic device further comprising instructions for generating the extended image such that the generated image portion of the extended image includes a region extending in a first direction from a first boundary of the two-dimensional image and a region extending in a second direction different from the first direction from a second boundary different from the first boundary.

4. In paragraph 1, The background area of ​​the above three-dimensional virtual space is It is configured in a curved shape with a distance from a point set within the above three-dimensional virtual space to the first depth, The one or more computer programs, when individually or collectively executed by the one or more processors, cause the electronic device to: An electronic device further comprising instructions for correcting the extended image to correspond to the curved shape.

5. In paragraph 1, The one or more computer programs, when individually or collectively executed by the one or more processors, cause the electronic device to: An electronic device further comprising instructions for generating, as part of the operation of generating the extended image, an area corresponding to a first range belonging to a main movable range of the field of view based at least on information related to the field of view, by performing out-painting based on the two-dimensional image, and generating an area corresponding to a second range different from the first range by performing in-painting based on an image generated in the area corresponding to the first range.

6. In paragraph 1, The one or more computer programs, when individually or collectively executed by the one or more processors, cause the electronic device to: An electronic device further comprising instructions for generating the image portion so as to be symmetrical about at least one of a virtual line or boundary line at one or more points of an image portion generated based on the two-dimensional image among the extended image, as part of the operation of generating the extended image.

7. In paragraph 1, The one or more computer programs, when individually or collectively executed by the one or more processors, cause the electronic device to: An electronic device further comprising instructions for generating the image portion by performing a fade-in process based on at least one of a virtual line or boundary line at one or more points of an image portion generated based on the two-dimensional image among the extended image, as part of the operation of generating the extended image.

8. In paragraph 1, The one or more computer programs, when individually or collectively executed by the one or more processors, cause the electronic device to: An electronic device further comprising instructions for receiving the two-dimensional image via an external electronic device associated with a user account of the electronic device.

9. In paragraph 1, The above electronic device, An electronic device that is a head-mounted display, HMD, device.

10. In paragraph 1, Further comprising a communication circuit (190), The one or more computer programs, when individually or collectively executed by the one or more processors, cause the electronic device to: Through the above communication circuit, the two-dimensional image is acquired from the first external electronic device, Transmitting the two-dimensional image acquired from the first external electronic device to the second external electronic device through the communication circuit, An electronic device further comprising instructions for receiving, in response to said transmission, said extended image from said second external electronic device via said communication circuit.

11. In paragraph 1, The one or more computer programs, when individually or collectively executed by the one or more processors, cause the electronic device to: Determining the properties of the object of interest included in the second part, and Further comprising instructions for adding a moving graphic effect to the object of interest based at least in part on properties of the object of interest, and The one or more computer programs, when individually or collectively executed by the one or more processors, cause the electronic device to: Obtain environmental information, and An electronic device further comprising instructions for determining properties of the graphic effect based at least on the environmental information.

12. In paragraph 11, The one or more computer programs, when individually or collectively executed by the one or more processors, cause the electronic device to: Identify contact information related to the above object of interest, In response to receiving user input for said object of interest, further comprising instructions for providing said contact information; and The one or more computer programs, when individually or collectively executed by the one or more processors, cause the electronic device to: Receive events related to the above contact information, An electronic device further comprising instructions for adding or changing a graphic effect associated with the object of interest based on the event.

13. In paragraph 1, The one or more computer programs, when individually or collectively executed by the one or more processors, cause the electronic device to: An electronic device further comprising instructions for determining, as part of the operation of generating the three-dimensional virtual image, a property of an object of interest included in the second portion, and selectively placing the second portion at one of a first point set to be interactable with the user or a second point set not to be interactable with the user, based at least in part on the property of the object of interest.

14. In a method for controlling an electronic device (101), An operation (1010) of obtaining information related to the user's field of view through a sensor (176) of the electronic device; An operation for acquiring a two-dimensional image (1020); An operation (1030) of confirming a first part of the two-dimensional image to be applied as a background image in a three-dimensional virtual space; An operation (1040) of identifying a second portion of the two-dimensional image corresponding to the object of interest; An operation (1050) of generating an extended image corresponding to the first portion of the two-dimensional image; An operation (1060) of applying a first depth to the extended image and applying a second depth to the second part of the two-dimensional image to generate a three-dimensional virtual image; and A method of controlling an electronic device, comprising an action (1070) of displaying a portion of the three-dimensional virtual image based on information related to the field of view of the user.

15. A non-transitory computer-readable storage medium storing one or more programs, wherein the one or more programs include computer-executable instructions, which, when individually or collectively executed by one or more processors (120) of an electronic device (101), cause the electronic device to perform operations of any one of claims 1 to 13, the operations comprising: An operation (1010) of obtaining information related to the user's field of view through a sensor of the electronic device; An operation for acquiring a two-dimensional image (1020); An operation (1030) of confirming a first part of the two-dimensional image to be applied as a background image in a three-dimensional virtual space; An operation (1040) of identifying a second portion of the two-dimensional image corresponding to the object of interest; An operation (1050) of generating an extended image corresponding to the first portion of the two-dimensional image; An operation (1060) of applying a first depth to the extended image and applying a second depth to the second part of the two-dimensional image to generate a three-dimensional virtual image. A storage medium including an operation (1070) of displaying a portion of the three-dimensional virtual image based on information related to the field of view of the user.

Images