Method for displaying three-dimensional image, storage medium supporting same, and electronic device supporting same

Abstract

An electronic device disclosed in the present document may comprise: a display; a memory; and at least one processor. The memory may store instructions that, when executed individually or collectively by the at least one processor, instruct at least one electronic device to: display a 3D image in a first mode; identify a request to switch from the first mode to a second mode; determine whether switching is possible within a specified time; generate a first intermediate image on the basis of the switching being possible within the specified time; generate a second intermediate image on the basis of the switching being possible after the specified time has elapsed; and output the first intermediate image or the second intermediate image during switching from the first mode to the second mode.

Description

Method for displaying three-dimensional images, storage medium supporting the same, and electronic device supporting the same

[0001] Embodiments disclosed in this document relate to a method for displaying a three-dimensional image and an electronic device supporting the same.

[0002] Electronic devices worn on the user's body that provide three-dimensional images, such as augmented reality (AR) glasses and head-mounted displays (HMDs), are becoming increasingly popular. These devices can perceive the user's surroundings and provide augmented reality, virtual reality, or mixed reality experiences that reflect this perception.

[0003] Augmented reality (AR) is a technology that superimposes virtual information onto real-world images displayed on electronic devices. Virtual reality (VR) is a technology that displays virtual information and / or pre-recorded preview images on electronic devices. Mixed reality (MR) is a technology that outputs complex content that combines VR and AR.

[0004] The electronic device may display a loading screen or provide simple effects when switching between 3D modes.

[0005] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0006] An electronic device according to one embodiment may include at least one processor including a display, a memory, and processing circuitry. The memory may store instructions that, when individually or collectively executed by the at least one processor, cause the electronic device to display a 3D image in a first mode, confirm a request to switch from the first mode to a second mode, determine whether switching is possible within a specified time period, process a first image in the first mode and a second image in the second mode according to a first image processing method to generate a first intermediate image based on whether switching is possible within the specified time period, process the first image in the first mode and the second image in the second mode according to a second image processing method to generate a second intermediate image based on whether switching is possible beyond the specified time period, and output the first intermediate image or the second intermediate image during the switching process from the first mode to the second mode.

[0007] An electronic device according to one embodiment may include at least one processor including a display, a memory, and processing circuitry. The memory may store instructions that, when individually or collectively executed by the at least one processor, cause the electronic device to display a 3D image on the display in a first mode, confirm a request to switch from the first mode to a second mode, create at least one virtual object related to the second mode, and, during the process of switching from the first mode to the second mode, add and output the at least one virtual object to a first image in the first mode.

[0008] A method for displaying a three-dimensional image according to one embodiment may be performed in an electronic device. The method may include an operation of displaying a 3D image in a first mode, an operation of confirming a request to switch from the first mode to a second mode, an operation of determining whether switching is possible within a specified time, an operation of processing a first image in the first mode and a second image in a second mode according to a first image processing method based on whether switching is possible within the specified time, and generating a first intermediate image, an operation of processing the first image in the first mode and the second image in the second mode according to a second image processing method based on whether switching is possible beyond the specified time, and an operation of outputting the first intermediate image or the second intermediate image during the switching from the first mode to the second mode.

[0009] A computer-readable non-transitory storage medium according to one embodiment may store instructions executable by a processor. When the instructions are executed, the electronic device may perform the following operations: displaying a 3D image in a first mode; confirming a request to switch from the first mode to a second mode; determining whether switching is possible within a specified time; generating a first intermediate image by processing a first image in the first mode and a second image in the second mode according to a first image processing method based on whether switching is possible within the specified time; generating a second intermediate image by processing the first image in the first mode and the second image in the second mode according to a second image processing method based on whether switching is possible beyond the specified time; and outputting the first intermediate image or the second intermediate image during the switching from the first mode to the second mode.

[0010] FIG. 1A is a block diagram of an electronic device within a network environment according to various embodiments.

[0011] FIG. 1b is a schematic diagram of an electronic device that outputs a three-dimensional image according to one embodiment.

[0012] Figure 2 illustrates a display of an intermediate image according to one embodiment.

[0013] Figure 3 illustrates an intermediate screen transition section according to one embodiment.

[0014] Figure 4 is a flowchart illustrating a three-dimensional image display method according to one embodiment.

[0015] Figure 5 illustrates the generation of real object and virtual object based intermediate images according to one embodiment.

[0016] Figure 6 shows changes in the intermediate image according to the loading progress of the second mode according to one embodiment.

[0017] Fig. 7 is a flowchart illustrating a method for displaying a three-dimensional image in an animated manner according to one embodiment.

[0018] Figure 8a illustrates the generation of an intermediate image by an alpha blending technique according to one embodiment.

[0019] Figure 8b illustrates the generation of an intermediate image using generative AI according to one embodiment.

[0020] Figure 9 illustrates the display of an intermediate image according to a user's position according to one embodiment.

[0021] Figure 10 illustrates a transition from a first VR mode to a second VR mode according to one embodiment.

[0022] Figures 11a and 11b illustrate the generation of intermediate images using real objects according to one embodiment.

[0023] Figures 12a and 12b illustrate the generation of intermediate images using virtual objects according to one embodiment.

[0024] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0025] Hereinafter, various embodiments of this document will be described with reference to the attached drawings. However, this is not intended to limit the technology described in this document to specific embodiments, and it should be understood that various modifications, equivalents, and / or alternatives of the embodiments of this document are included. In connection with the description of the drawings, similar reference numerals may be used for similar components.

[0026]

[0027] FIG. 1A is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1A, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with the electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0028] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0029] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0030] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0031] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0032] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0033] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0034] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0035] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0036] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0037] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0038] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0039] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0040] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0041] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0042] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0043] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0044] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0045] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0046] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0047] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0048] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0049]

[0050] FIG. 1b is a schematic diagram of an electronic device that outputs a three-dimensional image according to one embodiment.

[0051] Referring to FIG. 1B, the electronic device (101) may be a device that provides augmented reality (AR), virtual reality (VR), or mixed reality (MR). For example, the electronic device (101) may be a smart glass or head-mounted display (HMD) device that supports video see-through (VST).

[0052] An electronic device (101) (e.g., the electronic device (101) of FIG. 1A) may include a processor (120) (e.g., the processor (120) of FIG. 1A), a memory (130) (e.g., the memory (130) of FIG. 1A), a display (160) (e.g., the display module (160) of FIG. 1A), a sensor (176) (e.g., the sensor module (176) of FIG. 1A), and / or a camera (180) (e.g., the camera module (180) of FIG. 1A). The configuration of the electronic device (101) of FIG. 1B is exemplary, and the electronic device (101) may further include other configurations. For example, the electronic device (101) may further include a wireless communication circuit or an object recognition device.

[0053] The processor (120) (e.g., the processor (120) of FIG. 1A) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140) of FIG. 1A), and may perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components in a volatile memory (e.g., the volatile memory (132) of FIG. 1A), process the commands or data stored in the volatile memory (132), and store result data in a nonvolatile memory (e.g., the nonvolatile memory (134) of FIG. 1A).

[0054] The processor (120) can process operations for providing augmented reality, virtual reality, or mixed reality. The processor (120) can share the operations for providing augmented reality, virtual reality, or mixed reality with an external device (e.g., the server (108) of FIG. 1A) and process them (e.g., cloud computing). For example, if the electronic device (101) further includes a wireless communication circuit (e.g., the communication module (190) of FIG. 1A), the processor (120) can exchange data for providing augmented reality, virtual reality, or mixed reality with the external device via the wireless communication circuit. The external device can process operations related to the data and then transmit the data to the electronic device (101).

[0055] The memory (130) (e.g., the memory (130) of FIG. 1A) can store various data used by at least one component (e.g., the processor (120) or the sensor (176)) of the electronic device (101). The data can include, for example, software (e.g., the program (140) of FIG. 1A) and input data or output data for commands related thereto. The memory (130) can include a volatile memory (132) or a non-volatile memory (134). The memory (130) can store data for providing augmented reality, virtual reality, or mixed reality. For example, the memory (100) can store data for augmented reality elements, virtual reality elements, or mixed reality elements mapped to a 3D space.

[0056] A sensor (176) (e.g., a sensor module (176) of FIG. 1A) can detect an operating state (e.g., power) of the electronic device (101) or an external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor (176) may include, for example, a gesture sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, and / or a biometric sensor. The electronic device (101) can detect a user's motion using the sensor (176) and provide augmented reality, virtual reality, or mixed reality based on the user's motion.

[0057] In one embodiment, the display (160) (e.g., the display (160) of FIG. 1A) can visually provide information to an external party (e.g., a user) of the electronic device (101). For example, if AR is supported, the display (160) can be a see-through display. The electronic device (101) can output augmented reality elements, virtual reality elements, or mixed reality elements to the display (160). The user can view the augmented reality elements, virtual reality elements, or mixed reality elements through the display (160).

[0058] The electronic device (101) can provide augmented reality, virtual reality, or mixed reality to the user through the display (160). Augmented reality, virtual reality, or mixed reality can be understood as an output of augmented reality elements, virtual reality elements, or mixed reality elements corresponding to the screen viewed by the user.

[0059] According to one embodiment, the electronic device (101) can recognize objects and environments in a three-dimensional space, and map corresponding augmented reality elements, virtual reality elements, or mixed reality elements to the recognized objects or environments. The electronic device (101) can store mapping data in the memory (130). For example, if the electronic device (101) recognizes a TV in the AR space, it can map an application (e.g., a video application) related to the TV to the TV. The mapping data can be stored in the memory (130). According to another embodiment, if the electronic device (101) further includes a wireless communication circuit, the electronic device (101) can receive mapping data for the AR space from a server (e.g., the server (108) of FIG. 1A) using the wireless communication circuit. For example, if the AR space has already been provided, the mapping data for the AR space can be stored in the server (e.g., the cloud). The electronic device (101) can receive mapping data from the server and store the received mapping data in the memory (130).

[0060] The electronic device (101) can retrieve data from the memory (130) to output an augmented reality element, a virtual reality element, or a mixed reality element. The electronic device (101) can receive a user input (e.g., a gesture) for the augmented reality element, the virtual reality element, or the mixed reality element, and provide augmented reality, virtual reality, or mixed reality in response to the user input. For example, the electronic device (101) can output an application (e.g., a video application) on a specific object in the AR space, and can execute (e.g., play) the application (e.g., the video application) by receiving a user input (e.g., a touch gesture).

[0061] The electronic device (101) can obtain data related to the user's motion (e.g., information related to the user's movement direction, movement speed, gaze direction, and / or head direction) using a sensor (176) or a camera (180). For example, the user's motion can be understood as the user moving a body part (e.g., a hand or a foot). The electronic device (101) can recognize the user's motion as a user input if the user's motion corresponds to a preset gesture. For example, when the user performs a specific motion (e.g., a touch gesture) with respect to an augmented reality element, a virtual reality element, or a mixed reality element (e.g., an application), the electronic device (101) can recognize the user's motion as a user input. The electronic device (101) can execute the augmented reality element, the virtual reality element, or the mixed reality element (e.g., an application) in response to the user input.

[0062]

[0063] Figure 2 illustrates a display of an intermediate image according to one embodiment.

[0064] Referring to FIG. 2, the processor (120) can display a three-dimensional image in a first mode (hereinafter, referred to as a first mode image). The first mode may be an augmented reality (AR) mode. The processor (120) can capture an image in real time through a camera (180) and display the captured image on a display (160) (video see through; VST).

[0065] For example, in the first mode image (210), the processor (120) can virtually display additional information (e.g., object name, object description) on the real object (211, 212). The first mode image (210) can include the first real object (211) or the second real object (212).

[0066] When there is a separate user input (e.g., selecting a display object (e.g., an icon) of a VR application) or according to a specified condition (e.g., user movement, gesture input, entry into a specific area), the processor (120) may display a three-dimensional image of a second mode (hereinafter, referred to as a second mode image). The second mode may be a virtual reality (VR) mode. In the second mode, the processor (120) may display a virtual object or a virtual background, and may not display an actual object. The second mode may be a mode that provides the user with an immersive feeling as if he or she is in a separate space different from the actual space.

[0067] For example, in the second mode image (230), the processor (120) may display a first virtual object (231), a second virtual object (232), or a third virtual object (233). The second mode image (230) may not include an object corresponding to a separate real object.

[0068] When switching directly from a first mode image (210) to a second mode image (230), the user may experience discomfort due to sudden changes in environment, lighting, or spatial perception. Alternatively, when switching directly from a first mode image (210) to a second mode image (230), a simple loading screen may be displayed, causing the user to feel bored.

[0069] When the processor (120) converts from the first mode to the second mode, it can display an intermediate scene (hereinafter, “intermediate scene”) (220) in which the first mode and the second mode are mixed. The intermediate scene (220) can have a form in which the first mode image (210) and the second mode image (230) are mixed.

[0070] For example, the intermediate image (220) may include a first change object (221) and a second deformation object (222). The first change object (221) and the second deformation object (222) may each be a combination of the properties of an object in the first mode and the background properties of the second mode. The first change object (221) and the second deformation object (222) may each have substantially the same shape as the first real object (211) and the second real object (212) included in the first mode image (210). The first change object (221) and the second deformation object (222) may have a color or surface that is the same as or matches the background style of the second mode image (230).

[0071] When switching between 3D modes, the processor (120) can provide the user with an experience of gradually entering the virtual space of the second mode (VR) while retaining the characteristics of the first mode (AR) in reality. Thereafter, when the second mode is fully loaded, the user can immersively enter the space of the second mode.

[0072]

[0073] Figure 3 illustrates an intermediate screen transition section according to one embodiment.

[0074] Referring to FIG. 3, the intermediate screen switching unit (or style transfer module) (250) can generate an intermediate image (220) using first information extracted from a first mode image (210) and second information regarding a second mode image (230). The intermediate screen switching unit (250) can generate the intermediate image (220) using a generative AI model. The operation of the intermediate screen switching unit (250) may be a part of the operation of the processor (120) of FIG. 1B.

[0075] The intermediate screen switching unit (250) can obtain recognition information about a real object (hereinafter, “real object information”) from a first mode program (e.g., an AR application) that outputs a first mode image (210). The real object information may include information about the location, size, or properties of the real object. The real object information may be extracted internally through an internal operation of the electronic device (101) or received from an external server.

[0076] The mid-screen switching unit (250) can pre-store information about entities (objects, structures) included in the start frame of the second mode video (230). Information (library) about entities of the second mode can be pre-stored by the producer of the second application (VR program). Alternatively, if the second application (VR program) has a history of being executed previously, the mid-screen switching unit (250) can pre-store information about entities of the second mode after they are generated through a generative AI (e.g., a generative AI (252) of FIG. 5).

[0077] When switching from the first mode to the second mode, the intermediate screen switching unit (250) can display an intermediate image (220) that is a mixture of the first mode and the second mode. The intermediate screen switching unit (250) can generate the intermediate image (220) in different ways by distinguishing between a state in which switching is required within a specified time (hereinafter, an emergency state) or a state in which a switching time longer than a specified time is secured (hereinafter, a non-emergency state).

[0078] For example, in a non-emergency state, the intermediate screen switching unit (250) can generate an intermediate image (220) through generative AI. The intermediate image (220) can include a first change object (221) and a second transformation object (222). The first change object (221) and the second transformation object (222) can each be a form in which the properties of an object in the first mode and the background properties of the second mode are combined. The first change object (221) and the second transformation object (222) can each have a shape substantially the same as the first real object (211) and the second real object (212) included in the first mode image (210). The first change object (221) and the second transformation object (222) can have a color or surface that is the same as or matches the background style of the second mode image (230).

[0079] For another example, in an emergency situation, the intermediate screen transition unit (250) can generate the intermediate image (220) through simple alpha blending without using generative AI.

[0080]

[0081] Figure 4 is a flowchart illustrating a three-dimensional image display method according to one embodiment.

[0082] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0083] Referring to FIG. 4, in operation 410, according to one embodiment, the processor (120) may display a three-dimensional image in a first mode. For example, this may be an augmented reality (AR) mode. The processor (120) may capture an image in real time through a camera (180) and display the captured image on a display (160) (video see through; VST).

[0084] In operation 420, according to one embodiment, the processor (120) may confirm a request to switch from the first mode to the second mode. For example, the request may occur when the user selects a display object (e.g., an icon) of an application in the second mode. In another example, the request may occur when the electronic device (101) or the user moves and enters a specific area, or when the user generates a designated gesture input associated with the second mode.

[0085] According to one embodiment, when the above request occurs, the processor (120) may proceed with loading the second mode program in the background. For example, the processor (120) may proceed with loading the VR application program in the background.

[0086] In operation 430, according to one embodiment, the processor (120) may determine whether a transition from the first mode to the second mode is required (or requires a transition) within a specified time period. For example, the processor (120) may determine a non-emergency state if loading the second mode requires a significant amount of time (e.g., approximately 1 minute). In another example, the processor (120) may determine an emergency state if a user enters a specified area and an immediate transition to the second mode (e.g., within approximately 10 seconds) is required.

[0087] In operation 440, according to one embodiment, if a transition within a specified time is required (operation 430 - YES), the processor (120) may process the last frame of the first mode image (or a frame within a specified time from the last frame, hereinafter the same) and the start frame of the second mode image (or a frame within a specified time from the start frame, hereinafter the same) according to a first image processing method to generate a first intermediate image. For example, the first image processing method may be an alpha blending method.

[0088] In operation 450, according to one embodiment, if a transition is required beyond a specified time (operation 430 - NO), the processor (120) may process the last frame of the first mode and the start frame of the second mode according to a second image processing method to generate a second intermediate image. For example, the second image processing method may be a method using generative AI.

[0089] In operation 460, according to one embodiment, the processor (120) may output a first intermediate image or a second intermediate image during the process of switching from the first mode to the second mode. When the loading of the second mode is completed, the processor (120) may terminate the output of the first intermediate image or the second intermediate image and switch to the second mode.

[0090]

[0091] Figure 5 illustrates the generation of real object and virtual object based intermediate images according to one embodiment.

[0092] Referring to FIG. 5, the processor (120) can display a three-dimensional image in a first mode. For example, the first mode may be an augmented reality (AR) mode. In the first mode, the processor (120) can virtually display additional information (e.g., object name, object description) on an actual object (211, 212).

[0093] The processor (120) can capture images in real time through the camera (180) and display the captured images on the display (160) (video see through; VST). For example, the first mode image (210) may include a first real object (211) and a second real object (212).

[0094] In the first mode, the processor (120) may display a display object (e.g., an icon) (215) related to the execution of the second mode. The second mode may be a virtual reality (VR) mode. When a display object (e.g., an icon) (215) is selected by a user, the processor (120) may generate and display an intermediate image (220) corresponding to the first mode and the second mode.

[0095] The VR entity (251) can store a library regarding the start screen (start frame) of the second application (VR program) in the second mode that is being executed. The library regarding the second mode can be stored in advance by the developer of the second application (VR program). If the second application (VR program) has a history of being executed previously, the processor (120) can store the library regarding the second mode in advance after generating it through the generative AI (or generative AI module) (252).

[0096] The processor (120) can analyze the first mode image (210) to obtain information (location, size, properties) about objects included in the first mode image (210) (VST environment). For example, the processor (120) can obtain information about the location, size, and properties of the first real object (211) and the second real object (212).

[0097] The processor (120) can input i) information about an object included in a first mode image (210) and ii) information (style characteristics) about a start screen of a second mode obtained from a VR entity (251) into the generative AI (252). The generative AI (252) can generate an intermediate image (220).

[0098] The generative AI (252) may include an AI model with a simple structure and may quickly generate an intermediate image (220) during a mode switching process. According to one embodiment, the generative AI (252) may generate an intermediate image (220) in real time based on input information without the intervention of an external server.

[0099] The intermediate image (220) may include a first change object (221) based on a first real object (211) and a second deformation object (222) based on a second real object (212). The first change object (221) and the second deformation object (222) may each be a combination of the properties of an object in the first mode and the background properties of the second mode. The first change object (221) and the second deformation object (222) may have substantially the same shape as the first real object (211) and the second real object (212) included in the first mode image (210). The first change object (221) and the second deformation object (222) may have a color or surface that is the same as or matches the background style of the second mode image (230).

[0100] The intermediate image (220) may additionally include a first virtual object (225) and a second virtual object (226). When the VR program is loaded, the first virtual object (225) and the second virtual object (226) may be positioned substantially identically to the start frame of the second mode. The first virtual object (225) and the second virtual object (226) may be positioned and changed through the generative AI (252) to match the real objects displayed in the first mode, or may be projected onto the real objects.

[0101] The intermediate image (220) may be displayed while a second mode application (VR application) (e.g., Beat Saber) is loading in the background. The intermediate image (220) may be outputted and may be naturally switched to the second mode start screen when the second mode application is fully loaded.

[0102]

[0103] Figure 6 illustrates changes in an intermediate image according to the loading progress of the second mode according to one embodiment. Figure 6 is exemplary and is not limited thereto.

[0104] Referring to FIG. 6, the intermediate image may be composed of multiple frames. The processor (120) may generate and output intermediate images (601 to 603) that change sequentially over time.

[0105] The first frame (601) of the intermediate video may be an image displayed in the first loading state (e.g., about 30%) of the second mode. The first frame (601) may include a first change object (611) based on a first real object and a second deformation object (612) based on a second real object. The first change object (611) and the second deformation object (612) may each be a form in which the properties of the object of the first mode and the background properties of the second mode are combined. The first change object (611) and the second deformation object (612) may have substantially the same shape as the real object and may have a color or surface that is the same as or matches the background style of the second mode. The first frame (601) may include a first virtual object (615) and a second virtual object (616). The first virtual object (615) and the second virtual object (616) may be objects that are identically arranged on the start screen of the second mode when a VR program is loaded.

[0106] The second frame (602) of the intermediate image may be an image displayed in the second loading state (e.g., about 60%) of the second mode. The second frame (602) of the intermediate image may have the first modified object (611) based on the first real object removed and the second deformed object (612) based on the second real object maintained. The second frame (602) may maintain the first virtual object (615) and the second virtual object (616). The second frame (602) may further include a third virtual object (617). The first virtual object (615), the second virtual object (616), and the third virtual object (617) may be objects that are substantially identically arranged on the start screen of the second mode when the VR program is loaded.

[0107] The third frame (603) of the intermediate image may be an image displayed in the third loading state (e.g., about 90%) of the second mode. The third frame (603) of the intermediate image may have the first modified object (611) based on the first real object and the second deformed object (612) based on the second real object removed. The third frame (603) may maintain the first virtual object (615), the second virtual object (616), and the third virtual object (617). The third frame (603) may further include a fourth virtual object (618). The first virtual object (615), the second virtual object (616), the third virtual object (617), and the fourth virtual object (618) may be objects that are substantially identically arranged on the start screen of the second mode when the VR program is loaded.

[0108] According to one embodiment, the number of virtual objects placed may increase and the number of real objects may decrease depending on the degree to which the application (VR application) of the second mode is loaded, and when loading is complete, the screen may be switched to a VR screen.

[0109]

[0110] Fig. 7 is a flowchart illustrating a method for displaying a three-dimensional image in an animated manner according to one embodiment.

[0111] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0112] Referring to FIG. 7, in operation 710, according to one embodiment, the processor (120) may display a three-dimensional image in a first mode. For example, this may be an augmented reality (AR) mode. The processor (120) may capture an image in real time through a camera (180) and display the captured image on a display (160) (video see through; VST).

[0113] In operation 720, according to one embodiment, the processor (120) may confirm a request to switch from the first mode to the second mode.

[0114] For example, the request may be an input generated by a user selecting a display object (e.g., an icon) of an application in the second mode. This may be generated when the user generates a designated gesture input associated with the second mode. In another example, the request may be generated when the electronic device (101) or the user moves and enters a specific area.

[0115] In operation 730, according to one embodiment, the processor (120) may determine whether there is an emergency condition requiring a mode transition within a specified time period.

[0116] For example, if a user enters a designated area and needs to switch to the second mode immediately (e.g., within about 10 seconds), the request can be determined as the first attribute (urgent state). In another example, if the second mode requires a loading time (e.g., about 1 minute) and switching from the first mode to the second mode within the designated time is impossible, or if the difference in illuminance between the first mode and the second mode is greater than a designated value and glare to the user needs to be prevented or reduced, the request can be determined as the second attribute (non-urgent state).

[0117] In operation 740, according to one embodiment, if there is an emergency (operation 730 - YES), the processor (120) may alpha blend the last frame of the first mode (or a frame within a certain time from the last frame) and the start frame of the second mode (or a frame within a certain time from the start frame) to generate an intermediate image in an animated manner. Alpha blending may be a method of switching transparency between the two images (see FIG. 8A).

[0118] In operation 750, according to one embodiment, in a non-emergency situation (operation 730 - NO), the processor (120) may input the last frame of the first mode and the start frame of the second mode to the animation generation AI. In a non-emergency situation, the processor (120) sets the animation playback time (number of frames) using a time parameter that reflects the difference in lighting and the content loading delay (see FIG. 8b).

[0119] In operation 760, according to one embodiment, the processor (120) may output an intermediate image during the process of switching from the first mode to the second mode.

[0120] In operation 770, according to one embodiment, when loading of the second mode is completed, the processor (120) may switch to the second mode.

[0121]

[0122] Figure 8a illustrates the generation of an intermediate image by an alpha blending technique according to one embodiment.

[0123] Referring to FIG. 8A, the processor (120) can generate and output an intermediate image (820) in the process of changing from a first mode image (810) to a second mode image (830). If there is not enough time to generate the intermediate image (820) using generative AI, the processor (120) can quickly generate and display an intermediate image (820) having a fixed number of frames using an alpha blending technique.

[0124] For example, the processor (120) can generate an intermediate frame (820a) by simply combining the last frame of the first mode image (810) and the start frame of the second mode image (830).

[0125] The processor (120) can generate first animation frames (822) using an alpha blending technique using the first mode image (810) and the intermediate frame (820a). The first animation frames (822) can be generated such that the transparency of the last frame of the first mode image (810) sequentially increases over time, and the transparency of the intermediate frame (820a) sequentially decreases over time.

[0126] The processor (120) can generate second animation frames (823) using an alpha blending technique using an intermediate frame (820a) and a second mode image (830). The second animation frames (823) can be generated such that the transparency of the intermediate frame (820a) sequentially increases over time, and the transparency of the start frame of the second mode image (830) sequentially decreases over time.

[0127] In the process of switching from the first mode to the second mode, the processor (120) can combine the first animation frames (822) and the second animation frames (823) to generate and output an intermediate image.

[0128]

[0129] Figure 8b illustrates the generation of an intermediate image using generative AI according to one embodiment.

[0130] Referring to FIG. 8B, the processor (120) may generate and output an intermediate image (820) in the process of changing from a first mode image (810) to a second mode image (830). If time is secured to generate the intermediate image (820) using the generative AI (252), the processor (120) may generate and display the intermediate image (820) in an animated form using the generative AI (252). For example, if it takes about 3 minutes to load an application in the second mode, the processor (120) may generate an intermediate image (820) in an animated form using the generative AI (252) and display it for about 3 minutes.

[0131] For example, the processor (120) can generate an intermediate frame (820a) by simply combining the last frame of the first mode image (810) and the start frame of the second mode image (830).

[0132] The processor (120) can input the end frame and the middle frame (820a) of the first mode image (810) into the generation AI (252) to generate the first animation frames (825). The processor (120) can also input a time parameter (λ) into the generation AI (252). The time parameter (λ) can be determined according to the difference in illumination between the first mode and the second mode, the difference in spatial structure, and / or the delay time of program loading.

[0133] The start frame of the first animation frames (825) may be the end frame of the first mode image (810), and the end frame of the first animation frames (825) may be the intermediate frame (820a). The first animation frames (825) may be configured in a manner such that the characteristics of the first mode decrease over time and the characteristics of the intermediate frame (820a) increase over time, rather than in a manner of simply changing transparency. The number (N) of frames constituting the first animation frames (825) may be determined in proportion to the time parameter (λ).

[0134] The processor (120) can input the intermediate frame (820a) and the start frame of the second mode image (830) into the generation AI (252) to generate second animation frames (826). The processor (120) can also input a time parameter (λ) into the generation AI (252). The time parameter (λ) can be determined according to the difference in illumination between the first mode and the second mode, the difference in spatial structure, and / or the delay time of program loading.

[0135] The start frame of the second animation frames (826) may be the intermediate frame (820a), and the end frame of the second animation frames (826) may be the start frame of the second mode image (830). The second animation frames (826) may be configured in a manner such that the characteristics of the intermediate frame (820a) decrease over time, and the characteristics of the second mode image (830) increase over time, rather than in a manner of simply changing transparency. The number (N) of frames constituting the second animation frames (826) may be determined in proportion to the time parameter (λ).

[0136] The processor (120) can combine the first animation frames (825) and the second animation frames (826) to generate and output an intermediate image.

[0137] According to one embodiment, the generative AI (252) may receive a time parameter (λ). The time parameter (λ) may be determined based on at least one of a difference in illumination between the first mode and the second mode, a difference in spatial structure, and a delay time for program loading. For example, in the case of a difference in illumination, the processor (120) may determine that a larger difference in illumination indicates a larger difference by comparing the average Y values ​​when viewing the scene in YUV color representation. As another example, in the case of a difference in spatial structure, the processor (120) may determine that a smaller number of matching points indicates a larger structure by applying a feature matching algorithm.

[0138] According to one embodiment, the generative AI (252) can generate an intermediate image (820) including many frames so that the screen transition can be performed for a sufficient amount of time so that the user does not experience discomfort from a sudden change in field of view, as the difference in illumination or the difference in spatial structure between the first mode and the second mode is large.

[0139] The generative AI (252) can generate an intermediate image (820) by setting the number of frames so as to provide a transition experience equivalent to the amount of time when a loading delay in the second mode occurs.

[0140]

[0141] Figure 9 illustrates the display of an intermediate image according to a user's position according to one embodiment.

[0142] Referring to FIG. 9, the processor (120) can divide the surroundings of the user (905) into multiple areas (or multiple zones) and output in different ways according to each area. The processor (120) can determine the zone to which the user (905) belongs through a sensor or a camera. For example, the processor (120) can divide the zone into a first area (safety zone) that can be operated in VR mode, a second area (mixed zone) that is output as an intermediate image, and a third area (only AR zone) that is operated in AR mode.

[0143] The first zone (safety zone) may be an area where users can utilize VR without any or minimal risk factors, as the surrounding environment can be assessed in advance. The second zone (mixed zone) may be an area where users must be aware of surrounding objects when using VR. The third zone (only AR zone) may be an area where users need to check their surroundings due to obstacles, requiring them to exit VR mode.

[0144] For example, in the first state (901) where the user (905) is in the first zone (safety zone), the processor (120) may display a VR mode image (910). In the second state (902) where the user (905) moves from the first zone (safety zone) to the second zone (mixed zone), the processor (120) may determine the state as an emergency state and display an intermediate image (920) using an alpha blending method. The intermediate image (920) may include a first change object (921) and a second deformation object (922). The first change object (921) may have a shape substantially the same as the first real object (931), and the second deformation object (922) may have a shape substantially the same as the second real object (932). The first change object (921) and the second transformation object (922) may have a color or surface that is the same as or matches the background style of the VR mode image (910).

[0145] When the user (905) stays in the second zone (mixed zone), the processor (120) can continuously display the intermediate image (920). In the third state (903) where the user (905) moves from the second zone (mixed zone) to the third zone (only AR zone), the processor (120) determines that it is an emergency state and can display the AR mode image (930) using an alpha blending method. The AR mode image (930) can include a first real object (931) and a second real object (932). The user (905) can check the surrounding situation, such as an obstacle, in the third zone (only AR zone).

[0146]

[0147] Figure 10 illustrates a transition from a first VR mode to a second VR mode according to one embodiment.

[0148] Referring to FIG. 10, the processor (120) may generate and output an intermediate image (1020) in the process of changing from a first mode image (1010) to a second mode image (1030). For example, the first mode image (1010) may be an image output by the execution of a first VR application (program). The second mode image (1030) may be an image output by the execution of a second VR application (program). For example, the processor (120) may be outputting a first mode image (1010) including a first virtual object (1011) and a second virtual object (1012).

[0149] While outputting the first mode image (1010), the processor (120) may confirm a request to switch from the first mode to the second mode. For example, the request may occur when a user selects a display object (e.g., an icon) of the second VR application.

[0150] If loading of the second VR application requires time, the processor (120) may load the second VR application (program) in the background and generate and output an intermediate image (1020) using generative AI. The intermediate image (1020) may include virtual objects (1021, 1022) having substantially the same shape as the first virtual object (1011) and the second virtual object (1012) included in the first mode image (1010). In the intermediate image (1020), the third virtual object (1021) and the fourth virtual object (1022) may have a color or surface that is the same as or matches the background style of the second mode image (1030).

[0151]

[0152] Figures 11a and 11b illustrate the generation of an intermediate image using a real object according to one embodiment. Figures 11a and 11b are exemplary and not limiting.

[0153] Referring to FIGS. 11A and 11B , the processor (120) may display a three-dimensional image in a first mode. For example, the first mode may be an augmented reality (AR) mode. In the first mode, the processor (120) may virtually display additional information (e.g., object name, object description) on a real object (1111). For example, the first mode image (1110) may include the real object (1111).

[0154] The processor (120) may display a display object (e.g., an icon) (1115) related to the execution of the second mode. The second mode may be a virtual reality (VR) mode. When a display object (e.g., an icon) (1115) is selected by a user, the processor (120) may generate and display an intermediate image (1120) corresponding to the first mode and the second mode.

[0155] The processor (120) can analyze the first mode image (1110) to obtain information (location, size, properties) about an object included in the first mode image (1110). For example, the processor (120) can obtain information about the location, size, and properties of an actual object (1111). If the properties of the actual object (1111) are display properties, the processor (120) can detect the display area of ​​the actual object (1111) and generate an intermediate image (1120) that reflects (maps and renders) the start screen of the second mode on the display area of ​​the actual object (1111). The processor (120) can generate the intermediate image (1120) using the generative AI (252).

[0156] When loading of the second mode is completed, the processor (120) can complete the transition to the second mode by expanding the image of the display area of ​​the actual object (1111) or changing an area other than the display area to a second mode image.

[0157]

[0158] Figures 12a and 12b illustrate the generation of intermediate images using virtual objects according to one embodiment.

[0159] Referring to FIGS. 12A and 12B , the processor (120) can display a three-dimensional image in a first mode. For example, the first mode may be an augmented reality (AR) mode. The processor (120) can display a display object (e.g., an icon) (1215) related to the execution of the second mode. The second mode may be a virtual reality (VR) mode. When the display object (e.g., an icon) (1215) is selected by the user, the processor (120) can generate and display an intermediate image (1220) corresponding to the first mode and the second mode.

[0160] The VR entity (251) can store a library regarding the start frame of the second application (VR program) in the second mode being executed. The library regarding the start frame of the second mode can be stored in advance by the creator of the second application (VR program). Alternatively, if the second application (VR program) has a history of being executed previously, the processor (120) can store the library regarding the second mode in advance after generating it through the generative AI (252).

[0161] The processor (120) can generate first to third virtual objects (1221 to 1223) that reflect the style of the second mode. The processor (120) can generate an intermediate image (1220) by combining the first to third virtual objects (1221 to 1223) in various ways.

[0162] According to one embodiment, the processor (120) may analyze the first mode image (210) to obtain information (location, size, properties) about objects included in the first mode image (1210). For example, the processor (120) may cause the first to third virtual objects (1221 to 1223) to be placed appropriately on various real objects.

[0163] In FIG. 12B, the processor (120) can display the first to third virtual objects (1121 to 1123) differently over time. Depending on the loading level of the second application (VR program), the processor (120) can gradually increase the number of virtual objects loaded from the VR entity (251) or generated through the generation AI (252).

[0164] For example, the first frame (1220a) of the intermediate image (1220) may be an image displayed in a first loading state (e.g., about 50%). The first frame (1220a) may include a first virtual object (1221) and a second virtual object (1222).

[0165] The second frame (1220b) of the intermediate image (1220) may be an image displayed in a second loading state (e.g., about 80%). The first frame (1220a) may include a first virtual object (1221), a second virtual object (1222), and a third virtual object (1223).

[0166] According to one embodiment, the processor (120) can arrange the first virtual object (1221), the second virtual object (1222), or the third virtual object (1223) so as to blend in with the surrounding environment. The processor (120) can analyze the first mode image (1210) to obtain information (location, size, properties) about the real object included in the first mode image (1210). The processor (120) can arrange the first to third virtual objects (1121 to 1123) by changing the location, size, and shape so as to be arranged so as to blend in with various real objects.

[0167]

[0168] When an electronic device switches between 3D modes, it may display a loading screen or provide simple effects. This can be frustrating for the user. Alternatively, if the mode is switched immediately, the sudden changes in environment, lighting, or spatial perception can be uncomfortable for the user.

[0169] An electronic device according to one embodiment may include a display, a memory, and at least one processor. The memory may store instructions that, when executed by the at least one processor, cause the electronic device to display a 3D image in a first mode, check a request to switch from the first mode to a second mode, determine whether switching is possible within a specified time, and if switching is possible within the specified time, process a first image in the first mode and a second image in the second mode according to a first image processing method to generate a first intermediate image, and if switching is possible beyond the specified time, process the first image in the first mode and the second image in the second mode according to a second image processing method to generate a second intermediate image, and output the first intermediate image or the second intermediate image during the switching from the first mode to the second mode.

[0170] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to generate the first intermediate image or the second intermediate image using the first object displayed in the first mode or the second object displayed in the second mode.

[0171] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to generate the second intermediate image using generative AI.

[0172] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to input a time parameter to the generative AI to generate the second intermediate image.

[0173] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to determine the time parameter based on at least one of a difference in illumination between the first image and the second image, a difference in structure between the first mode and the second mode, and a loading time of the second mode.

[0174] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to generate the second intermediate image, the second intermediate image comprising a number of frames proportional to the time parameter.

[0175] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to generate the first intermediate image by an alpha blending method.

[0176] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to generate one frame of the first intermediate image, alpha-blend the last frame of the first image and the one frame for a first time, alpha-blend the one frame and the start frame of the second image for a second time, and generate the first intermediate image by the first and second alpha blending.

[0177] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to generate and output the first intermediate image or the second intermediate image having a plurality of frames that change over time.

[0178] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to store information about entities constituting a start frame of the second image in the memory.

[0179] In one embodiment, the first mode may be an AR mode and the second mode may be a VR mode.

[0180] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to recognize a real object included in the first image and display an object in the second intermediate image that has the same shape as the real object and reflects the style of the second mode.

[0181] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to detect a display area of ​​the real object and to reflect a style of the second mode in the display area.

[0182] In one embodiment, the first mode may be a first VR mode, and the second mode may be a second VR mode.

[0183] In one embodiment, the first mode may be a VR mode and the second mode may be an AR mode.

[0184] According to one embodiment, the electronic device may further include a sensor and a camera. The instructions, when executed by the at least one processor, may cause the electronic device to divide the surroundings of the electronic device into a plurality of regions, use the sensor and the camera to detect whether the electronic device enters a designated region, and, if the electronic device enters the designated region, generate and output the first intermediate image.

[0185] An electronic device according to one embodiment may include a display, a memory, and at least one processor. The memory may store instructions that, when executed by the at least one processor, cause the electronic device to display a 3D image on the display in a first mode, confirm a request to switch from the first mode to a second mode, create at least one virtual object related to the second mode, and, during the process of switching from the first mode to the second mode, add and output the at least one virtual object to a first image in the first mode.

[0186] According to one embodiment, the instructions, when executed by the at least one processor, may cause the electronic device to add the at least one virtual object to the first image so as to reflect a layout of a real object displayed in the first mode.

[0187] According to one embodiment, the at least one virtual object may be plural. The instructions, when executed by the at least one processor, may cause the electronic device to increase the number of the at least one virtual object added to the first image according to the loading progress status of the application of the second mode.

[0188] A method for displaying a three-dimensional image according to one embodiment may be performed in an electronic device. The method may include: displaying a 3D image in a first mode; confirming a request to switch from the first mode to a second mode; determining whether switching is possible within a specified time; if switching is possible within the specified time, processing a first image in the first mode and a second image in a second mode according to a first image processing method to generate a first intermediate image; if switching is possible beyond the specified time, processing the first image in the first mode and the second image in the second mode according to a second image processing method to generate a second intermediate image; and, in a process of switching from the first mode to the second mode, outputting the first intermediate image or the second intermediate image.

[0189]

[0190] An electronic device according to one embodiment disclosed in this document can generate and output an intermediate image when switching between 3D modes. The intermediate image can have both the characteristics of the 3D mode before the transition and the characteristics of the 3D mode after the transition. This can provide the user with a stable sense of immersion without sudden changes in the environment, lighting, or spatial perception.

[0191] An electronic device according to one embodiment disclosed in this document can provide an intermediate image that can naturally handle a delay situation occurring in program loading when switching between 3D modes.

[0192] An electronic device according to one embodiment disclosed in this document can distinguish between an emergency transition state and a non-emergency transition state, and generate and provide intermediate images in real time according to each situation. In a non-emergency transition state, generative AI can be utilized.

[0193] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0194] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0195] Various embodiments of the present document may be implemented as software (e.g., program (10)) including one or more instructions stored in a storage medium (e.g., built-in memory (1436) or external memory (138)) readable by a machine (e.g., electronic device (1401)). For example, a processor (e.g., processor (1420)) of the machine (e.g., electronic device (1401)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0196] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0197] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, Display; Memory; and At least one processor operatively connected to the display and the memory, the processor including processing circuitry; The above memory, when executed individually or collectively by the at least one processor, causes the electronic device to: On the above display, a 3D image is displayed in the first mode, Confirm the request to switch from the above first mode to the second mode, Determine whether a transition is possible within a specified time frame, Based on the fact that switching is possible within the above-mentioned specified time, the first image of the first mode and the second image of the second mode are processed according to the first image processing method to generate a first intermediate image, Based on the fact that the transition is possible beyond the above-mentioned specified time, the first image of the first mode and the second image of the second mode are processed according to the second image processing method to generate a second intermediate image, An electronic device storing instructions for outputting the first intermediate image or the second intermediate image during a process of switching from the first mode to the second mode.

2. In the first paragraph, the at least one processor, individually or collectively, the electronic device An electronic device that generates the first intermediate image or the second intermediate image by using the first object displayed in the first mode or the second object displayed in the second mode.

3. In the first paragraph, the at least one processor, individually or collectively, the electronic device An electronic device that generates the second intermediate image using generative AI.

4. In the third paragraph, the at least one processor, individually or collectively, the electronic device An electronic device that inputs time parameters into the generative AI to generate the second intermediate image.

5. In the third paragraph, the at least one processor, individually or collectively, the electronic device An electronic device that determines the time parameter based on at least one of a difference in illuminance between the first image and the second image, a difference in structure between the first mode and the second mode, and a loading time of the second mode.

6. In the third paragraph, the at least one processor, individually or collectively, the electronic device An electronic device for generating the second intermediate image including a number of frames proportional to the time parameter.

7. In the first paragraph, the at least one processor, individually or collectively, the electronic device An electronic device for generating the first intermediate image by alpha blending.

8. In the 7th paragraph, the at least one processor, individually or collectively, the electronic device Generate one frame of the first intermediate image, Alpha blending the last frame of the first image and the one frame first, Alpha blending the above one frame and the start frame of the second image for the second time, An electronic device for generating the first intermediate image by the first and second alpha blending.

9. In the first paragraph, the at least one processor, individually or collectively, the electronic device An electronic device that generates and outputs the first intermediate image or the second intermediate image including a plurality of frames that change over time.

10. In the first paragraph, the at least one processor, individually or collectively, the electronic device An electronic device for storing information about entities constituting a start frame of said second image in said memory.

11. In the first paragraph, the at least one processor, individually or collectively, the electronic device Recognize the actual object contained in the first image above, An electronic device that displays an object having the same shape as the actual object and reflecting the style of the second mode in the second intermediate image.

12. In the 11th paragraph, the at least one processor, individually or collectively, the electronic device Detect the display area of the above real object, An electronic device that reflects the style of the second mode in the above display area.

13. In paragraph 1, Including more sensors and cameras, The at least one processor, individually or collectively, is configured to: Divide the area surrounding the above electronic device into multiple areas, Using the above sensor and / or the above camera, detecting whether the electronic device enters a designated area, An electronic device that generates and outputs the first intermediate image based on the electronic device entering the designated area.

14. In electronic devices, Display; Memory; and At least one processor operatively connected to the display and the memory, the processor including processing circuitry; The above memory, when individually or collectively executed by the at least one processor, causes the at least one electronic device to: Displaying a 3D image in the first mode on the above display, Confirm the request to switch from the above first mode to the second mode, Create at least one virtual object associated with the second mode; An electronic device storing instructions for adding and outputting at least one virtual object to a first image of the first mode during a process of switching from the first mode to the second mode.

15. A method for displaying a three-dimensional (3D) image performed on an electronic device, Action to display 3D images in mode 1; An action for confirming a request to switch from the first mode to the second mode; An action that determines whether a transition is possible within a specified time; An operation of generating a first intermediate image by processing a first image of the first mode and a second image of the second mode according to a first image processing method based on the possibility of switching within the above-mentioned specified time; An operation of generating a second intermediate image by processing the first image of the first mode and the second image of the second mode according to a second image processing method based on the fact that the switching is possible beyond the above-mentioned specified time; and A method comprising: an operation of outputting the first intermediate image or the second intermediate image during a process of switching from the first mode to the second mode.

Images