Electronic device and data processing method using same

Abstract

According to various embodiments, an electronic device wearable on a user's head may comprise: a housing; a display which is supported by the housing and outputs visual information; a communication module including a communication circuit; a processor including a processing circuit; and a memory for storing instructions. When the instructions are individually or collectively executed by the processor, the electronic device (101) may: identify a usage record based on a virtual environment program in response to an execution command of the virtual environment program; load event information and information on a virtual object, which are included in the usage record; for results obtained by performing operations on the virtual object in the electronic device (101), generate, on the basis of the event information and the information on the virtual object, a virtual environment interface in which a time interface enabling results corresponding to each time to be separately displayed for each time and a space interface enabling results corresponding to each space to be separately displayed for each space are at least partially provided together; and visually output the generated virtual environment interface through the display (160). Various other embodiments may be possible.

Description

Electronic devices and data processing methods using the same

[0001] Embodiments of the present disclosure relate to an electronic device and a method for processing data using the same.

[0002] With the recent advancement of digital technology, various types of electronic devices (user devices) capable of communication and personal information processing (e.g., mobile terminals, Personal Digital Assistants (PDAs), electronic organizers, smartphones, tablets, wearable electronic devices, and / or Personal Computers (PCs)) are being released. For example, electronic devices are gradually evolving into wearable electronic devices that can be worn on certain parts of the body to enhance portability and user accessibility.

[0003] A wearable electronic device may include a head-mounted display (HMD) device that can be worn on the head, such as glasses. For example, the wearable electronic device may include augmented reality (AR) glasses and / or smart glasses in the form of glasses that implement various contents on transparent glasses (e.g., lenses). As another example, the wearable electronic device may include a video see-through (VST) device that is an HMD device and uses a camera to capture a real-world environment and superimposes the captured image onto a virtual image. The wearable electronic device, HMD device, and / or VST device may use the camera to provide a virtual reality service and / or an augmented reality service (e.g., an augmented reality world, an augmented reality function) to the user. For example, when the HMD device is worn on the user's head, the HMD device may implement a virtual reality and / or an augmented reality in response to the execution of an augmented reality-related application on a connected electronic device, and may provide the virtual reality service and / or the augmented reality service to the user.

[0004] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0005] Electronic devices (e.g., head mounted display (HMD) devices, wearable electronic devices) can provide users with augmented reality services (e.g., virtual reality services) based on augmented reality (e.g., virtual reality (VR), augmented reality (AR), mixed reality (MR), extended reality (XR)) technology. The electronic devices can implement a virtual environment interface within an augmented reality area and display the virtual environment interface through a display. The electronic devices can provide a virtual environment interface implemented as a visual image to a user wearing the electronic devices.

[0006] Based on a virtual environment interface, a user may wish to continue a previously performed task (e.g., a task, process, or program) or perform a task at a specific point in time in the past. An electronic device may provide an augmented reality service to the user by considering a specific point in time, location, or task in the past. In one embodiment, the electronic device may classify the user's tasks by past time or past space, and generate a virtual environment interface so that the user can intuitively recognize the classified tasks.

[0007] According to one embodiment, an electronic device (e.g., an HMD device) can categorize previously processed task-related data by time or space, and generate a virtual environment interface containing the task-related data so that the user can intuitively check it. The electronic device can select at least one task desired by the user and provide the user with a virtual environment interface for processing the selected at least one task.

[0008] The technical tasks to be achieved in this document are not limited to the technical tasks mentioned above, and other technical tasks not mentioned will be clearly understood by those with ordinary skill in the technical field to which this document pertains from the description below.

[0009] According to one embodiment, an electronic device wearable on a user's head may include a housing, a display supported by the housing and outputting visual information, a communication module including a communication circuit, a processor including a processing circuit, and a memory storing instructions. When the instructions are individually or collectively executed by the processor, the electronic device (101) may, in response to an execution command of a virtual environment program, check a usage record based on the virtual environment program, load event information and information on virtual objects included in the usage record, and, based on the event information and the information on the virtual object, generate a virtual environment interface in which a time interface in which results corresponding to each time can be displayed separately and in a space interface in which results corresponding to each space can be displayed separately and in a space, at least partially together, among results performed on the virtual object by the electronic device, and visually output the generated virtual environment interface through the display (160).

[0010] According to one embodiment, a method for processing data in an electronic device wearable on a user's head may include, in response to an execution command of a virtual environment program, an operation of checking a usage record based on the virtual environment program, an operation of loading event information and information on a virtual object included in the usage record, an operation of generating a virtual environment interface in which a time interface in which results corresponding to each time are displayed separately and a space interface in which results corresponding to each space are displayed separately and at least partially together are provided, among results performed on the virtual object in the electronic device (101) based on the event information and the information on the virtual object, and an operation of visually outputting the generated virtual environment interface.

[0011] According to one embodiment, a non-transitory computer-readable storage medium (or, a computer program product) storing one or more programs for performing a method of processing data in an electronic device wearable on a user's head may be described. According to one embodiment, the one or more programs may include instructions that, when executed by a processor of the electronic device, perform, in response to an execution command of a virtual environment program, an operation of checking a usage record based on the virtual environment program, an operation of loading event information and information on a virtual object included in the usage record, an operation of generating a virtual environment interface in which, based on the event information and the information on the virtual object, a time interface in which results corresponding to each time are displayed separately and a space interface in which results corresponding to each space are displayed separately and in which results corresponding to each space are displayed separately, at least partially together with results performed on the virtual object in the electronic device (101), and an operation of visually outputting the generated virtual environment interface.

[0012] According to one embodiment, the electronic device may include an HMD device worn on a part of the user's body (e.g., the head) and may provide the user with an augmented reality service (e.g., a virtual reality service). The electronic device may implement a virtual environment interface within the augmented reality area and, based on the virtual environment interface, may continuously or discontinuously process a specific task (e.g., a program, a task).

[0013] According to one embodiment, an electronic device can recognize specific tasks processed in the past, categorized by time, and generate a temporal interface based on a virtual environment interface. The electronic device can recognize specific tasks processed in the past, categorized by space (e.g., location), and generate a spatial interface based on a virtual environment interface. The electronic device can implement a virtual environment interface that at least partially integrates the temporal interface and the spatial interface. For example, the electronic device can provide a user with tasks at a specific point in time in the past or tasks at a specific location in the past.

[0014] According to one embodiment, an electronic device can display specific tasks by time or space, distinguishing them when a user performs a specific task in augmented reality. This allows the user to perform specific tasks more efficiently. According to one embodiment, this can enhance user convenience in augmented reality situations.

[0015] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present disclosure belongs from the description below.

[0016] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. The features and advantages described above will be clearly understood based on the attached drawings and the description of the drawings.

[0017] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment of the present disclosure.

[0018] FIG. 2 is a perspective view schematically illustrating the configuration of an XR device supporting an extended reality (XR) service according to one embodiment of the present disclosure.

[0019] FIG. 3 is a block diagram of an electronic device according to one embodiment of the present disclosure.

[0020] FIG. 4 is a flowchart illustrating a method for creating a virtual environment interface in which a time interface and a space interface are integrated according to one embodiment of the present disclosure.

[0021] FIG. 5 is a flowchart illustrating a method for storing a data block including a background image and a virtual object according to one embodiment of the present disclosure.

[0022] FIG. 6A is a first example diagram illustrating a first task processed at a first point in time according to one embodiment of the present disclosure.

[0023] FIG. 6b is a second example diagram illustrating a second task processed at a second point in time according to one embodiment of the present disclosure.

[0024] FIG. 6c is a third example diagram illustrating a third task processed at a third point in time according to one embodiment of the present disclosure.

[0025] FIG. 6d is a fourth example diagram illustrating a fourth task processed at a fourth time point according to one embodiment of the present disclosure.

[0026] FIG. 7A is an exemplary diagram illustrating a prompt signal for loading past work in augmented reality according to one embodiment of the present disclosure.

[0027] FIG. 7b is an exemplary diagram illustrating a virtual environment interface in which a time interface and a space interface are integrated according to one embodiment of the present disclosure.

[0028] FIG. 8A is a first exemplary diagram illustrating a first task processed in a first space according to one embodiment of the present disclosure.

[0029] FIG. 8b is a second example diagram illustrating a second task processed in a second space according to one embodiment of the present disclosure.

[0030] FIG. 9 is an exemplary diagram illustrating a virtual environment interface generated based on a plurality of tasks corresponding to a plurality of time points according to one embodiment of the present disclosure.

[0031] FIG. 10A is a first example diagram illustrating a first object at a first point in time according to one embodiment of the present disclosure.

[0032] FIG. 10b is a second example diagram illustrating a first object at a first point in time and a second object at a second point in time according to one embodiment of the present disclosure.

[0033] FIG. 11A is an exemplary diagram illustrating a virtual environment interface in which a plurality of objects corresponding to a plurality of viewpoints are arranged according to one embodiment of the present disclosure.

[0034] FIG. 11b is an exemplary diagram illustrating a virtual environment interface in which multiple objects corresponding to multiple viewpoints are arranged in various forms according to one embodiment of the present disclosure.

[0035] FIG. 12A is an example diagram of generating a task in an external electronic device as a task based on a virtual environment interface according to one embodiment of the present disclosure.

[0036] FIG. 12b is an exemplary diagram illustrating a virtual environment interface including objects generated based on a viewpoint and space according to one embodiment of the present disclosure.

[0037] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness.

[0038] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0039] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0040] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0041] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0042] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0043] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0044] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0045] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0046] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0047] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0048] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0049] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0050] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0051] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0052] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0053] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0054] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0055] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0056] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0057] In one embodiment, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0058] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0059] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0060] FIG. 2 is a perspective view schematically illustrating a configuration of an XR device supporting an extended reality (XR) service according to one embodiment of the present disclosure. FIG. 2 illustrates an XR device (201) implemented as AR glasses of the glasses type, but this is merely an example and may be replaced with an electronic device supporting an extended reality (XR) service or an immersive media service.

[0061] Referring to FIG. 2, an electronic device (101) according to one embodiment may be an XR device (e.g., the XR device (201) of FIG. 2) that provides a user with an image related to an XR (extended reality) service. XR or XR service may be defined as a service that collectively refers to virtual reality (VR), augmented reality (AR), or mixed reality (MR). The XR device (201) may be a head-mounted device (HMD), a head-mounted display (HMD), or AR glasses worn on a user's body or head, but is not limited thereto. FIG. 2 illustrates an XR device (201) implemented as AR glasses of the glass type, but this is merely an example.

[0062] The XR device (201) may include an OST (optical see-through) type configured to allow external light to reach the user's eyes through glass when worn, or a VST (video see-through) type configured to block external light so that, when worn, light emitted from the display reaches the user's eyes but external light does not reach the user's eyes.

[0063] The XR device (201) may provide a user with an image related to an extended reality (XR) service. For example, the XR device (201) may provide an XR service image (or immersive media) in which at least one virtual content (e.g., an image, a video, or an object) is displayed in a region (FoV (field of view) region) determined to be the user's field of view. The region determined to be the user's field of view is a region determined to be recognizable by the user through the XR device (201), and may be a region including all or at least a portion of the display of the XR device (201).

[0064] The XR device (201) may further include at least some of the configurations and / or functions of FIG. 1, and in the case of configurations overlapping with FIG. 1, the configurations may be substantially the same.

[0065] For example, the XR device (201) may include a display unit (210), a transparent member (220), an optical member (230), a camera unit (240), a lighting unit (250), a microphone unit (260), a speaker unit (270), a battery unit (280), and a PCB unit (290), but this is merely an example and is not limited thereto. The XR device (201) may include a housing including the above-described components, and each component may be at least partially supported or fixed to the housing. For example, the XR device (201) may include a glass-shaped housing.

[0066] The display unit (210) may include a first display (211) positioned on the right eye and a second display (212) positioned on the left eye. The parts / components constituting the first display (211) and the second display (212) may be the same. For example, the display unit (210) may be referred to as including a first display (211), a second display (212), a transparent member (220), a screen display unit (2210) (e.g., combiner optics), a display driver (213), and a lens (e.g., a projection lens) (not shown), but this is merely an example.

[0067] The first display (211) and the second display (212) may include a liquid crystal on silicon (LCoS), a light emitting diode (LED) on silicon (LEDoS), an organic light emitting diode (OLED), a micro light emitting diode (micro LED), and / or a digital mirror device (DMD). The first display (211) and the second display (212) may include a display area composed of pixels for displaying an image, and light-receiving pixels (e.g., photo sensor pixels) that receive light reflected from an eye and convert it into electrical energy and output it between the pixels.

[0068] The transparent member (220) is positioned on the front of the display (211, 212) and can protect the display (211, 212). The transparent member (220) can be formed of a glass plate, a plastic plate, or a polymer, and can be manufactured to be transparent or translucent. The transparent member (220) can control the transmission of light incident on the display (211, 212). The transparent member (220) or the screen display unit (2210) can include a lens including a waveguide (e.g., a waveguide) and / or a reflective lens.

[0069] Light emitted from the first display (211) and the second display (212) may pass through a lens (not shown) and a waveguide (e.g., a waveguide) and be reflected by the input optical member (222) and the grating area formed in the screen display unit (2210) and transmitted to the user's eyes. The waveguide may be made of glass, plastic, and / or polymer, and may include a nano-pattern formed on one surface of the inner or outer surface, for example, a grating structure having a polygonal or curved shape. The waveguide may include at least one diffractive element, for example, a diffractive optical element (DOE), a holographic optical element (HOE), or a reflective element (e.g., a reflective mirror). In one embodiment, the waveguide may guide the light emitted from the displays (211, 212) to the user's eyes by using at least one diffractive element or reflective element included in the waveguide. For example, in the case of an AR device (201) that provides augmented reality, a user can perceive the actual space (or actual environment) behind the display (211, 212) by looking through the display (211, 212).

[0070] The camera unit (240) may include a first camera (241), a second camera (242), and a third camera (243). The first camera (241) may be referred to as HR (high resolution) or PV (photo video) and may include a high-resolution camera. The first camera (211) may include a color camera equipped with functions for obtaining high-quality images, such as an AF (auto focus) function and an OIS (optical image stabilizer). The first camera (241) may also include a GS (global shutter) camera or an RS (rolling shutter) camera. The second camera (242) and the third camera (243) can perform at least one of 3 degrees of freedom (3DoF), 6DoF head tracking, hand detection and tracking, pose estimation & prediction, gesture recognition, slam function through depth shooting, spatial recognition, and / or gaze tracking.

[0071] For example, the second camera (242) and the third camera (243) may be implemented as at least one of a stereo camera for head tracking and spatial recognition, a gesture camera for detecting the user's movement (or hand), an eye tracking camera for tracking the gaze direction by tracking the movement of the user's left eye and / or right eye, a distance measuring camera (e.g., a time of flight camera (TOF) or depth camera) for measuring the distance to an object located in front of the XR device (201), a simultaneous localization and mapping camera (SLAM camera) for recognizing information related to the surrounding space (e.g., location and / or direction), and / or an RGB camera for detecting color-related information of an object and distance information from the object, or may be implemented as a camera in which some of them are integrated.

[0072] The lighting unit (250) may be placed in the housing (e.g., frame) of the XR device (201). The lighting unit (250) may be used as a means of supplementing the surrounding brightness when taking pictures with a camera. For example, the lighting unit (250) may be used when subject detection is not easy due to at least one of a dark environment, an environment where multiple light sources are mixed, or an environment where reflected light occurs. As another example, the lighting unit (250) may irradiate infrared wavelengths to facilitate detection of the pupil when detection of the pupil is not easy.

[0073] The microphone unit (260) includes multiple microphones and can process external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function being performed (or the application being executed) in the XR device (201). The speaker unit (270) includes multiple speakers and can output audio data under the control of the processor (120).

[0074] The battery unit (280) can supply power to drive components (e.g., the processor (120), memory (130), sensor module (176), and communication module (190) shown in FIG. 1) arranged in the display unit (210), camera unit (240), lighting unit (250), microphone unit (260), speaker unit (270), and PCB unit (290).

[0075] The sensor module (176) may include various sensors for detecting measurements related to the movement of the XR device (201) (e.g., velocity, acceleration, angular velocity, angular acceleration, and / or geographic location). For example, the sensor module (176) may include, but is not limited to, a proximity sensor, an ambient light sensor, a geomagnetic sensor, a gesture sensor, an acceleration sensor, and / or a gyro sensor. The proximity sensor may detect an object adjacent to the XR device (201). The ambient light sensor may measure the level of brightness around the XR device (201). The gyro sensor may detect the state (or posture, direction) and location of the XR device (201). The gyro sensor may detect the movement of the XR device (201) or a user wearing the XR device (201). For example, the XR device (201) can use a light sensor to check the brightness level around the XR device (201) and change brightness-related setting information of the display unit (210) based on the brightness level.

[0076] The memory (130) can store instructions executable by the processor (120) or the electronic device (101). The processor (120) can execute the instructions stored in the memory (130) to implement a software module and control components included in the XR device (201). The operation of the processor (120) described below can be performed when executing the instructions stored in the memory (130).

[0077] The processor (120) can obtain XR service image information related to a space corresponding to the field of view of a user wearing the XR device (201), and recognize an area (hereinafter, FOV) determined as the user's field of view (FOV). The processor (120) can control the display unit (210) so that at least a part of the XR service image is displayed through the recognized FOV. For example, a user wearing the XR device (201) can recognize an external environment (e.g., a background image) that is visible while passing through the display unit (210) and a virtual object (e.g., virtual content) output through the display unit (210) as being at least partially mixed. The user can distinguish and recognize an actual external environment image and a virtual object.

[0078] The processor (120) can measure measurements related to the movement of the XR device (201) (e.g., velocity, acceleration, angular velocity, angular acceleration, and geographical location), and can obtain movement information (e.g., rotation angle) of the XR device (201) using the measurements or a combination thereof. The processor (120) can analyze the movement information and XR service image information of the XR device (201) in real time and control the processing of operations required to provide an XR service, for example, head tracking and / or eye tracking operations.

[0079] According to one embodiment, the XR device (201) may provide an XR service video (or immersive media) in which at least one virtual content (e.g., an image, a video, or an object) is displayed based on a region determined by the user's field of view (FoV region). The XR device (201) may implement a virtual environment interface, output it through the display unit (210), and process various tasks within the virtual environment interface. The XR device (201) may recognize information related to tasks occurring within the virtual environment interface by dividing it by time or space (location), and store it in the form of a data block. According to real-time, when loading a specific task that was processed in the past, the XR device (201) may display the specific task by dividing it by time, or display the specific task by dividing it by space.

[0080] According to one embodiment, the XR device (201) may implement a time interface that appears divided by time and a space interface that appears divided by space in relation to a specific task, and may implement a virtual environment interface in which the time interface and the space interface are at least partially integrated. According to one embodiment, the XR device (201) may provide an augmented reality service to a user, thereby improving the convenience of the user using the augmented reality service.

[0081] FIG. 3 is a block diagram of an electronic device according to one embodiment of the present disclosure.

[0082] The electronic device (101) of FIG. 3 may be at least partially similar to the electronic device (101) of FIG. 1 and the XR device (201) of FIG. 2 (e.g., a wearable electronic device), or may further include other embodiments of the electronic device (101) and the XR device (201). According to one embodiment, the electronic device (101) may include at least partially similar components to the XR device (201) of FIG. 2. For example, the electronic device (101) may include one of a head-mounted device (HMD), a head-mounted display (HMD), or AR glasses worn on a user's body or head, as illustrated in FIG. 2, and may include the XR device (201) that provides an image related to an XR (extended reality) service to the user.

[0083] Referring to FIG. 3, the electronic device (101) may include a processor (120) (e.g., the processor (120) of FIG. 1), a memory (130) (e.g., the memory (130) of FIG. 1), a display module (160) (e.g., the display module (160) of FIG. 1, the display unit (210) of FIG. 2), a camera (180) (e.g., the camera module (180) of FIG. 1, the camera unit (240) of FIG. 2), and / or a communication module (190) (e.g., the communication module (190) of FIG. 1). The memory (130) of the electronic device (101) may store an augmented reality-related application (311) for implementing an augmented reality space (e.g., a virtual environment interface) and usage history information (312) which is information related to a specific task within the augmented reality space. According to one embodiment, the memory (130) may store various information related to augmented reality space and virtual content (e.g., virtual objects).

[0084] According to one embodiment, the processor (120) of the electronic device (101) may execute a program (e.g., program (140) of FIG. 1, augmented reality-related application (311)) stored in the memory (130) to control at least one other component (e.g., hardware and / or software component) and perform various data processing or operations. According to one embodiment, the processor (120) may include at least one processor including a processing circuit. According to one embodiment, the memory (130) may store instructions that the processors (120) (e.g., at least one processor) execute individually or collectively. According to one embodiment, the processor (120) may be operatively, functionally, and / or electrically connected to the memory (130), the display module (160), the camera (180), and / or the communication module (190).

[0085] According to one embodiment, the processor (120) can execute an augmented reality-related application (311) stored in the memory (130) and implement an augmented reality space (e.g., a virtual environment interface) based on the augmented reality-related application (311). The processor (120) can display the augmented reality space (e.g., a virtual environment interface) through the display module (160). According to one embodiment, the processor (120) can automatically execute the augmented reality-related application (311) in response to a situation in which the electronic device (101) is worn on the user's head and provide an augmented reality service to the user.

[0086] According to one embodiment, the electronic device (101) may be operatively connected to a dedicated controller for providing an augmented reality service to a user within an augmented reality space. For example, the dedicated controller may include an input means for at least partially controlling an augmented reality-related application (311). For example, the dedicated controller may include a hardware-implemented input device (e.g., a control device, a remote control, a stylus pen), or may include an input module and input means implemented in software (e.g., visually implemented) within the augmented reality space. As another example, the electronic device (101) may recognize a user's hand (e.g., a finger) as a dedicated controller, and may control the augmented reality-related application (311) based on input and movement using the user's hand. According to one embodiment, the processor (120) may detect a user's input (e.g., a prompt input, an audio input, a touch input, and / or a gesture input) and movement based on the dedicated controller, and may perform functions and / or operations related to augmented reality based on the user's input. For example, the processor (120) may detect an XR event signal for capturing a virtual environment image based on an augmented reality service. In response to detecting the XR event signal, the processor (120) may capture the virtual environment image.

[0087] According to one embodiment, the processor (120) may execute various functions and programs related to augmented reality within the augmented reality space. For example, the processor (120) may implement an augmented reality space in response to the execution of a first virtual environment program, and may execute a second virtual environment program while the augmented reality space is implemented. The second virtual environment program may be executed based on the augmented reality space implemented by the first virtual environment program, and an execution screen may be displayed in a form that at least partially overlaps the augmented reality space. For example, the processor (120) may execute a program (e.g., the second virtual environment program) for processing a specific task (e.g., document work, image work, audio work, task, process) within the augmented reality space, and may display a virtual object based on the program. For example, when a program related to document work is executed, a virtual page for document work may be displayed, and when a program related to image work is executed, a virtual page for image work may be displayed. For example, when a program related to audio work is executed, a virtual audio object for audio work may be displayed. According to one embodiment, the processor (120) may periodically or aperiodically store data related to the tasks (e.g., results and result data performed based on virtual objects) in the memory (130) while performing various tasks related to augmented reality (e.g., a second virtual environment program). For example, data related to tasks in augmented reality may be stored in usage history information (312).

[0088] According to one embodiment, the processor (120) may store event information and virtual object information (e.g., information about a virtual object) as a single data block when storing data (e.g., results, result data) related to a task in augmented reality. For example, the event information may include at least one of location information (e.g., GPS information, address information, and / or latitude / longitude information) of the electronic device (101) in a situation where the task is performed, time series information (e.g., time information, point-in-time information, and / or captured point-in-time information), task information (e.g., file name of the task, file type, and / or file-related information), and / or territory information (e.g., feature information, characteristic information, and / or background information of a place). For example, virtual object information (e.g., information about a virtual object) may include coordinate information about an object generated based on the above task (e.g., coordinate information related to a display location, posture information of the object, and / or shape information of the object) and path information (e.g., movement information and / or movement information of the object), and / or crawling information (e.g., when the virtual content is a web page, web page-related information, scroll information, login information, and / or search information). According to one embodiment, the processor (120) may store event information and information about a virtual object as one data block in usage history information (312) of the memory (130). The usage history information (312) may store data related to an task performed on augmented reality (e.g., a virtual environment interface). The usage history information (312) may include event information and information about a virtual object.

[0089] According to one embodiment, the camera (180) may be positioned so that the direction of the lens faces the front of the electronic device (101) (e.g., the direction of the user's gaze) and may capture an actual environment. For example, the processor (120) may use the camera (180) to capture an external environment corresponding to the front of the electronic device (101) and obtain a captured image based on the external environment. The processor (120) may obtain territory information (e.g., location feature information, characteristic information, background information) based on the captured image and store the obtained territory information in the usage history information (312) of the memory (130).

[0090] According to one embodiment, the display module (160) can visually display a program running on the electronic device (101) and an activated function. For example, when an augmented reality-related application (311) is executed, the processor (120) can implement an augmented reality space (e.g., a virtual environment interface) and display the augmented reality space through the display module (160). According to one embodiment, the electronic device (101) can visually display an augmented reality space implemented based on an augmented reality service to the user through the display module (160). In one embodiment, a user wearing the electronic device (101) on his / her head can feel as if he / she is living in an actual real space.

[0091] According to one embodiment, the communication module (190) can perform a communication connection with an external electronic device (e.g., the electronic device (102, 104) of FIG. 1). For example, the electronic device (101) (e.g., the HMD device) and the external electronic device (102, 104) (e.g., the portable electronic device) can be connected to each other through the communication module (190) according to various communication methods (e.g., a wired communication channel, a wireless communication channel).

[0092] According to one embodiment, the electronic device (101) can independently implement an augmented reality space and provide an augmented reality service to a user. According to one embodiment, the electronic device (101) may be operatively connected to an external electronic device (102) (e.g., a portable electronic device, a smartphone) through a communication module (190) and may be at least partially controlled by the external electronic device (102). For example, in a state where an augmented reality space has already been implemented in the external electronic device (102), the electronic device (101) may perform an operation of displaying the implemented augmented reality space through the display module (160). The electronic device (101) may implement an augmented reality space in response to the execution of the first virtual environment program.

[0093] According to one embodiment, a processor (120) of an electronic device (101) (e.g., an HMD device) can detect a situation in which the electronic device (101) is worn on a user's head, and execute an augmented reality-related application (311) (e.g., a first virtual environment program) stored in a memory (130). The processor (120) can implement a virtual environment interface (e.g., an augmented reality space) based on the executed augmented reality-related application (311), and display the implemented virtual environment interface through a display module (160).

[0094] According to one embodiment, the electronic device (101) may implement a virtual environment interface (e.g., an augmented reality space) in response to the execution of a first virtual environment program, and may execute a second virtual environment program within the virtual environment interface. For example, the first virtual environment program and the second virtual environment program may be the same program or different programs. According to one embodiment, the electronic device (101) may perform a specific task (e.g., document work, image work, audio work, the second virtual environment program) within the virtual environment interface, and, in an intermediate step of processing the specific task, may record and store a result (e.g., result data) for the specific task. For example, when storing the result (e.g., result data) for the specific task, the processor (120) may store event information (e.g., location information, time series information, task information, and / or spatial information) and information (e.g., coordinate information, posture information, and / or shape information) related to the specific task as a single data block. The processor (120) can store the results (e.g., result data) for the specific task by dividing them by time (e.g., point in time) or by space.

[0095] According to one embodiment, when loading results (e.g., result data) for a specific task processed in the past, the electronic device (101) may display results (e.g., the first task) corresponding to each time, divided by time (point in time), or display results (e.g., the second task) corresponding to each space, divided by space (location), and / or display the first task and the second task together within a single interface (e.g., a virtual environment interface). According to one embodiment, the electronic device (101) may select and load results for a specific task that a user processed at a specific point in time. The user may continue processing one result among the results for a specific task at a specific point in time. According to one embodiment, the electronic device (101) may select and load a specific task that a user processed in a specific space (location). The user may continue processing the specific task in the specific space. According to one embodiment, the electronic device (101) may implement a task situation at a specific point in time or in a specific space for tasks that the user processed in the past. Users can continue to process specific tasks at a specific point in time or space, and can more efficiently process tasks within a virtual environment interface. This can improve user convenience in augmented reality situations.

[0096] According to one embodiment, an electronic device (101) wearable on a user's head may include a housing, a display (160) supported by the housing and outputting visual information, a communication module (190) including a communication circuit, a processor (120) including a processing circuit, and a memory (130) storing instructions. When the instructions are individually or collectively executed by the processor (120), the electronic device (101) may, in response to an execution command of a virtual environment program, check a usage record based on the virtual environment program, load event information and information on virtual objects included in the usage record, and, based on the event information and the information on the virtual object, generate a virtual environment interface in which a time interface in which results corresponding to each time can be displayed separately and in a space interface in which results corresponding to each space can be displayed separately and in a space, at least partially together, among results performed on the virtual object in the electronic device (101), and visually output the generated virtual environment interface through the display (160).

[0097] According to one embodiment, the electronic device (101) may further include a camera (180) that photographs the surroundings of the electronic device (101). When the instructions are individually or collectively executed by the processor (120), the electronic device (101) may check a background image photographed using the camera (180), generate at least one virtual object that is positioned closer to the user than the background image within a field of view of the user based on the virtual environment program, generate a data block including the background image and the at least one virtual object, and store the generated data block as the usage record in the memory (130).

[0098] According to one embodiment, when the instructions are individually or collectively executed by the processor (120), the electronic device (101) communicates with an external server through the communication module (190), transmits the generated data block to the external server, and based on the external server, the generated data block can be stored.

[0099] According to one embodiment, the event information may include at least one of location information of the electronic device (101), task information for the virtual environment program, time series information when performing a task of the virtual environment program, or spatial information based on a background image.

[0100] According to one embodiment, the spatial information may include location information indicating characteristics of the background image based on at least one object included in the background image.

[0101] According to one embodiment, the information about the virtual object may include at least one of coordinate information about the location of at least one object generated based on the virtual environment program, path information about the movement of the at least one object, or crawling information related to the at least one object.

[0102] According to one embodiment, when the instructions are individually or collectively executed by the processor (120), the electronic device (101) can determine one display mode among a spatiotemporal integration mode, a time-based display mode, and a space-based display mode based on a spatiotemporal mode setting icon included in the virtual environment interface.

[0103] According to one embodiment, when the instructions are individually or collectively executed by the processor (120), the electronic device (101) generates at least one time interface divided by time when the time-based display mode is determined based on the space-time mode setting icon, and the results performed according to the at least one time interface can be arranged in the order of time elapsed.

[0104] According to one embodiment, when the instructions are individually or collectively executed by the processor (120), when an object included in the virtual environment interface is selected in the time-based display mode, the electronic device (101) can execute a history mode that shows changes in the selected object over time, and output the object transformed over time in the history mode.

[0105] According to one embodiment, when the instructions are individually or collectively executed by the processor (120), the electronic device (101) can generate at least one space interface divided by space when the space-specific display mode is determined based on the space-time mode setting icon, and arrange the performed results in correspondence with the location according to the at least one space interface.

[0106] According to one embodiment, when the instructions are individually or collectively executed by the processor (120), the electronic device (101) can communicate with an external electronic device through the communication module (190), obtain a program running on the external electronic device and data related to the program, and visually output the program and data related to the program based on the external electronic device based on the virtual environment interface.

[0107] FIG. 4 is a flowchart illustrating a method for creating a virtual environment interface that integrates a time interface and a space interface according to one embodiment of the present disclosure. FIG. 5 is a flowchart illustrating a method for storing a data block containing a background image and a virtual object according to one embodiment of the present disclosure.

[0108] In the following examples, the operations may be performed sequentially, but are not necessarily sequential. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0109] According to one embodiment, operations 401 to 409, or operations 501 to 507 may be understood to be performed by a processor (e.g., the processor 120 of FIG. 3, a processing circuit, at least one processor) of an electronic device (e.g., the electronic device (101) of FIG. 3). The electronic devices of FIGS. 4 and 5 may be at least partially similar to the XR device (201) of FIG. 2 and the electronic device (101) of FIG. 3, or may further include other embodiments of the electronic device (101). For example, the electronic device (101) may include an XR device (201), an electronic device in the form of glasses, and / or a wearable electronic device, as illustrated in FIG. 2. The electronic device (101) may include a wearable electronic device (e.g., a head-mounted display (HMD) device) that operates while being at least partially worn on a part of a user's body (e.g., a head).

[0110] In operation 401, the processor (120) of the electronic device (101) may execute a virtual environment program (e.g., an augmented reality-related application (311) of FIG. 3) stored in a memory (e.g., a memory (130) of FIG. 3) in a situation where the electronic device (101) is worn on a part of the user's body (e.g., a head). For example, the electronic device (101) may automatically execute a virtual environment program (e.g., a first virtual environment program) in response to the situation where the electronic device (101) is worn on the user's head, or may execute the virtual environment program in response to a user input. The processor (120) may implement an augmented reality space in response to the execution of the virtual environment program, and output a visualized virtual environment interface. The user may visually confirm the virtual environment interface (e.g., an augmented reality space).

[0111] In operation 403, the processor (120) may, while a virtual environment program (e.g., a first virtual environment program) is running, check a data block indicating a usage record (e.g., a usage record for past tasks). For example, in response to the execution of the first virtual environment program, the processor (120) may implement a virtual environment interface (e.g., an augmented reality space) and execute a second virtual environment program within the virtual environment interface. The second virtual environment program may include a specific program (e.g., a program for performing a specific task (e.g., a document task, an image task, an audio task)) that is executed within the augmented reality space. The processor (120) may, in response to the execution of the second virtual environment program, check a data block indicating a usage record (e.g., a usage record for past tasks) related to the second virtual environment program. According to one embodiment, the first virtual environment program and the second virtual environment program may be the same program or may be different programs. For example, the electronic device (101) may store information related to tasks processed in the past in a memory (130) (e.g., usage history information (312) of FIG. 3) based on a virtual environment program (e.g., a first virtual environment program and a second virtual environment program). The processor (120) may check a data block representing a usage record based on the information stored in the usage history information (312).

[0112] In operation 405, the processor (120) may load event information and virtual object information (e.g., information about a virtual object) included in the data block. For example, the event information may include at least one of location information (e.g., GPS information, address information, latitude / longitude information), time series information (e.g., point-in-time information, and captured point-in-time information) of the electronic device (101) in a situation where a past task was being processed, task information (e.g., file name of the task, file type, file-related information), and / or territory information (e.g., feature information of a place, characteristic information, background information). For example, virtual object information (e.g., information about a virtual object) may include coordinate information about the object (e.g., coordinate information related to a display location, posture information about the object, shape information about the object) and path information (e.g., movement information and / or movement information about the object), and / or crawling information (e.g., when the virtual content is a web page, web page-related information, scroll information, login information, and / or search information) generated based on the past task. For example, when storing a past task, the processor (120) may store event information and information about a virtual object as one data block. According to one embodiment, a past task may include a result (e.g., result data) for a specific task performed within a virtual environment interface. When storing the result (e.g., result data) for the specific task, the processor (120) may store event information (e.g., location information, time series information, task information, and / or spatial information) and information about a virtual object (e.g., coordinate information, posture information, and / or shape information) related to the specific task as one data block.

[0113] In operation 407, the processor (120) may generate a virtual environment interface (e.g., an integrated virtual environment interface) in which a time interface and a space interface are provided together based on event information and information about virtual objects. For example, the processor (120) may divide past tasks into results corresponding to each time by time (e.g., a point in time when the past task was processed), and may generate a time interface based on the results (e.g., a first task) divided by time. For example, the processor (120) may divide past tasks into results corresponding to each space by space (e.g., a space or location when the past task was processed), and may generate a space interface based on the results (e.g., a second task) divided by space. The processor (120) may display only the time interface or only the space interface within the augmented reality space. The processor (120) may also generate and display a virtual environment interface in which the time interface and the space interface are at least partially integrated. The processor (120) can generate a virtual environment interface (e.g., an integrated virtual environment interface) that allows both a time interface and a space interface to be recognized within a single screen. According to one embodiment, the electronic device (101) can provide an option to select one interface from among the time interface and the space interface based on the integrated virtual environment interface, and can selectively display one interface based on the user's selection.

[0114] In one embodiment, the processor (120) may display a time interface based on a first criterion. For example, the processor (120) may display results sequentially, starting with the most recently processed result, according to the chronological order in which the results were stored. For another example, the processor (120) may preferentially display the most recently processed result. The first criterion may be variously set by the user. In one embodiment, the processor (120) may display a spatial interface based on a second criterion. For example, the processor (120) may sequentially display the spatial interfaces, starting with the spatial interface containing the most results, according to the number of stored results. For another example, the processor (120) may preferentially display the spatial interface containing the most recently processed result. The second criterion may be variously set by the user.

[0115] According to one embodiment, the integrated virtual environment interface may display stored results arranged in chronological order based on the first spatial interface. For example, results of work performed in the past by a user within a specific space (e.g., an office) may be stored on an electronic device (101). When implementing the integrated virtual environment interface, the electronic device (101) may be able to check the stored results within the office (e.g., the first spatial interface) and display the results in a form in which the results are arranged in chronological order.

[0116] In operation 409, the processor (120) may output a virtual environment interface (e.g., an integrated virtual environment interface) through a display (e.g., a display module (160) of FIG. 3). For example, the processor (120) may visually display the generated virtual environment interface through the display module (160). In one embodiment, a user wearing the electronic device (101) on his / her head may check a virtual environment interface (e.g., an integrated virtual environment interface) within a real environment, and may check past tasks classified by time or space based on the virtual environment interface. The user may select one interface from among the time interface and / or the space interface based on the integrated virtual environment interface, and may check results corresponding to past tasks centered on the selected interface.

[0117] According to one embodiment, when loading results (e.g., result data) for a specific task processed in the past, the electronic device (101) may display results (e.g., a first task) corresponding to each time, divided by time (point in time), or display results (e.g., a second task) corresponding to each space, divided by space (location), and / or display the first task and the second task together within a single interface (e.g., an integrated virtual environment interface). The electronic device (101) may display results (e.g., result data) for a plurality of past tasks within the integrated virtual environment interface. The electronic device (101) may implement a task situation at a specific time or in a specific space for tasks that the user has processed in the past. The user may continue to process a specific task at a specific time or in a specific space, and may process the task processing on the virtual environment interface more efficiently. User convenience in an augmented reality situation may be improved.

[0118] In operation 501, the processor (120) may check a background image captured using a camera (e.g., the camera (180) of FIG. 3). For example, the processor (120) may use the activated camera (180) to capture a front view of the electronic device (101) (e.g., a direction of the user's gaze, the surroundings of the electronic device (101)) and check the captured background image. The processor (120) may periodically or non-periodically check the background image captured using the camera (180). According to one embodiment, the processor (120) may collect event information (e.g., location information of the electronic device (101) (e.g., GPS information, address information, latitude / longitude information), time series information (e.g., point-in-time information, and captured point-in-time information), task information (e.g., file name of the task, file type, file-related information), and / or territory information (e.g., feature information of a place, characteristic information, background information)) based on the background image.

[0119] In operation 503, the processor (120) may generate a virtual object based on a virtual environment program. For example, the processor (120) may generate a virtual object that is at least partially overlapped with a background image. The processor (120) may generate at least one virtual object that is positioned relatively closer to the user than the background image within a field of view of the user. For example, the at least one virtual object may be positioned closer to the user than the background image in terms of depth, centered on the user's field of view. The virtual object may include a virtual window for a specific task in the virtual environment program. According to one embodiment, the processor (120) may collect coordinate information (e.g., coordinate information related to a display position, posture information of the virtual object, shape information of the virtual object) and path information (e.g., movement information and / or movement information of the virtual object) and / or crawling information (e.g., when the virtual content (virtual object) is a web page, web page-related information, scroll information, login information, and / or search information) for the virtual object based on the virtual object.

[0120] In operation 505, the processor (120) may generate a data block including a background image and a virtual object. For example, the processor (120) may generate a data block based on event information collected based on the background image and information about the virtual object collected based on the virtual object. For example, if there are multiple virtual objects, multiple data blocks may be generated. The processor (120) may generate a data block corresponding to each virtual object. According to one embodiment, when loading a virtual object on a virtual environment interface, the processor (120) may identify a data block corresponding to the virtual object to be loaded, and load the virtual object based on the identified data block.

[0121] In operation 507, the processor (120) may store the generated data block in the memory (130). For example, the processor (120) may store the data block in the usage history information of the memory (130) (e.g., the usage history information (312) of FIG. 3).

[0122] According to one embodiment, the electronic device (101) may generate a data block including event information and virtual object information (e.g., information about the virtual object) when storing data (e.g., results of the task, result data) related to a task (e.g., a virtual object) in augmented reality (e.g., a virtual environment interface), and may store the data block in the memory (130) (e.g., usage history information (312) of the memory (130). For example, the event information may be collected based on a background image captured through a camera (180) of the electronic device (101), and the information about the virtual object may be collected based on a virtual object generated on a virtual environment program. The electronic device (101) may manage the virtual object in the form of a data block.

[0123] FIG. 6A is a first exemplary diagram illustrating a first task processed at a first time point according to an embodiment of the present disclosure. FIG. 6B is a second exemplary diagram illustrating a second task processed at a second time point according to an embodiment of the present disclosure. FIG. 6C is a third exemplary diagram illustrating a third task processed at a third time point according to an embodiment of the present disclosure. FIG. 6D is a fourth exemplary diagram illustrating a fourth task processed at a fourth time point according to an embodiment of the present disclosure.

[0124] The electronic device of FIGS. 6A to 6D may be at least partially similar to the XR device (201) of FIG. 2 and the electronic device (101) of FIG. 3, or may further include other embodiments of the electronic device (101). The electronic device (101) may include a wearable electronic device (e.g., a head-mounted display (HMD) device) that operates while being at least partially worn on a part of a user's body (e.g., a head). The operations in FIGS. 6A to 6D may be understood to be performed by a processor of the electronic device (101) (e.g., the processor (120) of FIG. 3, a processing circuit, at least one processor).

[0125] Referring to FIG. 6A, the processor (120) may display a first virtual environment interface (601) implemented based on a virtual environment program (e.g., a first virtual environment program) through a display module (160) (e.g., the display module (160) of FIG. 3). For example, the first virtual environment interface (601) may include a background image (611) captured using a camera (180) of the electronic device (101) (e.g., the camera (180) of FIG. 3) and a first virtual object (612) generated based on a virtual environment program (e.g., a second virtual environment program). The first virtual object (612) may be generated in a form that is positioned relatively closer to the user than the background image (611) within the user's field of view. The processor (120) can generate a first virtual environment interface (601) based on a background image (611) captured at a first point in time and a first virtual object (612) being worked on at the first point in time. The first virtual object (612) can display information related to a file worked on at the first point in time (e.g., file name and file type). The processor (120) can generate a first virtual environment interface (601) including the background image (611) and the first virtual object (612) in response to an event signal at the first point in time. The processor (120) can store the first virtual environment interface (601) in the memory (130).

[0126] Referring to FIG. 6B, the processor (120) may display a second virtual environment interface (602) implemented based on a virtual environment program (e.g., a first virtual environment program) through the display module (160). For example, the second virtual environment interface (601) may include a background image (611) captured using a camera (180) of the electronic device (101) and a second virtual object (613) generated based on the virtual environment program (e.g., the second virtual environment program). The processor (120) may generate the second virtual environment interface (602) based on the background image (611) captured at a second point in time and the second virtual object (613) being worked on at the second point in time. The second virtual object (613) may display information related to a file worked on at the second point in time (e.g., an audio file icon). The processor (120) may generate a second virtual environment interface (602) including the background image (611) and the second virtual object (613) in response to an event signal at a second point in time. The processor (120) may store the second virtual environment interface (602) in the memory (130).

[0127] Referring to FIG. 6C, the processor (120) may display a third virtual environment interface (603) implemented based on a virtual environment program (e.g., a first virtual environment program) through the display module (160). For example, the third virtual environment interface (603) may include a background image (611) captured using a camera (180) of the electronic device (101) and a third virtual object (614) generated based on a virtual environment program (e.g., a second virtual environment program). The processor (120) may generate the third virtual environment interface (603) based on the background image (611) captured at a third point in time and the third virtual object (614) being worked on at the third point in time. The third virtual object (614) may display information (e.g., content, 3D content) related to a file worked on at the third point in time together. The processor (120) may generate a third virtual environment interface (603) including the background image (611) and the third virtual object (614) in response to an event signal at a third point in time. The processor (120) may store the third virtual environment interface (603) in the memory (130).

[0128] Referring to FIG. 6D, the processor (120) may display a fourth virtual environment interface (604) implemented based on a virtual environment program (e.g., a first virtual environment program) through the display module (160). For example, the fourth virtual environment interface (604) may include a background image (611) captured using a camera (180) of the electronic device (101) and a fourth virtual object (615) generated based on a virtual environment program (e.g., a second virtual environment program). The processor (120) may generate the fourth virtual environment interface (604) based on the background image (611) captured at the fourth point in time and the fourth virtual object (615) being worked on at the fourth point in time. The fourth virtual object (615) may display information related to a file worked on at the fourth point in time (e.g., a file name and a file type). The processor (120) may generate a fourth virtual environment interface (604) including the background image (611) and the fourth virtual object (615) in response to an event signal at a fourth point in time. The processor (120) may store the fourth virtual environment interface (604) in the memory (130).

[0129] Referring to FIGS. 6A to 6D , when specific tasks are performed at different points in time (e.g., first to fourth points in time) on the same background image (611) (e.g., same place, same location), the processor (120) can individually distinguish each specific task according to the point in time. Each specific task may be processed at the same place, but may be processed so that different results are implemented at different points in time. According to one embodiment, the electronic device (101) can distinguish and manage virtual objects worked on at different points in time. The first virtual environment interface (601) to the fourth virtual environment interface (604) may be distinguished by a time interface. Although not illustrated, the electronic device (101) can also distinguish and manage virtual objects worked on at different places according to each place.

[0130] FIG. 7A is an exemplary diagram illustrating a prompt signal for loading past work in augmented reality according to one embodiment of the present disclosure. FIG. 7B is an exemplary diagram illustrating a virtual environment interface that integrates a temporal interface and a spatial interface according to one embodiment of the present disclosure.

[0131] The electronic device of FIGS. 7A and 7B may be at least partially similar to the XR device (201) of FIG. 2 and the electronic device (101) of FIG. 3, or may further include other embodiments of the electronic device (101). The operations in FIGS. 7A and 7B may be understood to be performed by a processor of the electronic device (101) (e.g., the processor (120) of FIG. 3, a processing circuit, at least one processor).

[0132] Referring to FIG. 7A, the processor (120) may display a virtual environment area implemented based on a virtual environment program (e.g., a first virtual environment program) through a display module (160) (e.g., the display module (160) of FIG. 3). For example, the virtual environment area may include a background image (611) captured using a camera (180) of the electronic device (101) (e.g., the camera (180) of FIG. 3). A user wearing the electronic device (101) may check the background image (611) based on the virtual environment program.

[0133] Referring to FIG. 7A, the electronic device (101) can verify a user's input (711) (e.g., prompt input, touch input, and / or gesture input) based on a virtual environment area, and can activate a microphone for voice recognition in response to the user input (711). For example, the processor (120) can display a voice input window (712) for entering text according to the voice input. Referring to FIG. 7A, the processor (120) can receive a voice input of "Find the flagship store design work" by the user based on the activated microphone, and can input text corresponding to the voice input into the voice input window (712).

[0134] According to one embodiment, the electronic device (101) can identify the user's intention corresponding to the user's voice input based on an artificial intelligence model (e.g., an AI model) and collect a data block based on the user's intention. Referring to FIG. 7A, the processor (120) can identify virtual objects that have been processed or stored in the past in relation to the "flagship store design." For another example, the processor (120) can identify distinct virtual objects based on the current location or another location. For example, the processor (120) can input the user's voice data based on a multi-modal interface and obtain result data based on the multi-modal interface. The multi-modal interface may be a method of processing various types of data, such as text, images, voices, and videos, together. The processor (120) can obtain result data reflecting the user's intention based on the multi-modal interface and can identify at least one data block to be loaded based on the obtained result data. The processor (120) can load at least one data block and individually implement a virtual object corresponding to at least one data block.

[0135] Referring to FIG. 7B, the processor (120) may implement a first virtual object (741) based on a first data block, a second virtual object (742) based on a second data block, a third virtual object (743) based on a third data block, and a fourth virtual object (745) based on a fourth data block. For example, the first virtual object (741) to the fourth virtual object (745) may be tasks related to “flagship store design” and may be recognized as having been performed at different times in different locations (e.g., at home, a cafe, an office). Referring to FIG. 7B, the processor (120) may implement a fifth virtual object (746) based on a fifth data block. For example, the fifth virtual object (746) may correspond to a task related to “flagship store design,” but may be perceived as having been performed in a different location. In one embodiment, the virtual objects (741, 742, 743, 745, 746) may be created in a form that is positioned relatively closer to the user than the background image, within the user’s field of view.

[0136] According to one embodiment, the processor (120) may load virtual objects (741, 742, 743, 745, 746) corresponding to each data block based on a plurality of data blocks, and may generate a virtual environment interface including the loaded virtual objects (741, 742, 743, 745, 746). For example, the virtual objects (741, 742, 743, 745, 746) may be arranged in a form that at least partially overlaps each other. The virtual objects (741, 742, 743, 745, 746) may be arranged at a position relatively closer to the user than the background image within the user's field of view. The virtual object may be implemented based on the position and size at a time point during processing in the past.

[0137] Referring to FIG. 7B, the electronic device (101) may select either a spatiotemporal integrated display mode (721) or a spatially integrated display mode (722) when displaying a virtual environment interface. For example, the spatiotemporal integrated display mode (721) may be displayed as a spatiotemporal mode setting icon, and the spatially integrated display mode (722) may be displayed as a spatial mode setting icon. For example, when the spatiotemporal integrated display mode (721) is selected, the processor (120) may display a virtual environment interface (e.g., an integrated virtual environment interface) in which a time interface in which results corresponding to each time are displayed separately by time (e.g., a point in time) and a place interface in which results corresponding to each space are displayed separately by space (e.g., a place) are at least partially provided together. As another example, when the spatially integrated display mode (722) is selected, the processor (120) may display a virtual environment interface that is divided by space based on a plurality of virtual objects. The virtual environment interface may include a space option (730) for distinguishing virtual objects based on a space (territory) (731, 732, 733) (e.g., a place). For example, the space option (730) may include a first option (731) for selecting results based on past work at “home,” a second option (732) for selecting results based on past work at “cafe,” and / or a third option (733) for selecting results based on past work at “work.”

[0138] According to one embodiment, the electronic device (101) can select virtual objects classified by time and virtual objects classified by space based on a virtual environment program. When in a spatiotemporal integrated display mode (721), the electronic device (101) can generate a virtual environment interface in which all virtual objects are integrated, and output the virtual environment interface (e.g., an integrated virtual environment interface) through the display module (160).

[0139] FIG. 8A is a first exemplary diagram illustrating a first task processed in a first space according to one embodiment of the present disclosure. FIG. 8B is a second exemplary diagram illustrating a second task processed in a second space according to one embodiment of the present disclosure.

[0140] The electronic device of FIGS. 8A and 8B may be at least partially similar to the XR device (201) of FIG. 2 and the electronic device (101) of FIG. 3, or may further include other embodiments of the electronic device (101). The operations in FIGS. 8A and 8B may be understood to be performed by a processor of the electronic device (101) (e.g., the processor (120) of FIG. 3, a processing circuit, at least one processor).

[0141] Referring to FIG. 8A, the processor (120) can display a virtual environment area implemented based on a virtual environment program through the display module (160) (e.g., the display module (160) of FIG. 3). The electronic device (101) operates in a space-specific integrated display mode (811), and can display virtual objects (813, 814) related to a task processed in “home” (812) based on a first space (first space) (821). For example, in response to the selection of “home” (812), the processor (120) can select a first virtual object (813) and a second virtual object (814) processed in the past in “home”, and display a first virtual environment interface including the first virtual object (813) and the second virtual object (814). A user (801) of an electronic device (101) may be in a state of gazing at a first space (821) and may check a first virtual environment interface corresponding to the first space (821). According to one embodiment, the electronic device (101) may be implemented with a plurality of spaces (821, 822, 823) centered around the user (801) and surrounding the user. For example, when a user (801) wearing the electronic device (101) is gazing at the first space (821), the user (801) may check a first virtual environment interface including virtual objects (813, 814) that were worked on at the first place (location) (812).

[0142] Referring to FIG. 8B, the electronic device (101) operates in a space-specific integrated display mode (811) and, based on a second space (822), can display virtual objects (832, 833) related to a task processed in “park” (831). For example, in response to the selection of “park” (831), the processor (120) can select a virtual object (832, 833) corresponding to a task processed in the past in “park” and display a second virtual environment interface including the selected virtual object (832, 833). The user (801) of the electronic device (101) can be in a state of gazing at the second space (822) and can check the second virtual environment interface corresponding to the second space (822). For example, when a user (801) wearing an electronic device (101) gazes at a second space (822), the user (801) can check a second virtual environment interface including virtual objects (832, 833) that were worked on in the second place (location) (831).

[0143] FIG. 9 is an exemplary diagram illustrating a virtual environment interface generated based on a plurality of tasks corresponding to a plurality of time points according to one embodiment of the present disclosure.

[0144] The electronic device of FIG. 9 may be at least partially similar to the XR device (201) of FIG. 2 and the electronic device (101) of FIG. 3, or may further include other embodiments of the electronic device (101).

[0145] Referring to FIG. 9, a processor (e.g., processor (120) of FIG. 3, processing circuit, at least one processor) of an electronic device (101) may display a virtual environment area implemented based on a virtual environment program through a display module (160) (e.g., display module (160) of FIG. 3). The electronic device (101) may be in a state of operating in a time-based display mode. For example, the time-based display mode may be a mode in which tasks processed in the past are displayed in a chronological order.

[0146] Referring to FIG. 9, a first virtual environment interface (911) may include a first virtual object (e.g., a virtual window) related to a task processed at a first point in time. A second virtual environment interface (912) may include a second virtual object (e.g., an audio icon) related to a task processed at a second point in time. A third virtual environment interface (913) may include a third virtual object (e.g., a three-dimensional chair image) related to a task processed at a third point in time. A fourth virtual environment interface (914) may include a fourth virtual object (e.g., a virtual window) related to a task processed at a fourth point in time. For example, time may flow sequentially from a first point in time to a fourth point in time. The first point in time may be a relatively oldest point in time, and the fourth point in time may be a relatively newest point in time.

[0147] Referring to FIG. 9, the electronic device (101) may be implemented with a plurality of spaces (921, 922, 923, 924, 925, 926) centered around the user (901) and surrounding the user (901). Referring to FIG. 9, it is illustrated that about six spaces (921, 922, 923, 924, 925, 926) are formed centered around the user (901), but the present invention is not limited thereto. When the user (901) wearing the electronic device (101) gazes at the first space (921), the user (901) can check the first virtual environment interface (911) including the first virtual object that was worked on at the first point in time. For example, when a user (901) gazes at a second space (922), the user (901) can check a second virtual environment interface (912) according to a second viewpoint, and when the user (901) gazes at a third space (923), the user (901) can check a third virtual environment interface (913) according to a third viewpoint. For example, the screen switching from the first virtual environment interface (911) to the second virtual environment interface (912) may include the user (901) wearing the electronic device (101) directly changing the gaze direction from the first space (921) to the second space (922), or the screen changing by inputting a set gesture.

[0148] FIG. 10A is a first exemplary diagram illustrating a first object at a first point in time according to one embodiment of the present disclosure. FIG. 10B is a second exemplary diagram illustrating a first object at a first point in time and a second object at a second point in time together according to one embodiment of the present disclosure.

[0149] The electronic device of FIGS. 10A and 10B may be at least partially similar to the XR device (201) of FIG. 2 and the electronic device (101) of FIG. 3, or may further include other embodiments of the electronic device (101). The operations in FIGS. 10A and 10B may be understood to be performed by a processor of the electronic device (101) (e.g., the processor (120) of FIG. 3, a processing circuit, at least one processor).

[0150] Referring to FIGS. 10A and 10B, the processor (120) of the electronic device (101) can display a virtual environment area implemented based on a virtual environment program through a display module (160) (e.g., the display module (160) of FIG. 3).

[0151] Referring to FIG. 10A, the electronic device (101) may select either a history viewing mode (1001) or a timeline viewing mode (e.g., the timeline viewing mode (1101) of FIG. 11A) when displaying a virtual environment interface. For example, when the history viewing mode (1001) is selected, the processor (120) may display time series information (1002) related to past operations on the first virtual object (1011). The time series information (1002) may include records stored in a memory (e.g., the memory (130) of FIG. 3) during the past processing of the first virtual object (1011). The time series information (1002) may be arranged at multiple points in time according to the passage of time.

[0152] Referring to FIG. 10A, the electronic device (101) may verify a first user input (1010) (e.g., a prompt input and / or a gesture input) based on a virtual environment area, and in response to the first user input (1010), select a first point in time (1004) (e.g., 24.04.26, a specific date). For example, the first virtual object (1011) may be displayed in the form in which it was stored at the first point in time (1004). When the first point in time (1004) is selected, the processor (120) may display a time bar (1003) indicating a change process of the first virtual object (1011) that was processed at the first point in time (1004).

[0153] Referring to FIG. 10B, the electronic device (101) may select a second point in time (1021) while the first virtual object (1011) is being displayed at the first point in time (1004). The second point in time (1021) may be a point in the past relative to the first point in time (1004). For example, the second virtual object (1031) may be displayed in the form in which it was stored at the second point in time (1021). The processor (120) may display the first virtual object (1011) at the first point in time (1004) and the second virtual object (1031) at the second point in time (1021) together. When the second point in time (1021) is selected, the processor (120) can display a time bar (1022) indicating the change process of the second virtual object (1031) that was processed at the second point in time (1021).

[0154] FIG. 11a is an exemplary diagram illustrating a virtual environment interface in which multiple objects corresponding to multiple viewpoints are arranged according to one embodiment of the present disclosure. FIG. 11b is an exemplary diagram illustrating a virtual environment interface in which multiple objects corresponding to multiple viewpoints are arranged in various configurations according to one embodiment of the present disclosure.

[0155] The electronic device of FIGS. 11A and 11B may be at least partially similar to the XR device (201) of FIG. 2 and the electronic device (101) of FIG. 3, or may further include other embodiments of the electronic device (101). The operations in FIGS. 11A and 11B may be understood to be performed by a processor of the electronic device (101) (e.g., the processor (120) of FIG. 3, a processing circuit, at least one processor).

[0156] Referring to FIG. 11A, the electronic device (101) may select either a history viewing mode (e.g., the history viewing mode (1001) of FIG. 10A) or a timeline viewing mode (1101) when displaying a virtual environment interface. For example, when the timeline viewing mode (1101) is selected, the processor (120) may display time series information (1121-1130) and the shape (1111-1120) of the virtual object corresponding to the time series information (1121-1130) when processing a virtual object (e.g., a 3D chair content).

[0157] For example, the electronic device (101) can store a first virtual object (1120) at a first time point (1121) (e.g., 24.04.10) and store a second virtual object (1119) at a second time point (1122) (e.g., the first processing time point of 24.04.16). The electronic device (101) can arrange and display past processing forms (1111-1120) of virtual objects corresponding to past time series information (1121-1130). According to one embodiment, when a specific time point is selected from the past time series information (1121-1130), the processor (120) can visually highlight and display the virtual object at the selected specific time point.

[0158] Referring to FIG. 11A, the electronic device (101) may provide a 360 view mode (1131) or a flat view mode (1132) when displaying a past processing form (1111-1120) of a virtual object. For example, when the 360 ​​view mode (1131) is selected, a plurality of virtual objects may be displayed in a form surrounding the user, as in the first form (1140) of FIG. 11B. When the flat view mode (1132) is selected, a plurality of virtual objects may be displayed in a form in which they are sequentially arranged within a space that the user gazes at, as in the second form (1150) of FIG. 11B.

[0159] Referring to FIG. 11b, a first form (1140) according to a 360 view mode (1131) may be a form in which a plurality of virtual objects (1141, 1142, 1143, … 1149) are arranged in time sequence. For example, a user wearing an electronic device (101) may check a plurality of virtual objects (1141, 1142, 1143, … 1149) while changing a gaze direction. Referring to FIG. 11b, a second form (1150) according to a flat view mode (1132) may be a form in which a plurality of virtual objects (1151, 1152, 1153, … 1159) are arranged in time sequence within a specific space (e.g., a space included in the user's field of view). For example, a user wearing an electronic device (101) can check multiple virtual objects (1151, 1152, 1153, … 1159) at a glance.

[0160] FIG. 12a is an exemplary diagram illustrating a task generated from an external electronic device as a task based on a virtual environment interface according to one embodiment of the present disclosure. FIG. 12b is an exemplary diagram illustrating a virtual environment interface including objects generated based on a viewpoint and space according to one embodiment of the present disclosure.

[0161] The electronic device of FIGS. 12A and 12B may be at least partially similar to the XR device (201) of FIG. 2 and the electronic device (101) of FIG. 3, or may further include other embodiments of the electronic device (101). The operations in FIGS. 12A and 12B may be understood to be performed by a processor of the electronic device (101) (e.g., the processor (120) of FIG. 3, a processing circuit, at least one processor).

[0162] Referring to FIGS. 12a and 12b, the processor (120) of the electronic device (101) can display a virtual environment area implemented based on a virtual environment program through a display module (160) (e.g., the display module (160) of FIG. 3).

[0163] Referring to FIG. 12A, when displaying a first virtual environment interface (1201), the electronic device (101) may obtain information (1212) related to a task processed in an external electronic device (1211), and generate a virtual object (1221) based on the obtained information (1212). For example, the electronic device (101) may be in a state of communication connection with an external electronic device (1211) through a communication module (e.g., the communication module (190) of FIG. 3).

[0164] According to one embodiment, the electronic device (101) can recognize a task being displayed on a display of an external electronic device (1211), obtain information (1212) related to the recognized task from the external electronic device (1211), generate a virtual object (1221) based on the obtained information (1212), and display a first virtual screen interface (1201) including the generated virtual object (1221). The electronic device (101) can implement the task being processed by the external electronic device (1211) in virtual reality and provide it to the user.

[0165] Referring to FIG. 12B, when displaying a second virtual environment interface (1202), the electronic device (101) can obtain information (1233) related to a task processed in an external electronic device (1231) and display the obtained information (1233) in chronological order. Referring to FIG. 12B, the electronic device (101) can implement a virtual object (1232) based on a task processed in an external electronic device (1231) and display a second virtual environment interface (1202) including the virtual object (1232).

[0166] According to one embodiment, the electronic device (101) may display a first task classified by time (point in time) or a second task classified by space (location) when loading a specific task that has been processed in the past, and / or may integrate the first task and the second task and display them together in a single interface. The electronic device (101) may display a plurality of past tasks in a classified manner within a virtual environment interface. The electronic device (101) may implement a task situation at a specific point in time or in a specific space for tasks that the user has processed in the past. The user may continue to process a specific task at a specific point in time or in a specific space, and may more efficiently process the task on the virtual environment interface. User convenience in an augmented reality situation may be improved.

[0167] A method for processing data in a head-wearable electronic device (101) by a user according to one embodiment may include, in response to an execution command of a virtual environment program, an operation of checking a usage record based on the virtual environment program, an operation of loading event information and information on a virtual object included in the usage record, an operation of generating a virtual environment interface in which, based on the event information and the information on the virtual object, a time interface in which results corresponding to each time are displayed separately and a space interface in which results corresponding to each space are displayed separately and at least partially together are provided, among results performed on the virtual object in the electronic device (101), and an operation of visually outputting the generated virtual environment interface.

[0168] A method according to one embodiment may further include an operation of checking a background image captured using a camera (180) of the electronic device (101), an operation of generating, based on the virtual environment program, at least one virtual object positioned closer to the user than the background image within a field of view of the user, an operation of generating a data block including the background image and the at least one virtual object, and an operation of storing the generated data block as the usage record in the memory (130).

[0169] A method according to one embodiment may further include an operation of communicating with an external server through a communication module (190) of the electronic device (101), an operation of transmitting the generated data block to the external server, and an operation of storing the generated data block based on the external server.

[0170] According to one embodiment, the event information includes at least one of location information of the electronic device (101), task information for the virtual environment program, time series information when performing a task of the virtual environment program, or spatial information based on a background image, and the spatial information may include location information indicating characteristics of the background image based on at least one object included in the background image.

[0171] The method according to one embodiment may further include an operation of determining one display mode from among a spatiotemporal integration mode, a time-based display mode, and a space-based display mode, based on a spatiotemporal mode setting icon included in the virtual environment interface.

[0172] A method according to one embodiment may further include an operation of generating at least one time interface divided by time when the time-based display mode is determined based on the spatiotemporal mode setting icon, and an operation of arranging the results performed according to the at least one time interface in order of time elapse.

[0173] The method according to one embodiment may further include an operation of executing a history mode that shows changes in the selected object over time when an object included in the virtual environment interface is selected in the time-based display mode, and an operation of outputting the object transformed over time in the history mode.

[0174] According to one embodiment, the method may further include, when the space-specific display mode is determined based on the space-time mode setting icon, an operation of generating at least one space interface divided by space, and an operation of arranging the performed results corresponding to a location according to the at least one space interface.

[0175] According to one embodiment, a non-transitory computer-readable storage medium (or, a computer program product) storing one or more programs for performing a method of processing data in an electronic device (101) wearable on a user's head may be described. According to one embodiment, the one or more programs may include instructions that, when executed by a processor (120) of the electronic device (101), perform, in response to an execution command of a virtual environment program, an operation of checking a usage record based on the virtual environment program, an operation of loading event information and information on a virtual object included in the usage record, an operation of generating a virtual environment interface in which, based on the event information and the information on the virtual object, results corresponding to each time are displayed separately and a space interface corresponding to each space are displayed separately and at least partially together, among results performed on the virtual object in the electronic device (101), and an operation of visually outputting the generated virtual environment interface.

[0176] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0177] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0178] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0179] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0180] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0181] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101) wearable on the head by a user, housing; A display (160) supported on the housing and outputting visual information; A communication module (190) including a communication circuit; a processor (120) including a processing circuit; and A memory (130) for storing instructions; When the above instructions are individually or collectively executed by the processor (120), the electronic device (101) causes: In response to the execution command of the virtual environment program, check the usage history based on the virtual environment program, Loading event information and virtual object information included in the above usage records, Based on the above event information and information about the virtual object, a virtual environment interface is created in which a time interface in which results corresponding to each time can be displayed separately and in each space and a space interface in which results corresponding to each space can be displayed separately and in each space are at least partially provided together with the results performed on the virtual object in the electronic device (101). An electronic device that visually outputs the generated virtual environment interface through the display (160).

2. In paragraph 1, Further comprising a camera (180) for photographing the surroundings of the electronic device; When the above instructions are individually or collectively executed by the processor (120), the electronic device (101) causes: Check the background image captured using the above camera (180), Based on the above virtual environment program, at least one virtual object is generated that is positioned closer to the user than the background image within a field of view of the user, Generating a data block including the background image and at least one virtual object, An electronic device that stores the above-mentioned generated data block in the memory (130) as the above-mentioned usage record.

3. In paragraph 2, When the above instructions are individually or collectively executed by the processor (120), the electronic device (101) causes: Connecting to an external server through the above communication module (190), Transmit the above generated data block to the external server, An electronic device that stores the generated data block based on the external server.

4. In paragraph 1, An electronic device in which the event information includes at least one of location information of the electronic device (101), task information for the virtual environment program, time series information when performing a task of the virtual environment program, or spatial information based on a background image.

5. In paragraph 4, An electronic device including location information representing characteristics of the background image based on at least one object included in the background image.

6. In paragraph 1, An electronic device in which information about the virtual object includes at least one of coordinate information about the location of at least one object generated based on the virtual environment program, path information about the movement of the at least one object, or crawling information related to the at least one object.

7. In paragraph 1, When the above instructions are individually or collectively executed by the processor (120), the electronic device (101) causes: An electronic device that determines one display mode among a spatiotemporal integration mode, a time-based display mode, and a space-based display mode based on a spatiotemporal mode setting icon included in the above virtual environment interface.

8. In paragraph 7, When the above instructions are individually or collectively executed by the processor (120), the electronic device (101) causes: Based on the above space-time mode setting icon, when the time-based display mode is determined, at least one time interface distinguished by time is created, An electronic device that arranges the results of the above-described operations in order of time elapse according to at least one time interface.

9. In paragraph 8, When the above instructions are individually or collectively executed by the processor (120), the electronic device (101) causes: When an object included in the virtual environment interface is selected in the above time-based display mode, a history mode is executed that shows changes in the selected object over time. An electronic device that outputs the object transformed according to the passage of time in the above history mode.

10. In paragraph 7, When the above instructions are individually or collectively executed by the processor (120), the electronic device (101) causes: Based on the above space-time mode setting icon, when the space-specific display mode is determined, at least one space interface distinguished by space is created, An electronic device that arranges the results of the above-described performance in accordance with the at least one spatial interface in a location.

11. In paragraph 1, When the above instructions are individually or collectively executed by the processor (120), the electronic device (101) causes: Communicate with an external electronic device through the above communication module (190), Obtaining a program running on the external electronic device and data related to the program, An electronic device that visually outputs the program and data related to the program based on the external electronic device based on the virtual environment interface.

12. A method for processing data in an electronic device (101) wearable on the head by a user, An action to check the usage history based on the virtual environment program in response to an execution command of the virtual environment program; An action to load event information and information about virtual objects included in the above usage records; An operation of generating a virtual environment interface in which, based on the above event information and information about the virtual object, a time interface in which results corresponding to each time are displayed separately and a space interface in which results corresponding to each space are displayed separately and at least partially together are provided for the results performed on the virtual object in the electronic device (101); and A method comprising: visually outputting the generated virtual environment interface; 13. In paragraph 12, An action of checking a background image captured using a camera (180) of the above electronic device (101); An operation of generating at least one virtual object that is positioned closer to the user than the background image within a field of view of the user based on the virtual environment program; An operation of generating a data block including the background image and the at least one virtual object; and A method further comprising: an operation of storing the generated data block in memory (130) as the usage record; 14. In paragraph 12, An operation of communicating with an external server through the communication module (190) of the above electronic device (101); An operation of transmitting the generated data block to the external server; and A method further comprising: an operation of storing the generated data block based on the external server; 15. A non-transitory computer-readable storage medium storing one or more programs for performing a method of processing data in an electronic device (101) wearable on a user's head, When the above one or more programs are executed by the processor (120) of the electronic device (101), An action to check the usage history based on the virtual environment program in response to an execution command of the virtual environment program; An action to load event information and information about virtual objects included in the above usage records; An operation of generating a virtual environment interface in which, based on the above event information and information about the virtual object, a time interface in which results corresponding to each time are displayed separately and a space interface in which results corresponding to each space are displayed separately and at least partially together are provided for the results performed on the virtual object in the electronic device (101); and A computer-readable storage medium comprising commands for performing an action of visually outputting the generated virtual environment interface.

Images