Electronic device, method, and non-transitory storage medium for managing plurality of windows of virtual screen in virtual reality space

Abstract

The present disclosure relates to an electronic device, a method, and a non-transitory storage medium for managing a plurality of windows of a virtual screen in a virtual reality space. According to an embodiment, the electronic device may comprise: a display; a memory including instructions; and at least one processor including the display and a processing circuit, wherein the instructions are configured to, when executed individually or collectively by the at least one processor, cause the electronic device to control the display to display a virtual screen in a virtual reality space, and on the basis of an arrangement condition configured using user context information, control the display to display a plurality of windows of at least one application on the virtual screen without overlapping between the plurality of windows. Another embodiment is also possible.

Description

Electronic device, method and non-transitory storage medium for managing multiple windows of a virtual screen in a virtual reality space

[0001] The present disclosure relates to an electronic device, method and non-transitory storage medium for managing multiple windows of a virtual screen in a virtual reality space.

[0002] With the advancement of digital technology, electronic devices are now available in various forms, such as smartphones, tablet personal computers (PCs), and personal digital assistants (PDAs). Electronic devices are also being developed into wearable devices to enhance portability and accessibility.

[0003] Electronic devices developed to be wearable by users are being developed in the form of wearable electronic devices such as augmented reality glasses (AR glasses), video see-through (VST) devices, and head-mounted displays (HMDs) to provide virtual spaces in virtual environments. The variety of services and additional functions offered by wearable electronic devices is steadily increasing. To enhance the utility of these electronic devices and satisfy the needs of diverse users, telecommunication service providers and electronic device manufacturers are competitively developing electronic devices that offer diverse functions and differentiate themselves from other companies. Accordingly, the various functions offered by wearable electronic devices are also becoming increasingly sophisticated.

[0004] AR glasses or VST devices, worn on the user's body, can provide a realistic experience by displaying virtual images. AR glasses or VST devices can replace smartphones in various fields, such as gaming, entertainment, education, and social networking services (SNS). Through AR glasses or VST devices, users can receive realistic content and, through interaction, experience the sensation of being immersed in a virtual world.

[0005] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0006] According to one embodiment of the present disclosure, an electronic device may include a display, a memory including instructions, and at least one processor including a processing circuit.

[0007] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to display a virtual screen in a virtual reality space.

[0008] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to execute a plurality of windows of at least one application and to select a first window among the plurality of windows based on user context information.

[0009] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may be configured to control the display to cause the electronic device to arrange and display the selected first window in a designated area on the virtual screen based on an arrangement condition set using user context information, and to arrange and display other windows among the plurality of windows in the remaining area of ​​the virtual screen without overlapping based on the first window.

[0010] According to one embodiment, a method of operating in an electronic device may include controlling a display to display a virtual screen in a virtual reality space.

[0011] According to one embodiment, a method of operating in an electronic device may include executing a plurality of windows of at least one application and selecting a first window from among the plurality of windows based on user context information.

[0012] According to one embodiment, an operating method in an electronic device may include an operation of arranging and displaying the selected first window in a designated area on a virtual screen based on an arrangement condition set using user context information, and arranging and displaying other windows among the plurality of windows in a remaining area of ​​the virtual screen without overlapping based on the first window.

[0013] According to one embodiment, a non-transitory storage medium storing one or more programs may include instructions that, when executed by at least one processor of an electronic device, cause the electronic device to perform an operation of controlling the display to display a virtual screen in a virtual reality space, an operation of executing a plurality of windows of at least one application and selecting a first window among the plurality of windows based on user context information, and an operation of arranging and displaying the selected first window in a designated area on the virtual screen based on an arrangement condition set using the user context information, and an operation of displaying the plurality of windows of at least one application on the virtual screen without overlapping between the plurality of windows.

[0014] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0015] FIG. 2 is a diagram showing the configuration of a wearable electronic device according to one embodiment.

[0016] FIGS. 3A, 3B, and 3C are drawings showing the front and back of a wearable electronic device according to one embodiment.

[0017] FIG. 4 is a diagram showing the configuration of an electronic device according to one embodiment.

[0018] FIG. 5 is a diagram illustrating an example for displaying multiple windows of a virtual screen in a virtual reality space in an electronic device according to one embodiment.

[0019] FIG. 6 is a diagram illustrating an example for displaying multiple windows of a virtual screen in a virtual reality space in an electronic device according to one embodiment.

[0020] FIGS. 7A, 7B, and 7C are diagrams illustrating examples of displaying multiple windows of a virtual screen in a virtual reality space in an electronic device according to one embodiment.

[0021] FIGS. 8A and 8B are drawings illustrating examples for changing the arrangement of a plurality of windows in an electronic device according to one embodiment.

[0022] FIG. 9 is a diagram illustrating an example of arranging windows based on a user's gaze in an electronic device according to one embodiment.

[0023] FIG. 10 is a diagram illustrating an example of arranging windows based on a user's gaze in an electronic device according to one embodiment.

[0024] FIG. 11 is a diagram illustrating an example of executing multiple virtual screens in an electronic device according to one embodiment.

[0025] FIG. 12 is a drawing showing an example of an operating method in an electronic device according to one embodiment.

[0026] FIGS. 13a, 13b, and 13c are drawings illustrating examples for changing the arrangement of windows in an electronic device according to one embodiment.

[0027] FIGS. 14a, 14b, and 14c are drawings illustrating examples for changing the arrangement of windows in an electronic device according to one embodiment.

[0028] FIGS. 15A and 15B are diagrams illustrating examples for changing the arrangement of windows in an electronic device according to one embodiment.

[0029] FIG. 16 is a diagram illustrating an example for executing multiple virtual screens in an electronic device according to one embodiment.

[0030] FIG. 17 is a diagram illustrating an example for executing multiple virtual screens in an electronic device according to one embodiment.

[0031] FIGS. 18A, 18B, and 18C are diagrams illustrating examples of executing multiple virtual screens in an electronic device according to one embodiment.

[0032] FIGS. 19A, 19B, and 19C are diagrams illustrating examples for managing multiple windows in an electronic device according to one embodiment.

[0033] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0034] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components. In addition, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and conciseness. The term "user" used in the embodiments of the present disclosure may refer to a person using an electronic device or a device (e.g., an artificial intelligence electronic device) using an electronic device.

[0035] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0036] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0037] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0038] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0039] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0040] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0041] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0042] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0043] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0044] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0045] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0046] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0047] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0048] The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0049] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0050] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0051] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0052] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for realizing 1eMBB, a loss coverage (e.g., 164 dB or less) for realizing mMTC, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for realizing URLLC.

[0053] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0054] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0055] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0056] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0057] FIG. 2 is a diagram showing the configuration of a wearable electronic device according to one embodiment.

[0058] Referring to FIG. 2, a wearable electronic device (200) according to one embodiment of the present disclosure may include at least one of a light output module (211), a display member (201), and a camera module (250).

[0059] According to one embodiment of the present disclosure, the light output module (211) may include a light source capable of outputting an image, and a lens for guiding the image to the display member (201). According to one embodiment of the present disclosure, the light output module (211) may include at least one of a liquid crystal display (LCD), a digital mirror device (DMD), a liquid crystal on silicon (LCoS), an organic light emitting diode (OLED), or a micro light emitting diode (micro LED).

[0060] According to one embodiment of the present disclosure, the display member (201) may include an optical waveguide (e.g., a waveguide). According to one embodiment of the present disclosure, an output image of an optical output module (211) incident on one end of the optical waveguide may be propagated inside the optical waveguide and provided to a user. According to one embodiment of the present disclosure, the optical waveguide may include at least one diffractive element (e.g., a diffractive optical element (DOE), a holographic optical element (HOE)) or at least one reflective element (e.g., a reflective mirror). For example, the optical waveguide may guide an output image of the optical output module (211) to a user's eye by using at least one diffractive element or reflective element.

[0061] According to one embodiment of the present disclosure, the camera module (250) can capture still images and / or moving images. According to one embodiment, the camera module (250) is disposed within a lens frame and can be disposed around the display member (201).

[0062] According to one embodiment of the present disclosure, the first camera module (251) can capture and / or recognize the trajectory of the user's eye (e.g., pupil, iris) or gaze. According to one embodiment of the present disclosure, the first camera module (251) can periodically or aperiodically transmit information related to the trajectory of the user's eye or gaze (e.g., trajectory information) to a processor (e.g., processor (120) of FIG. 1).

[0063] According to one embodiment of the present disclosure, the second camera module (253) can capture an external image.

[0064] According to one embodiment of the present disclosure, the third camera module (255) can be used for hand detection and tracking, and user gesture (e.g., hand movement) recognition. According to one embodiment of the present disclosure, the third camera module (255) can be used for 3 degrees of freedom (3DoF), 6DoF head tracking, position (space, environment) recognition, and / or movement recognition. According to one embodiment of the present disclosure, the second camera module (253) can also be used for hand detection and tracking, and user gesture recognition. According to one embodiment of the present disclosure, at least one of the first camera module (251) to the third camera module (255) can be replaced with a sensor module (e.g., a LiDAR sensor). For example, the sensor module can include at least one of a vertical cavity surface emitting laser (VCSEL), an infrared sensor, and / or a photodiode.

[0065] FIGS. 3A and 3C are drawings showing the front and back of a wearable electronic device according to one embodiment.

[0066] Referring to FIGS. 3A and 3B, in one embodiment, camera modules (311, 312, 313, 314, 315, 316) and / or depth sensors (317) for obtaining information related to the surrounding environment of the wearable electronic device (300) may be arranged on the first surface (310) of the housing.

[0067] In one embodiment, the camera modules (311, 312) can acquire images related to the environment surrounding the wearable electronic device.

[0068] In one embodiment, the camera modules (313, 314, 315, 316) can acquire images while the wearable electronic device is worn by a user. The camera modules (313, 314, 315, 316) can be used for hand detection and tracking, and user gesture (e.g., hand movement) recognition. The camera modules (313, 314, 315, 316) can be used for 3DoF, 6DoF head tracking, position (spatial, environmental) recognition, and / or movement recognition. In one embodiment, the camera modules (311, 312) can also be used for hand detection and tracking, and user gesture.

[0069] In one embodiment, a depth sensor (317) may be configured to transmit a signal and receive a signal reflected from a subject, and may be used for purposes such as time of flight (TOF) to determine the distance to an object. Instead of or in addition to the depth sensor (217), the camera modules (313, 314, 315, 316) may determine the distance to an object.

[0070] According to one embodiment, camera modules (325, 326) for facial recognition and / or a display (321) (and / or a lens) may be arranged on the second side (320) of the housing.

[0071] In one embodiment, the face recognition camera modules (325, 326) adjacent to the display may be used to recognize the user's face, or may recognize and / or track the user's two eyes.

[0072] In one embodiment, the display (321) (and / or lens) may be disposed on the second side (320) of the wearable electronic device (300). In one embodiment, the wearable electronic device (300) may not include camera modules (315, 316) among the plurality of camera modules (313, 314, 315, 316). Although not illustrated in FIGS. 3A and 3B , the wearable electronic device (300) may further include at least one of the configurations illustrated in FIG. 2 .

[0073] Referring to FIG. 3C, according to one embodiment, a wearable electronic device (300) may have a form factor (e.g., a Head-mounted Display: HMD) for being worn on a user's head. The wearable electronic device (300) may further include a strap for being fixed on a body part of the user, and / or a wearing member. The wearable electronic device (300) may include a volume button (331), a ventilation hole (333), a status indicator light (335), and a power button (e.g., including a fingerprint recognition sensor) (317), and these configurations may be equally included in the wearable electronic device (300) illustrated in FIGS. 3A and 3B. While worn on the user's head, the wearable electronic device (300) may provide a user experience based on augmented reality, virtual reality, and / or extended reality (or mixed reality). The wearable electronic device (300) configured in the form of an HMD may include configurations identical or similar to the components of FIGS. 3A and 3B described above.

[0074] In the above-described FIGS. 1, 2a, 2b, 3a, 3b and 3c, the term 'module' in the electronic device (101, 200, 300) refers to a unit that processes at least one function or operation, and may be implemented by hardware, software or a combination of hardware and software. Although it is described as 'module' in the electronic device, it may be replaced with terms such as 'circuitry', 'unit', or 'device', for example.

[0075] FIG. 4 is a diagram illustrating a configuration of an electronic device according to one embodiment. In FIG. 4, terms such as "circuitry" within the electronic device refer to a unit that processes at least one function or operation, which may be implemented by hardware, software, or a combination of hardware and software. Although the term "module" is used in the electronic device, it may be replaced with terms such as "module," "unit," or "device."

[0076] Referring to FIG. 4, an electronic device (401) according to one embodiment (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, and / or the wearable electronic device (300) of FIGS. 3A and 3C) includes a processor (410) (e.g., the processor (120) of FIG. 1), a display (420) (e.g., the display module (160) of FIG. 1, the light output module (211) of FIG. 2, the display member (201), the display (321) of FIGS. 3A to 3C), a camera circuit (430) (e.g., the camera module (180) of FIG. 1, the camera module (250) of FIG. 2, the camera modules (311, 312, 313, 314, 315, 316) of FIGS. 3A to 3C), and a communication circuit (440) (e.g., the communication circuit of FIG. 1). It may be configured to include a module (190)) and a memory (450) (e.g., memory (130) of FIG. 1). Without being limited thereto, the electronic device (401) may further include other components illustrated in FIG. 1.

[0077] An electronic device (401) according to one embodiment can connect to an external electronic device (403) (e.g., a PC, a mobile device, a laptop, or an input interface (e.g., a keyboard)) using a communication method (e.g., BLE communication) designated by a communication circuit (440). An electronic device (401) according to one embodiment can be connected to at least one controller (e.g., a handler) through a designated communication method (e.g., BLE communication). Here, at least one controller can be a pointing device or an input control device, and can transmit a user interaction related to a virtual object displayed in a virtual reality space to the electronic device (401). However, the input interface that transmits the user interaction to the electronic device (401) can be replaced with at least one of a keyboard, a mouse, an electronic pen, a smartphone, or an artificial intelligence (AI) device.

[0078] FIG. 5 is a diagram illustrating an example for displaying a plurality of windows of a virtual screen in a virtual reality space in an electronic device according to one embodiment, FIG. 6 is a diagram illustrating an example for displaying a plurality of windows of a virtual screen in a virtual reality space in an electronic device according to one embodiment, and FIGS. 7a, 7b, and 7c are diagrams illustrating examples for displaying a plurality of windows of a virtual screen in a virtual reality space in an electronic device according to one embodiment.

[0079] Referring to FIGS. 4, 5, 6, 7a, 7b, and 7c, according to one embodiment, the processor (410) of the electronic device (401) may control the display (420) to display a virtual screen (520) on a virtual reality space (510) in a virtual reality (e.g., XR) environment. The virtual reality space (510) may display a virtual screen (520) having a larger size than the actual environment. According to one embodiment, the processor (410) may display the virtual screen (520) when a user wears the electronic device (401) on a part of the body (e.g., the head), and a designated interaction (e.g., an interaction for a user's gesture or menu input) occurs for a user interaction target (e.g., an external input device) (601), an interaction for the occurrence of a content or notification event (603), or an interaction for the selection of an object (605) displayed in the virtual reality space (510). According to one embodiment, the processor (410) may control the display (420) to display one or more windows (e.g., execution screens) (530) for at least one application running on the electronic device (401) or the external electronic device (403) on the virtual screen (520) (or, if the virtual screen (520) is not displayed, in an area of ​​the virtual reality space (510) where the user's gaze is directed). According to one embodiment, when a plurality of applications are running, the processor (410) may control the display (420) to display one or more windows for each of the plurality of applications on the virtual screen (520) or in an area of ​​the virtual reality space. According to one embodiment, the processor (410) may display the plurality of applications in a mapped or differentiated manner. For example, the processor (410) may control the display (420) to display one or more windows of an application currently being used by the user on the virtual screen (520) and to display the remaining running applications in an area of ​​the virtual reality space.For example, the processor (410) may control the display (420) to display one or more windows of an application currently being used by the user at a location where the user's gaze is directed on the virtual reality space (510) or the virtual screen (520), and to display the remaining applications being executed at another location where the user's gaze is not directed on the virtual reality space (510) or the virtual screen (520). For example, the processor (410) may control the display (420) to arrange one or more windows mapped or separated for each of a plurality of applications in the Z direction and display them in an area corresponding to the user's gaze range (e.g., an area of ​​the virtual screen (520) or an area on the virtual reality space (510).

[0080] According to one embodiment, when displaying a plurality of windows (530) on a virtual screen (520), the processor (401) may set an arrangement condition so as to display the plurality of windows (530) without overlapping (e.g., without interference). Here, the arrangement condition may be a condition for setting the size, arrangement position, or arrangement interval of the plurality of windows (530) based on user context information or execution status information of the virtual screen (520) stored in the memory (450). According to one embodiment, the processor (410) may adjust the arrangement position and / or the arrangement interval so that all of the plurality of windows (530) displayed on the virtual screen do not overlap based on the arrangement condition. According to one embodiment, the processor (410) may control the display (420) so as to display the windows (530) displayed within the user's field of view on the virtual screen by adjusting the arrangement position and / or the arrangement interval so that they do not overlap based on the arrangement condition. Here, some of the windows outside the field of view may be displayed overlappingly, and when the user turns his or her head and the user's field of view changes, the processor (410) may control the display (420) to change the arrangement of the overlapping windows (530) displayed in the changed field of view so that they are displayed without overlapping. According to one embodiment, the processor (410) may arrange some of the windows (530) that are far from the user's gaze or located in a designated area of ​​the virtual screen or far from the user's field of view so as to overlap. According to one embodiment, the processor (410) may arrange the windows (530) based on an arrangement condition to the left / right or up / down with respect to a designated area of ​​the virtual screen, and the windows (530) arranged in the edge area of ​​the virtual screen (520) may be arranged so as to overlap.According to one embodiment, when some of a plurality of windows (530) are arranged to overlap on a virtual screen, the processor (410) may erase some of the overlapping windows and not display them on the virtual screen (520), and instead display objects corresponding to the erased windows on the virtual reality space (510). Here, the designated area may be a central area of ​​the virtual screen (520) designated based on the user's field of vision when the user wearing the electronic device (401) looks straight ahead.

[0081] According to one embodiment, when the processor (410) displays the virtual screen (520), the processor (410) may change (or adjust) at least one of the size, distance, or type of the virtual screen (520). According to one embodiment, the processor (410) may allow the user to directly adjust the size of the virtual screen (e.g., expand or reduce) based on a portion (e.g., a corner) of the virtual screen (520), as shown in 710 of FIG. 7A. The processor (410) may adjust the size of the virtual screen (520) by selecting one of the magnifications (e.g., 1.5X, 2.0X, and 3.0X) specified through a user interface (UI) (721), as shown in 720 of FIG. 7A. The processor (410) may automatically adjust the size of the virtual screen (520) in proportion to the distance (701) between the virtual screen and the user, as shown in 730 of FIG. 7A. For example, the processor (410) may reduce the size of the virtual screen (520) when the user's distance (701) is less than (e.g., closer to) a specified distance from the virtual screen. For example, the processor (410) may enlarge the size of the virtual screen (520) when the user's distance (701) is greater than (e.g., farther from) a specified distance from the virtual screen. The processor (410) may automatically adjust the size of the virtual screen to one of the specified magnifications based on the size or type of one or more windows that were last executed before displaying the virtual screen. For example, when the number of windows that were previously executed is one, the processor (410) may display the one window in a specified area (e.g., the center area) of the virtual screen without adjusting the size of the virtual screen. For example, the processor (410) may automatically adjust the size of the virtual screen displayed at a specified size to one of the specified magnifications based on the total size of one or more windows that were last executed before.

[0082] According to one embodiment, when displaying a virtual screen (520), the processor (410) may display the virtual screen (520) by changing the distance (703) between the user and the virtual screen, as shown in FIG. 7B. For example, the processor (410) may adjust the distance (703) based on the Z-axis of the virtual reality space to bring the virtual screen closer to or further away from the user.

[0083] According to one embodiment, the processor (410) can change the type (721, 723, or 725) of the virtual screen based on the type of the external electronic device (403) to be linked, as shown in FIG. 7C. If the external electronic device (403) to be linked is, for example, a mobile phone, the processor (410) can change the virtual screen to the first type (721), display the virtual screen in a size set for the first type (721), adjust the size of a first window among a plurality of windows (530) and display it on the virtual screen, delete the remaining windows, and then display a second window among the remaining windows on the virtual screen according to a request for arrangement change or movement, and delete the first window. When the external electronic device (403) to be linked is, for example, a tablet, the processor (410) can change to the second type (723), display the virtual screen in a size set for the second type (723), display a first window among a plurality of windows (530) on the virtual screen, erase the remaining windows, and then display a second window among the remaining windows on the virtual screen according to a request for arrangement change or movement, and erase the first window. When the external electronic device (403) to be linked is, for example, a PC, the processor (410) can change to the third type (725), display the virtual screen in a size set for the third type (725), and display a plurality of windows (530) on the virtual screen.

[0084] According to one embodiment, the processor (410) can control the display (420) to display an object (711) for changing the type in an adjacent area of ​​the virtual screen (520), and when the object (711) is selected, to display an extended object (713) including contextual menus. When a user selects one of the contextual menus displayed in the extended object (713), the processor (410) can change the type of the virtual screen (520) in response to the selected menu.

[0085] FIGS. 8A and 8B are drawings illustrating examples for changing the arrangement of a plurality of windows in an electronic device according to one embodiment.

[0086] Referring to FIG. 8A, according to one embodiment, the processor (410) may display one or more windows (531, 532, 533) of an application running on the virtual screen (520) based on the horizontal width of the virtual screen (520) based on user context information. Here, the user context information may include at least one of user gaze information, distance information from the virtual screen, user movement information, user input information (e.g., voice, text, and / or gesture input), frequently used pattern information (e.g., window arrangement pattern or virtual screen arrangement pattern), or information on an external electronic device used by the user (e.g., an external electronic device connected to the electronic device). For example, when the processor (410) displays a virtual screen, if a new first window (531) is executed (810) in a state where no window is displayed, the processor (410) may control the display (420) to display the first window (531) in a designated area (e.g., a central area based on a center line). When a second window (532) is executed while a first window (531) is displayed (820), the processor (410) can control the display (420) to change the arrangement between the first window (531) and the second window (532) without interference (e.g., so as not to overlap), move the first window (531) to another area (e.g., toward the left) outside the central area according to the changed arrangement, and display the second window (532) in the central area. When a new third window (533) is executed again (830) while the first window (531) and the second window (532) are displayed, the processor (410) can control the display (420) to change the arrangement between the first window (531), the second window (532), and the third window (533) without interference (e.g., so as not to overlap), move the second window (532) to another area (e.g., toward the right) outside the central area according to the changed arrangement, and display the third window (533) in the central area.When a new fourth window (534) is executed (840) while the first window (531), the second window (532), and the third window (533) are displayed, the processor (410) checks whether there is an empty area where the displayed plurality of windows (531, 532, 533) and the newly executed fourth window (534) can be displayed without interference (e.g., not overlapping) based on the size of the virtual screen (e.g., horizontally), and if there is an empty area, the display (420) can be controlled to change the arrangement between the plurality of windows (531, 532, 533) without interference in the empty area, move some of the plurality of windows (531, 532, 533) to the empty area (e.g., in the direction of the empty area) and display them, and display the fourth window (534) in the central area. If there is no blank area where the fourth window (534) can be displayed without interference (e.g., without overlapping), the processor (410) may display the fourth window (534) based on an offset between the fourth window (534) and the third window (533) so as not to obscure (e.g., without overlapping) the toolbar (801) of the third window (533) displayed in the central area. For example, if there is no blank area horizontally and there is a blank area of ​​a size corresponding to or similar to the size of the fourth window (534) vertically, the processor (410) may control the display (420) to move the third window (533) displayed in the central area to the identified blank area so that the third window (533) is arranged in the blank area identified vertically. If the blank area identified based on the vertical axis is larger than the size of the fourth window (534) by a threshold value, the processor (410) can control the display (420) to move the displayed plurality of windows (531, 532, 533) in the opposite direction of the blank area to secure a size of the blank area corresponding to the size of the fourth window (534) and display the fourth window (534) in the blank area.

[0087] Referring to FIG. 8B, according to one embodiment, the processor (410) can automatically arrange a plurality of windows (531, 532, 533, 534, 535) running on a virtual screen (520). According to one embodiment, when the processor (410) selects a first context menu (e.g., a button, a graphic, or a function) (803) included in an extended object (713), the processor (410) can automatically arrange the windows (534, 533, 535) in a most recently executed order based on the width of the virtual screen so that there is no interference between the windows (534, 533, 535), and can eliminate (e.g., provide in a minimized state) the remaining windows (531, 532) that are not arranged (e.g., cause interference) without terminating execution. Here, the arrangement order of the windows (534, 533, 535) can be arranged in the order of most recently executed based on the central area (e.g., arranged in the order of central area, left area, right area). The processor (410) can display an object representing the windows (531, 532) by including it in a menu screen (803) displayed in a part of the virtual screen or an adjacent area of ​​the virtual screen so that the deleted windows (531, 532) can be displayed again on the virtual screen according to a specified situation or event. When the user changes the gaze (e.g., field of view (FoV)) to a specific window (e.g., window (app5) (535)) on the automatically arranged virtual screen and maintains it for a specified period of time, the processor (410) can change the arrangement position to the central area (e.g., the area between window (533) and window (534)) to change the arrangement position or arrangement order of the windows (534, 533, 535). The processor (410) can display windows (534, 533, 535) according to the changed array position or array order.

[0088] FIG. 9 is a diagram illustrating an example of arranging windows based on a user's gaze in an electronic device according to one embodiment. FIG. 10 is a diagram illustrating an example of arranging windows based on a user's gaze in an electronic device according to one embodiment.

[0089] Referring to FIGS. 9 and 10 , according to one embodiment, the processor (410) may track the gaze (e.g., field of view (FoV)) (901) of a user wearing the electronic device (401) while displaying a plurality of windows (530) on a virtual screen (520), and determine the gaze position in the tracked gaze (901). When the user's gaze (901) gazes at the central area (521) of the virtual screen with respect to the center line (903) and then changes the gaze position to another position and gazes for a specified period of time, the processor (410) may change the arrangement position (e.g., arrangement order) or arrangement interval of the windows displayed at the changed gaze position.

[0090] According to one embodiment, when the position of the virtual screen (520) toward which the electronic device (401) is directed is changed (e.g., left / right / up / down) to a position outside the central area (521) (e.g., the user turns his / her head to the left or raises it upward), and the user's gaze (901) is outside the central area (521), as shown in (a) and (b) of FIG. 10, the processor (410) may identify that a change in the arrangement of windows is required, identify a gaze position at which the current user's gaze (901) is maintained for a specified period of time, and identify a window (window A) displayed at the identified gaze position. The processor (410) may change the arrangement position to display the identified window (window A) in the central area (521), and change the arrangement position to display the window (window A) displayed in the central area (521) in another area. At this time, if there are other windows on the virtual screen (520) and the arrangement position of the other windows needs to be changed, the processor (410) can change the arrangement position of the other windows according to the arrangement conditions.

[0091] According to one embodiment, the processor (410) may determine, as shown in (c) of FIG. 10, whether the position of the virtual screen (520) facing the electronic device (401) is the central area (521), and if the user's gaze (901) moves out of the central area (521), whether the window displayed at the gaze position where the user's gaze (901) is currently maintained for a specified period of time is out of the central area (521). If the window (window A) is in a position out of the central area (521), the processor (410) may identify that the arrangement of the windows needs to be changed, and may change the arrangement position so that it is displayed in the central area (521). If the window (window A) is included in the central area (521), the arrangement position of the window (window A) may not be changed. According to one embodiment, the processor (410) can identify that there is no need to change the arrangement positions of windows if the position of the virtual screen (520) toward which the electronic device (401) is directed is changed up / down / left / right, as shown in (d) of FIG. 10, but the user's line of sight (901) does not leave the central area (521).

[0092] FIG. 11 is a diagram illustrating an example of executing multiple virtual screens in an electronic device according to one embodiment.

[0093] Referring to FIG. 11, a processor (410) of an electronic device (401) according to an embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (202) of FIG. 2, the wearable electronic device (300) of FIGS. 3A to 3C, or the electronic device (401) of FIG. 4) may control a display (420) to display a plurality of virtual screens (520) executed on a virtual reality space (510) when executing a plurality of virtual screens. The processor (410) may display a plurality of virtual screens (520) within a range of a user's field of view (FoV), and may arrange further executed virtual screens in another space area outside the range of the user's field of view. Here, the field of view may be a range of a virtual reality space in which a user wearing the electronic device (401) can move his / her head up / down or left / right without inconvenience based on the user's field of view. The electronic device (401) can display a virtual screen (520) used by a user at a central location of a virtual reality space (510) (e.g., a location corresponding to a field of view that the user looks straight ahead). The processor (410) can control the display (420) to display a plurality of windows (530) running on the virtual screen (520). According to one embodiment, the processor (410) can arrange the plurality of virtual screens (520) on the virtual reality space (510) based on at least one of a specified arrangement criterion, a recent execution order, or user context information.

[0094] According to one embodiment, the processor (410) may control the display (420) to display a user interface (1111) including objects (A, B, C, …) representing each of a plurality of virtual screens (520) on the virtual reality space (510). The processor (410) may control the display to move input objects (A, B, C, …) (e.g., menus, buttons, graphic objects, or input interfaces) (1111) representing each of the plurality of virtual screens (520) to display a selected virtual screen at a central position viewed by the user.

[0095] In the description of the above-described drawings 5 ​​to 11, a virtual screen (520) is displayed on a virtual reality space, but the present invention is not limited thereto, and in another embodiment, a virtual screen (e.g., an object or image corresponding to the virtual screen) may not be displayed (e.g., provided), but a plurality of windows (e.g., objects) may be displayed on a virtual reality space.

[0096] According to an embodiment, the processor (410) of an electronic device (401) (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, and / or the wearable electronic device (300) of FIGS. 3A and 3C) may, when virtual windows (530) are displayed on a virtual screen (520) (or a virtual reality space (510)), arrange or change the arrangement of the virtual windows (530) by a specific button (e.g., a menu, a function, a graphic element, or an object) provided in the virtual reality space (510) or by a specific input by an external electronic device, without selecting at least one window or an object corresponding to at least one window among the virtual windows (530). When a specific button or a specific input is received, the electronic device (401) may arrange or change the arrangement of the virtual windows (530) under different arrangement conditions than when a window or an object corresponding to a window is selected. For example, when a specific button or a specific input is received, the processor (410) may arrange virtual windows (530) with a gap between the virtual windows (530) in the Z-axis direction based on the user's line of sight. Here, the gap between the virtual windows (530) may be set to a constant specified distance value or may be set to different distance values. For example, the processor (410) may set the gap from a wide gap to a narrow gap or from a narrow gap to a wide gap starting from the window arranged at the front based on the user's line of sight. For example, the processor (410) may change a specific window (e.g., a window in which a notification or alert has occurred) to a larger distance value than the distance value set for the remaining windows to arrange the gap so that it becomes wider. The present invention is not limited thereto, and the gap may be set in various ways.According to one embodiment, when at least one window among virtual windows (530) arranged in the Z-axis direction based on the line of sight is selected, the processor (410) can adjust (e.g., reflect or set) the Z-direction spacing of the remaining windows based on a Z-axis distance value set for the at least one selected window (e.g., a preset distance value or a distance value adjusted by a user interaction).

[0097] An electronic device according to an embodiment (e.g., an electronic device (101) of FIG. 1, an electronic device (202) of FIG. 2, a wearable electronic device (300) of FIGS. 3A to 3C, or an electronic device (401) of FIG. 4) may implement a software module (e.g., a program (140) of FIG. 1) for managing a plurality of windows running on a virtual screen. A memory of the electronic device (e.g., a memory (130) of FIG. 1 and a memory (450) of FIG. 4) may store commands (e.g., instructions) to implement the software module illustrated in FIG. 2. At least one processor may execute commands stored in the memory to implement the software module, and control hardware (e.g., a sensor module (176), a camera module (180), or a communication module (190) of FIG. 1) associated with a function of the software module.

[0098] According to one embodiment, a software module of an electronic device may be configured to include a kernel (or HAL), a framework (e.g., middleware (144) of FIG. 1), and an application (e.g., application (146) of FIG. 1). At least a portion of the software module may be preloaded on the electronic device or may be downloadable from a server (e.g., server (108)). The components and names of the components of the software module according to the embodiment may vary depending on the type of operating system. According to one embodiment, at least a portion of the software module may be implemented as software, firmware, hardware, or a combination of at least two or more thereof. At least a portion of the software module may be implemented (e.g., executed) by, for example, a processor (e.g., an AP). At least a portion of the software module may include, for example, a module, a program, a routine, a set of instructions, or a process for performing at least one function.

[0099] As such, in one embodiment, the main components of the electronic device have been described through the electronic device described above (e.g., the electronic device (101) of FIG. 1, the electronic device (202) of FIG. 2, the wearable electronic device (300) of FIGS. 3A to 3C, or the electronic device (401) of FIG. 4). However, in various embodiments, not all of the components illustrated through FIGS. 1 to 4 are essential components, and the electronic device may be implemented with more components than the illustrated components, or may be implemented with fewer components. In addition, the positions of the main components of the electronic device described above through FIGS. 1 to 4 may be changed according to various embodiments.

[0100] According to one embodiment, an electronic device (e.g., an electronic device (101) of FIG. 1, an electronic device (202) of FIG. 2, a wearable electronic device (300) of FIGS. 3A to 3C, or an electronic device (401) of FIG. 4) may include a display (e.g., a display module (160) of FIG. 1, a display member (201) of FIG. 2, a display (321) of FIG. 3B, or a display (420) of FIG. 4), a memory including instructions (e.g., a memory (130) of FIG. 1 or a memory (450) of FIG. 4), and at least one processor including a processing circuit (e.g., a processor (120) of FIG. 1 or a processor (410) of FIG. 4).

[0101] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to control the display to display a virtual screen in a virtual reality space.

[0102] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to execute a plurality of windows of at least one application and to select a first window among the plurality of windows based on user context information.

[0103] According to one embodiment, the instructions, when executed by the at least one processor, may be configured to control the display to cause the electronic device to arrange and display the selected first window in a designated area of ​​the virtual screen based on an arrangement condition set using user context information, and to arrange and display other windows among the plurality of windows in the remaining area of ​​the virtual screen without overlapping based on the first window.

[0104] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may be configured to cause the electronic device to change at least one of a size of the virtual screen, a distance between the user and the virtual screen, or a type of the virtual screen.

[0105] According to one embodiment, the arrangement condition may be a condition for setting the size, arrangement position, or arrangement interval of the plurality of windows based on the user context information or the execution status information of the virtual screen stored in the memory.

[0106] According to one embodiment, the user context information may include at least one of gaze information, distance information, movement information, user input information, frequently used pattern information, or information of a connected external electronic device.

[0107] According to one embodiment, the designated area may be a central area of ​​the virtual screen designated based on the user's field of view when the user wearing the electronic device looks straight ahead.

[0108] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may be configured to cause the electronic device to, while displaying the plurality of windows on the virtual screen, arrange a new window of the new application in a central area of ​​the virtual screen based on which new application is executed, and change the arrangement of the plurality of windows to display the plurality of windows without interference with the new window.

[0109] According to one embodiment, the instructions, when executed by the at least one processor, may be configured to cause the electronic device to track a user's gaze using at least one camera circuit of the electronic device (e.g., the camera module (251) of FIG. 2 or the camera circuit (430) of FIG. 4), identify a gaze position on the virtual screen, and change an arrangement position of a first window displayed at the gaze position among the plurality of windows to the central area based on whether the gaze position is outside the central area of ​​the virtual screen.

[0110] In one embodiment, the instructions, when individually or collectively executed by the at least one processor, may be configured to cause the electronic device to control the display to display an object to move the first window displayed at the gaze location to the central area based on the gaze being maintained outside the central area and in another area for a specified period of time.

[0111] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may be configured to cause the electronic device to, based on the movement of the gaze after selecting one of the plurality of windows, display an area of ​​the virtual screen corresponding to the moved gaze position as a movable area of ​​the selected window, change the arrangement positions of the plurality of windows to move the selected window to the movable area, and control the display to display the selected window in the central area based on the changed arrangement positions of the plurality of windows.

[0112] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to store execution status information of the virtual screen in the memory in response to a request to terminate the virtual screen.

[0113] According to one embodiment, the execution status information may include at least one of the size, distance, or type of the virtual screen and at least one of the sizes, arrangement positions, arrangement intervals of the plurality of windows or identification information of applications corresponding to the plurality of windows.

[0114] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may be configured to cause the electronic device to, in response to a request to re-execute the virtual screen, obtain the execution state information from the memory, and, based on the execution state information, control the display to re-execute the virtual screen and display the plurality of windows.

[0115] According to one embodiment, the instructions, when individually or collectively executed by the at least one processor, may be configured to cause the electronic device to: control the display to display virtual screen objects representing each of the plurality of virtual screens in the virtual reality space based on displaying a plurality of virtual screens in the virtual reality space; control the display to display a first virtual screen displayed at a gaze position of the user among the plurality of virtual screens at a central position of the virtual reality space; and change the arrangement of the plurality of virtual screens so as to display the plurality of virtual screens without overlapping when displaying the first virtual screen at the central position.

[0116] Figure 12 is a diagram illustrating an example of an operating method in an electronic device according to one embodiment. In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0117] Referring to FIG. 12, in operation 1201, an electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (202) of FIG. 2, the wearable electronic device (300) of FIGS. 3A to 3C, or the electronic device (401) of FIG. 4) can identify a wearing state of the electronic device by a user.

[0118] In operation 1203, the electronic device may display a virtual screen in a virtual reality space in a virtual reality environment based on identifying that the user is wearing the electronic device. The electronic device may display the virtual screen based on previously set virtual screen execution information (e.g., size, distance, or type) or default virtual screen execution information. The electronic device may change the size, distance, or type of the virtual screen based on direct manipulation by the user or user context information. Here, the user context information may include at least one of the user's gaze information, distance information from the virtual screen, user movement information, user input information (e.g., voice, text, and / or gesture input), frequently used pattern information (e.g., window arrangement pattern or virtual screen arrangement pattern), or information on an external electronic device used by the user (e.g., an external electronic device connected to the electronic device). Without being limited thereto, the electronic device according to one embodiment may display a plurality of windows (e.g., objects) in the virtual reality space without displaying (e.g., providing) a virtual screen (e.g., an object or image corresponding to the virtual screen), in which case operation 1205 may be omitted.

[0119] In operation 1205, the electronic device may display multiple windows of at least one application on the virtual screen without interference (e.g., without overlapping) based on arrangement conditions set according to user context information. The electronic device may display the multiple windows arranged based on the arrangement conditions on the virtual screen with a center line as the reference. According to one embodiment, the electronic device may display multiple recently executed windows with the previous arrangement conditions. According to one embodiment, when displaying multiple windows as at least one new window is executed while there are windows displayed on the virtual screen, the electronic device may arrange and display the multiple windows according to the arrangement conditions. According to one embodiment, when displaying one window on the virtual screen, the electronic device may display one window in the center area without based on the arrangement conditions. Here, the arrangement conditions may be set according to the usage context (e.g., customized). The arrangement conditions may be conditions for setting the size, arrangement positions, or arrangement intervals of the multiple windows based on user context information or execution status information of the virtual screen stored in a memory (e.g., memory 130 of FIG. 1, memory 450 of FIG. 4).

[0120] In operation 1207, the electronic device can determine whether the virtual screen has finished executing. If the determination indicates that the virtual screen has finished executing, the electronic device can perform operation 1209. Otherwise, the electronic device can perform operation 1205.

[0121] In operation 1209, the electronic device may store the execution status information of the current virtual screen in memory based on the virtual screen being terminated, and terminate the operation. Here, the execution status information of the virtual screen may include at least one of the size, distance, or type of the virtual screen, and at least one of the sizes, arrangement positions, arrangement intervals of the plurality of windows, or identification information of applications corresponding to the plurality of windows.

[0122] In the operating method of FIG. 12 described above, an operation for checking whether a change in the arrangement of multiple windows is required may be further performed. For example, the electronic device may check whether a change in the arrangement of multiple windows is required when performing operation 1205, or before operation 1207 after performing operation 1205.

[0123] According to one embodiment, the electronic device can determine whether a state requiring a change in the arrangement of a plurality of windows is present based on user context information. The state requiring a change in the arrangement may include at least one of a state in which the user's gaze deviates from a central area of ​​a virtual screen and gazes at another area for a specified period of time without movement of the electronic device (e.g., without turning the head), a state in which an object for moving a window displayed on the virtual screen is selected, a state in which one of the plurality of windows is selected by a handler or a user gesture, a state in which a new window is executed, or a state in which some of the plurality of windows are terminated.

[0124] According to one embodiment, when the electronic device is in a state where it is necessary to change the arrangement of a plurality of windows, the electronic device identifies a window to be displayed in a central area, sets the arrangement position to the central area so that the identified window is displayed in the central area, switches another window currently displayed in the central area to the arrangement position of the identified window, or changes the arrangement positions of other windows other than the identified windows according to an arrangement criterion.

[0125] FIGS. 13a, 13b, and 13c are drawings illustrating examples for changing the arrangement of windows in an electronic device according to one embodiment.

[0126] Referring to FIGS. 13A, 13B, and 13C, an electronic device (401) according to an embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (202) of FIG. 2, the wearable electronic device (300) of FIGS. 3A to 3C, or the electronic device (401) of FIG. 4) displays a plurality of windows (531, 532, 533) on a virtual screen (520) by arranging them without interference (e.g., without overlapping) with each other, as in the operation 1205 of FIG. 12 described above, when the user's gaze (e.g., field of view (FoV)) (901) is maintained at a gaze position (1311) outside a central area set based on a center line (1301) for a specified period of time, the electronic device (401) identifies a second window (532) at the gaze position (1311) where the gaze (901) is maintained, and arranges the identified second window (532). You can change the location to the central area.

[0127] According to one embodiment, when the user's gaze (901) is maintained at a location (1311) outside the central area for a specified period of time, as shown in FIG. 13A, the electronic device (401) may display an object (e.g., a menu, a graphic element, or a function) (1313) for automatically arranging a window to a specified area (e.g., a central area) at the gaze location (1311) where the gaze (901) is maintained (e.g., an adjacent area of ​​the second window (532)). When the object (1313) is selected by the user, the electronic device (401) may exchange the arrangement positions of the second window (532) and the third window (533) displayed in the central area with each other, and may display the second window (532) by moving it to the central area of ​​the virtual screen based on the changed arrangement position, and may display the third window (533) in the area (e.g., a left area) of the virtual screen where the second window (532) was displayed.

[0128] According to one embodiment, when the user's gaze (901) is maintained at a position (1311) outside the central area for a specified period of time, as shown in FIG. 13b, the electronic device (401) identifies a second window (532) displayed at the gaze position where the gaze (901) is maintained, and without changing the arrangement of a plurality of windows currently displayed on the virtual screen (520), all windows (e.g., the first window (531) and the third window (533)) arranged in the direction of the center line (1301) based on the second window (532) can be displayed by moving in the direction of the center line (1301) (e.g., the right direction) according to the movement of the gaze (901) so as to display the second window (532) in the central area. According to one embodiment, when some moved windows (e.g., the first window (531)) go out of the range of the virtual screen (520), the electronic device (401) may erase some windows (e.g., switch to a minimized state) and display objects (e.g., menus, graphic elements, or functions) to be displayed on the virtual screen (520) again in the virtual reality space. According to one embodiment, the electronic device (401) may display the windows (e.g., the first window (531)) that go out of the range of the virtual screen (520) in an empty area (e.g., a left area) created by the movement. At this time, the arrangement position of the windows (e.g., the first window (531)) may be changed (e.g., to the left of the second window (532).

[0129] According to one embodiment, when the user's gaze (901) is maintained at a position outside the central area for a specified period of time, as illustrated in FIG. 13C, the electronic device (401) may identify the first window (window 1) (531) displayed at the gaze position where the gaze (901) is maintained, and display an object (1312) for automatically arranging the window to a specified area (e.g., the central area). When the object (1312) is selected by the user, the electronic device (401) may display all windows (e.g., the first window (532) to the fifth window (535)) arranged in the direction of the center line (1301) based on the first window (532) in accordance with the movement of the gaze (901) so as to display the first window (531) in the central area without changing the arrangement of the plurality of windows currently displayed on the virtual screen (520). According to one embodiment, when some of the moved windows (e.g., the fourth window (534) and the fifth window (535)) go out of the range of the virtual screen (520), the electronic device (401) can erase some of the windows (e.g., switch to a minimized state) and display an object (e.g., a menu, a graphic element, or a function) in the virtual reality space to request display on the virtual screen (520) again.

[0130] FIGS. 14a, 14b, and 14c are drawings illustrating examples for changing the arrangement of windows in an electronic device according to one embodiment.

[0131] Referring to FIGS. 14A, 14B, and 14C, an electronic device (401) according to an embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (202) of FIG. 2, the wearable electronic device (300) of FIGS. 3A to 3C, or the electronic device (401) of FIG. 4) may, as in operation 1205 of FIG. 12 described above, display a plurality of windows (531, 532, 533) on a virtual screen (520) without interference with each other (e.g., without overlapping), and when a second window (532), which is one of the plurality of windows, is selected by a handler (e.g., a controller) (1401) or a user's gesture, the electronic device may display an object (1411) indicating selection of the second window (532) and a visual effect (e.g., a visual affordance) (1413) indicating that the selected second window (532) is movable.

[0132] Referring to FIG. 14A, according to one embodiment, the electronic device (401) may track the user's gaze (901) to identify the gaze position toward which the gaze (901) is directed, and change the arrangement (e.g., arrangement position, arrangement order, or arrangement interval) of the windows (531, 532, 533) currently displayed on the virtual screen (520) to move the selected second window (532) at the gaze position. Based on the gaze position, the electronic device (401) may change the arrangement position of the selected second window (531) (e.g., change it to be arranged in a central area between the third window (533) and the first window (511). If there is a window displayed at the gaze position, the electronic device (401) may move the corresponding window (e.g., the first window (531)) and display a movable area (1421) based on the size of the second window (532). If there is no window displayed at the gaze position and the area is empty, the electronic device (401) can display a movable area (1421) based on the size of the second window (532) in the empty area. Based on the changed arrangement position, the electronic device (401) can move the second window (532) to the movable area (1421) and display all of the plurality of windows (531, 532, 531) so that the selected second window (532) is displayed in the central area set based on the center line (1301).

[0133] Referring to FIG. 14B, according to one embodiment, the electronic device (401) can track the user's gaze (901) to identify the location where the gaze (901) is directed, and change the arrangement order of the windows (531, 532, 533) currently displayed on the virtual screen (520) to display the second window (532) selected at the gaze location. Based on the gaze location, the electronic device (401) can change the arrangement location of the selected second window (531) (e.g., change it to be arranged between the third window (533) and the first window (511). If there is a window displayed at the gaze location, the electronic device (401) can move the corresponding window (e.g., the first window (531)) and display a movable area (1421) based on the size of the second window (532). If there is no window displayed at the gaze position and the electronic device (401) is an empty area, the electronic device (401) can display a movable area (1421) based on the size of the second window (532) in the empty area. Based on the changed arrangement position, the electronic device (401) can move and display the second window (532) in the movable area (1421) while maintaining the arrangement position of the third window (533) displayed in the central area without changing it. At this time, if the empty area in which the moved first window (531) is to be displayed is smaller than the size of the first window (531), the size of the first window (531) can be reduced and displayed or the first window (531) can be deleted.

[0134] Referring to FIG. 14C, according to one embodiment, the electronic device (401) can track the user's gaze (901) to identify the gaze position toward which the gaze (901) is directed, and change the arrangement position of the second window (532) to display the selected second window (532) at the gaze position (e.g., an upper region based on the center line). The electronic device (401) can check whether there is a window displayed at the gaze position, and if it determines that the gaze position is an empty region, it can display a movable area (1423) based on the size of the second window (532). Based on the changed arrangement position, the electronic device (401) can move and display the second window (532) in the movable area (1423) while maintaining the arrangement positions of the remaining windows (e.g., the first window (531) and the third window (533)) without changing them. If the blank area at the gaze position is smaller than the size of the second window (532) and larger than a specified ratio, the electronic device (401) can display the remaining windows (e.g., the first window (531) and the third window (533)) by moving them to the blank area in the opposite direction to the gaze position to secure an area of ​​a size capable of displaying the second window (532).

[0135] FIGS. 15A and 15B are diagrams illustrating examples for changing the arrangement of windows in an electronic device according to one embodiment.

[0136] Referring to FIGS. 15A and 15B , an electronic device (401) according to an embodiment (e.g., the electronic device (101) of FIG. 1 , the electronic device (202) of FIG. 2 , the wearable electronic device (300) of FIGS. 3A to 3C , or the electronic device (401) of FIG. 4 ) may store execution status information of the current virtual screen (520) in a memory (e.g., the memory (130) of FIG. 1 , the memory (450) of FIG. 4 ) at a user request or automatically when the virtual screen (520) is terminated, as in operation 1209 of FIG. 12 described above. The electronic device (401) may display an object (e.g., a menu, a graphic element, or a function) (1501) for storing the execution status information of the virtual screen (520) in a virtual reality space. The electronic device (401) may display a menu screen (1503) displayed in an area adjacent to the virtual screen (520), and may display an object (1505) indicating execution status information stored in the menu screen (1503). Here, the object (1501) may be displayed separately or may be included in an object including context menus. The execution status information of the virtual screen (520) may include at least one of the size, distance, or type of the virtual screen (520), and at least one of the sizes, arrangement positions, arrangement intervals of a plurality of windows (e.g., a first window (531) and a second window (532)) currently displayed on the virtual screen (520), or identification information of applications corresponding to the plurality of windows.

[0137] According to one embodiment, an electronic device (401) may store execution status information of a virtual screen (520) when an object (1501) is selected by a user, as shown in FIG. 15A. When re-executing the virtual screen (520), the electronic device (401) may re-execute the virtual screen (520) in its previous execution state based on the execution status information when an object (1505) indicating execution status information is selected from a menu screen (1503) displayed in a virtual reality space, and may display a plurality of windows on the re-executed virtual screen (520).

[0138] According to one embodiment, the electronic device (401) may change the type of the virtual screen (520) based on user context information (e.g., current context) when re-executing the virtual screen (520), as illustrated in FIG. 15B. For example, if the external electronic device (403) to be linked is, for example, a mobile phone, the electronic device may change the virtual screen to the first type (721), display the virtual screen in a size set for the first type (721), adjust the size of the first window (531) among a plurality of windows (530) to minimize the blank area on the virtual screen (520), and erase the second window (532), which is the remaining window. Afterwards, if it is confirmed that a change in arrangement is required, the electronic device (401) may display the second window (532) on the virtual screen and erase the first window (531). If the external electronic device (403) to be linked is, for example, a tablet, the electronic device (401) can change to the second type (723), display the virtual screen (520) in a size set for the second type (723), and adjust the sizes or arrangement intervals of a plurality of windows (531, 532) to display them on the virtual screen. For example, the electronic device (401) can display the first window (531) as is without resizing it, delete the second window (532), which is the remaining window, and if it is confirmed that the arrangement needs to be changed, display the second window (532) on the virtual screen (520), and delete the first window (531). If the external electronic device (403) to be linked is, for example, a PC, the electronic device (401) can change to the third type (725), and display the virtual screen (520) in a size set for the third type (725). Since the electronic device (401) is identical to the type information included in the execution status information acquired from the memory by the third type (725), it can display multiple windows (531, 532) on the virtual screen without interference in the previous execution status based on the acquired execution status information.

[0139] FIG. 16 is a diagram illustrating an example for executing multiple virtual screens in an electronic device according to one embodiment.

[0140] An electronic device according to one embodiment (e.g., an electronic device (101) of FIG. 1, an electronic device (202) of FIG. 2, a wearable electronic device (300) of FIGS. 3A to 3C, or an electronic device (401) of FIG. 4) may, when a designated specific gesture (1603) is input, expand a virtual screen (520) being used by a user to the entire area of ​​a virtual reality space (1601) (e.g., a virtual reality space (510) of FIGS. 4 and 11), and display a plurality of windows (530) included in the virtual screen (520) in the virtual reality space (1601).

[0141] FIG. 17 is a diagram illustrating an example for executing multiple virtual screens in an electronic device according to one embodiment.

[0142] Referring to FIG. 17, an electronic device (401) according to an embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (202) of FIG. 2, the wearable electronic device (300) of FIGS. 3A to 3C, or the electronic device (401) of FIG. 4) may, when executing a plurality of virtual screens, display a plurality of virtual screens (1711, 1712, 1713) executed on a virtual reality space (1601). The electronic device (401) may display a plurality of virtual screens (1711, 1712, 1713) within a user's field of vision, and may arrange additionally executed virtual screens (1714, 1715, 1716) in another space area outside the user's field of vision. The electronic device (401) can display one or more windows executed on each of the plurality of virtual screens (1711, 1712, 1713) on each of the plurality of virtual screens (1711, 1712, 1713). The electronic device (401) can arrange the plurality of virtual screens (1711, 1712, 1713) based on a specified arrangement standard. For example, the most recently executed virtual screen (e.g., 1712) can be arranged at the center of the virtual reality space (1601), and the remaining virtual screens (1711, 1713 to 1716) can be arranged in the order of most recent execution in areas on both sides of the center. For example, the virtual reality screens can be arranged according to the user's situation based on user situation information.

[0143] According to one embodiment, the electronic device (401) may display a user interface (UI) (1720) including objects representing virtual screens (1711 to 1716) in the form of a list (e.g., a mini map) that can be moved left and right. When the user's gaze (901) moves to a first virtual screen (1711), which is a virtual screen located at a different location from the central position set based on the central line (1701), and is maintained there for a specified period of time, the electronic device (401) may display the central line (1701) on the first virtual screen (1711).

[0144] FIGS. 18A, 18B, and 18C are diagrams illustrating examples of executing multiple virtual screens in an electronic device according to one embodiment.

[0145] Referring to FIG. 18A, when an electronic device (401) according to one embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (202) of FIG. 2, the wearable electronic device (300) of FIGS. 3A to 3C, or the electronic device (401) of FIG. 4) executes a plurality of virtual screens (1711, 1712, 1713, 1714, 1715), the electronic device may display the virtual screen (1712) being executed among the plurality of virtual screens (1711, 1712, 1713, 1714, 1715) executed on a virtual reality space (1601) within a user's field of view, and display the remaining virtual screens (1711, 1713, 1714, 1715) outside the field of view. A first virtual screen (1711) may include a plurality of windows (531, 532), a second virtual screen (1712) may include a plurality of windows (533, 534), a third virtual screen (1713) may include a plurality of windows (535, 536), and a fourth virtual screen (1714) may include a plurality of windows (537, 538). According to one embodiment, the electronic device (401) may display a user interface (UI) (1720) including objects (e.g., A, B, C, and D) representing each of the virtual screens (1711, 1712, 1713, 1714, 1715) in the form of a list (e.g., a mini map) that can be moved left / right. Here, the field of view may be a range of a virtual reality space in which the user wearing the electronic device (401) can comfortably move his / her head up / down or left / right based on the user's field of view. For example, the electronic device (401) may display a virtual screen (third virtual screen (screen C) (1713)) by moving it to the central position toward which the user's gaze (901) is directed as the user interface (UI) (1720) moves to the left (1801).

[0146] Referring to FIG. 18B, an electronic device (401) according to an embodiment may, when a user selects (1803) an object (A) corresponding to a virtual screen (e.g., a first virtual screen (1711)) through a handler (1401) in a user interface (1720), change the arrangement position of the object (A) to a central position through a swipe or shift, move the object (B) corresponding to the virtual screen (e.g., a second virtual screen (1712)) displayed at the central position to another position, and display the object (A) by moving it to the central position. Accordingly, the electronic device (401) may move the first virtual screen (1711) corresponding to the object (A) to a central position in the virtual reality space and display it, and move the second virtual screen (1712) corresponding to the object (B) displayed at the central position to another position and display it, as the arrangement positions of the objects (A) and (B) are changed by moving them.

[0147] Referring to FIG. 18C, an electronic device (401) according to an embodiment may display at least one window included in each of the virtual screens (1711, 1712, 1713) displayed within a user's field of view by moving the window (532) between the virtual screens (1711, 1712, 1713). For example, the electronic device (401) may move (1811) the window (532) included in the second virtual screen (1712) to an object (C) corresponding to the third virtual screen (1713) included in the user interface (UI) (1720) so as to display the window (532) on another virtual screen (e.g., the third virtual screen (1713)). At this time, the electronic device (401) may change the arrangement positions of the plurality of windows displayed on the third virtual screen (1713) to which the window (532) has been moved so that the plurality of windows are arranged without interference with each other.

[0148] FIGS. 19A, 19B, and 19C are diagrams illustrating examples for managing multiple windows in an electronic device according to one embodiment.

[0149] Referring to FIG. 19A, an electronic device (401) according to an embodiment (e.g., the electronic device (101) of FIG. 1, the electronic device (202) of FIG. 2, the wearable electronic device (300) of FIGS. 3A to 3C, or the electronic device (401) of FIG. 4) may provide a plurality of virtual screens in a virtual reality space (1601). When the electronic device (401) arranges each of the plurality of virtual screens in the x-axis direction (e.g., left / right), the electronic device (401) may expand the size in the y-axis direction (e.g., up / down). As the virtual screen (520) expands in size in the y-axis direction (e.g., up / down), the electronic device (401) may set an area within the user's field of view range as a display area (1911), and set an area outside the field of view (FoV) range as an abstract area (1913). Here, the field of view range may be a range of a virtual reality space in which the user wearing the electronic device (401) can comfortably move his / her head up / down or left / right based on the user's field of view. The electronic device (401) may move a plurality of windows (531, 532, 533, 534) included in the virtual screen (520) up / down (1901, 1903) to display a specific window in a central area toward which the user's gaze (901) is directed. The electronic device (401) may display a visual effect (e.g., a dim effect, a blur effect, a gray out effect, a gradation, a shadow effect, transparency adjustment, or a highlight) in an abstract area (1913) that is outside the user's field of view range.

[0150] Referring to FIGS. 19B and 19C , according to one embodiment, the electronic device (401) may allow a user to move a plurality of windows (531, 532, 533, 534) up / down (1901) or left / right (1905), and may display a user interface (1920) including objects representing each of the plurality of windows (531, 532, 533, 534). The electronic device (401) may use the user interface (1920) to move (e.g., swipe or scroll) the plurality of windows included in the expanded virtual screen (520).

[0151] An electronic device according to an embodiment (e.g., the electronic device (101) of FIG. 1, the wearable electronic device (200) of FIG. 2, the wearable electronic device (300) of FIGS. 3A and 3C, and / or the electronic device (410) of FIG. 4) may, when virtual windows are displayed on a virtual screen (or virtual reality space), arrange or change the arrangement of the virtual windows by a specific button (e.g., a menu, a function, a graphic element, or an object) provided in the virtual reality space or a specific input by an external electronic device, without selecting at least one window among the virtual windows or an object corresponding to at least one window. When a specific button or a specific input is received, the electronic device may arrange or change the arrangement of the virtual windows under different arrangement conditions than when a window or an object corresponding to a window is selected. As an example, when a specific button or a specific input is received, the electronic device may arrange the virtual windows (530) with a gap between the virtual windows in the Z-axis direction based on the user's line of sight. Here, the spacing between virtual windows (530) can be set to a constant distance value or to different distance values. For example, the electronic device can set the spacing from wide to narrow, or from narrow to wide, starting from the window arranged at the front based on the user's line of sight. For example, the electronic device can arrange a specific window (e.g., a window in which a notification or alert is generated) to a distance value greater than the distance value set for the remaining windows, thereby widening the spacing. The present invention is not limited thereto, and the spacing can be set in various ways.According to one embodiment, when at least one window among virtual windows (530) arranged in the Z-axis direction based on the line of sight is selected, the electronic device can adjust (e.g., reflect or set) the Z-axis distance of the remaining windows based on a Z-direction distance value set for the at least one selected window (e.g., a preset distance value or a distance value adjusted by a user's interaction).

[0152] According to one embodiment, an operating method in an electronic device (e.g., an electronic device (101) of FIG. 1, an electronic device (202) of FIG. 2, a wearable electronic device (300) of FIGS. 3A to 3C, or an electronic device (401) of FIG. 4) may include an operation of controlling a display (e.g., a display module (160) of FIG. 1, a display member (201) of FIG. 2, a display (321) of FIG. 3B, or a display (420) of FIG. 4) of the electronic device to display a virtual screen in a virtual reality space.

[0153] According to one embodiment, a method of operating in an electronic device may include executing a plurality of windows of at least one application and selecting a first window from among the plurality of windows based on user context information.

[0154] According to one embodiment, an operating method in an electronic device may include an operation of arranging and displaying the selected first window in a designated area on a virtual screen based on an arrangement condition set using user context information, and arranging and displaying other windows among the plurality of windows in a remaining area of ​​the virtual screen without overlapping based on the first window.

[0155] According to one embodiment, the method may further include an operation of changing at least one of a size of the virtual screen, a distance between the user and the virtual screen, or a type of the virtual screen.

[0156] According to one embodiment, the arrangement condition may be a condition for setting the size, arrangement position, or arrangement interval of the plurality of windows based on the user context information or the execution status information of the virtual screen stored in the memory of the electronic device (e.g., the memory (130) of FIG. 1 or the memory (450) of FIG. 4).

[0157] According to one embodiment, the user context information may include at least one of gaze information, distance information, movement information, user input information, frequently used pattern information, or information of a connected external electronic device.

[0158] According to one embodiment, the designated area may be a central area of ​​the virtual screen designated based on the user's field of view when the user wearing the electronic device looks straight ahead.

[0159] According to one embodiment, the method may further include an operation of arranging a new window of the new application in a central area of ​​the virtual screen based on whether a new application is executed while displaying the plurality of windows on the virtual screen, and changing the arrangement of the plurality of windows so as to display the plurality of windows without interference with the new window.

[0160] According to one embodiment, the method may further include an operation of tracking a user's gaze using at least one camera circuit of the electronic device (e.g., the camera module (251) of FIG. 2 or the camera circuit (430) of FIG. 4), an operation of identifying a gaze position on the virtual screen, and an operation of changing an arrangement position of a first window displayed at the gaze position among the plurality of windows to the central area based on whether the gaze position is outside the central area of ​​the virtual screen.

[0161] According to one embodiment, the method may further include an action of displaying an object for moving the first window displayed at the gaze location to the central area based on the gaze being maintained outside the central area and in another area for a specified period of time.

[0162] According to one embodiment, the method may further include an operation of displaying an area of ​​the virtual screen corresponding to a moved gaze position as a movable area of ​​the selected window based on the movement of the gaze after selecting one of the plurality of windows, an operation of changing the arrangement positions of the plurality of windows to move the selected window to the movable area, and an operation of displaying the selected window in the central area based on the changed arrangement positions of the plurality of windows.

[0163] According to one embodiment, the method may further include an operation of storing execution status information of the virtual screen in a memory of the electronic device in response to a request to terminate the virtual screen.

[0164] According to one embodiment, the execution status information may include at least one of the size, distance, or type of the virtual screen and at least one of the sizes, arrangement positions, arrangement intervals, or identification information of applications corresponding to the plurality of windows.

[0165] According to one embodiment, the method may further include, in response to a request for re-execution of the virtual screen, an operation of obtaining the execution state information from the memory, and an operation of re-executing the virtual screen and displaying the plurality of windows based on the execution state information.

[0166] According to one embodiment, the method may further include an operation of displaying virtual screen objects representing each of the plurality of virtual screens in the virtual reality space based on displaying a plurality of virtual screens in the virtual reality space, and an operation of displaying a first virtual screen displayed at a gaze position of the user among the plurality of virtual screens at a central position of the virtual reality space.

[0167] According to one embodiment, the operation of displaying the first virtual screen at the central position may further include an operation of changing the arrangement of the plurality of virtual screens so as to display the plurality of virtual screens without overlapping.

[0168] According to one embodiment, in a non-transitory storage medium storing one or more programs, the one or more programs, when executed by at least one processor (e.g., the processor (120) of FIG. 1 or the processor (410) of FIG. 4) of an electronic device (e.g., the electronic device (101) of FIG. 1, the electronic device (202) of FIG. 2, the wearable electronic device (300) of FIGS. 3A to 3C or the electronic device (401) of FIG. 4), cause the electronic device to control a display (e.g., the display module (160) of FIG. 1, the display member (201) of FIG. 2, the display (321) of FIG. 3B or the display (420) of FIG. 4)) of the electronic device to display a virtual screen in a virtual reality space, execute a plurality of windows of at least one application, and select a first window among the plurality of windows based on user context information, and display the selected first window by arranging it in a designated area on a virtual screen based on an arrangement condition set using the user context information, and It may include commands that cause an operation to display a plurality of windows of at least one application on the screen without overlapping between the windows.

[0169] According to the above-described embodiment, the electronic device can display a virtual screen in a virtual reality space according to the user's situation, and automatically arrange and display multiple windows on the virtual screen according to the user's situation, thereby reducing the user's inconvenience in using the virtual screen. In addition, various effects that can be directly or indirectly understood through this document can be provided. The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by those skilled in the art to which the present disclosure pertains from the description below.

[0170] The embodiments disclosed in this document are presented for the purpose of explaining and understanding the disclosed technical content, and do not limit the scope of the technology described in this document. Therefore, the scope of this document should be interpreted to include all modifications or various other embodiments based on the technical concepts of this document.

[0171] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0172] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0173] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0174] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0175] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0176] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices (101, 200, 300, 401), display(160, 201, 321, 420); Memory containing instructions (130, 450); and At least one processor (120, 410) comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Control the display to display a virtual screen in a virtual reality space, Running multiple windows of at least one application, and selecting a first window among the multiple windows based on user context information, An electronic device that controls the display to arrange and display the selected first window in a designated area of ​​a virtual screen based on arrangement conditions set using user context information, and to arrange and display other windows among the plurality of windows in the remaining area of ​​the virtual screen without overlapping based on the first window.

2. In the first paragraph, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: To change at least one of the size of the virtual screen, the distance between the user and the virtual screen, or the type of the virtual screen; The above arrangement condition is a condition for setting the size, arrangement position or arrangement interval of the plurality of windows based on the user situation information or the execution status information of the virtual screen stored in the memory. An electronic device, wherein the user context information includes at least one of the user's gaze information, the user's distance information, the user's movement information, user input information, frequently used pattern information, or information of a connected external electronic device.

3. In paragraph 1 or 2, The above designated area is the central area of ​​the virtual screen designated based on the user's field of vision when the user wearing the electronic device looks straight ahead, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: An electronic device that, while displaying the plurality of windows on the virtual screen, arranges a new window of the new application in a central area of ​​the virtual screen based on whether a new application is executed, and changes the arrangement of the plurality of windows so as to display the plurality of windows without interference with the new window.

4. In any one of paragraphs 1 to 3, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Tracking the user's gaze using at least one camera circuit (251, 430) of the electronic device, Identify the gaze position on the above virtual screen, An electronic device that changes the arrangement position of a first window displayed at the gaze position among the plurality of windows to the central area based on whether the gaze position is outside the central area of ​​the virtual screen.

5. In any one of paragraphs 1 to 4, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: An electronic device that controls the display to display an object for moving the first window displayed at the gaze location to the central area based on the gaze being maintained outside the central area and in another area for a specified period of time.

6. In any one of paragraphs 1 to 5, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: After selecting one of the above multiple windows, based on the movement of the gaze, an area of ​​the virtual screen corresponding to the moved gaze position is displayed as a movable area of ​​the selected window, Changing the arrangement positions of the plurality of windows so as to move the selected window to the movable area; Based on the changed arrangement positions of the plurality of windows, the display is controlled to display the selected window in the central area, In response to a request to terminate the above virtual screen, store the execution status information of the above virtual screen in the memory, In response to a request for re-execution of the above virtual screen, the execution status information is obtained from the above memory, Based on the above execution status information, the virtual screen is re-executed and the display is controlled to display the plurality of windows. An electronic device, wherein the execution status information includes at least one of the size, distance or type of the virtual screen and at least one of the sizes, arrangement positions, arrangement intervals of the plurality of windows or identification information of applications corresponding to the plurality of windows.

7. In any one of paragraphs 1 to 6, the instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Based on displaying a plurality of virtual screens in the virtual reality space, the display is controlled to display virtual screen objects representing each of the plurality of virtual screens in the virtual reality space, Controlling the display so that the first virtual screen displayed at the position where the user's gaze is directed among the plurality of virtual screens is displayed at the central position of the virtual reality space, An electronic device that changes the arrangement of the plurality of virtual screens so that the plurality of virtual screens are displayed without overlapping when the first virtual screen is displayed at the central location.

8. In a method of operation in an electronic device (101, 200, 300, 401), An action of controlling a display (160, 201, 321, 420) of the electronic device to display a virtual screen in a virtual reality space; An operation of executing multiple windows of at least one application and selecting a first window among the multiple windows based on user context information; and A method comprising: arranging and displaying the selected first window in a designated area of ​​a virtual screen based on an arrangement condition set using user context information; and arranging and displaying other windows among the plurality of windows in a remaining area of ​​the virtual screen without overlapping based on the first window.

9. In paragraph 8, the method, Further comprising an action of changing at least one of the size of the virtual screen, the distance between the user and the virtual screen, or the type of the virtual screen; The above arrangement conditions are conditions for setting the size, arrangement position or arrangement interval of the plurality of windows based on the user situation information or the execution status information of the virtual screen stored in the memory (130, 450) of the electronic device. A method wherein the user context information includes at least one of gaze information, distance information, movement information, user input information, frequently used pattern information, or information of a connected external electronic device.

10. In clause 8 or 9, The above designated area is the central area of ​​the virtual screen designated based on the user's field of vision when the user wearing the electronic device looks straight ahead, The above method, A method further comprising: while displaying the plurality of windows on the virtual screen, arranging a new window of the new application in a central area of ​​the virtual screen based on whether a new application is executed, and changing the arrangement of the plurality of windows so as to display the plurality of windows without interference with the new window.

11. In any one of clauses 8 to 10, the method, An operation of tracking a user's gaze using at least one camera circuit (251, 430) of the electronic device; An action of identifying a gaze position on the virtual screen; and A method further comprising an action of changing the arrangement position of a first window displayed at the gaze position among the plurality of windows to the central area based on whether the gaze position is outside the central area of ​​the virtual screen.

12. In any one of clauses 8 to 11, the method, A method further comprising an action of displaying an object for moving the first window displayed at the gaze location to the central area based on the gaze being maintained outside the central area and in another area for a specified period of time.

13. In any one of clauses 8 to 12, the method, An action of selecting one of the plurality of windows and then displaying an area of ​​the virtual screen corresponding to the moved gaze position as a movable area of ​​the selected window based on the movement of the gaze; An operation of changing the arrangement positions of the plurality of windows so as to move the selected window to the movable area; and A method further comprising an action of displaying the selected window in the central area based on the changed arrangement positions of the plurality of windows.

14. In any one of clauses 8 to 13, the method, An action of storing execution status information of the virtual screen in the memory in response to a request to terminate the virtual screen; In response to a request for re-execution of the virtual screen, an operation of obtaining the execution status information from the memory; and Further comprising an action of re-executing the virtual screen based on the above execution status information and displaying the plurality of windows, A method wherein the execution status information includes at least one of the size, distance or type of the virtual screen and at least one of the sizes, arrangement positions, arrangement intervals of the plurality of windows or identification information of applications corresponding to the plurality of windows.

15. In a non-transitory storage medium storing one or more programs, the one or more programs, when executed by at least one processor (120, 410) of an electronic device (101, 200, 300, 401), cause the electronic device to: An action of controlling a display (160, 201, 321, 420) of the electronic device to display a virtual screen in a virtual reality space; An operation of executing multiple windows of at least one application and selecting a first window among the multiple windows based on user context information; and A non-transitory storage medium including commands for arranging and displaying the selected first window in a designated area on a virtual screen based on an arrangement condition set using user context information, and executing an operation of displaying a plurality of windows of at least one application on the virtual screen without overlapping the windows.

Images