An application display method, device and computer readable storage medium

Abstract

An application display method, apparatus, and computer-readable storage medium are disclosed. The method includes: creating a virtual window container for each application installed in an electronic device, the electronic device's system being configured to display virtual windows for each application, the virtual window container having a state variable and being configured to display applications one-to-one in the system's virtual windows; upon receiving an input event, determining the target virtual window container for the input event; setting the state variable of the target virtual window container to an active state; and displaying the application the user wishes to open corresponding to the input event in the virtual window corresponding to the target virtual window container. This allows multiple applications to be displayed simultaneously in a VR system and makes all applications controllable by the user, improving the user experience.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to virtual display technology, in particular to a method, device and computer readable storage medium for displaying multiple application programs in a system. BACKGROUND

[0002] Virtual Reality (VR) refers to a virtual environment generated by computer technology as the core and modern high-tech means. A user obtains the same feeling as in the real world through vision, hearing and touch by means of special input / output devices. VR technology is a high-level human-computer interaction technology that integrates computer graphics, human-computer interface technology, sensor technology and artificial intelligence, and can effectively simulate various perceptions of people in a natural environment.

[0003] Currently, only one application program can be run in the VR system, and multiple application programs cannot be run simultaneously. When other application programs are to be run, the function keys of the terminal need to be manually operated to switch the application program to be run to the foreground, so the user experience is poor. SUMMARY

[0004] In a first aspect, the present disclosure provides an application program display method, comprising:

[0005] creating a virtual window container for each application program installed in an electronic device, the system of the electronic device being configured with a virtual window for displaying each application program, the virtual window container being provided with a state variable, and the virtual window container being configured to display the application programs one by one in the virtual window of the system;

[0006] when an input event is received, determining a target virtual window container of the input event; the input event being a trigger operation of a user to start an application program;

[0007] setting the state variable of the target virtual window container to an active state;

[0008] displaying the application program to be started by the user corresponding to the input event in the virtual window corresponding to the target virtual window container.

[0009] In some embodiments, the virtual window container is configured to display the application programs one by one in the virtual window of the system, comprising:

[0010] establishing a one-to-one correspondence between the application programs and the created virtual window containers;

[0011] starting the application program to the corresponding virtual window container;

[0012] The content in the virtual window container is sent to the corresponding virtual window in the system for display.

[0013] In some embodiments, the electronic device's system is configured with virtual windows for displaying each application, including:

[0014] Create a number of virtual windows in the system equal to the number of applications, and establish a one-to-one correspondence between the applications and the created virtual windows;

[0015] A first texture and a second texture are created for each virtual window container, and the created first texture and second texture are bound to the set virtual window container; wherein, the first texture is applicable to the application, and the second texture is applicable to the system of the electronic device;

[0016] The content of the corresponding virtual window container is obtained through the first texture;

[0017] The obtained content is drawn onto the corresponding second texture;

[0018] The content of the second texture is sent to the corresponding virtual window of the system for displaying the content of the second texture in the virtual window of the system.

[0019] In some embodiments, determining the target virtual window container for the input event includes:

[0020] Obtain the location information of the input event;

[0021] The target virtual window for the application to be displayed in the system is determined based on the location information of the input event.

[0022] The virtual window container corresponding to the target virtual window is determined based on the one-to-one correspondence between the application and the set virtual window, and the one-to-one correspondence between the application and the set virtual window container.

[0023] In some embodiments, displaying the application the user wants to open corresponding to the input event in the virtual window corresponding to the target virtual window container includes:

[0024] Determine the target window handle corresponding to the target virtual window container;

[0025] The code corresponding to the input event is injected into the target window handle to display the application that the user wants to open corresponding to the input event in the virtual window corresponding to the target virtual window container.

[0026] Secondly, this application provides an apparatus for displaying multiple applications in a system, comprising:

[0027] A virtual window container creation module is used to create a virtual window container for each application installed in an electronic device. The system of the electronic device is configured to display a virtual window for each application. The virtual window container is set with a state variable and is configured to display the application one by one in the system's virtual window.

[0028] The event receiving module is used to determine the target virtual window container of the input event when an input event is received; the input event is a triggering operation that the user wants to open the application.

[0029] A state variable setting module is used to set the state variable of the target virtual window container to an active state.

[0030] The processing module is used to display the application that the user wants to open, corresponding to the input event, in the virtual window corresponding to the target virtual window container.

[0031] In some embodiments, the virtual window container creation module is specifically used for:

[0032] Establish a one-to-one correspondence between the application and the created virtual window container;

[0033] Launch the application into the corresponding virtual window container;

[0034] The content in the virtual window container is sent to the corresponding virtual window in the system for display.

[0035] In some embodiments, the virtual window container creation module is further configured to:

[0036] Create a number of virtual windows in the system equal to the number of applications, and establish a one-to-one correspondence between the applications and the created virtual windows;

[0037] A first texture and a second texture are created for each virtual window container, and the created first texture and second texture are bound to the set virtual window container; wherein, the first texture is applicable to the application, and the second texture is applicable to the system of the electronic device;

[0038] The content of the corresponding virtual window container is obtained through the first texture;

[0039] The obtained content is drawn onto the corresponding second texture;

[0040] The content of the second texture is sent to the corresponding virtual window of the system for displaying the content of the second texture in the virtual window of the system.

[0041] In some embodiments, the event receiving module is specifically used for:

[0042] Obtain the location information of the input event;

[0043] The target virtual window for the application to be displayed in the system is determined based on the location information of the input event.

[0044] The virtual window container corresponding to the target virtual window is determined based on the one-to-one correspondence between the application and the set virtual window, and the one-to-one correspondence between the application and the set virtual window container.

[0045] In some embodiments, the processing module is specifically used for:

[0046] Determine the target window handle corresponding to the target virtual window container;

[0047] The code corresponding to the input event is injected into the target window handle to display the application that the user wants to open corresponding to the input event in the virtual window corresponding to the target virtual window container.

[0048] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when executed by the processor, the computer program in the memory is used to perform the method as described in any of the first aspects.

[0049] Fourthly, this application also provides a computer-readable storage medium storing computer-executable commands for performing the methods described below as described in any of the first aspects.

[0050] Compared with related technologies, the application display method provided in this application creates a virtual window container for each application installed in an electronic device. The system of the electronic device is configured with a virtual window for displaying each application. The virtual window container has a state variable and is configured to display each application one-to-one in the system's virtual windows. When an input event is received, the target virtual window container of the input event is determined; the state variable of the target virtual window container is set to an active state; finally, the application that the user wants to open corresponding to the input event is displayed in the virtual window corresponding to the target virtual window container. Therefore, this application allows multiple applications to be displayed simultaneously in a virtual reality system, thereby improving the user experience.

[0051] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the application. Other advantages of this application can be realized and obtained by means of the solutions described in the description and the accompanying drawings. Attached Figure Description

[0052] The accompanying drawings are used to provide an understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0053] Figure 1 A flowchart illustrating a method for displaying multiple applications in a system, provided as an embodiment of this application;

[0054] Figure 2 A schematic diagram illustrating the display of multiple applications in a system, provided as an embodiment of this application;

[0055] Figure 3 This is a schematic diagram of the structure of a device for displaying multiple application windows in a system, provided in an embodiment of this application. Detailed Implementation

[0056] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0057] This article provides a method for displaying an application, such as... Figure 1 As shown, it includes:

[0058] Step 101: Create a virtual window container for each application installed in the electronic device. The system of the electronic device is configured to display a virtual window for each application. The virtual window container has a state variable and is configured to display each application in a corresponding virtual window of the system. The state variable includes an active state.

[0059] In one exemplary instance, the virtual container window is equivalent to a register in the background of the virtual environment interface, used to temporarily store the application's display content.

[0060] In one exemplary instance, the system is an existing operating system, including but not limited to Windows, Android, and iOS; the electronic device is a VR / AR headset.

[0061] Step 102: When an input event is received, determine the target virtual window container of the input event; the input event is a trigger operation that the user wants to open the application.

[0062] Step 103: Set the state variable of the target virtual window container to the active state.

[0063] Step 104: Display the application that the user wants to open, corresponding to the input event, in the virtual window corresponding to the target virtual window container.

[0064] In one exemplary instance, a schematic diagram of multiple applications in the system can be shown as follows: Figure 2 As shown.

[0065] The method for displaying multiple applications in a system provided in this application embodiment enables multiple applications to be displayed in the system simultaneously and to be in a state that can be controlled by the user, thus realizing the simultaneous operation of multiple applications in the system and improving the user experience.

[0066] In one exemplary instance, the virtual window container is configured to display applications one-to-one in a virtual window of the system, including:

[0067] First, establish a one-to-one correspondence between the application and the created virtual window container.

[0068] Next, launch the application into the corresponding virtual window container.

[0069] Finally, the content in the virtual window container is sent to the corresponding virtual window in the virtual environment interface for display.

[0070] In one exemplary instance, the electronic device's system is configured with virtual windows for displaying each application, including:

[0071] First, create an equal number of virtual windows in the system as the number of applications, and establish a one-to-one correspondence between the applications and the created virtual windows.

[0072] Next, a first texture and a second texture are created for each virtual window container, and the created first texture and second texture are bound to the set virtual window container; wherein, the first texture is applicable to the application, and the second texture is applicable to the system of the electronic device.

[0073] Next, the content of the corresponding virtual window container is obtained through the first texture.

[0074] Next, the content of the corresponding virtual window container is obtained through the first texture.

[0075] Finally, the content of the second texture is sent to the corresponding virtual window of the system for displaying the content of the second texture in the system's virtual window.

[0076] In one exemplary instance, the first texture includes a GL_TEXTURE_EXTERNAL_OES texture, and the second texture includes a GL_TEXTURE_2D texture.

[0077] In one exemplary instance, displaying multiple applications simultaneously in a system can be achieved through the following steps:

[0078] Step 1: Create a virtual display (in this embodiment, the virtual screen corresponds to the virtual container window in other embodiments).

[0079] Since the VirtualDisplay class represents a virtual display, and the content of the virtual display is rendered on the GL_TEXTURE_EXTERNAL_OES texture, the createVirtualDisplay() method of the DisplayManager class is called to create the GL_TEXTURE_EXTERNAL_OES texture and bind it to the virtual display.

[0080] Step 2: The app launches to the virtual display.

[0081] The setLaunchDisplayId function provided by ActivityOptions can specify which virtual display the app should be displayed on after it is launched. When launching the app, the setLaunchDisplayId function is used to specify the ID of the virtual display to launch the app on the virtual screen.

[0082] Step 3: Unity displays the virtual display content.

[0083] The content displayed on the virtual display is obtained through GL_TEXTURE_EXTERNAL_OES, but Unity only supports traditional GLES textures (GL_TEXTURE_2D) and not GL_TEXTURE_EXTERNAL_OES. Therefore, a GL_TEXTURE_2D texture is created, and the GL_TEXTURE_EXTERNAL_OES texture is drawn onto the GL_TEXTURE_2D texture using a frame buffer object (FBO). Then, the TextureID is passed to Unity, and the RawImage control in Unity receives and displays the content of the GL_TEXTURE_2D texture.

[0084] In one exemplary instance, the method for displaying multiple applications simultaneously in a system further includes the following steps:

[0085] Modify the Activity management logic in AMS to keep multiple launched Activities alive by managing the Activity state in the ActivityDisplay class corresponding to each virtual window container.

[0086] The existing virtual environment interface has only one Activity Management System (AMS), and each AMS has only one variable recording which Activity is in the Resumed state. Therefore, there is only one ResumedActivity. Whenever a new Activity is launched, the ResumedActivity is replaced, and the previous Activity enters the OnPause state. In one exemplary instance, the modification of the AMS's Activity management logic, which manages the Activity state in the ActivityDisplay class corresponding to each virtual window container to keep multiple launched Activities alive, includes:

[0087] First, modify the Activity management logic of AMS by adding a variable to manage the Activity state in the ActivityDisplay class corresponding to each virtual window container.

[0088] Secondly, set the variable managing the Activity state in the ActivityDisplay class corresponding to all virtual window containers to the Resumed state.

[0089] In one exemplary instance, the method for displaying multiple applications in the system provided in this application modifies the Activity management logic of AMS, reducing the granularity of Activity management to the ActivityDisplay level. This way, when a new Activity is launched, it first determines which virtual window container the Activity is launched in, and then modifies the corresponding ActivityDisplay state, ensuring that it does not interfere with applications in other virtual window containers.

[0090] In one exemplary instance, determining the target virtual window container for the input event includes:

[0091] First, obtain the location information of the input event.

[0092] Secondly, the target virtual window for the application to be displayed in the system is determined based on the location information of the input event.

[0093] Finally, the target virtual window container corresponding to the target virtual window is determined based on the one-to-one correspondence between the application to be run and the set virtual window, and the one-to-one correspondence between the application and the set virtual window container.

[0094] In one exemplary instance, displaying the application the user wishes to open corresponding to the input event in the virtual window corresponding to the target virtual window container includes:

[0095] First, determine the target window handle corresponding to the target virtual window container.

[0096] Secondly, the code corresponding to the input event is injected into the target window handle to display the application that the user wants to open corresponding to the input event in the virtual window corresponding to the target virtual window container.

[0097] In existing technologies, InputFlinger, which is responsible for distributing input events, only stores one FocusedWindowHandle. When the system receives an input event, it will only inject it into this Windows window.

[0098] The method for displaying multiple applications in the system provided in this application involves drawing the content of a virtual window container within a full-screen application on a default virtual display. Therefore, the focus is always on this full-screen application, preventing input events from being injected into the virtual window container. Consequently, it's necessary to set focus for each display in InputFlinger. However, the input system lacks a data structure similar to ActivityDisplay corresponding to a display; a hashmap or queue for setting focus is required, along with adding a DisplayId attribute to InputEvent. Simultaneously, input from input devices (such as gamepads and buttons) is sent to the main full-screen application. Upon receiving the input, the full-screen application determines which window container has focus and injects the input event into the system using the InputForwarder interface. InputFlinger then distributes the input event to the corresponding window handle (window) based on the displayId.

[0099] This article also provides a display device for an application, such as Figure 3 As shown, the device 2 that displays multiple applications in the system includes:

[0100] The virtual window container creation module 21 is used to create a virtual window container for each application installed in the electronic device. The system of the electronic device is configured to display a virtual window for each application. The virtual window container is set with a state variable and is configured to display the application one by one in the system's virtual window.

[0101] The event receiving module 22 is used to determine the target virtual window container of the input event when an input event is received; the input event is a trigger operation that the user wants to open the application.

[0102] The state variable setting module 23 is used to set the state variable of the target virtual window container to the active state.

[0103] Processing module 24 is used to display the application that the user wants to open corresponding to the input event in the virtual window corresponding to the target virtual window container.

[0104] In one exemplary instance, the virtual window container creation module 21 is specifically used for:

[0105] Establish a one-to-one correspondence between the application and the created virtual window container.

[0106] Launch the application into the corresponding virtual window container.

[0107] The content in the virtual window container is sent to the corresponding virtual window in the virtual environment interface for display.

[0108] In one exemplary instance, the virtual window container creation module 21 is further configured to:

[0109] Create a number of virtual windows in the system equal to the number of applications, and establish a one-to-one correspondence between the applications and the created virtual windows.

[0110] A first texture and a second texture are created for each virtual window container, and the created first texture and second texture are bound to the set virtual window container; wherein, the first texture is applicable to the application, and the second texture is applicable to the system of the electronic device.

[0111] The content of the corresponding virtual window container is obtained through the first texture.

[0112] The obtained content is drawn onto the corresponding second texture.

[0113] The content of the second texture is sent to the corresponding virtual window of the system for displaying the content of the second texture in the virtual window of the system.

[0114] In one exemplary instance, the event receiving module 22 is specifically used for:

[0115] Obtain the location information of the input event.

[0116] The target virtual window that the application displays in the system is determined based on the location information of the input event.

[0117] The virtual window container corresponding to the target virtual window is determined based on the one-to-one correspondence between the application and the set virtual window, and the one-to-one correspondence between the application and the set virtual window container.

[0118] In one exemplary instance, the processing module 24 is specifically used for:

[0119] Determine the target window handle corresponding to the target virtual window container.

[0120] The code corresponding to the input event is injected into the target window handle to display the application that the user wants to open corresponding to the input event in the virtual window corresponding to the target virtual window container.

[0121] The device for displaying multiple applications in the system provided in this application embodiment enables multiple applications to be displayed in the system at the same time and to be in a state that can be controlled by the user. Therefore, it realizes that multiple applications can run simultaneously in a virtual environment interface, thereby improving the user experience.

[0122] This document also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, which, when executed by the processor, is used to perform the methods described in any of the above embodiments.

[0123] This document also provides a computer-readable storage medium storing computer-executable commands for performing the methods described in any of the above embodiments.

[0124] This application describes several embodiments, but these descriptions are exemplary and not restrictive, and it will be apparent to those skilled in the art that many more embodiments and implementations are possible within the scope of the embodiments described herein. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with, or may replace, any feature or element of any other embodiment.

[0125] This application includes and contemplates combinations of features and elements known to those skilled in the art. The embodiments, features, and elements disclosed in this application may also be combined with any conventional features or elements to form a unique inventive scheme as defined by the claims. Any feature or element of any embodiment may also be combined with features or elements from other inventive schemes to form another unique inventive scheme as defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in this application may be implemented individually or in any suitable combination. Therefore, the embodiments are not limited except by the limitations imposed by the appended claims and their equivalents. Furthermore, various modifications and changes may be made within the scope of the appended claims.

[0126] Furthermore, in describing representative embodiments, the specification may have presented methods and / or processes as a specific sequence of steps. However, the method or process should not be limited to the specific order of steps described herein, to the extent that it does not depend on such a specific order. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific order of steps set forth in the specification should not be construed as a limitation of the claims. Moreover, the claims concerning the method and / or process should not be limited to the steps performed in the written order, and those skilled in the art will readily understand that these orders can be varied and still remain within the spirit and scope of the embodiments of this application.

[0127] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

Claims

1. An application display method, characterized in that, include: A virtual window container is created for each application installed in an electronic device. The system of the electronic device is configured to display a virtual window for each application. The virtual window container has a state variable and is configured to display each application in a corresponding virtual window of the system. The virtual window container is a register in the background of the virtual environment interface for temporarily storing the display content of the application. When an input event is received, the target virtual window container of the input event is determined; the input event is a triggering operation that the user wants to open the application. Set the state variable of the target virtual window container to the active state; The application that the user wants to open, corresponding to the input event, is displayed in the virtual window corresponding to the target virtual window container. The electronic device's system is configured with virtual windows for displaying each application, including: Create a number of virtual windows in the system equal to the number of applications, and establish a one-to-one correspondence between the applications and the created virtual windows; A first texture and a second texture are created for each virtual window container, and the created first texture and second texture are bound to the set virtual window container; wherein, the first texture is applicable to the application, and the second texture is applicable to the system of the electronic device; The content of the corresponding virtual window container is obtained through the first texture; The obtained content is drawn onto the corresponding second texture; The content of the second texture is sent to the corresponding virtual window of the system for displaying the content of the second texture in the virtual window of the system.

2. The method according to claim 1, characterized in that, The virtual window container is configured to display applications one-to-one in the system's virtual windows, including: Establish a one-to-one correspondence between the application and the created virtual window container; Launch the application into the corresponding virtual window container; The content in the virtual window container is sent to the corresponding virtual window in the system for display.

3. The method according to claim 1, characterized in that, The determination of the target virtual window container for the input event includes: Obtain the location information of the input event; The target virtual window for the application to be displayed in the system is determined based on the location information of the input event. The virtual window container corresponding to the target virtual window is determined based on the one-to-one correspondence between the application and the set virtual window, and the one-to-one correspondence between the application and the set virtual window container.

4. The method according to claim 3, characterized in that, Displaying the application the user wants to open corresponding to the input event in the virtual window corresponding to the target virtual window container includes: Determine the target window handle corresponding to the target virtual window container; The code corresponding to the input event is injected into the target window handle to display the application that the user wants to open corresponding to the input event in the virtual window corresponding to the target virtual window container.

5. An application display device, comprising: A virtual window container creation module is used to create a virtual window container for each application installed in an electronic device. The system of the electronic device is configured to display a virtual window for each application. The virtual window container is set with a state variable and is configured to display the application one by one in the system's virtual window. The virtual window container is a register in the background of the virtual environment interface and is used to temporarily store the display content of the application. The event receiving module is used to determine the target virtual window container of the input event when an input event is received; the input event is a triggering operation that the user wants to open the application. A state variable setting module is used to set the state variable of the target virtual window container to an active state. The processing module is used to display the application that the user wants to open corresponding to the input event in the virtual window corresponding to the target virtual window container; The virtual window container creation module is also specifically used for: Create a number of virtual windows in the system equal to the number of applications, and establish a one-to-one correspondence between the applications and the created virtual windows; A first texture and a second texture are created for each virtual window container, and the created first texture and second texture are bound to the set virtual window container; wherein, the first texture is applicable to the application, and the second texture is applicable to the system of the electronic device; The content of the corresponding virtual window container is obtained through the first texture; The obtained content is drawn onto the corresponding second texture; The content of the second texture is sent to the corresponding virtual window of the system for displaying the content of the second texture in the virtual window of the system.

6. The apparatus according to claim 5, characterized in that, The virtual window container creation module is specifically used for: Establish a one-to-one correspondence between the application and the created virtual window container; Launch the application into the corresponding virtual window container; The content in the virtual window container is sent to the corresponding virtual window in the system for display.

7. The apparatus according to claim 5, characterized in that, The event receiving module is specifically used for: Obtain the location information of the input event; The target virtual window for the application to be displayed in the system is determined based on the location information of the input event. The virtual window container corresponding to the target virtual window is determined based on the one-to-one correspondence between the application and the set virtual window, and the one-to-one correspondence between the application and the set virtual window container.

8. The apparatus according to claim 7, characterized in that, The processing module is specifically used for: Determine the target window handle corresponding to the target virtual window container; The code corresponding to the input event is injected into the target window handle to display the application that the user wants to open corresponding to the input event in the virtual window corresponding to the target virtual window container.

9. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program, and the computer program in the memory, when executed by the processor, is used to perform the method as described in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that, The storage medium stores computer-executable commands for performing the method described in any one of claims 1-4.

Images