Window display method, device, computer equipment and storage medium

Abstract

The invention discloses a window display method, device, computer equipment and storage medium, which loads an OpenGL ES rendering library; an Android system encodes an OpenGL ES rendering instruction and sends the encoded OpenGL ES rendering instruction to a Linux system; the Linux system receives the encoded OpenGL ES rendering instruction, decodes it, and calls the loaded OpenGL ES rendering library; an Android application is started, a QT window is created, an EGL Surface is created according to a window identifier, the Android side performs graphics drawing, and the Linux side outputs the OpenGL ES rendering data to the EGL Surface. The invention realizes displaying an Android application on a Linux system using a QT window based on an X11 protocol or a Wayland protocol.

Description

Technical Field

[0001] The present invention relates to the field of computer operating systems, and in particular to a window display method, device, computer equipment and storage medium. Background Art

[0002] In the Linux operating system compatible with Android environment, Linux operating system manufacturers need to implement the standard OpenGL ES (OpenGL for Embedded Systems) interface when adapting the GPU (graphics card). The application can call the GPU graphics card hardware by calling the system's OpenGL ES interface. Different Linux operating systems have different OpenGL ES version numbers and different graphics servers (X11 or Wayland) for different GPU graphics cards. These differences in Linux operating systems lead to various graphics card compatibility issues when displaying Android application windows on Linux systems. For example:

[0003] Scenario 1: The Linux system uses the OpenGL ES interface version 2.0 to call the GPU hardware, while the Android 10 operating system relies on the OpenGL ES version 3.0 or higher. Due to version incompatibility and differences, the Linux system cannot process all OpenGL rendering requests sent by the Android system, which ultimately results in the inability to display the Android application window on the Linux system.

[0004] Scenario 2: The Linux system calls the GPU hardware OpenGL ES interface using the Wayland display server protocol. However, when displaying the Android application window, the Linux system only supports the X11 display server protocol and is not adapted to the Wayland display server protocol. As a result, the Android application window cannot be displayed on the Linux system.

[0005] The existing solutions cannot solve the problems in the above two scenarios, that is, the version number of the OpenGL ES interface on the Linux side cannot meet the requirements of running the Android system, or the window of the Android application cannot be displayed on the display server protocols of X11 and Wayland at the same time. Summary of the invention

[0006] In order to solve the problem of displaying Android applications using a QT window based on the X11 protocol or the Wayland protocol on a Linux system, the present invention provides a window display method, apparatus, computer equipment and storage medium.

[0007] In a first aspect, the present invention provides a window display method, the method comprising:

[0008] Load the OpenGL ES rendering library;

[0009] The Android system encodes the OpenGL ES rendering instructions and sends the encoded OpenGL ES rendering instructions to the Linux system;

[0010] The Linux system receives the encoded OpenGL ES rendering instructions, decodes them, and calls the loaded OpenGLES rendering library;

[0011] Start the Android application, create a QT window according to the display server protocol supported by the Linux system, create an EGL Surface according to the window identifier, draw graphics on the Android side, and output OpenGL ES rendering data to EGLSurface on the Linux side.

[0012] Furthermore, before loading the OpenGL ES rendering library, the method further includes:

[0013] Check whether the Linux system supports X11 protocol OpenGL ES rendering;

[0014] Check whether the Linux system supports Wayland protocol OpenGL ES rendering;

[0015] Check whether the Linux system supports OpenGL ES version greater than or equal to 3.0;

[0016] Based on the detection results, different OpenGL ES rendering libraries are loaded.

[0017] Furthermore, the method creates a QT window according to a display server protocol supported by the Linux system, including:

[0018] Create a QT window based on the X11 protocol or Wayland protocol according to the display server protocol supported by the Linux system.

[0019] Furthermore, the Android application is started and a QT window is created according to the display server protocol supported by the Linux system, specifically:

[0020] When the Android application is started on the Linux side, the Linux side sends a message to Android, the Android side starts the application process, notifies the Linux side to start the Android application display window, and the Linux side creates a QT window according to the display server protocol supported by the Linux system.

[0021] Furthermore, an EGL Surface is created according to the window identifier, the Android side performs graphics drawing, and the Linux side outputs the OpenGL ES rendering data to the EGL Surface, specifically:

[0022] After the QT window is created, an EGL Surface bound to the window is created according to the window identifier of the QT window. The SurfaceFlinger process on the Android side performs graphics drawing and sends OpenGL ES rendering instructions to the Linux side. After receiving the instructions, the Linux side outputs the OpenGL ES rendering data to the EGL Surface.

[0023] In a second aspect, the present invention provides a window display device, the device comprising:

[0024] Memory, used to store programs;

[0025] A processor, configured to execute a program stored in the memory. When the program stored in the memory is executed, the processor is configured to execute:

[0026] Load the OpenGL ES rendering library;

[0027] The Android system encodes the OpenGL ES rendering instructions and sends the encoded OpenGL ES rendering instructions to the Linux system;

[0028] The Linux system receives the encoded OpenGL ES rendering instructions, decodes them, and calls the loaded OpenGLES rendering library;

[0029] Start the Android application, create a QT window according to the display server protocol supported by the Linux system, create an EGL Surface according to the window identifier, draw graphics on the Android side, and output OpenGL ES rendering data to EGLSurface on the Linux side.

[0030] In a third aspect, the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes:

[0031] Load the OpenGL ES rendering library;

[0032] The Android system encodes the OpenGL ES rendering instructions and sends the encoded OpenGL ES rendering instructions to the Linux system;

[0033] The Linux system receives the encoded OpenGL ES rendering instructions, decodes them, and calls the loaded OpenGLES rendering library;

[0034] Start the Android application, create a QT window according to the display server protocol supported by the Linux system, create an EGL Surface according to the window identifier, draw graphics on the Android side, and output OpenGL ES rendering data to EGLSurface on the Linux side.

[0035] In a fourth aspect, the present invention provides a storage medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors execute:

[0036] Load the OpenGL ES rendering library;

[0037] The Android system encodes the OpenGL ES rendering instructions and sends the encoded OpenGL ES rendering instructions to the Linux system;

[0038] The Linux system receives the encoded OpenGL ES rendering instructions, decodes them, and calls the loaded OpenGLES rendering library;

[0039] Start the Android application, create a QT window according to the display server protocol supported by the Linux system, create an EGL Surface according to the window identifier, draw graphics on the Android side, and output OpenGL ES rendering data to EGLSurface on the Linux side.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] In a Linux-compatible Android environment, the problem of displaying Android applications using QT windows based on the X11 protocol or Wayland protocol on Linux systems can be completely solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a window display method provided by an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of the composition of a window display device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] For environments that are compatible with running Android on the Linux operating system, the Android runtime environment runs in a container.

[0046] Through container technology, a new Linux system (including other systems customized based on Linux system, such as Android) can be run on the Linux system. The container isolates the client system and the host system from each other. Different from the traditional solution based on virtual machine technology, in container technology, the client system and the host system share CPU, memory, storage and other resources, and the resource overhead is very low. In addition, through proper configuration, the client system can also directly access the hardware resources of the host, further improving the performance of the client system.

[0047] Figure 1 A schematic diagram of a window display method provided by an embodiment of the present invention, the method is:

[0048] Load the OpenGL ES rendering library;

[0049] The Android system encodes the OpenGL ES rendering instructions and sends the encoded OpenGL ES rendering instructions to the Linux system;

[0050] The Linux system receives the encoded OpenGL ES rendering instructions, decodes them, and calls the loaded OpenGLES rendering library;

[0051] Start the Android application, create a QT window according to the display server protocol supported by the Linux system, create an EGL Surface according to the window identifier, draw graphics on the Android side, and output OpenGL ES rendering data to EGLSurface on the Linux side.

[0052] Explanation: EGL Surface is a memory area used to store images, which can be understood as a backend rendering target window. EGL is the interface between OpenGL ES and the underlying Native platform window system.

[0053] Furthermore, before loading the OpenGL ES rendering library, the method further includes:

[0054] Check whether the Linux system supports X11 protocol OpenGL ES rendering;

[0055] Check whether the Linux system supports Wayland protocol OpenGL ES rendering;

[0056] Check whether the Linux system supports OpenGL ES version greater than or equal to 3.0;

[0057] Based on the detection results, different OpenGL ES rendering libraries are loaded.

[0058] Specifically, it is detected whether the Linux system supports GPU hardware OpenGL ES rendering of the X11 protocol.

[0059] After starting the OpenGL ES proxy program opengl_proxy, it will first detect whether the current Linux system supports hardware OpenGL ES rendering of the X11 protocol. It mainly completes the detection by calling the system's EGL interface.

[0060] The relevant code is as follows:

[0061] #include<EGL / egl.h>

[0062] #include<EGL / eglext.h>

[0063] EGLDisplay mDisplay=eglGetDisplay(EGL_DEFAULT_DISPLAY)

[0064] eglInitialize(mDisplay,nullptr,nullptr);

[0065] const char*extensions=eglQueryString(mDisplay,

[0066] EGL_EXTENSIONS

[0067] EGLDisplay is a general data type associated with the system's physical screen. For a PC, Display is the handle of the display. Get the EGLDisplay of the X11 protocol through the eglGetDisplay function with the EGL_DEFAULT_DISPLAY parameter, and then initialize EGLDisplay through eglInitialize. Finally, get the EGL extension implemented by EGLDisplay through the eglQueryString function with the EGL_EXTENSIONS parameter. If the returned string contains EGL_EXT_platform_x11 or EGL_KHR_platform_x11, it means that the Linux system supports GPU hardware OpenGL ES rendering of the X11 protocol.

[0068] Check whether the Linux system supports GPU hardware OpenGL ES rendering of the Wayland protocol.

[0069] If it is detected that the Linux system does not support GPU hardware OpenGL ES rendering of the X11 protocol, it will start to detect whether the Linux system supports GPU hardware OpenGL ES rendering of the Wayland protocol. The detection is mainly completed by calling the system's EGL interface, but there are some differences in the called APIs. The main code is as follows:

[0070] #include<EGL / egl.h>

[0071] #include<EGL / eglext.h>

[0072] static PFNEGLGETPLATFORMDISPLAYEXTPROC get_platform_display

[0073] =

[0074] (PFNEGLGETPLATFORMDISPLAYEXTPROC)eglGetProcAddress("eglGet

[0075] PlatformDisplayEXT");

[0076] EGLDisplay mDisplay=

[0077] get_platform_display(EGL_PLATFORM_WAYLAND_KHR,

[0078] m_wayland_display,NULL);

[0079] eglInitialize(mDisplay,nullptr,nullptr);

[0080] const char*extensions=eglQueryString(mDisplay,

[0081] EGL_EXTENSIONS

[0082] Get the symbolic address of eglGetPlatformDisplayEXT through the eglGetProcAddress function, and then call its symbolic address with the EGL_PLATFORM_WAYLAND_KHR parameter to get the EGLDisplay of the wayland protocol. Then initialize EGLDisplay through eglInitialize. Finally, get the EGL extension implemented by EGLDisplay through the eglQueryString function with the EGL_EXTENSIONS parameter. If the returned string contains EGL_EXT_platform_wayland or EGL_KHR_platform_wayland, it means that the Linux system supports GPU hardware OpenGL ES rendering of the wayland protocol.

[0083] Checks whether Linux supports OpenGL ES rendering with version 3.0 or greater.

[0084] If it is detected that the Linux system supports GPU hardware OpenGL ES rendering of the X11 protocol or GPU hardware OpenGL ES rendering of the wayland protocol, since the required Android version is 10.0, and Android 10.0 relies on OpenGL ES version 3.0 and above, it is also necessary to detect whether the currently supported OpenGL ES version number is greater than or equal to 3.0. The EGL interface is also called to query the OpenGL ES version. The relevant code is as follows:

[0085] eglQueryString(EGL_NO_DISPLAY,EGL_VERSION)

[0086] After obtaining EGLDisplay and calling the eglInitialize function to initialize EGLDisplay, you can use the eglQueryString function with the EGL_VERSION parameter to obtain the OpenGL ES version number to determine whether the OpenGL ES version number is greater than or equal to 3.0.

[0087] Load different OpenGL ES rendering libraries based on the detection results.

[0088] The OpenGL ES rendering library mainly includes libEGL.so, libGLESv1.so, and libGLESv2.so. These three shared libraries implement the standard EGL and OpenGL ES interfaces upwards and directly call the GPU (graphics card) downwards. They are generally provided by GPU (graphics card) manufacturers.

[0089] If it is detected that the Linux system supports GPU hardware OpenGL ES rendering of the X11 protocol or GPU hardware OpenGL ES rendering of the wayland protocol, and the currently supported OpenGL ES version number is greater than or equal to 3.0, you can directly load the OPENGLES rendering library that comes with the Linux system.

[0090] If it is detected that the Linux system does not support GPU hardware OpenGL ES rendering of the X11 protocol and does not support GPU hardware OpenGL ES rendering of the wayland protocol, or the Linux system supports GPU hardware OpenGL ES rendering of the X11 protocol or GPU hardware OpenGL ES rendering of the wayland protocol, but the currently supported OpenGL ES version is less than 3.0, then load the library SwiftShader that uses software to simulate the EGL and OpenGL ES interfaces. SwiftShader is a software library for high-performance graphics rendering on the CPU. It uses standard EGL and OpenGL ES interfaces upward and calls the CPU instead of the GPU downward, that is, it uses software simulation instead of calling hardware resources. Therefore, SwiftShader does not rely on graphics card hardware. When it is detected that the hardware interface cannot meet the requirements, the SwiftShader library is used to complete the OpenGL ES call. SwiftShader currently supports the X11 display server protocol.

[0091] After loading the OpenGL ES rendering library, the OpenGL ES agent will initialize EGL and start a process to receive OpenGL ES instructions sent by Android. Then start the Android system, which will start a SurfaceFlinger display process. SurfaceFlinger will send OpenGL ES instructions to the OpenGLES agent to complete the display initialization of the Android system. The OpenGL ES agent will also start a process to receive Android application startup messages sent by the Android system, waiting for the Android application to start.

[0092] The Android system encodes the OpenGL ES rendering instructions and sends them to the OpenGL ES agent of the Linux system. After receiving the OpenGL ES instructions, the OpenGL ES receiving process of the OpenGL ES agent decodes the OpenGL ES instructions and calls the loaded OpenGL ES rendering library, thereby achieving the purpose of the Android system calling the Linux system hardware GPU.

[0093] The OpenGL ES rendering of the Android system is ultimately implemented through the Linux system, but a local window is still required in the Linux system, and the rendering results of OpenGL ES are output to the local window of the Linux system in order to realize the function of using the Linux system window to display Android applications.

[0094] If it is detected that the Linux system supports GPU hardware OpenGL ES rendering of the X11 protocol, you can start a QT program based on the X11 protocol. You can set the QT program's display server protocol to X11 by setting the QT_QPA_PLATFORM environment variable to xcb, or by adding the -platform xcb parameter when starting the QT program.

[0095] After starting the QT program, start a QWindow class window in the QT program, and then pass the QWindow window identifier winId to the OpenGL ES agent. The OpenGL ES agent calls the eglCreateWindowSurface function with EGLDisplay and winId as parameters to create an EGLSurface bound to QWindow. The surfaceFlinger process on the Android side will draw graphics, and at this time it will send the encoded OpenGL ES instructions to the OpenGL ES agent on the Linux side. The OpenGL ES agent will output the OpenGL ES rendered data to the EGLSurface. Since the EGLSurface is bound to the QWindow, the content in the EGLSurface will eventually be output to the local QWindow window, completing the purpose of displaying Android applications using the QT window of the X11 protocol on the Linux system.

[0096] Furthermore, the method creates a QT window according to a display server protocol supported by the Linux system, including:

[0097] Create a QT window based on the X11 protocol or Wayland protocol according to the display server protocol supported by the Linux system.

[0098] Furthermore, the Android application is started and a QT window is created according to the display server protocol supported by the Linux system, specifically:

[0099] When the Android application is started on the Linux side, the Linux side sends a message to Android, the Android side starts the application process, notifies the Linux side to start the Android application display window, and the Linux side creates a QT window according to the display server protocol supported by the Linux system.

[0100] Furthermore, an EGL Surface is created according to the window identifier, the Android side performs graphics drawing, and the Linux side outputs the OpenGL ES rendering data to the EGL Surface, specifically:

[0101] After the QT window is created, an EGL Surface bound to the window is created according to the window identifier of the QT window. The SurfaceFlinger process on the Android side performs graphics drawing and sends OpenGL ES rendering instructions to the Linux side. After receiving the instructions, the Linux side outputs the OpenGL ES rendering data to the EGL Surface.

[0102] Figure 2 A schematic diagram of a window display device provided by an embodiment of the present invention, the device comprising:

[0103] Memory, used to store programs;

[0104] The processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the steps of the above method.

[0105] In addition, an embodiment of the present invention provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the above method.

[0106] An embodiment of the present invention provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the above method.

[0107] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A window display method, characterized in that: The method is applied in an environment where Linux is compatible with Android, and the method is: S1. Detect the OpenGL ES rendering library version supported by the Linux system. According to the detection result, load different OpenGLES rendering libraries. Specifically: S11, detect whether the Linux system supports GPU hardware OpenGL ES rendering of X11 protocol, obtain EGLDisplay of X11 protocol by using eglGetDisplay function with EGL_DEFAULT_DISPLAY parameter, initialize EGLDisplay by using eglInitialize, and finally obtain EGL extension implemented by EGLDisplay by using eglQueryString function with EGL_EXTENSIONS parameter; if the returned string contains EGL_EXT_platform_x11 or EGL_KHR_platform_x11, it means that the Linux system supports GPU hardware OpenGL ES rendering of X11 protocol; if it is detected that the Linux system supports GPU hardware OpenGL ES rendering of X11 protocol, execute step S14; Otherwise, execute step S12; S12, check whether the Linux system supports GPU hardware OpenGL ES rendering of the wayland protocol, obtain the symbolic address of eglGetPlatformDisplayEXT through the eglGetProcAddress function, and then call its symbolic address with the EGL_PLATFORM_WAYLAND_KHR parameter to obtain the EGLDisplay of the wayland protocol, and then initialize the EGLDisplay through eglInitialize, and finally obtain the EGL extension implemented by EGLDisplay through the eglQueryString function with the EGL_EXTENSIONS parameter. If the returned string contains EGL_EXT_platform_wayland or EGL_KHR_platform_wayland, it means that the Linux system supports GPU hardware OpenGL ES rendering of the wayland protocol; if the Linux system supports GPU hardware OpenGL ES rendering of the wayland protocol, execute step S14; Otherwise, execute step S13; S13, detecting whether the Linux system supports OpenGL ES version greater than or equal to 3.0; S14, loading different OpenGL ES rendering libraries according to the detection results; If it is detected that the Linux system supports GPU hardware OpenGL ES rendering of the X11 protocol or GPU hardware OpenGL ES rendering of the wayland protocol, and the currently supported OpenGL ES version number is greater than or equal to 3.0, the OPENGLES rendering library that comes with the Linux system can be directly loaded; if it is detected that the Linux system does not support GPU hardware OpenGL ES rendering of the X11 protocol and does not support GPU hardware OpenGL ES rendering of the wayland protocol, or the Linux system supports GPU hardware OpenGL ES rendering of the X11 protocol or GPU hardware OpenGL ES rendering of the wayland protocol, but the currently supported OpenGL ES version number is less than 3.0, then load the library SwiftShader that uses software to simulate EGL and OpenGL ES interfaces; S2. The Android system encodes the OpenGL ES rendering instructions and sends the encoded OpenGL ES rendering instructions to the Linux system; S3, the Linux system receives the encoded OpenGL ES rendering instruction, decodes it, and calls the loaded OpenGLES rendering library; S4. When the Android application is started on the Linux side, the Linux side sends a message to Android, the Android side starts the application process, notifies the Linux side to start the Android application display window, and the Linux side creates a QT window based on the X11 protocol or the Wayland protocol according to the display server protocol supported by the Linux system; S5. Create an EGL Surface according to the window identifier, the Android side performs graphics drawing, and the Linux side outputs the OpenGL ES rendering data to the EGL Surface.

2. The window display method according to claim 1, characterized in that: Create an EGL Surface based on the window ID, draw graphics on the Android side, and output OpenGL ES rendering data to the EGL Surface on the Linux side. Specifically: After the QT window is created, an EGL Surface bound to the window is created according to the window identifier of the QT window. The SurfaceFlinger process on the Android side performs graphics drawing and sends OpenGL ES rendering instructions to the Linux side. After receiving the instructions, the Linux side outputs the OpenGL ES rendering data to the EGL Surface.

3. A window display device, characterized in that: The device comprises: Memory, used to store programs; A processor, configured to execute the program stored in the memory; when the program stored in the memory is executed, the processor is configured to execute the steps of the method as claimed in any one of claims 1 to 2.

4. A computer device, comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 2.

5. A storage medium storing computer-readable instructions, wherein when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the method according to any one of claims 1 to 2.

Images