Application Screen Processing Method, Apparatus and System

Abstract

The present invention discloses an application screen processing method, apparatus and system. Among them, the application screen processing system includes: a predefined image processor and a terminal session management module. The predefined image processor includes a front end and a back end. Wherein, the front end of the predefined image processor is used to send an encapsulated rendering instruction stream; the back end of the predefined image processor is used to receive the encapsulated rendering instruction stream, determine an off-screen rendering session according to the encapsulated rendering instruction stream, and perform off-screen rendering based on a physical image processor; the terminal session management module is used to collect the application screen that has completed off-screen rendering from the off-screen rendering buffer corresponding to the off-screen rendering session and distribute it to the terminal. The present invention solves the technical problem that the distribution of the application screen of the virtual machine in the related art is likely to cause waste of the physical resources of the server or a relatively high cost.

Description

Technical Field

[0001] The present invention relates to the technical field of virtual machines, and in particular, to an application screen processing method, apparatus, and system. Background Art

[0002] Cloud desktop applications based on servers + zero terminals are becoming more and more widespread. For example, Figure 1 As shown, generally, the number of virtual machines to be started on the server side is the same as the number of output channels required on the user side, that is, a one-to-one correspondence.

[0003] Each virtual machine needs to occupy corresponding server configurations. For example, for daily desktop office work, a configuration of 4-core CPU + 4GB memory is required. In some specific scenarios, mainly for output display, such as playing a video on one channel, showing slides on one channel, and looping static pictures on one channel. If the architecture shown in Figure 1 is adopted, three virtual machines need to be configured, and the CPU and memory configuration requirements of the server also increase correspondingly, causing relatively large waste of the physical resources of the server.

[0004] In response to the above problems, another solution is also adopted, as shown in Figure 2 As shown. In the client operating system, it is distributed in units of applications. This solution only requires one virtual machine to correspond to multiple display terminals, saving server resources.

[0005] However, this scenario is only applicable to lightweight applications. If the target application needs to use the graphics card for 3D rendering, a physical graphics card needs to be configured on the server side. Whether it is graphics card passthrough or a dedicated graphics card virtualization solution similar to Nvidia / AMD vGPU, the cost is relatively high.

[0006] In response to the problem that the application screen distribution of virtual machines in the above related technologies is likely to cause waste of server physical resources or high costs, no effective solution has been proposed yet. Summary of the Invention

[0007] Embodiments of the present invention provide an application screen processing method, apparatus, and system to at least solve the technical problem that the application screen distribution of virtual machines in related technologies is likely to cause waste of server physical resources or high costs.

[0008] According to another aspect of the embodiments of the present invention, there is provided an application screen processing system, including: a predefined image processor and a terminal session management module. The predefined image processor includes a front end and a back end. Among them, the front end of the predefined image processor is used to send the encapsulated rendering instruction stream; the back end of the predefined image processor is used to receive the encapsulated rendering instruction stream, determine an off-screen rendering session according to the encapsulated rendering instruction stream, and perform off-screen rendering based on a physical image processor; the terminal session management module is used to collect the application screen that has completed off-screen rendering from the off-screen rendering buffer corresponding to the off-screen rendering session and distribute it to the terminal.

[0009] Optionally, the front end of the predefined image processor further includes: an initiation module, configured to initiate a rendering request by the application of the virtual machine invoking a rendering application programming interface; a creation module, configured to create a rendering session for the application according to the rendering request; and an obtaining module, configured to encapsulate the rendering instruction stream, the application to which the rendering instruction stream belongs, and the rendering session according to a preset encapsulation format to obtain the encapsulated rendering instruction stream.

[0010] Optionally, the obtaining module includes: a marking unit, configured to mark the application and the rendering session to which the rendering instruction stream belongs, where the application ID is used to identify the application, and the rendering session ID is used to identify the rendering session; and an encapsulation unit, configured to encapsulate according to the application ID associated with the application to which the rendering instruction stream belongs, the rendering session ID associated with the rendering session to which the rendering instruction stream belongs, and the rendering instruction stream to obtain the encapsulated rendering instruction stream.

[0011] Optionally, the back end of the predefined image processor includes: a de-encapsulation module, configured to de-encapsulate the encapsulated rendering instruction stream by the back end of the predefined image processor to obtain the application ID, the rendering session ID, and the rendering instruction stream; a judgment module, configured to judge whether an off-screen rendering session corresponding to the rendering session of the application has been created; and a new creation module, configured to create an off-screen rendering session corresponding to the rendering session of the application when it is determined that no corresponding off-screen rendering session has been created.

[0012] Optionally, the system further includes: a generation module, configured to receive an access request sent by the terminal, identify the application to be started by the terminal according to the access request, and generate a rendering request based on the operation on the application.

[0013] According to one aspect of an embodiment of the present invention, there is also provided an application screen processing method, including: receiving a packaged rendering instruction stream, where the packaged rendering instruction stream is generated by the front end of a predefined image processor; determining an off-screen rendering session according to the packaged rendering instruction stream, and performing off-screen rendering based on a physical image processor; and sending the application screen that has completed off-screen rendering in the off-screen rendering buffer corresponding to the off-screen rendering session to a terminal.

[0014] Optionally, before determining an off-screen rendering session according to the packaged rendering instruction stream and performing off-screen rendering based on a physical image processor, it further includes: unpacking the packaged rendering instruction stream to obtain an application ID, a rendering session ID, and a rendering instruction stream, where the application ID is used to identify an application, and the rendering session ID is used to identify a rendering session; determining whether an off-screen rendering session corresponding to the rendering session of the application has been created; and creating an off-screen rendering session corresponding to the rendering session of the application when it is determined that no corresponding off-screen rendering session has been created.

[0015] According to another aspect of an embodiment of the present invention, there is also provided an application screen processing device, including: a receiving module, configured to receive a packaged rendering instruction stream, where the packaged rendering instruction stream is generated by the front end of a predefined image processor; a processing module, configured to determine an off-screen rendering session according to the packaged rendering instruction stream and perform off-screen rendering based on a physical image processor; and a distribution module, configured to send the application screen that has completed off-screen rendering in the off-screen rendering buffer corresponding to the off-screen rendering session to a terminal.

[0016] According to another aspect of an embodiment of the present invention, there is also provided a computer-readable storage medium, where the computer-readable storage medium includes a stored program, and when the program runs, it controls a device where the computer-readable storage medium is located to execute the application screen processing method described in any one of the above.

[0017] According to another aspect of an embodiment of the present invention, there is also provided a processor, where the processor is used to run a program, and when the program runs, it executes the application screen processing method described in any one of the above.

[0018] In an embodiment of the present invention, the application screen processing system includes a predefined image processor and a terminal session management module. The predefined image processor includes a front end and a back end. Specifically, the front end of the predefined image processor is configured to send an encapsulated rendering instruction stream; the back end of the predefined image processor is configured to receive the encapsulated rendering instruction stream, determine an off-screen rendering session based on the encapsulated rendering instruction stream, and perform off-screen rendering based on a physical image processor; the terminal session management module is configured to collect the application screen that has completed off-screen rendering from the off-screen rendering buffer corresponding to the off-screen rendering session and distribute it to the terminal. By using this application screen processing system, multiple applications can be run simultaneously on one virtual machine, and all rendering tasks of one application are performed for off-screen rendering on one physical image processor, and the application screen required by the terminal in the off-screen rendering buffer is sent to the terminal, thereby achieving the technical effect of saving the physical resources of the application screen distribution server of the virtual machine and reducing related costs, and further solving the technical problem in the related art that the distribution of the application screen of the virtual machine is likely to cause waste of the physical resources of the server or high costs. Description of the Drawings

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings:

[0020] Figure 1 is an architecture diagram of starting multiple virtual machines on the server side according to the prior art;

[0021] Figure 2 is an architecture diagram of starting one virtual machine on the server side according to the prior art;

[0022] Figure 3 is a schematic diagram of an application screen processing system according to an embodiment of the present invention;

[0023] Figure 4 is an overall block diagram of an application screen processing system according to an optional embodiment of the present invention;

[0024] Figure 5 is a flowchart of an application screen processing method according to an optional embodiment of the present invention;

[0025] Figure 6 is a flowchart of an application screen processing method according to an embodiment of the present invention;

[0026] Figure 7 is a schematic diagram of an application screen processing device according to an embodiment of the present invention. Detailed Embodiments

[0027] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment 1

[0030] According to another aspect of the embodiments of the present invention, an application screen processing system is provided. Figure 3 is a schematic diagram of the application screen processing system according to the embodiments of the present invention. As Figure 3 shown, the application screen processing system includes: a predefined image processor 30 and a terminal session management module 32. The predefined image processor 30 includes a front end 301 and a back end 303. Among them, the front end 301 of the predefined image processor 30 is used to send the encapsulated rendering instruction stream; the back end 303 of the predefined image processor 30 is connected to the front end 301 of the above-mentioned predefined image processor 30, and is used to receive the encapsulated rendering instruction stream, determine an off-screen rendering session according to the encapsulated rendering instruction stream, and perform off-screen rendering based on a physical image processor; the terminal session management module 32 is connected to the back end 303 of the above-mentioned predefined image processor 30, and is used to collect the application screen that has completed off-screen rendering from the off-screen rendering buffer corresponding to the off-screen rendering session and distribute it to the terminal.

[0031] The front end of the above-mentioned predefined image processor runs in the virtual machine Guest OS, and the back end of the above-mentioned predefined image processor and the terminal session management module run in the host machine Host OS. Among them, the terminal session management module can be used for session management of the terminal, including operations such as access requests, capturing, encoding, and transmitting the screens of the requested applications.

[0032] In the above-described embodiment, the application screen processing system can determine an off-screen rendering session based on the encapsulated rendering instruction stream, and use the physical image processor to perform off-screen rendering. Subsequently, the application screen that has completed off-screen rendering in the off-screen rendering buffer corresponding to the off-screen rendering session is sent to the terminal. This enables multiple applications to run simultaneously on a single virtual machine, and all rendering tasks of one application are performed for off-screen rendering on a single physical image processor. Then, the application screen required by the terminal in the off-screen rendering buffer is sent to the terminal, thereby achieving the technical effect of saving the physical resources of the application screen distribution server of the virtual machine and reducing related costs. Furthermore, it solves the technical problem in the related art that the distribution of application screens of virtual machines easily causes waste of the physical resources of the server or relatively high costs.

[0033] Optionally, the front end 301 of the above-mentioned predefined image processor 30 further includes: an initiation module for the application of the virtual machine to call the rendering application programming interface to initiate a rendering request; a creation module for creating a rendering session for the application according to the rendering request; and an obtaining module for encapsulating the rendering instruction stream and the application and rendering session to which the rendering instruction stream belongs according to a preset encapsulation format to obtain the encapsulated rendering instruction stream.

[0034] Among them, one application can create at least one rendering session. The front end of the above-mentioned predefined image processor can record the subordinate relationship between the application and the rendering session in the manner of application ID + rendering session ID.

[0035] In an alternative embodiment, the application of the virtual machine can call the rendering application programming interface to initiate a rendering request, then create a rendering session for the application according to the rendering request, and further encapsulate the rendering instruction stream and the application and rendering session to which the rendering instruction stream belongs according to a preset encapsulation format to obtain the encapsulated rendering instruction stream. It should be noted that the above-mentioned preset encapsulation format includes but is not limited to application ID + rendering session ID + original rendering instruction stream.

[0036] Optionally, the obtaining module includes: a marking unit for marking the application and the rendering session to which the rendering instruction stream belongs, where the application ID is used to identify the application and the rendering session ID is used to identify the rendering session; and an encapsulation unit for encapsulating according to the application ID associated with the application to which the rendering instruction stream belongs, the rendering session ID associated with the rendering session to which the rendering instruction stream belongs, and the rendering instruction stream to obtain the encapsulated rendering instruction stream.

[0037] In an alternative embodiment, when encapsulating the rendering instruction stream and the application and rendering session to which the rendering instruction stream belongs according to a preset encapsulation format, it is necessary to mark the application and rendering session to which the rendering instruction stream belongs. Among them, the application ID is used to identify the application, and the rendering session ID is used to identify the rendering session. Then, encapsulation is performed based on the application ID associated with the application to which the rendering instruction stream belongs, the rendering session ID associated with the rendering session to which the rendering instruction stream belongs, and the rendering instruction stream, and the encapsulated rendering instruction stream is obtained.

[0038] Optionally, the backend 303 of the predefined image processor 30 includes: a decompression module, configured to decompress the encapsulated rendering instruction stream of the backend of the predefined image processor to obtain the application ID, the rendering session ID, and the rendering instruction stream; a determination module, configured to determine whether a corresponding off-screen rendering session has been created for the rendering session of the application; and a creation module, configured to create a corresponding off-screen rendering session for the rendering session of the application when it is determined that no corresponding off-screen rendering session has been created.

[0039] In an alternative embodiment, the encapsulated rendering instruction stream can be decompressed to obtain the rendering session of the application, and then it is further determined whether a corresponding off-screen rendering session has been created for the rendering session of the application. When it is determined that no corresponding off-screen rendering session has been created, a corresponding off-screen rendering session for the rendering session of the application is created; when it is determined that a corresponding off-screen rendering session has been created, the off-screen rendering session is directly used. After determining the off-screen rendering session, off-screen rendering can be performed based on the physical image processor, and the application screen after off-screen rendering is stored in the corresponding off-screen rendering buffer.

[0040] Optionally, the above system further includes: a generation module, configured to receive an access request sent by the terminal, identify the application to be launched by the terminal according to the access request, and generate a rendering request based on the operation on the application.

[0041] In the specific implementation process, after receiving the access request sent by the terminal, the application to be launched by the terminal can be identified according to the access request, and a rendering request can be generated based on the operation on the application.

[0042] A detailed description of an alternative embodiment of the present invention will be given below.

[0043] Figure 4 is an overall block diagram of an application screen processing system according to an alternative embodiment of the present invention, as Figure 4As shown in the figure, the software-defined GPU (equivalent to the above-mentioned predefined image processor) is divided into a front end and a back end. The front end runs in the virtual machine Guest OS. When the virtual machine Guest OS starts, it is enumerated as a 3D graphics card. The applications in the virtual machine use it as a physical graphics card normally. The front end is responsible for managing all application rendering sessions and sending them to the back end in the form of a combination of application ID + rendering session ID + rendering instruction stream. The back end runs in the host machine Host OS. After receiving and unpacking the rendering instructions, the back end creates corresponding off-screen rendering buffers for each application and completes the rendering based on the physical GPU. The terminal session management module runs in the host machine Host OS and is responsible for managing the terminal sessions, including access requests, capturing, encoding, and transmitting the screens of the requested applications. In the present invention, the host machine also includes a terminal session management module, and this management module stores a terminal session management table.

[0044] Figure 5 is a flowchart of an application screen processing method according to an optional embodiment of the present invention. As Figure 5 shown, the specific implementation steps are as follows:

[0045] Step 501, the virtual machine application calls the rendering application programming interface (Application Programming Interface, abbreviated as API) to initiate a rendering request.

[0046] Step 502, the software-defined GPU front-end module creates a rendering session for the application, that is, a rendering context.

[0047] An application may create one rendering session or multiple rendering sessions. This is determined by the implementation of the application itself. The software-defined GPU front-end module records the subordinate relationship between the application and the rendering session in the form of application ID + rendering session ID.

[0048] Taking OpenGL as an example, the rendering session is also the rendering context Context. Context records all the information required for OpenGL rendering. It can be understood as a large structure, which records the colors used for the current drawing, whether there is lighting calculation, and the enabled light sources, etc., which are very many states and state attributes set by using OpenGL function calls.

[0049] Step 503, the software-defined GPU front-end module encapsulates the rendering instruction stream.

[0050] When encapsulating, mark the application and the rendering session to which the rendering instruction stream belongs.

[0051] For example, the encapsulation format: application ID + rendering session ID + original rendering instruction stream;

[0052] The application ID is a one-to-one correspondence with an application and is used to uniquely identify the application. For example, application 1 corresponds to ID1, and application 2 corresponds to ID2. The corresponding relationship is stored in the rendering session management table.

[0053] The rendering session ID is used to identify different rendering sessions, and the correspondence between the rendering session and the rendering session ID is also stored in the rendering session management table in advance.

[0054] Original rendering instruction stream:

[0055] If there are multiple rendering instruction streams, the encapsulation format is as follows:

[0056] Application ID1+rendering session ID1+original rendering instruction stream 1;

[0057] Application ID1+rendering session ID2+original rendering instruction stream 2;

[0058] Each of the multiple rendering sessions corresponding to the application ID1 is encapsulated into an independent instruction stream.

[0059] Step 504: the software-defined GPU front-end module sends the packaged rendering instruction stream to the back-end of the software-defined GPU.

[0060] Step 505: The software-defined GPU backend module receives a rendering instruction stream.

[0061] Step 506, the software-defined GPU backend module unpacks the rendering instruction stream, and obtains the application ID, the rendering session ID, and the original rendering instruction stream in sequence.

[0062] Step 507: the software-defined GPU backend module determines whether the rendering session of the application has created a corresponding off-screen rendering session, and if not, creates a new one.

[0063] It should be noted that the above creation of an off-screen rendering session is to save the management process of the foreground window in the host machine, and secondly, because in the actual running environment, the host machine generally provides a windowless system environment. Therefore, off-screen rendering can well solve the problem of a windowless system during rendering.

[0064] Step 508: The software-defined GPU backend module performs rendering based on the physical GPU.

[0065] Step 509: The terminal session management module processes the terminal access request.

[0066] Step 510: The terminal session management module collects images from the corresponding off-screen rendering buffer according to the application requested by the terminal.

[0067] Step 511, the terminal session management module encodes and sends the original screen.

[0068] If multiple display devices request the screen of the same application instance, the virtual machine only starts one application instance. After the terminal session management module completes the screen encoding process, it sends them to multiple display devices respectively.

[0069] If multiple display devices request the screens of different applications, the virtual machine starts multiple applications, such as Application 1, Application 2, and Application 3. The software-defined GPU backend creates off-screen rendering buffers for Application 1, Application 2, and Application 3 respectively and completes off-screen rendering. The terminal session management module captures, encodes, and sends the screens from different off-screen rendering buffers to the corresponding display devices respectively.

[0070] If the same application is independently displayed separately, it is regarded as a scenario of multiple applications. For example, Display 1 and Display 2 independently display different interfaces of Application 1, two Application 1s are started, and two off-screen rendering buffers are created.

[0071] In the above embodiment, the virtual machine obtains 3D rendering support by loading the software-defined GPU. When the virtual machine runs a 3D application, the software-defined GPU encapsulates the rendering instructions and sinks them to the host machine, and hands them over to the physical GPU for off-screen rendering. The software-defined GPU isolates and manages each application rendering session of the virtual machine to ensure that all rendering tasks of an application are off-screen rendered on a physical GPU, and then captures the application screen from the off-screen rendering buffer and distributes it to the client. In addition, by adopting the software-defined GPU, the virtual machine can run applications that require 3D rendering; multiple 3D applications can be run on one virtual machine at the same time, and the application screens can be captured from different off-screen rendering buffers respectively.

[0072] Embodiment 2

[0073] According to an embodiment of the present invention, an embodiment of an application screen processing method is further provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0074] Figure 6 is a flowchart of an application screen processing method according to an embodiment of the present invention. As Figure 6 shown, the application screen processing method includes the following steps:

[0075] Step S602, receive the encapsulated rendering instruction stream, where the encapsulated rendering instruction stream is generated by the front end of a predefined image processor;

[0076] The above-mentioned predefined image processor includes but is not limited to a software-defined image processor. It should be noted that the above-mentioned predefined image processor includes a front end and a back end, wherein the front end runs in the virtual machine Guest OS and the back end runs in the host machine Host OS.

[0077] In an optional embodiment, the front end of the predefined image processor can generate an encapsulated rendering instruction stream, wherein the encapsulation format of the encapsulated rendering instruction stream adopts application ID+rendering session ID+rendering instruction stream; the back end of the predefined image processor is communicatively connected with the front end of the predefined image processor, and the back end can be used to receive the encapsulated rendering instruction stream.

[0078] It should be noted that the front-end and back-end of the predefined image processor generally run on the same host, and the communication process can be completed through memory address mapping, thereby avoiding actual copying.

[0079] Step S604, determining an off-screen rendering session according to the encapsulated rendering instruction stream, and performing off-screen rendering based on a physical image processor;

[0080] The above off-screen rendering session is to save the management process of the foreground window in the host machine, and generally in the actual operating environment, the host machine provides a windowless system environment.

[0081] Step S606: Send the application screen that has completed off-screen rendering in the off-screen rendering buffer corresponding to the off-screen rendering session to the terminal.

[0082] In an optional implementation, the application screen that has completed off-screen rendering in the corresponding off-screen rendering buffer may be sent to the corresponding terminal according to the application requested by the terminal.

[0083] Through the above steps, it is possible to determine the off-screen rendering session through the encapsulated rendering instruction stream, and use the physical image processor to perform off-screen rendering, and then send the application screen that has completed off-screen rendering in the off-screen rendering buffer corresponding to the off-screen rendering session to the terminal, so that multiple applications can be run simultaneously on one virtual machine, and all rendering tasks of an application are off-screen rendered on a physical image processor, and the application screen required by the terminal in the off-screen rendering buffer is sent to the terminal, thereby achieving the purpose of saving the physical resources of the virtual machine's application screen distribution server and reducing related costs, and thus solving the technical problem that the application screen distribution of the virtual machine in the related technology is likely to waste the server's physical resources or cause high costs.

[0084] Optionally, before determining the off-screen rendering session based on the encapsulated rendering instruction stream and performing off-screen rendering based on the physical image processor, the method further includes: unpacking the encapsulated rendering instruction stream, obtaining the application ID, the rendering session ID, and the rendering instruction stream, wherein the application ID is used to identify the application and the rendering session ID is used to identify the rendering session; determining whether the rendering session of the application has created a corresponding off-screen rendering session; and if it is determined that the corresponding off-screen rendering session has not been created, creating a new off-screen rendering session corresponding to the rendering session of the application.

[0085] In an optional implementation, the encapsulated rendering instruction stream can be unpacked to obtain the rendering session of the application, and then determine whether the rendering session of the application has created a corresponding off-screen rendering session. If it is determined that the corresponding off-screen rendering session has not been created, a new off-screen rendering session corresponding to the rendering session of the application is created; if it is determined that the corresponding off-screen rendering session has been created, the off-screen rendering session is directly used. After the off-screen rendering session is determined, off-screen rendering can be performed based on the physical image processor, and the application screen that has completed the off-screen rendering is stored in the corresponding off-screen rendering buffer.

[0086] Example 3

[0087] According to another aspect of an embodiment of the present invention, there is also provided an application screen processing device. Figure 7 is a schematic diagram of an application screen processing device according to an embodiment of the present invention. Figure 7 As shown, the application screen processing device includes: a receiving module 72, a processing module 74 and a distribution module 76. The application screen processing device is described in detail below.

[0088] A receiving module 72 is used to receive an encapsulated rendering instruction stream, wherein the encapsulated rendering instruction stream is generated by the front end of a predefined image processor; a processing module 74 is connected to the above-mentioned receiving module 72, and is used to determine an off-screen rendering session based on the encapsulated rendering instruction stream, and perform off-screen rendering based on a physical image processor; a distribution module 76 is connected to the above-mentioned processing module 74, and is used to send the application screen that has completed off-screen rendering in the off-screen rendering buffer corresponding to the off-screen rendering session to the terminal.

[0089] The above-mentioned predefined image processor includes but is not limited to a software-defined image processor. It should be noted that the above-mentioned predefined image processor includes a front end and a back end, wherein the front end runs in the virtual machine Guest OS and the back end runs in the host machine Host OS.

[0090] In an alternative embodiment, the front end of the predefined image processor may generate an encapsulated rendering instruction stream, where the encapsulation format of the encapsulated rendering instruction stream is Application ID + Rendering Session ID + Rendering Instruction Stream; the back end of the predefined image processor is communicatively connected to the front end of the predefined image processor, and the back end can be used to receive the encapsulated rendering instruction stream.

[0091] In the above embodiment, the application screen processing device may determine an off-screen rendering session through the encapsulated rendering instruction stream, and use the physical image processor to perform off-screen rendering, and then send the application screen that has completed off-screen rendering in the off-screen rendering buffer corresponding to the off-screen rendering session to the terminal, achieving the purpose of running multiple applications simultaneously on one virtual machine, and performing off-screen rendering of all rendering tasks of one application on one physical image processor, and sending the application screen required by the terminal in the off-screen rendering buffer to the terminal, thereby realizing the technical effect of saving the physical resources of the application screen distribution server of the virtual machine and reducing related costs, and further solving the technical problem that the application screen distribution of the virtual machine in the related art is likely to cause waste of the physical resources of the server or high costs.

[0092] It should be noted that the above-mentioned various modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following manner: the above-mentioned various modules can be located in the same processor; or, the above-mentioned various modules are located in different processors in any combination.

[0093] It should be noted here that the above-mentioned receiving module 72, processing module 74, and distribution module 76 correspond to steps S602 to S606 in Embodiment 2. The examples and application scenarios implemented by the above-mentioned modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 2 above.

[0094] Optionally, the above-mentioned device is further configured to, before determining an off-screen rendering session according to the encapsulated rendering instruction stream and performing off-screen rendering based on the physical image processor, decrypt the encapsulated rendering instruction stream to obtain an application ID, a rendering session ID, and a rendering instruction stream, where the application ID is used to identify an application, and the rendering session ID is used to identify a rendering session; determine whether an off-screen rendering session corresponding to the rendering session of the application has been created; and create an off-screen rendering session corresponding to the rendering session of the application when it is determined that no corresponding off-screen rendering session has been created.

[0095] Embodiment 4

[0096] According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, which includes a stored program, where, when the program runs, it controls the device where the computer-readable storage medium is located to execute the application screen processing method in any one of the above.

[0097] Example 5

[0098] According to another aspect of the embodiments of the present invention, a processor is further provided, which is used to run a program. When the program runs, it executes the application screen processing method of any one of the above.

[0099] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0100] In the above embodiments of the present invention, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0101] In the several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the units or modules can be in an electrical or other form.

[0102] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple units. The purpose of the solution of this embodiment can be achieved based on some or all of the units according to actual needs.

[0103] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0104] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0105] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. An application screen processing system, characterized in that, including: a predefined image processor and a terminal session management module, the predefined image processor including a front end and a back end, wherein, the front end of the predefined image processor is configured to send an encapsulated rendering instruction stream; the back end of the predefined image processor is configured to receive the encapsulated rendering instruction stream, determine an off-screen rendering session according to the encapsulated rendering instruction stream, and perform off-screen rendering based on a physical image processor; the terminal session management module is configured to collect an application screen that has completed off-screen rendering from an off-screen rendering buffer corresponding to the off-screen rendering session and distribute it to a terminal, the front end of the predefined image processor further includes: an initiation module configured to initiate a rendering request by an application of a virtual machine calling a rendering application programming interface; a creation module configured to create a rendering session for the application according to the rendering request, and create multiple such rendering sessions for one application, the rendering session being a rendering context; a obtaining module including: a marking unit configured to mark the application and the rendering session to which a rendering instruction stream belongs, wherein an application ID is used to identify an application, and a rendering session ID is used to identify different rendering sessions; an encapsulation unit configured to encapsulate according to the application ID associated with the application to which the rendering instruction stream belongs, the rendering session ID associated with the rendering session to which the rendering instruction stream belongs, and the rendering instruction stream, to obtain the encapsulated rendering instruction stream, and each rendering session among multiple rendering sessions corresponding to the application ID is encapsulated into an independent rendering instruction stream, and the front end of the predefined image processor records the subordinate relationship between the application and the rendering session by means of the application ID plus the rendering session ID, the back end of the predefined image processor includes: a de-encapsulation module configured to de-encapsulate the encapsulated rendering instruction stream by the back end of the predefined image processor to obtain an application ID, a rendering session ID, and a rendering instruction stream; a judgment module configured to judge whether an off-screen rendering session corresponding to a rendering session of an application has been created; a new creation module configured to newly create an off-screen rendering session corresponding to the rendering session of the application when it is determined that no corresponding off-screen rendering session has been created.

2. The system according to claim 1, wherein The system further includes: a generation module configured to receive an access request sent by a terminal, identify an application to be launched by the terminal according to the access request, and generate a rendering request based on an operation on the application.

3. A method for processing an application screen, characterized in that, including: Receive the encapsulated rendering instruction stream, where the encapsulated rendering instruction stream is generated by the front end of a predefined image processor. The front end of the predefined image processor further includes: an initiation module for an application of a virtual machine to call a rendering application programming interface to initiate a rendering request; a creation module for creating a rendering session for the application according to the rendering request, and multiple such rendering sessions are created for one application, and the rendering session is a rendering context; a obtaining module including: a marking unit for marking the application and the rendering session to which the rendering instruction stream belongs, where the application ID is used to identify the application, and the rendering session ID is used to identify different rendering sessions; an encapsulation unit for encapsulating according to the application ID associated with the application to which the rendering instruction stream belongs, the rendering session ID associated with the rendering session to which the rendering instruction stream belongs, and the rendering instruction stream to obtain the encapsulated rendering instruction stream. Each rendering session in the multiple rendering sessions corresponding to the application ID is encapsulated into an independent rendering instruction stream. The front end of the predefined image processor records the subordinate relationship between the application and the rendering session by the method of adding the rendering session ID to the application ID; the back end of the predefined image processor includes: a de-encapsulation module for the back end of the predefined image processor to de-encapsulate the encapsulated rendering instruction stream to obtain the application ID, the rendering session ID, and the rendering instruction stream; a judgment module for judging whether a corresponding off-screen rendering session has been created for the rendering session of the application; a new creation module for creating a corresponding off-screen rendering session for the rendering session of the application when it is determined that no corresponding off-screen rendering session has been created. Determine an off-screen rendering session based on the encapsulated rendering instruction stream and perform off-screen rendering based on a physical image processor. Send the application picture that has completed off-screen rendering in the off-screen rendering buffer corresponding to the off-screen rendering session to the terminal.

4. The method according to claim 3, wherein Before determining an off-screen rendering session based on the encapsulated rendering instruction stream and performing off-screen rendering based on a physical image processor, it further includes: De-encapsulate the encapsulated rendering instruction stream to obtain the application ID, the rendering session ID, and the rendering instruction stream, where the application ID is used to identify the application, and the rendering session ID is used to identify the rendering session. Judge whether a corresponding off-screen rendering session has been created for the rendering session of the application. When it is determined that no corresponding off-screen rendering session has been created, create a corresponding off-screen rendering session for the rendering session of the application.

5. An application screen processing device, characterized in that, Include: A receiving module, configured to receive the encapsulated rendering instruction stream, wherein the encapsulated rendering instruction stream is generated by the front end of a predefined image processor, and the front end of the predefined image processor further includes: an initiating module, configured to initiate a rendering request by an application of a virtual machine invoking a rendering application programming interface; a creating module, configured to create a rendering session for the application according to the rendering request, and create multiple such rendering sessions for one application, where the rendering session is a rendering context; a obtaining module, including: a marking unit, configured to mark the application and the rendering session to which the rendering instruction stream belongs, where the application ID is used to identify the application, and the rendering session ID is used to identify the rendering session; an encapsulating unit, configured to encapsulate the rendering instruction stream based on the application ID associated with the application to which the rendering instruction stream belongs, the rendering session ID associated with the rendering session to which the rendering instruction stream belongs, and the rendering instruction stream, to obtain the encapsulated rendering instruction stream, and each rendering session in the multiple rendering sessions corresponding to the application ID is encapsulated into an independent rendering instruction stream, and the front end of the predefined image processor records the subordination relationship between the application and the rendering session by the way of adding the rendering session ID to the application ID; the back end of the predefined image processor includes: a de-encapsulating module, configured to de-encapsulate the encapsulated rendering instruction stream by the back end of the predefined image processor to obtain the application ID, the rendering session ID, and the rendering instruction stream; a judging module, configured to judge whether a corresponding off-screen rendering session has been created for the rendering session of the application; a creating-new module, configured to create a corresponding off-screen rendering session for the rendering session of the application when it is determined that no corresponding off-screen rendering session has been created. A processing module, configured to determine an off-screen rendering session based on the encapsulated rendering instruction stream, and perform off-screen rendering based on a physical image processor. A distributing module, configured to send the application picture that has completed off-screen rendering in the off-screen rendering buffer corresponding to the off-screen rendering session to a terminal.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein when the program runs, it controls the device where the computer-readable storage medium is located to execute the application picture processing method according to any one of claims 3 to 4.

7. A processor, characterized in that, The processor is configured to run a program, wherein when the program runs, it executes the application picture processing method according to any one of claims 3 to 4.

Images