A method and system for two-dimensional and three-dimensional integration remote rendering based on X protocol

Abstract

The application discloses a two-three-dimensional integrated remote rendering method and system based on X protocol and belongs to the technical field of computer graphics. First, compared with the original X Window remote window scheme, three-dimensional information of the application can be rendered through hardware acceleration and transmitted through coding, so that network bandwidth is small and rendering efficiency is improved. Second, compared with the original X Window remote window scheme, the application splits image information and transmits the image information through coding, so that bandwidth resources required are reduced, the influence of network bandwidth delay is small, and user experience is improved. Finally, compared with a similar VNC scheme, the remote rendering of the application provides a window operation experience consistent with local operation to users. The application utilizes the distributed characteristics of X Window, provides a two-three-dimensional window experience consistent with the local, reduces transmission bandwidth demand, and reduces the influence of network delay on rendering.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics technology, and more specifically, relates to a two-dimensional and three-dimensional integrated remote rendering method based on the X protocol. Background Technology

[0002] The X Window System is a system that creates a graphical user interface independent of system hardware and can be fully expanded over a network. X adopts a client / server architecture, consisting of an X server and an X client.

[0003] X Window remote windows are suitable for applications with a large amount of text and vector graphics, and can bring the same operating experience as local applications. However, they do not support 3D hardware acceleration, and the native X Window system was not designed to take into account the large amount of video and image information contained in the two-dimensional information transmitted by current applications, resulting in high bandwidth consumption and greater susceptibility to network latency.

[0004] Currently, remote windows in Linux systems often use a solution similar to VNC, where images are generated and captured locally on the server, then encoded and transmitted to the remote display. The advantages of this approach are that it transmits images via image encoding, requires less network bandwidth, is less affected by network latency, and improves the user experience. However, this solution is essentially a video player and cannot provide the same interactive experience as local applications. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a unified 2D / 3D remote rendering method based on the X protocol, which solves the problem that existing methods cannot simultaneously ensure a consistent interactive experience for local applications and minimize the impact of network latency on rendering.

[0006] To achieve the above objectives, in a first aspect, the present invention provides a method for integrated 2D and 3D remote rendering based on the X protocol, which is applied to a user terminal and includes:

[0007] S1. Send a display request to the application server via X11 Event;

[0008] S2. Receive text and drawing instructions via the X11 protocol; decode the received 2D video stream information and convert it into a 2D image; receive 3D video stream information via VirtualGL Client and decode it into a 3D image.

[0009] S3. Combine the text, drawing instruction information, and decoded 2D image into a new X11 instruction;

[0010] S4. Draw and display two-dimensional or three-dimensional images using VirtualGLXServer.

[0011] Preferably, the step of receiving text and drawing instruction information via the X11 protocol, and decoding the received two-dimensional video stream information into a two-dimensional image includes:

[0012] (1) After receiving the X11 protocol message, the XProxy module parses it and obtains the reserved position information of the text, drawing instruction information and two-dimensional video stream information in the original X11 protocol packet;

[0013] (2) The PicProxy module receives two-dimensional video stream information and decodes and decompresses it into rendered two-dimensional image information.

[0014] Preferably, step S3 specifically includes:

[0015] The ComMod module, based on the reserved position information of the 2D video stream parsed from the XProxy module, repackages the 2D image information decoded and decompressed by the PicProxy module, along with text and drawing instruction information, into a new X11 protocol message.

[0016] Preferably, both the two-dimensional video stream information and the three-dimensional video stream information are decoded using H.264 encoding.

[0017] To achieve the above objectives, in a second aspect, the present invention provides a method for integrated 2D and 3D remote rendering based on the X protocol. This method is applied to an application server and includes:

[0018] T1. Receives display requests sent by user terminals; the graphics application calls the Xlib library to generate 2D image rendering information of the requested object based on the X11 protocol; and generates 3D graphics rendering information of the requested object through the OpenGL and GLX interfaces.

[0019] T2. Hijack two-dimensional graphic instruction information and split the information into text, drawing instruction information, and image information;

[0020] T3. Transmits text and drawing instructions to the user terminal via the X11 protocol; encodes image information and sends it to the user terminal; encodes 3D graphics rendering information and sends it to the user terminal via VirtualGL XServer.

[0021] Preferably, step T2 includes:

[0022] T21. Using the SplitMod module to hijack the two-dimensional graphics instruction stream information generated by the application through Xlib;

[0023] The T22.SplitMod module parses the two-dimensional graphics command stream information according to the standard X11 protocol, and splits and reassembles the information into two new X11 protocol messages: one is the text command stream information, which reserves the position information of the image information; the other is the image information.

[0024] Preferably, in step T3, the text command stream information is transmitted through the XProxy module; the image information is encoded and then transmitted through the PicProxy module.

[0025] Preferably, after obtaining the two-dimensional image information, the PicProxy module compresses the two-dimensional image information using H264 encoding, reassembles it into an X11 protocol packet, and then sends it to the user terminal; after obtaining the three-dimensional image information, the VirtualGLXServer uses H264 encoding to assemble it into an X11 protocol packet and then sends it to the user terminal.

[0026] To achieve the above objectives, in a third aspect, the present invention provides a two-dimensional and three-dimensional integrated remote rendering system based on the X protocol, including a user terminal and an application server.

[0027] The user terminal includes: a processor and a memory; the memory is used to store computer execution instructions; the processor is used to execute the computer execution instructions, causing the method described in the first aspect to be executed;

[0028] The application server includes a processor and a memory; the memory is used to store computer execution instructions; the processor is used to execute the computer execution instructions, causing the method described in the second aspect to be executed.

[0029] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0030] This invention provides a unified 2D / 3D remote rendering method and system based on the X protocol. First, compared to the native X Window remote windowing solution, the 3D information in this invention can be rendered using 3D hardware acceleration and transmitted via encoding, thus consuming less network bandwidth and improving rendering efficiency. Second, compared to the native X Window remote windowing solution, this invention separates the 2D video image information and transmits it via encoding, reducing the required bandwidth resources and minimizing the impact of network latency, thereby improving the user experience. Finally, compared to solutions like VNC, the remote rendering of this invention provides users with a window operation experience consistent with local operation. This invention utilizes the distributed nature of X Window to provide a consistent 2D / 3D windowing experience with local operation; it reduces transmission bandwidth requirements and minimizes the impact of network latency on rendering. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of a two-dimensional and three-dimensional integrated remote rendering method based on the X protocol provided in an embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0033] This invention provides a method for integrated 2D and 3D remote rendering based on the X protocol. This method is applied to a user terminal and includes:

[0034] S1. Send a display request to the application server via X11 Event, for example, keyboard and mouse events;

[0035] S2. Receive text and drawing instructions via the X11 protocol; decode the received 2D video stream information and convert it into a 2D image; receive 3D video stream information via VirtualGL Client and decode it into a 3D image.

[0036] S3. Combine the text, drawing instruction information, and decoded 2D image into a new X11 instruction;

[0037] S4. Draw and display two-dimensional or three-dimensional images using VirtualGLXServer.

[0038] Preferably, the step of receiving text and drawing instruction information via the X11 protocol, and decoding the received two-dimensional video stream information into a two-dimensional image includes:

[0039] (1) After receiving the X11 protocol message, the XProxy module parses it and obtains the reserved position information of the text, drawing instruction information and two-dimensional video stream information in the original X11 protocol packet;

[0040] (2) The PicProxy module receives two-dimensional video stream information and decodes and decompresses it into rendered two-dimensional image information.

[0041] Preferably, step S3 specifically includes:

[0042] The ComMod module, based on the reserved position information of the 2D video stream parsed from the XProxy module, repackages the 2D image information decoded and decompressed by the PicProxy module, along with text and drawing instruction information, into a new X11 protocol message.

[0043] Preferably, both the two-dimensional video stream information and the three-dimensional video stream information are decoded using H.264 encoding.

[0044] This invention provides a method for integrated 2D and 3D remote rendering based on the X protocol. This method is applied to an application server and includes:

[0045] T1. Receives display requests sent by user terminals; the graphics application calls the Xlib library to generate 2D image rendering information of the requested object based on the X11 protocol; and generates 3D graphics rendering information of the requested object through the OpenGL and GLX interfaces.

[0046] T2. Hijack two-dimensional graphic instruction information and split the information into text, drawing instruction information, and image information;

[0047] T3. Transmits text and drawing instructions to the user terminal via the X11 protocol; encodes image information and sends it to the user terminal; encodes 3D graphics rendering information and sends it to the user terminal via VirtualGL XServer.

[0048] Preferably, step T2 includes:

[0049] T21. Using the SplitMod module to hijack the two-dimensional graphics instruction stream information generated by the application through Xlib;

[0050] The T22.SplitMod module parses the two-dimensional graphics command stream information according to the standard X11 protocol, and splits and reassembles the information into two new X11 protocol messages: one is the text command stream information, which reserves the position information of the image information; the other is the image information.

[0051] Preferably, in step T3, the text command stream information is transmitted through the XProxy module; the image information is encoded and then transmitted through the PicProxy module.

[0052] Preferably, after obtaining the two-dimensional image information, the PicProxy module compresses the two-dimensional image information using H264 encoding, reassembles it into an X11 protocol packet, and then sends it to the user terminal; after obtaining the three-dimensional image information, the VirtualGLXServer uses H264 encoding to assemble it into an X11 protocol packet and then sends it to the user terminal.

[0053] This invention provides a 2D / 3D integrated remote rendering system based on the X protocol, comprising a user terminal and an application server; the user terminal includes a processor and a memory; the memory is used to store computer execution instructions; the processor is used to execute the computer execution instructions, causing the aforementioned method to be executed; the application server includes a processor and a memory; the memory is used to store computer execution instructions; the processor is used to execute the computer execution instructions, causing the aforementioned method to be executed.

[0054] Example

[0055] Figure 1 This diagram illustrates a method for integrated 2D and 3D remote rendering based on the X protocol, as provided in an embodiment of the present invention. It employs a client-server architecture based on the X window and uses the OpenGL graphics library for 3D rendering, separating and compressing the video image information from the 2D data for transmission. The specific working process is as follows: Figure 1 As shown:

[0056] Step 1: The graphics application calls the Xlib library to generate a 2D graphics instruction stream based on the X11 protocol, and calls the OpenGL and GLX interfaces to generate 3D graphics rendering information.

[0057] Step 2: Obtain the two-dimensional graphics command stream information by preloading the SplitMod module, and split the information into text command stream information and image information.

[0058] Step 3: The SplitMod module transmits the text command stream information through the XProtocol Proxy module via the X11 protocol, and reserves the display position information of the split image in the new X11 protocol message; the image information is transmitted as a video stream after being encoded by H264 through the PicProxy module.

[0059] Step 4: The 3D graphics rendering information is encoded into a video stream and transmitted via the VirtualGL Server.

[0060] Step 5: On the user terminal side, use the corresponding agents to receive the text, drawing instructions, and image information from the two-dimensional information respectively:

[0061] (1) Use the X Protocol Proxy module (XProxy) to receive text and drawing command information via the X11 protocol;

[0062] (2) The PicProxy module receives the encoded information and decodes it into a single frame image via streaming.

[0063] (3) The ComMod module combines text, drawing instruction information and decoded image information into new X11 instructions and transmits them to the XServer for drawing.

[0064] Step 6: Receive 3D video information through VirtualGL Client, decode it into image information, and transmit it to XServer for display.

[0065] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for integrated 2D and 3D remote rendering based on the X protocol, characterized in that, This method is applied to a user terminal, and the method includes: S1. Send a display request to the application server via X11 Event; After receiving the X11 protocol message, the S2.XProxy module parses it to obtain the reserved position information of the text, drawing command information and 2D video stream information in the original X11 protocol packet; the PicProxy module receives the 2D video stream information and decodes and decompresses it into rendered 2D image information; it receives the 3D video stream information through VirtualGL Client and decodes it into a 3D image. The S3. ComMod module, based on the reserved position information of the two-dimensional video stream parsed from the XProxy module, repackages the two-dimensional image information decoded and decompressed by the PicProxy module, along with text and drawing instruction information, into a new X11 protocol message, which is used by the standard X Server as drawing instructions for parsing and drawing. S4. Draw and display two-dimensional or three-dimensional images using VirtualGL XServer.

2. The method as described in claim 1, characterized in that, Both 2D and 3D video stream information are decoded using H.264 encoding.

3. A method for integrated 2D and 3D remote rendering based on the X protocol, characterized in that, This method is applied to the application server and includes: T1. Receives display requests sent by user terminals; the graphics application calls the Xlib library to generate 2D image rendering information of the requested object based on the X11 protocol; and generates 3D graphics rendering information of the requested object through the OpenGL and GLX interfaces. T2. Using the SplitMod module to hijack the 2D graphics command stream information generated by the application via Xlib; the SplitMod module parses the 2D graphics command stream information according to the standard X11 protocol, and splits and reassembles the information into two new X11 protocol messages: one is text command stream information, which reserves the position information of image information; the other is image information, wherein the text command stream information retains the text and vector graphics drawing commands in the X11 protocol; T3. Transmits text and drawing instructions to the user terminal via the X11 protocol; encodes image information and sends it to the user terminal; encodes 3D graphics rendering information and sends it to the user terminal via VirtualGL XServer.

4. The method as described in claim 3, characterized in that, In step T3, the text command stream information is transmitted through the XProxy module; the image information is encoded and then transmitted through the PicProxy module.

5. The method as described in claim 4, characterized in that, After obtaining the 2D image information, the PicProxy module compresses the 2D image information using H264 encoding, reassembles it into an X11 protocol packet, and then sends it to the user terminal; after obtaining the 3D image information, the VirtualGL XServer uses H264 encoding to assemble it into an X11 protocol packet and then sends it to the user terminal.

6. A two-dimensional and three-dimensional integrated remote rendering system based on the X protocol, characterized in that, This includes user terminals and application servers; The user terminal includes: a processor and a memory; the memory is used to store computer execution instructions; the processor is used to execute the computer execution instructions, causing the method of claim 1 or 2 to be executed; The application server includes: a processor and a memory; the memory is used to store computer execution instructions; the processor is used to execute the computer execution instructions, causing the method described in any one of claims 3 to 5 to be executed.

Images