Buffer zero-copy rendering and display method based on gfxstream

Abstract

The invention discloses a buffer zero-copy rendering and display method based on gfxstream, which is applied between a virtual machine guest and a physical machine host, and comprises the following steps: initializing a virtio GPU (Graphics Processing Unit) to establish a communication pipeline between the virtual machine guest and the physical machine host; a buffer 1 is applied through the virtual machine guest, a Scatter-Gather list A of the buffer 1 is obtained, and the Scatter-Gather list A is transmitted to the physical machine host; the buffer 1 is converted into dma-buffer 1 through the physical machine host, and the dma-buffer 1 is imported into an OpenGL (Open Graphics Library); applying for a buffer 2 through the virtual machine guest, and binding the buffer 2 to a texture to complete texture initialization; and sending a rendering instruction and a display instruction to the physical machine host through the virtual machine guest to complete data rendering and display. According to the method, the physical memory distribution of the guest of the virtual machine is described by using the Scatter-Gather list, and the Scatter-Gather list is converted into the continuous dma-buffer at the host end of the physical machine, so that the guest virtual machine and the host physical machine share the same physical memory, the zero-copy transmission of the buffer is realized, and the zero-copy display is further realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a gfxstream-based buffer zero-copy rendering and display method. Background Art

[0002] gfxstream is a set of open source GPU virtualization technologies developed by Google. It is implemented based on OpenGL API forwarding and is commonly used in Android studio, Qemu, Android virtual machines, Android game emulators, etc.

[0003] In the gfxstream GPU virtualization technology implementation, the guest (virtual machine) buffer needs to be rendered and displayed on the host (physical machine). This involves the issue of guest-to-host buffer transfer. Data copying or memory sharing can be used, but copying performs poorly and consumes significant CPU resources. In the gfxstream implementation architecture, gfxstream is treated as a separate domain. The framebuffer, pbuffer, eglimage, and gbmbuffer are all implemented using EGLImage in texture2d mode. Buffer transfers require copying via glTexImage2D. Each frame requires multiple, large-scale copy operations, significantly increasing the CPU burden. Summary of the Invention

[0004] The purpose of the present invention is to provide a buffer zero-copy rendering and display method based on gfxstream to solve the discrete memory and hardware import problems and realize memory sharing, thereby achieving the purpose of zero copy.

[0005] The present invention provides a gfxstream-based buffer zero-copy rendering and display method, which is applied between a virtual machine guest and a physical machine host, comprising:

[0006] Initialize the virtio GPU to establish a communication channel between the virtual machine guest and the physical machine host;

[0007] Apply for buffer1 through the virtual machine guest, obtain the Scatter-Gather list A of buffer1, and transmit the Scatter-Gather list A to the physical machine host;

[0008] Convert buffer1 to dma-buffer1 through the physical host and import it into OpenGL;

[0009] Apply for buffer2 through the virtual machine guest and bind it to texture to complete texture initialization;

[0010] The virtual machine guest sends rendering instructions and display instructions to the physical machine host to complete data rendering and display.

[0011] Furthermore, applying for buffer1 through the virtual machine guest, obtaining the Scatter-Gather list A of buffer1, and transmitting the Scatter-Gather list A to the physical machine host, including:

[0012] Applying buffer1 through the graphics memory allocator gralloc in the virtual machine guest, and generating a scatter-gather list A describing the physical memory distribution of buffer1;

[0013] Send the VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING command through the Virtio GPU protocol to transfer the Scatter-Gather list A to the physical machine host.

[0014] Furthermore, buffer1 is converted into dma-buffer1 by the physical host and imported into OpenGL, including:

[0015] Convert the Scatter-Gather list A to dma-buffer1 through the dma-heap allocator, and pass dma-buffer1 to the gfxstream rendering pipeline;

[0016] Convert dma-buffer1 to EGLImage1 recognizable by OpenGL through the EGLImage interface, and bind EGLImage1 as the rendering target of OpenGL.

[0017] Furthermore, the texture initialization is completed by applying for buffer2 through the virtual machine guest and binding it to texture, including:

[0018] Applying for buffer2 through the graphics memory allocator gralloc in the virtual machine guest, and generating a scatter-gather list B describing the physical memory distribution of buffer2;

[0019] Send the VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING command through the Virtio GPU protocol to transfer the Scatter-Gather list B to the physical machine host;

[0020] Convert the Scatter-Gather list B to dma-buffer2 via the dma-heap allocator and import it into OpenGL;

[0021] Convert buffer2 to EGLImage2 through the virtual machine guest, bind EGLImage2 to OpenGLtexture through the glEGLImageTargetTexture2DOES function, and send the binding information to the physical machine host;

[0022] The binding information is received through the physical machine host, and the dma-buffer2 of buffer2 is associated with the texture to complete the texture initialization.

[0023] Furthermore, the data rendering is completed by sending a rendering instruction to the physical machine host through the virtual machine guest, including:

[0024] Sending an OpenGL rendering command to the physical machine host through the virtual machine guest, wherein buffer1 is specified as a rendering target in the OpenGL rendering command;

[0025] Synchronize the rendering instructions of the virtual machine guest to the physical machine host and write them into buffer1;

[0026] The physical machine host parses the OpenGL rendering command and calls the OpenGL driver in the physical machine host to perform the rendering operation, uses the texture of buffer2 for mapping, and writes the rendering result into dma-buffer1 of buffer1 to complete the rendering.

[0027] Furthermore, the display is completed by sending a display instruction from the virtual machine guest to the physical machine host, including:

[0028] Sending a VIRTIO_GPU_CMD_RESOURCE_FLUSH command to the physical machine host through the virtual machine guest, wherein the VIRTIO_GPU_CMD_RESOURCE_FLUSH command includes a status of buffer1;

[0029] The DMA-buffer1 of buffer1 is sent to the display synthesizer through the physical machine host to be mixed with other layers, and is output to the screen for display through the display driver.

[0030] The present invention has at least the following beneficial effects:

[0031] The present invention ensures the reliability of data transmission and instruction synchronization by using virtio GPU, and can dynamically recycle resources at the same time; describes the physical memory distribution of the virtual machine guest by using a Scatter-Gather list, and converts the Scatter-Gather list into a continuous dma-buffer on the physical machine host side, so that the guest virtual machine and the host physical machine share the same physical memory, thereby realizing zero-copy transmission of the buffer, and then realizing zero-copy display. The present invention also realizes a double-buffer design by using multiple buffers (buffer1 and buffer2) to separate rendering and display targets. When the virtual machine guest renders to the BackBuffer, the physical machine host displays the FrontBuffer, thereby avoiding screen tearing, wherein the FrontBuffer represents the currently displayed picture and the BackBuffer represents the picture being rendered in the next frame. By converting the dma-buffer into an EGLImage that can be recognized by OpenGL, the graphics stack of the guest virtual machine and the host physical machine is seamlessly connected, thereby improving rendering efficiency.

[0032] This invention solves the copy bottleneck in traditional GPU virtualization, reduces CPU load, improves frame rate, and reduces latency. At the same time, the invention supports existing virtualization frameworks (such as QEMU and Android emulator) and can be expanded to more optimization scenarios, such as cloud gaming, in-vehicle systems, mobile development and other high-demand scenarios.

[0033] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of a gfxstream-based buffer zero-copy rendering and display method provided by the present invention;

[0035] Figure 2 This is a flowchart of a separate rendering process in the present invention;

[0036] Figure 3 It is a flowchart showing a separate process in the present invention. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0038] In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as limiting the present invention.

[0039] Example 1: Combination Figure 1 Description of this embodiment

[0040] The present invention provides a gfxstream-based buffer zero-copy rendering and display method, which is applied between a virtual machine guest and a physical machine host, comprising:

[0041] Initialize the virtio GPU to establish a communication channel between the virtual machine guest and the physical machine host;

[0042] Enable the virtio GPU virtual device in the virtual machine monitor QEMU, initialize the virtio GPU command queues, such as the control queue, rendering queue, display queue, and event notification mechanisms, such as MSI-X interrupts, and establish bidirectional communication between the virtual machine guest and the physical machine host through the PCI virtualization channel, thereby providing underlying communication support for GPU command and buffer transmission between the virtual machine guest and the physical machine host

[0043] Apply for buffer1 through the virtual machine guest, obtain the Scatter-Gather list A of buffer1, and transmit the Scatter-Gather list A to the physical machine host.

[0044] Buffer1 is the display buffer, which can be a framebuffer or pbuffer. It is used to store the frame data that is finally output to the screen. In the display process, it serves as the OpenGL rendering target FBO and is directly associated with the screen display.

[0045] Buffer 1's physical memory consists of multiple non-contiguous physical pages (discrete memory). Scatter-Gather list A records the addresses and lengths of these pages. For example, if buffer 1 consists of three physical pages, Scatter-Gather list A is described as [page1_addr, page1_len], [page2_addr, page2_len], and [page3_addr, page3_len]. The guest sends Scatter-Gather list A to the host using the VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING command.

[0046] To improve transmission efficiency and protect data security, data compression and encryption mechanisms can be introduced between the sender and receiver. For example, when transmitting scatter-gather lists, a compression algorithm (such as LZ4) can be added to reduce bandwidth usage, or sensitive data (such as game textures) can be encrypted to ensure the security of the virtualized environment.

[0047] Buffer1 is converted into dma-buffer1 by the physical machine host and imported into OpenGL.

[0048] Use the dma-heap allocator to convert the discrete scatter-gather list A into a continuous dma-buffer1. dma-buffer is a memory block that supports DMA operations, which can eliminate memory fragmentation and enable the host GPU to directly access the data. Then, convert dma-buffer1 to EGLImage and import it into OpenGL.

[0049] Texture initialization is completed by applying for buffer2 through the virtual machine guest and binding it to texture; buffer2 is rendering buffer2, which is used to store intermediate rendering data, such as textures, geometries, etc.

[0050] The virtual machine guest sends rendering instructions and display instructions to the physical machine host to complete data rendering and display. After the data rendering and display are completed, memory resources can be recovered by releasing texture, DMA-buffer and original buffer.

[0051] The present invention implements a double buffer design by using multiple buffers (buffer1 and buffer2) to separate rendering and display targets. When the virtual machine guest renders to the BackBuffer, the physical machine host displays the FrontBuffer, thereby avoiding screen tearing. The FrontBuffer represents the currently displayed screen, and the BackBuffer represents the screen being rendered in the next frame.

[0052] Furthermore, applying for buffer1 through the virtual machine guest, obtaining the Scatter-Gather list A of buffer1, and transmitting the Scatter-Gather list A to the physical machine host, including:

[0053] Buffer1 is requested through the graphics memory allocator gralloc in the virtual machine guest, and a scatter-gather list A describing the physical memory distribution of buffer1 is generated, thereby allocating dedicated memory for the display process.

[0054] Send the VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING command through the Virtio GPU protocol to transfer the Scatter-Gather list A to the physical machine host.

[0055] Furthermore, buffer1 is converted into dma-buffer1 by the physical host and imported into OpenGL, including:

[0056] Convert the Scatter-Gather list A to dma-buffer1 through the dma-heap allocator, and pass dma-buffer1 to the gfxstream rendering pipeline;

[0057] Convert dma-buffer1 to EGLImage1 recognizable by OpenGL through the EGLImage interface, and bind EGLImage1 as the rendering target of OpenGL.

[0058] Using the dma-heap allocator and the address remapping function of the IOMMU, discrete physical pages are converted into a continuous DMA address space. That is, the Scatter-Gather list A is converted into a dma-buffer1 that supports DMA operations. The dma-buffer1 is exported through the DMA_BUF interface and becomes a memory block accessible to the Host GPU. Then, the dma-buffer1 is passed to the gfxstream rendering pipeline, and the eglCreateImageKHR interface is called. The dma-buffer1 is converted into an EGLImage1 through the EGL_EXT_image_dma_buf_import extension and bound to the OpenGL rendering target (FramebufferObject, FBO), thereby mapping the discrete guest memory into continuous DMA-operable memory on the host side.

[0059] Furthermore, the texture initialization is completed by applying for buffer2 through the virtual machine guest and binding it to texture, including:

[0060] Applying for buffer2 through the graphics memory allocator gralloc in the virtual machine guest, and generating a scatter-gather list B describing the physical memory distribution of buffer2;

[0061] Send the VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING command through the Virtio GPU protocol to transfer the Scatter-Gather list B to the physical machine host;

[0062] Convert the Scatter-Gather list B to dma-buffer2 via the dma-heap allocator and import it into OpenGL;

[0063] Convert buffer2 to EGLImage2 through the virtual machine guest, bind EGLImage2 to OpenGLtexture through the glEGLImageTargetTexture2DOES function, and send the binding information to the physical machine host;

[0064] The binding information is received through the physical machine host, and the dma-buffer2 of buffer2 is associated with the texture to complete the texture initialization.

[0065] Furthermore, the data rendering is completed by sending a rendering instruction to the physical machine host through the virtual machine guest, including:

[0066] Sending an OpenGL rendering command to the physical machine host through the virtual machine guest, wherein buffer1 is specified as a rendering target in the OpenGL rendering command;

[0067] Synchronize the rendering instructions of the virtual machine guest to the physical machine host and write them into buffer1;

[0068] The physical machine host parses the OpenGL rendering command and calls the OpenGL driver in the physical machine host to perform the rendering operation, uses the texture of buffer2 for mapping, and writes the rendering result into dma-buffer1 of buffer1 to complete the rendering.

[0069] The virtual machine guest sends OpenGL rendering commands (such as glDrawElements) through the virtioGPU protocol, which are encapsulated as the VIRTIO_GPU_CMD_SUBMIT_3D instruction. After the physical machine host parses the VIRTIO_GPU_CMD_SUBMIT_3D instruction, the GPU directly operates dma-buffer1 of buffer1 to complete the rendering, and the data is synchronized to the virtual machine guest through shared memory.

[0070] Furthermore, the display is completed by sending a display instruction from the virtual machine guest to the physical machine host, including:

[0071] Sending a VIRTIO_GPU_CMD_RESOURCE_FLUSH command to the physical machine host through the virtual machine guest, wherein the VIRTIO_GPU_CMD_RESOURCE_FLUSH command includes a status of buffer1;

[0072] The DMA-buffer1 of buffer1 is sent to the display synthesizer through the physical machine host to be mixed with other layers, and is output to the screen for display through the display driver.

[0073] The guest sends the VIRTIO_GPU_CMD_RESOURCE_FLUSH command to notify the host that display buffer 1 is ready. The host then sends buffer 1's DMA-buffer 1 to the display compositor (such as Android's SurfaceFlinger). The compositor then blends buffer 1 with other layers (such as UI elements and video streams). The display driver then directly reads DMA-buffer 1 via DMA (Direct Memory Access) for display, eliminating CPU intervention and achieving zero-copy display. The DRM (DirectRenderingManager) interface manages display buffer switching and synchronization during this process.

[0074] like Figure 2 As shown in FIG, the independent rendering process of the present invention includes the following main steps:

[0075] S1: Initialize the virtual machine communication system virtio GPU;

[0076] S2: In the guest virtual machine, a buffer is requested through the allocator for rendering, and the buffer's scatter-gather list is transmitted to the host physical machine through the virtioGPU protocol;

[0077] S3: In the physical host, the buffer's scatter-gather list is converted into a dma-buffer through dma-heap; the dma-buffer is passed to gfxstream; in gfxstream, the dma-buffer is converted into an EGLImage using EGLImage in dma-buffer mode and imported into OpenGL;

[0078] S4: Convert the buffer to an Android native mode EGLImage in the virtual machine guest; bind the EGLImage to a texture using the glEGLImageTargetTexture2DOES function; load data into the texture using glTexImage2D; render the scene and texture map into the framebuffer using 3D commands;

[0079] S5: destroy texture, dma-buffer, buffer;

[0080] like Figure 3 As shown, it is a separate display process in the present invention, which includes the following main steps:

[0081] S1: Initialize the virtual machine communication system virtio GPU;

[0082] S2: Request a display buffer (framebuffer) in the virtual machine guest; transmit the buffer's scatter-gather list to the physical machine host through the virtio GPU protocol;

[0083] S3: Convert the buffer's scatter-gather list into a dma-buffer on the physical host through dma-heap; pass the dma-buffer to gfxstream; use EGLKHRImage in dma-buffer mode in gfxstream to import the dma-buffer into OpenGL and use it as a rendering target;

[0084] S4: In the virtual machine guest, the display buffer is used as the rendering target through the Virtio GPU protocol and the physical machine host is notified;

[0085] S5: The physical host receives the notification information;

[0086] S6: Render the scene into a buffer in the virtual machine guest and submit the buffer for display on the screen;

[0087] S7: Send the DMA buffer to the display synthesizer in the physical host for display;

[0088] S8: destroy dma-buffer, buffer;

[0089] It should also be noted that the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, product, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, product, or apparatus. Without further limitation, the phrase "comprises a..." does not preclude the presence of additional identical elements in the process, method, product, or apparatus comprising the elements. Terms such as "first," "second," and the like are used to designate names and do not imply any particular order. The above illustrative description of the present invention and its embodiments is non-limiting. The present invention may be embodied in other specific forms without departing from the spirit or essential features of the present invention. The figures shown in the accompanying drawings are merely one embodiment of the present invention; the actual structure is not limited thereto. Any reference numerals in the claims should not limit the claims to which they relate. Therefore, if a person of ordinary skill in the art is inspired by this and, without departing from the spirit of the present invention, devises structures and embodiments similar to the present invention without inventive means, they shall fall within the scope of protection of the present invention.

Claims

1. A gfxstream-based buffer zero-copy rendering and display method, applied between a virtual machine guest and a physical machine host, characterized in that: include: Initialize virtioGPU to establish a communication channel between the virtual machine guest and the physical machine host; Apply for buffer1 through the virtual machine guest, obtain the Scatter-Gather list A of buffer1, and transmit the Scatter-Gather list A to the physical machine host; Convert buffer1 to dma-buffer1 through the physical host and import it into OpenGL; Apply for buffer2 through the virtual machine guest and bind it to texture to complete texture initialization; The virtual machine guest sends rendering instructions and display instructions to the physical machine host to complete data rendering and display.

2. A buffer zero copy method based on gfxstream according to claim 1, characterized in that, Applying for buffer 1 through the virtual machine guest, obtaining the Scatter-Gather list A of buffer 1, and transmitting the Scatter-Gather list A to the physical machine host, including: Applying buffer1 through the graphics memory allocator gralloc in the virtual machine guest, and generating a scatter-gather list A describing the physical memory distribution of buffer1; Send the VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING command through the virtioGPU protocol to transfer the Scatter-Gather list A to the physical machine host.

3. A buffer zero copy method based on gfxstream according to claim 2, characterized in that, Convert buffer1 to dma-buffer1 through the physical host and import it into OpenGL, including: Convert the Scatter-Gather list A to dma-buffer1 through the dma-heap allocator, and pass dma-buffer1 to the gfxstream rendering pipeline; Convert dma-buffer1 to EGLImage1 recognizable by OpenGL through the EGLImage interface, and bind EGLImage1 as the rendering target of OpenGL.

4. A buffer zero copy method based on gfxstream according to claim 3, characterized in that, The texture initialization is completed by applying for buffer2 through the virtual machine guest and binding it to texture, including: Applying for buffer2 through the graphics memory allocator gralloc in the virtual machine guest, and generating a scatter-gather list B describing the physical memory distribution of buffer2; Send the VIRTIO_GPU_CMD_RESOURCE_ATTACH_BACKING command through the virtioGPU protocol to transfer the Scatter-Gather list B to the physical machine host; Convert the Scatter-Gather list B to dma-buffer2 via the dma-heap allocator and import it into OpenGL; Convert buffer2 to EGLImage2 through the virtual machine guest, bind EGLImage2 to OpenGLtexture through the glEGLImageTargetTexture2DOES function, and send the binding information to the physical machine host; The binding information is received through the physical machine host, and the dma-buffer2 of buffer2 is associated with the texture to complete the texture initialization.

5. A gfxstream-based buffer zero copy method according to claim 4, characterized in that: The data rendering is completed by sending a rendering instruction to the physical machine host through the virtual machine guest, including: Sending an OpenGL rendering command to the physical machine host through the virtual machine guest, wherein buffer1 is specified as a rendering target in the OpenGL rendering command; Synchronize the rendering instructions of the virtual machine guest to the physical machine host and write them into buffer1; The physical machine host parses the OpenGL rendering command and calls the OpenGL driver in the physical machine host to perform the rendering operation, uses the texture of buffer2 for mapping, and writes the rendering result into dma-buffer1 of buffer1 to complete the rendering.

6. A gfxstream-based buffer zero copy method according to claim 5, characterized in that: The display is completed by sending a display instruction from the virtual machine guest to the physical machine host, including: Sending a VIRTIO_GPU_CMD_RESOURCE_FLUSH command to the physical machine host through the virtual machine guest, wherein the VIRTIO_GPU_CMD_RESOURCE_FLUSH command includes a status of buffer1; The DMA-buffer1 of buffer1 is sent to the display synthesizer through the physical machine host to be mixed with other layers, and is output to the screen for display through the display driver.

Images