Video rendering method and system based on a compute shader and an Xpixmap

Abstract

The application provides a video rendering method and system based on a computing shader and Xpixmap, compared with an opengl display and a graphics pipeline mode, the computing shader can fully call general computing resources of a GPU, as long as the computing resources are sufficient, high resolution and high frame rate video rendering can be supported, and the processing capacity is greatly improved compared with the graphics pipeline mode. Compared with the opengl display mode, the use of the Xpixmap display mode reduces the multiple copying of the video frame, and further improves the efficiency of the video rendering and display. The scheme fully calls the computing resources of the GPU, renders YUV into RGBA and directly stores in the buffer area corresponding to the Xpixmap, and then uses the X function to submit the display, compared with the opengl display and the graphics pipeline mode, the scheme has higher efficiency, stronger processing capacity and adaptability, and is especially suitable for a graphics processing pipeline with insufficient capacity but powerful general computing capacity.

Description

Technical Field

[0001] This invention relates to the field of video rendering technology, and more specifically, to a video rendering method and system based on computational shaders and XPixmaps. Background Technology

[0002] Video playback is divided into two parts: video decoding and video rendering. Video decoding converts the video stream into YUV image data, while video rendering typically converts the YUV data into RGBA data and then pushes it to the display buffer for display. Therefore, video rendering is the core step in video playback, and insufficient performance in this part is often a major cause of video playback stuttering and screen tearing, especially when the video resolution and frame rate are high, such as 4K 60fps or 8K 60fps. In such cases, it is necessary to use a GPU or other dedicated video processing chips. Using the GPU's OpenGL display and shader language is one of the common ways to call GPU resources to complete video rendering. This method can efficiently complete video output tasks by calling GPU resources, improving the smoothness of video playback.

[0003] Currently, video output rendering functionality is implemented using GPUs through various methods. One such method involves users simply inputting YUV data via the standard OpenGL API, eliminating the need for manual texture resource creation. The hardware's built-in texture units, along with a pre-defined transformation matrix algorithm, directly generate RGBA data, avoiding the complex texture configuration and shader compilation and debugging processes at the user level, thus significantly improving rendering efficiency. However, this method has high hardware requirements, necessitating the GPU to integrate a YUV-to-RGBA texture unit; not all GPUs possess this hardware configuration. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a video rendering method and system based on computational shaders and XPixmaps, which can improve GPU rendering efficiency.

[0005] According to a first aspect of the present invention, a video rendering method based on computational shaders and XPixmaps is provided, comprising:

[0006] Step S1: Obtain the Xdisplay and Xwindow objects through the video output interface;

[0007] Step S2: Create a pixel map Xpixmap object based on the Xdisplay object and the Xwindow object;

[0008] Step S3: Initialize the EGL display connection object, create the EGL context and the EGL off-screen rendering target;

[0009] Step S4: Create an egl image object by using the DMABUF flag bit of the frame buffer storing YUV;

[0010] Step S5: Create an associated egl image object based on the flag bits of the Xpixmap object;

[0011] Step S6: Generate textures and bind them to associated EGL images;

[0012] Step S7: Perform parallel computation on the input YUV texture using a computation shader to generate an RGBA image and write it to the output texture;

[0013] Step S8: Submit and display the Xpixmap object containing the completed rendering via the XCB interface.

[0014] According to a second aspect of the present invention, a video rendering system based on computational shaders and XPixmaps is provided, comprising:

[0015] The acquisition module is used to obtain Xdisplay and Xwindow objects through the video output interface;

[0016] The module is used to create a pixel map Xpixmap object based on the Xdisplay object and the Xwindow object; initialize an egl display connection object, create an egl context and an egl off-screen rendering target; and create an egl image object by using the DMABUF flag of the frame buffer storing YUV; and also to create an associated egl image object based on the flag of the Xpixmap object.

[0017] The generation module is used to generate textures and bind them to associated EGL images;

[0018] The computation module is used to perform parallel computations on the input YUV texture using a computation shader to generate an RGBA image and write it to the output texture.

[0019] The display submission module is used to submit and display Xpixmap objects containing completed rendering via the XCB interface.

[0020] According to a third aspect of the present invention, an electronic device is provided, including a memory and a processor, the processor being configured to implement a video rendering method based on computational shaders and XPixmap when executing a computer management program stored in the memory.

[0021] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer management class program is stored, wherein when executed by a processor, the computer management class program implements the steps of a video rendering method based on computational shaders and XPixmaps.

[0022] This invention provides a video rendering method and system based on compute shaders and Xpixmaps. Compute shaders can fully utilize the general computing resources of the GPU. As long as computing resources are sufficient, high-resolution and high-frame-rate video rendering can be supported, significantly improving processing power compared to the graphics pipeline mode. The Xpixmap display mode reduces the number of video frame copies compared to the OpenGL display mode, further improving the efficiency of video rendering and display. This invention fully utilizes the GPU's computing resources, rendering YUV to RGBA and directly storing it in the corresponding buffer of the Xpixmap, then using the X function for display. Compared to OpenGL display and graphics pipeline methods, it has higher efficiency, stronger processing power, and greater adaptability, making it particularly suitable for graphics cards with insufficient graphics pipeline capabilities but strong general computing power. Attached Figure Description

[0023] Figure 1 A flowchart of a video rendering method based on computational shaders and XPixmaps is provided as an embodiment of the present invention;

[0024] Figure 2 This is a flowchart illustrating the process of calculating and generating an RGBA image from a YUV texture according to an embodiment of the present invention.

[0025] Figure 3 A structural block diagram of a video rendering system based on computational shaders and XPixmaps is provided in one embodiment of the present invention;

[0026] Figure 4 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0027] Figure 5 This is a schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined with each other to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0029] Video rendering is a crucial process in video playback, and its performance is a significant factor affecting the playback quality. When the decoding frame rate is constant, the stronger the video rendering performance, the higher the resolution and frame rate that the video playback can support.

[0030] The existing methods of rendering and displaying using GPUs generally rely on graphics rendering pipelines and OpenGL display. When the GPU's graphics performance is insufficient, it will be difficult to support high-resolution and high-frame-rate video rendering and display.

[0031] A compute shader is a general-purpose parallel computing program that runs on a GPU. It breaks free from the fixed workflow constraints of traditional graphics rendering pipelines and can directly utilize the GPU's massively parallel computing capabilities to process various types of data (including graphics, video, and numerical data). Using compute shaders for YUV to RGBA conversion fully utilizes GPU resources, greatly improving the efficiency of video rendering.

[0032] Xpixmap is a server-side graphics resource used in X Window systems (graphical window systems in operating systems such as Linux / Unix) to store pixel data. It is used to represent pixel information of images or drawing areas in the X server. It is one of the core graphics objects defined in the X11 protocol and is mainly used in scenarios such as window drawing, image transmission, and screen rendering.

[0033] Computational shader languages ​​break free from the fixed workflow constraints of traditional graphics rendering pipelines, directly leveraging the massively parallel computing capabilities of GPUs to process various types of data (including graphics, video, and numerical data), making them particularly suitable for GPUs with powerful general-purpose computing capabilities. In OpenGL display mode, video output requires swapping, meaning the rendered RGBA image needs to be read once and then copied to the buffer corresponding to the display frame. However, by creating an Xpixmap and directly rendering the RGBA image to the corresponding video memory area, intermediate copying steps and bandwidth consumption are reduced, further improving video rendering efficiency.

[0034] This invention utilizes compute shaders and Xpixmaps to implement video rendering and output functions on a GPU. Compared to OpenGL display and graphics pipeline methods, compute shaders can fully utilize the GPU's general-purpose computing resources. As long as computing resources are sufficient, high-resolution and high-frame-rate video rendering can be supported, resulting in a significant improvement in processing power compared to the graphics pipeline mode. Furthermore, by rendering YUV to RGBA and directly storing it in the corresponding buffer of the Xpixmap, and then using the X function for display, it achieves higher efficiency than OpenGL display and graphics pipeline methods, making it particularly suitable for graphics cards with insufficient graphics processing pipeline capabilities but powerful general-purpose computing capabilities.

[0035] Figure 1 A flowchart of a video rendering method based on computational shaders and XPixmaps is provided as an embodiment of the present invention, as follows: Figure 1 As shown, the method includes the following steps:

[0036] Step S1: Obtain the Xdisplay object and Xwindow object through the video output interface.

[0037] Step S1, which involves obtaining the Xdisplay and Xwindow objects through the video output interface, includes:

[0038] Step S101: Create an X window as the window for video output display;

[0039] Step S102: Obtain the Xdisplay display connection object and Xwindow window object of the X window through the X function.

[0040] Step S2: Create a pixel map Xpixmap object based on the Xdisplay object and the Xwindow object.

[0041] Step S3: Initialize the EGL display connection object, create the EGL context, and the EGL off-screen rendering target. Step S3, initializing the EGL display connection object, creating the EGL context, and the EGL off-screen rendering target, includes:

[0042] Step S301: Initialize the egl display connection object according to the Xdisplay object;

[0043] Step S302: Initialize the context of EGL according to the EGL display connection object and EGL-related configuration;

[0044] Step S303: Create a pixel buffer Pbuffer as an off-screen rendering target based on the egl display connection object, and specify the width and height of the pixel buffer Pbuffer for background rendering;

[0045] Step S304: Bind the egl context and egl surface to the current thread.

[0046] Step S4: Create an egl image object by using the DMABUF flag of the frame buffer storing YUV.

[0047] Step S4, which involves creating an egl image object by using the DMABUF flag of the frame buffer storing YUV data, includes:

[0048] Step S401: Obtain and save the frame buffer information of the YUV to be displayed;

[0049] Step S402: Obtain the DMABUF flag bit through the DRM interface based on the frame buffer information;

[0050] Step S403: Create an egl image object based on the YUV type, DMABUF flag, and the width and height of the YUV frame.

[0051] Step S5: Create an associated egl image object based on the flag bits of the Xpixmap object.

[0052] Step S5, which involves creating an associated egl image object based on the flag bits of the Xpixmap object, includes:

[0053] Step S501: Obtain the flag bits of Xpixmap;

[0054] Step S502: Use the EGL_NATIVE_PIXMAP_KHR flag to configure and import an egl image based on the native pixel map Xpixmap.

[0055] Step S6: Generate textures and bind them to associated EGL images.

[0056] Step S6, generating a texture and binding it to the associated EGL image, includes:

[0057] Step S601: Generate a corresponding number of textures according to the YUV type, and bind the created input egl image object to the generated texture as the input YUV texture. For example, NV12 generates two textures, and the texture type is ordinary 2D texture.

[0058] Step S602: Generate an external texture and bind the external texture to the egl image object created by the Xpixmap object as the output texture;

[0059] Step S603: After completing the above binding, unbind the texture and temporarily store the texture in the cache area.

[0060] Step S7: Perform parallel computation on the input YUV texture using a computation shader to generate an RGBA image and write it to the output texture.

[0061] Among them, see Figure 2 Step S7, performing parallel computation on the input YUV texture using a computation shader to generate an RGBA image and writing it to the output texture, includes:

[0062] Step S701: Create a compute shader program based on the YUV type and the YUV and RGBA algorithms;

[0063] Step S702: Activate and bind the input YUV texture in the buffer area, and pass non-texture parameters through the gl function. The non-texture parameters include size, flag and offset.

[0064] Step S703: Activate and bind the output texture in the buffer area;

[0065] Step S704: Calculate the number of shader workgroups based on the video width and height;

[0066] Step S705: Pixels are processed in parallel by multiple shader working groups, pixel values ​​are sampled from the YUV texture, and YUV to RGBA color space conversion is performed.

[0067] In step S706, the compute shader writes the output texture into the RGBA image data.

[0068] Step S8: Submit and display the Xpixmap object containing the completed rendering via the XCB interface.

[0069] The video rendering method based on computational shaders and XPixmaps provided by this invention will be described below using a specific implementation example.

[0070] vdpau (Video Decode and Presentation API for Unix) is a set of video decoding and presentation APIs developed by NVIDIA for Unix-like systems (such as Linux). Its main purpose is to accelerate the video decoding, post-processing (such as scaling, deinterlacing, etc.) and presentation process through GPU hardware, reduce the CPU load, and improve the smoothness and performance of video playback, especially suitable for high-definition (HD) and even ultra-high-definition (UHD) video content.

[0071] The video rendering method based on computational shaders and XPixmaps provided in this invention is used to complete the VDPAU video output function, and the following steps are performed:

[0072] Step 1: Create an Xpixmap object based on the Xdislay display connection and Xwindow window information provided by the vdpau output interface, and return the Xpixmap's flags;

[0073] Step 2: Based on the provided Xdislay display connection, initialize the EGL display connection, create the EGL context and the EGL off-screen rendering target;

[0074] Step 3: Obtain the DMABUF flag from the buffer frame storing YUV data, and use this flag to create the input EGL image object;

[0075] Step 4: Create an associated output egl image object based on the flags in the Xpixmap;

[0076] Step 5: Generate a corresponding number of ordinary textures according to the YUV type, use them as input textures, generate external textures, and associate them with the egl image created based on Xpixmap as output textures;

[0077] Step 6: Perform parallel computations on the input YUV texture using a computation shader to generate an RGBA image and write it to the output texture;

[0078] Step 7: Submit the rendered Xpixmap object for display using the XCB Present extension function.

[0079] The above steps complete the hardware-accelerated video rendering and display based on the vdpau video decoding framework. In other words, the GPU can be invoked to complete high-performance video rendering and output by using mpv -vo=vdpau file.

[0080] See Figure 3 A video rendering system based on computational shaders and XPixmaps is provided in one embodiment of the present invention. The system includes:

[0081] Module 301 is used to obtain Xdisplay and Xwindow objects through the video output interface;

[0082] The creation module 302 is used to create a pixel map Xpixmap object based on the Xdisplay object and the Xwindow object; initialize an egl display connection object, create an egl context and an egl off-screen rendering target; and create an egl image object by using the DMABUF flag bit of the frame buffer storing YUV; and also to create an associated egl image object based on the flag bit of the Xpixmap object.

[0083] The generation module 303 is used to generate textures and bind them to associated EGL images;

[0084] The calculation module 304 is used to perform parallel calculations on the input YUV texture through the calculation shader to generate an RGBA image and write it to the output texture;

[0085] The display submission module 305 is used to submit and display an Xpixmap object containing the completed rendering via the XCB interface.

[0086] It is understood that the video rendering system based on computational shaders and Xpixmaps provided by this invention corresponds to the video rendering methods based on computational shaders and Xpixmaps provided in the foregoing embodiments. The relevant technical features of the video rendering system based on computational shaders and Xpixmaps can be referred to the relevant technical features of the video rendering methods based on computational shaders and Xpixmaps, and will not be repeated here.

[0087] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 4 As shown, an embodiment of the present invention provides an electronic device, including a memory 410, a processor 420, and a computer program 411 stored in the memory 410 and executable on the processor 420. When the processor 420 executes the computer program 411, it implements the steps of a video rendering method based on computational shaders and Xpixmaps.

[0088] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. (See diagram below.) Figure 5 As shown, this embodiment provides a computer-readable storage medium 500 on which a computer program 511 is stored. When the computer program 511 is executed by a processor, it implements the steps of a video rendering method based on computational shaders and Xpixmaps.

[0089] This invention provides a video rendering method and system based on compute shaders and Xpixmaps. Compute shaders can fully utilize the general computing resources of the GPU. With sufficient computing resources, high-resolution and high-frame-rate video rendering can be supported, significantly improving processing power compared to the graphics pipeline mode. The Xpixmap display mode reduces the number of video frame copies compared to the OpenGL display mode, further improving the efficiency of video rendering and display. This invention fully utilizes the GPU's computing resources, rendering YUV to RGBA and directly storing it in the corresponding buffer of the Xpixmap, then using the X function for display. Compared to OpenGL display and graphics pipeline methods, it has higher efficiency, stronger processing power, and greater adaptability, making it particularly suitable for graphics cards with insufficient graphics pipeline capabilities but strong general computing power.

[0090] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0091] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0096] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A video rendering method based on computational shaders and XPixmaps, characterized in that, include: Step S1: Obtain the Xdisplay and Xwindow objects through the video output interface; Step S2: Create a pixel map Xpixmap object based on the Xdisplay object and the Xwindow object; Step S3: Initialize the EGL display connection object, create the EGL context and the EGL off-screen rendering target; Step S4: Create an egl image object by using the DMABUF flag bit of the frame buffer storing YUV; Step S5: Create an associated egl image object based on the flag bits of the Xpixmap object; Step S6: Generate textures and bind them to associated EGL images; Step S7: Perform parallel computation on the input YUV texture using a computation shader to generate an RGBA image and write it to the output texture; Step S8: Submit and display the Xpixmap object containing the completed rendering via the XCB interface; Step S6, generating a texture and binding it to the associated EGL image, includes: Step S601: Generate a corresponding number of textures according to the YUV type, and bind the created input egl image object to the generated texture as the input YUV texture; Step S602: Generate an external texture and bind the external texture to an egl image object created based on the Xpixmap object as the output texture; Step S603: After completing the above binding, unbind the texture and temporarily store the texture in the cache area; Step S7, which involves generating an RGBA image from the input YUV texture using a computation shader in parallel and writing it to the output texture, includes: Step S701: Create a compute shader program based on the YUV type and the YUV and RGBA algorithms; Step S702: Activate and bind the input YUV texture in the buffer area, and pass non-texture parameters through the gl function. The non-texture parameters include size, flag and offset. Step S703: Activate and bind the output texture in the buffer area; Step S704: Calculate the number of shader workgroups based on the video width and height; Step S705: Pixels are processed in parallel by multiple shader working groups, pixel values ​​are sampled from the YUV texture, and YUV to RGBA color space conversion is performed. In step S706, the compute shader writes the output texture into the RGBA image data.

2. The video rendering method according to claim 1, characterized in that, Step S1, obtaining the Xdisplay and Xwindow objects through the video output interface, includes: Step S101: Create an X window as the window for video output display; Step S102: Obtain the Xdisplay display connection object and Xwindow window object of the X window through the X function.

3. The video rendering method according to claim 1, characterized in that, Step S3: Initializing the EGL display connection object, creating the EGL context and the EGL off-screen rendering target, including: Step S301: Initialize the egl display connection object according to the Xdisplay object; Step S302: Initialize the context of EGL according to the EGL display connection object and EGL-related configuration; Step S303: Create a pixel buffer Pbuffer as an off-screen rendering target based on the egl display connection object, and specify the width and height of the pixel buffer Pbuffer for background rendering; Step S304: Bind the egl context and egl surface to the current thread.

4. The video rendering method according to claim 1, characterized in that, Step S4, which creates an egl image object by storing the DMABUF flag bit of the YUV frame buffer, includes: Step S401: Obtain and save the frame buffer information of the YUV to be displayed; Step S402: Obtain the DMABUF flag bit through the DRM interface based on the frame buffer information; Step S403: Create an egl image object based on the YUV type, DMABUF flag, and the width and height of the YUV frame.

5. The video rendering method according to claim 1, characterized in that, Step S5, which involves creating an associated egl image object based on the flag bits of the Xpixmap object, includes: Step S501: Obtain the flag bits of Xpixmap; Step S502: Use the EGL_NATIVE_PIXMAP_KHR flag to configure and import an egl image based on the native pixel map Xpixmap.

6. A video rendering system based on computational shaders and XPixmaps, characterized in that, include: The acquisition module is used to obtain Xdisplay and Xwindow objects through the video output interface; A module is created to generate a pixel map (Xpixmap) object based on the Xdisplay object and the Xwindow object; initialize the egl display connection object, create the egl context and egl off-screen rendering target; And for creating egl image objects by using the DMABUF flag bit of the frame buffer storing YUV; also for creating associated egl image objects based on the flag bits of the Xpixmap object; The generation module is used to generate textures and bind them to associated EGL images; The computation module is used to perform parallel computations on the input YUV texture using a computation shader to generate an RGBA image and write it to the output texture. The display submission module is used to submit and display an Xpixmap object that has been rendered through the XCB interface; The generation module is specifically used for: Generate a corresponding number of textures based on the YUV type, and bind the created input EGL image object to the generated textures as input YUV textures; Generate an external texture and bind the external texture to an egl image object created from the Xpixmap object as the output texture; After completing the above binding, unbind the texture and temporarily store the texture in the cache area; The computing module is specifically used for: Create a compute shader program based on the YUV type and the algorithms for YUV and RGBA; The input YUV texture in the buffer is activated and bound, and non-texture parameters, including size, flags and offset, are passed through the gl function. Activate and bind the output texture in the buffer; Calculate the number of shader workgroups based on the video's width and height; Pixels are processed in parallel by multiple shader working groups, sampling pixel values ​​from YUV textures and performing YUV to RGBA color space conversion; The computation shader writes the output texture into RGBA image data.

Images