Video rendering method and system based on calculation shader and Xpixmap

Abstract

According to the video rendering method and system based on the computing shader and the Xpixmap, compared with an opengl display and graph pipeline mode, the computing shader can fully call general computing resources of a GPU, high-resolution and high-frame-rate video rendering can be supported as long as the computing resources are sufficient, and compared with the graph pipeline mode, the processing capacity is greatly improved. Compared with an opengl display mode, the Xpixmap display mode has the advantages that multiple copies of the video frames are reduced, and the video rendering and display efficiency is further improved. According to the scheme, computing resources of the GPU are fully called, the YUV is rendered into RGBA to be directly stored in the buffer area corresponding to the Xpixmap, then the RGBA is submitted and displayed through the X function, compared with opengl display and a graphic pipeline mode, higher efficiency and higher processing capacity and adaptability are achieved, and the method is particularly suitable for a display card with insufficient graphic processing pipeline capacity but high general computing capacity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video rendering, and more particularly to a video rendering method and system based on a compute shader and an Xpixmap. BACKGROUND

[0002] A video playing process is divided into two parts of video decoding and video rendering. The video decoding is to decode a video code stream into YUV image data, and the video rendering is usually to convert the YUV data into RGBA data and then push the data to a display buffer for display. Therefore, the video rendering is a core step of the video playing, and a performance deficiency in the part of processing is an important reason for causing a video playing lag and a picture tearing, especially when the resolution and frame rate of the video are high, such as processing 4K60fps or 8k60fps, a GPU or other video special chip needs to be used for processing, and an opengl display and a shader language using the GPU are one of common ways for calling GPU resources to complete video rendering. The method can call GPU resources to efficiently complete a video output task and improve the fluency of the video playing.

[0003] At present, a video output rendering function is realized by using a GPU through different methods. One of the methods is that a user only needs to input YUV data information by calling a standard OpenGL API, and does not need to manually create a texture resource. RGBA data is directly generated by a texture unit of the hardware itself and a preset conversion matrix algorithm, so as to avoid a complex texture configuration and a Shader compiling and debugging process of a user layer and significantly improve rendering efficiency. The method has a high requirement for hardware, needs to integrate a YUV to RGBA texture unit in the GPU, and is not suitable for all GPUs. SUMMARY

[0004] In view of the technical problems in the prior art, the present application provides a video rendering method and system based on a compute shader and an Xpixmap, which can improve the GPU rendering efficiency.

[0005] According to a first aspect of the present application, a video rendering method based on a compute shader and an Xpixmap is provided, which comprises the following steps:

[0006] In step S1, an Xdisplay object and an Xwindow object are obtained through a video output interface.

[0007] In step S2, a pixel map Xpixmap object is created according to the Xdisplay object and the Xwindow object.

[0008] In step S3, an egl display connection object is initialized, an egl context and an egl off-screen rendering target are created.

[0009] Step S4, creating egl image object by DMABUF flag of storing YUV frame buffer;

[0010] Step S5, creating associated egl image object according to flag of Xpixmap object;

[0011] Step S6, generating texture and binding associated egl image;

[0012] Step S7, generating RGBA image by parallel calculation of input YUV texture through calculation shader and writing into output texture;

[0013] Step S8, submitting Xpixmap object containing completed rendering to display through XCB interface.

[0014] According to the second aspect of the present application, a video rendering system based on calculation shader and Xpixmap is provided, comprising:

[0015] An acquisition module is configured to acquire Xdisplay object and Xwindow object through video output interface;

[0016] A creating module is configured to create pixel map Xpixmap object according to the Xdisplay object and the Xwindow object; initialize egl display connection object, create egl context and egl off-screen rendering target; and create egl image object by DMABUF flag of storing YUV frame buffer; and create associated egl image object according to flag of Xpixmap object;

[0017] A generating module is configured to generate texture and bind associated egl image;

[0018] A calculating module is configured to generate RGBA image by parallel calculation of input YUV texture through calculation shader and write into output texture;

[0019] A display submitting module is configured to submit Xpixmap object containing completed rendering to display through XCB interface.

[0020] According to the third aspect of the present application, an electronic device is provided, comprising memory and processor, the processor is configured to execute computer management type program stored in the memory to realize steps of the video rendering method based on calculation shader and Xpixmap.

[0021] According to the fourth aspect of the present application, a computer readable storage medium is provided, which stores computer management type program, the computer management type program is executed by processor to realize steps of the video rendering method based on calculation shader and Xpixmap.

[0022] The application provides a video rendering method and system based on a computing shader and Xpixmap. The computing shader can fully call general computing resources of a GPU. As long as the computing resources are sufficient, the video rendering with high resolution and high frame rate can be supported, and the processing capacity is greatly improved compared with a graphic pipeline mode. The display mode using the Xpixmap reduces the multiple copying of the video frame compared with the opengl display mode, and further improves the efficiency of the video rendering and display. The application fully calls the computing resources of the GPU, stores YUV rendering as RGBA in a corresponding buffer of the Xpixmap, and then uses an X function to submit display. Compared with the opengl display and the graphic pipeline mode, the application has higher efficiency, stronger processing capacity and adaptability, and is especially suitable for a graphics processing pipeline with insufficient capacity but powerful general computing capacity. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 A flowchart of a video rendering method based on a computing shader and Xpixmap is provided for an embodiment of the application.

[0024] Figure 2 A flowchart of generating an RGBA image from YUV texture calculation for an embodiment of the application is provided.

[0025] Figure 3 A structural block diagram of a video rendering system based on a computing shader and Xpixmap is provided for an embodiment of the application.

[0026] Figure 4 A hardware structural schematic diagram of a possible electronic device is provided for the application.

[0027] Figure 5 A hardware structural schematic diagram of a possible computer readable storage medium is provided for the application. DETAILED DESCRIPTION

[0028] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the protection scope of the present application. In addition, the technical features in each of the embodiments or in a single embodiment provided by the present application can be combined with each other at will to form a feasible technical solution, and this combination is not restricted by the sequence of steps and / or the mode of structural composition, but should be based on the implementation by those of ordinary skill in the art. When the combination of technical solutions contradicts each other or cannot be implemented, it should be considered that the combination of technical solutions does not exist and is not within the protection scope of the present application.

[0029] Video rendering is an important process of video playback, and the performance of video rendering is an important factor affecting the effect of video playback. When the decoding frame rate is constant, the stronger the video rendering performance is, the higher the resolution and frame rate that the video playback can support are.

[0030] The existing rendering and display method using GPU is generally completed through a graphics rendering pipeline and an opengl display method. When the GPU graphics performance is insufficient, it is difficult to support high-resolution and high-frame-rate video rendering and display.

[0031] A compute shader is a general-purpose parallel computing program running on a GPU, which is free from the constraints of the fixed process of the traditional graphics rendering pipeline and can directly use the large-scale parallel computing capability of the GPU to process various data (including graphics, video, numerical values, etc.). Using a compute shader for YUV to RGBA conversion can fully call the resources of the GPU and greatly improve the efficiency of video rendering.

[0032] Xpixmap is a server-side graphics resource used to store pixel data in the XWindow system (a graphical window system of Linux / Unix operating systems, etc.), which is used to represent pixel information of an image or drawing area in the X server. It is one of the core graphics objects defined in the X11 protocol, mainly used in window drawing, image transmission and screen rendering scenarios.

[0033] The computing shader language is separated from the fixed flow constraint of the traditional graphics rendering pipeline, and can directly utilize the massive parallel computing capability of the GPU to process various data (including graphics, video, numerical value, etc.), and is especially suitable for the GPU with strong general computing capability. In the Opengl display mode, the video output needs to be performed in a swap mode, that is, the rendered RGBA image needs to be read once, and then copied to the buffer corresponding to the display frame, and the Xpixmap is created, and the RGBA is directly rendered to the memory area corresponding to the Xpixmap, so that the intermediate link and bandwidth consumption of copying are reduced, and the efficiency of video rendering is further improved.

[0034] The GPU video rendering and output function is realized by using the computing shader and the Xpixmap. Compared with the opengl display and the graphics pipeline mode, the computing shader can fully call the general computing resource of the GPU, and as long as the computing resource is sufficient, the high-resolution and high-frame-rate video rendering can be supported, the processing capability is greatly improved compared with the graphics pipeline mode, and the YUV is directly rendered into the RGBA and stored in the buffer corresponding to the Xpixmap, and then the X function is used for display submission, so that the efficiency is higher than that of the opengl display and the graphics pipeline mode, and the graphics processing pipeline capability is especially suitable for the graphics card with strong general computing capability.

[0035] Figure 1 A video rendering method based on a computing shader and an Xpixmap is provided for an embodiment of the application, as shown in Figure 1 The method comprises the following steps:

[0036] In step S1, the Xdisplay object and the Xwindow object are obtained through the video output interface.

[0037] In step S1, the Xdisplay object and the Xwindow object are obtained through the video output interface, including:

[0038] In step S101, the X window is created as the window for video output display.

[0039] In step S102, the Xdisplay display connection object and the Xwindow window object of the X window are obtained through the X function.

[0040] In step S2, the pixel map Xpixmap object is created according to the Xdisplay object and the Xwindow object.

[0041] In step S3, the egl display connection object is initialized, the egl context is created, and the egl off-screen rendering target is created. In step S3, the egl display connection object is initialized, the egl context is created, and the egl off-screen rendering target is created, including:

[0042] Step S301, initializing an egl display connection object according to the Xdisplay object;

[0043] Step S302, initializing a context of egl according to the egl display connection object and egl related configuration;

[0044] Step S303, creating a pixel buffer Pbuffer as an off-screen rendering target according to the egl display connection object, and specifying the width and height of the pixel buffer Pbuffer for background rendering;

[0045] Step S304, binding the egl context and egl surface to the current thread.

[0046] Step S4, creating an egl image object through DMABUF flag of the frame buffer storing YUV.

[0047] In step S4, the egl image object is created through DMABUF flag of the frame buffer storing YUV, comprising:

[0048] Step S401, obtaining frame buffer information storing YUV to be displayed;

[0049] Step S402, obtaining DMABUF flag through DRM interface according to the frame buffer information;

[0050] Step S403, creating an egl image object according to the type of YUV, DMABUF flag and width and height of YUV frame.

[0051] Step S5, creating an associated egl image object according to the flag of Xpixmap object.

[0052] In step S5, the associated egl image object is created according to the flag of Xpixmap object, comprising:

[0053] Step S501, obtaining the flag of Xpixmap;

[0054] Step S502, configuring import using the EGL_NATIVE_PIXMAP_KHR flag to create an egl image based on the original pixel map Xpixmap.

[0055] Step S6, generating a texture and binding the associated egl image.

[0056] In step S6, the texture is generated and the associated egl image is bound, comprising:

[0057] Step S601, generating corresponding number of textures according to YUV type, and binding created input egl image object to generated texture as input YUV texture, such as NV12, generating two textures, and texture type is normal 2D texture;

[0058] Step S602, generating external texture, binding external texture to Xpixmap object created egl image object as output texture;

[0059] Step S603, after completing the above binding, unbinding the above texture, and temporarily storing the texture in a buffer area.

[0060] Step S7, generating RGBA image by parallel calculation of input YUV texture through calculation shader and writing into output texture.

[0061] Wherein, referring to Figure 2 , step S7, generating RGBA image by parallel calculation of input YUV texture through calculation shader and writing into output texture, comprising:

[0062] Step S701, creating calculation shader program according to YUV type and YUV to RGBA algorithm;

[0063] Step S702, activating and binding input YUV texture of buffer area, and passing non-texture parameters through gl function, wherein the non-texture parameters include size, flag and offset;

[0064] Step S703, activating and binding output texture of buffer area;

[0065] Step S704, calculating number of calculation shader work groups according to video width and height;

[0066] Step S705, sampling pixel value from YUV texture by multiple shader work groups to parallel process pixels, and executing YUV to RGBA color space conversion;

[0067] Step S706, calculation shader writing output texture into RGBA image data.

[0068] Step S8, submitting Xpixmap object containing completed rendering to display through XCB interface.

[0069] The video rendering method based on calculation shader and Xpixmap provided by the application will be described below according to a specific implementation case.

[0070] Vdpau (Video Decode and Presentation API for Unix) is a video decoding and presentation API for Unix-like systems (such as Linux) developed by NVIDIA, which is mainly used to accelerate video decoding, post-processing (such as scaling, de-interlacing, etc.) and presentation process through GPU hardware, to reduce the burden of CPU, improve the smoothness and performance of video playback, and is especially suitable for HD (High Definition) or even UHD (Ultra High Definition) video content.

[0071] The video rendering method based on the computing shader and Xpixmap provided by the application is used to complete the vdpau video output function, and the following steps are executed:

[0072] Step 1: An Xpixmap object is created according to the Xdislay display connection and Xwindow window information provided by the vdpau output interface, and the flag bit of the Xpixmap is returned;

[0073] Step 2: According to the provided Xdislay display connection, the egl display connection is initialized, the egl context and the egl off-screen rendering target are created;

[0074] Step 3: The flag bit of the DMABUF is obtained from the cache frame storing the YUV, and the input egl image object is created by using the flag bit;

[0075] Step 4: The associated output egl image object is created according to the flag bit of the Xpixmap;

[0076] Step 5: A corresponding number of normal textures are generated according to the type of YUV as input textures, external textures are generated, and the egl image created according to the Xpixmap is associated as output textures;

[0077] Step 6: The input YUV texture is calculated by the computing shader to generate an RGBA image and write it into the output texture;

[0078] Step 7: The Xpixmap object containing the completed rendering is submitted to the display through the Present extension function of XCB.

[0079] Through the above steps, the hardware accelerated video rendering display based on the vdpau video decoding framework can be completed, that is, the GPU can be called to complete the high-performance rendering output of the video by mpv -vo=vdpau file.

[0080] Referring to Figure 3 The video rendering system based on the computing shader and Xpixmap provided by the embodiment of the application comprises:

[0081] Module 301 is used to obtain Xdisplay and Xwindow objects through the video output interface;

[0082] The creation module 302 is used to create a pixel map Xpixmap object based on the Xdisplay object and the Xwindow object; initialize an egl display connection object, create an egl context and an egl off-screen rendering target; and create an egl image object by using the DMABUF flag bit of the frame buffer storing YUV; and also to create an associated egl image object based on the flag bit of the Xpixmap object.

[0083] The generation module 303 is used to generate textures and bind them to associated EGL images;

[0084] The calculation module 304 is used to perform parallel calculations on the input YUV texture through the calculation shader to generate an RGBA image and write it to the output texture;

[0085] The display submission module 305 is used to submit and display an Xpixmap object containing the completed rendering via the XCB interface.

[0086] It is understood that the video rendering system based on computational shaders and Xpixmaps provided by this invention corresponds to the video rendering methods based on computational shaders and Xpixmaps provided in the foregoing embodiments. The relevant technical features of the video rendering system based on computational shaders and Xpixmaps can be referred to the relevant technical features of the video rendering methods based on computational shaders and Xpixmaps, and will not be repeated here.

[0087] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 4 As shown, an embodiment of the present invention provides an electronic device, including a memory 410, a processor 420, and a computer program 411 stored in the memory 410 and executable on the processor 420. When the processor 420 executes the computer program 411, it implements the steps of a video rendering method based on computational shaders and Xpixmaps.

[0088] Please see Figure 5 , Figure 5 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. (See diagram below.) Figure 5 As shown, this embodiment provides a computer-readable storage medium 500 on which a computer program 511 is stored. When the computer program 511 is executed by a processor, it implements the steps of a video rendering method based on computational shaders and Xpixmaps.

[0089] The video rendering method and system based on the computing shader and Xpixmap provided by the embodiment of the present application can fully call the general computing resource of the GPU, and can support high-resolution and high-frame-rate video rendering as long as the computing resource is sufficient, and the processing capability is greatly improved compared with the graphic pipeline mode. The display mode using Xpixmap reduces the multiple copying of the video frame compared with the opengl display mode, and further improves the efficiency of the video rendering and display. The present application fully calls the computing resource of the GPU, stores the YUV rendering as RGBA in the buffer corresponding to the Xpixmap, and then uses the X function to submit the display, and has higher efficiency, stronger processing capability, and adaptability compared with the opengl display and the graphic pipeline mode, and is especially suitable for the graphics processing pipeline with insufficient capability but powerful general computing capability.

[0090] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0091] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media containing computer usable program code (including but not limited to disk memory, CD-ROM, optical memory, etc.).

[0092] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The device that implements the functions specified in one flow or multiple flows and / or blocks.

[0093] These computer program instructions can also be stored in a computer readable memory that can guide the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer readable memory produce a product including instruction devices that implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1the function specified in one or more blocks.

[0094] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operations steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide processes for implementing the flows Figure 1 one or more flows and / or blocks Figure 1 the steps of the function specified in one or more blocks.

[0095] Although preferred embodiments of the application have been described, those skilled in the art will recognize that additional modifications and variations can be made thereto without departing from the spirit and scope of the application. It is therefore intended that the appended claims cover all such modifications and variations as fall within the scope of the application.

[0096] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore intended that the invention be covered within the scope of the appended claims, and their equivalents.

Claims

1. A method for video rendering based on a computing shader and Xpixmap, characterized in that, The application relates to a method for rendering YUV image to RGBA image and displaying the image on a display screen, comprising the following steps: Step S1, acquiring an Xdisplay object and an Xwindow object through a video output interface; Step S2, creating a pixel map Xpixmap object according to the Xdisplay object and the Xwindow object; Step S3, initializing an egl display connection object, creating an egl context and an egl off-screen rendering target; Step S4, creating an egl image object through a DMABUF flag of a frame buffer storing YUV; Step S5, creating an associated egl image object according to a flag of the Xpixmap object; Step S6, generating a texture and binding the associated egl image; Step S7, generating an RGBA image through parallel calculation of an input YUV texture by a calculation shader and writing the image into an output texture; Step S8, submitting the Xpixmap object containing the completed rendering to the display through an XCB interface.

2. The video rendering method of claim 1, wherein, The step S1, acquiring an Xdisplay object and an Xwindow object through a video output interface, comprises the following steps: Step S101, creating an Xwindow as a window for video output display; Step S102, acquiring an Xdisplay display connection object and an Xwindow object of the Xwindow through an X function.

3. The video rendering method of claim 1, wherein, The step S3, initializing an egl display connection object, creating an egl context and an egl off-screen rendering target, comprises the following steps: Step S301, initializing the egl display connection object according to the Xdisplay object; Step S302, initializing the context of the egl according to the egl display connection object and egl related configuration; Step S303, creating a pixel buffer Pbuffer as an off-screen rendering target according to the egl display connection object, and specifying the width and height of the pixel buffer Pbuffer for background rendering; Step S304, binding the egl context and egl surface to the current thread.

4. The video rendering method of claim 1, wherein, The step S4, creating an egl image object through a DMABUF flag of a frame buffer storing YUV, comprises the following steps: Step S401, acquiring frame buffer information of YUV to be displayed; Step S402, acquiring a DMABUF flag through a DRM interface according to the frame buffer information; Step S403, creating an egl image object according to the type of YUV, the DMABUF flag and the width and height of the YUV frame.

5. The video rendering method of claim 1, wherein, The step S5, creating an associated egl image object according to a flag of the Xpixmap object, comprises the following steps: Step S501, acquiring the flag of the Xpixmap; Step S502, using an EGL_NATIVE_PIXMAP_KHR flag configuration to create an egl image based on the original pixel map Xpixmap.

6. The video rendering method of claim 1, wherein, The step S6, generating a texture and binding the associated egl image, comprises the following steps: Step S601, generating a corresponding number of textures according to the type of YUV, and binding the created input egl image object to the generated texture as an input YUV texture; Step S602, generate an external texture, bind the external texture to an egl image object created according to an Xpixmap object as an output texture; Step S603, after the above binding is completed, unbind the texture, and temporarily store the texture in a buffer area.

7. The video rendering method of claim 1, wherein, The step S7, generating an RGBA image by parallel computing of the input YUV texture through a computing shader and writing the RGBA image into an output texture, includes: Step S701, creating a computing shader program according to a YUV type and an algorithm of YUV and RGBA; Step S702, activating and binding the input YUV texture of the buffer area, and passing non-texture parameters through a gl function, the non-texture parameters including size, flags and offset; Step S703, activating and binding the output texture of the buffer area; Step S704, calculating a number of computing shader work groups according to the width and height of the video; Step S705, sampling pixel values from the YUV texture through a plurality of computing shader work groups to perform color space conversion from YUV to RGBA; Step S706, writing the output texture into RGBA image data by the computing shader.

8. A video rendering system based on a compute shader and Xpixmap, characterized in that, It includes: An acquisition module, configured to acquire an Xdisplay object and an Xwindow object through a video output interface; A creation module, configured to create a pixel map Xpixmap object according to the Xdisplay object and the Xwindow object, initialize an egl display connection object, create an egl context and an egl off-screen rendering target, and create an egl image object through a DMABUF flag of a frame buffer storing YUV, and further configured to create an associated egl image object according to a flag of the Xpixmap object; A generation module, configured to generate a texture and bind the associated egl image; A computing module, configured to generate an RGBA image by parallel computing of the input YUV texture through a computing shader and write the RGBA image into an output texture; A display submission module, configured to submit an Xpixmap object containing completed rendering for display through an XCB interface. ​

Images