A method, apparatus, system and storage medium for video rendering

Abstract

The present invention provides a method and system for video rendering, including creating a window for D3D to perform video drawing in the Qt framework, shielding the rendering messages of Qt on the window, and creating D3D rendering facilities on the window so that D3D takes over the drawing of the window; making the decoded D3D texture be bindable to the D3D shader for rendering when creating the D3D texture for decoding; establishing a rendering intermediate layer, which is used to parse the OpenGL shader into a D3D shader and encapsulate the D3D function into the form of OpenGL, so that users can create and compile the D3D shader in the way of OpenGL. By using the above technical solutions, the video rendering efficiency can be improved, the learning cost of users can be reduced, the support scope for graphics card drivers is wider, and the business code can be easily ported to the platform that supports OpenGL.

Description

Technical Field

[0001] The present invention relates to video rendering, and in particular to a method, device, system and storage medium for video rendering. Background Art

[0002] Currently, there are mainly two common video hard decoding and rendering solutions on Qt in the industry:

[0003] 1. Copy the D3D texture data generated by DXVA hard decoding into memory, and then copy the data as an OpenGL texture into video memory for rendering. The disadvantage of this solution is that a large amount of CPU resources are consumed in the data copying process.

[0004] 2. Use the OpenGL extension WGL_NV_DX_interop to share the data generated by DXVA hard decoding with OpenGL, thus avoiding CPU data copying. The disadvantage of this solution is that it is limited by the graphics card driver. If the driver does not provide the WGL_NV_DX_interop extension, this function cannot be implemented. Summary of the Invention

[0005] Embodiments of the present invention provide a method and system for video rendering to achieve efficient video rendering through a combination of D3D and Qt.

[0006] To achieve the above object, on the one hand, a method for video rendering is provided, which uses D3D and Qt for hybrid video rendering, including the following steps:

[0007] S1: In the Qt framework, create a window for D3D to perform video drawing, block the rendering messages of Qt on the window, and create D3D rendering facilities on the window so that D3D takes over the drawing of the window;

[0008] S2: When creating a D3D texture for decoding, specify it as D3D11_BIND_SHADER_RESOURCE so that the decoded D3D texture can be bound to a D3D shader for rendering;

[0009] S3: Establish a rendering intermediate layer, which is used to parse an OpenGL shader into a D3D shader and encapsulate D3D functions into an OpenGL form so that users can create and compile D3D shaders in an OpenGL manner.

[0010] In a specific embodiment, the step of creating a window in the Qt framework that allows D3D to perform video drawing specifically includes:

[0011] Create a window using the class QWindow provided by Qt, and encapsulate the window into a QWidget control using the methods provided by QWidget. The QWidget control can be embedded into the UI implemented by QWidget.

[0012] In a specific embodiment, shield the rendering messages of Qt on the window, and create a D3D rendering facility on the window, so that D3D takes over the drawing of the window. Specifically, it includes:

[0013] Shield the rendering events of Qt by overloading the event method of QWindow, create a D3D rendering facility on the window, and create a SwapChain on the window to convert the rendering method from Qt to D3D;

[0014] Among them, the video part is all rendered by D3D, and the non-video part is drawn by Qt.

[0015] In a specific embodiment, the video part is all rendered by D3D, and the non-video part is drawn by Qt. Specifically, it includes:

[0016] Qt implements UI drawing, and D3D uses shaders to implement the generation and rendering of video, OSD, and video effects.

[0017] In a specific embodiment, enable the decoded D3D texture to be bound to the D3D shader for rendering. Specifically, it includes:

[0018] Map the texture data in YUV format generated after DXVA decoding, so as to implement the conversion from YUV format to RGB format of the texture data in the D3D shader:

[0019] Map the Y component of the texture data into a first texture containing only the R component, and map the UV components of the texture data into a second texture containing RG components; and, in the shader, access the Y data through the R component of the first texture, and access the UV data through the RG components of the second texture.

[0020] In order to reuse OpenGL code or allow users to directly use the OpenGL functions provided by Qt for image rendering, we also need to design an intermediate layer to encapsulate the D3D interface, design it into an interface similar to OpenGL, and be able to parse the OpenGL shader into a shader used by D3D, for translating OpenGL shader code and encapsulating D3D functions into the form of OpenGL. In a specific embodiment, the parsing of the OpenGL shader into a D3D shader specifically includes:

[0021] Translate OpenGL into the SPIR-V format, and then convert the SPIR-V into a D3D shader. Among them, the steps of translating OpenGL into the SPIR-V format and converting the SPIR-V into a D3D shader are encapsulated into the glCompileShader function to achieve the creation and compilation of D3D shaders in the way of OpenGL. This enables the system to reuse the code developed for OpenGL, and users only need to master OpenGL to develop.

[0022] In a specific embodiment, the encapsulation of D3D functions into the form of OpenGL specifically includes:

[0023] In the UI rendering thread, determine whether to use the OpenGL module provided by Qt:

[0024] If so, encapsulate the D3D functions into a complete OpenGL framework, and compile the OpenGL framework into a library file identical to OpenGL for dynamic loading by Qt during runtime;

[0025] If not, encapsulate the D3D functions into an OpenGL subset for the user to parse the OpenGL shader into a D3D shader through the rendering intermediate layer, so as to create and compile the D3D shader in the way of OpenGL.

[0026] On the other hand, a video rendering system is provided, including:

[0027] D3D rendering form establishment module: configured to create a form for D3D video drawing in the Qt framework, shield the rendering messages of Qt on the form, and create D3D rendering facilities on the form for D3D to take over the drawing of the form;

[0028] D3D shader binding module: configured to specify it as D3D11_BIND_SHADER_RESOURCE when creating a D3D texture for decoding, so that the decoded D3D texture can be bound to the D3D shader for rendering;

[0029] Rendering intermediate layer establishment module: configured to establish a rendering intermediate layer, which is used to parse the OpenGL shader into a D3D shader and encapsulate the D3D functions into the form of OpenGL, so that users can create and compile the D3D shader in the way of OpenGL.

[0030] On yet another hand, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer processor, the video rendering method described above is implemented.

[0031] In another aspect, a video rendering device is provided, including a memory and a processor. The memory stores at least one program, and the at least one program is executed by the processor to implement the video rendering method as described above.

[0032] The above technical solution has the following technical effects:

[0033] 1. By embedding a window in the Qt control, the hybrid rendering of Qt and D3D is achieved. Qt is specifically used to implement UI drawing, and D3D is combined with shaders to implement rendering. The YUV texture data generated by DXVA hard decoding can be directly used without being transferred through the CPU, improving the video rendering efficiency.

[0034] 2. Adopting the native D3D solution, it does not require the support of OpenGL extensions and has a wider range of support for graphics card drivers.

[0035] 3. By utilizing shader language translation technology and encapsulating the D3D interface, an interface similar to OpenGL is provided externally. Users only need to implement the code and shaders for OpenGL without understanding the details of D3D, thereby reducing the learning cost of users, and the business code can be easily ported to platforms that support OpenGL.

[0036] 4. This solution establishes a rendering intermediate layer, encapsulates the D3D interface, designs it into an interface similar to OpenGL, and can parse OpenGL shaders into shaders used by D3D. The system can reuse the code developed for OpenGL, and users can also directly use the OpenGL functions provided by Qt for image rendering, enabling users to develop only by mastering OpenGL. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic flowchart of a video rendering method according to an embodiment of the present invention;

[0038] Figure 2 It is a flowchart of YUV format texture data mapping according to an embodiment of the present invention;

[0039] Figure 3 It is a schematic flowchart of a video rendering method according to another embodiment of the present invention;

[0040] Figure 4 It is a schematic structural diagram of a video rendering device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] To further illustrate each embodiment, the present invention provides accompanying drawings. These drawings are part of the disclosure of the present invention, mainly used to illustrate the embodiments, and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, those of ordinary skill in the art should be able to understand other possible embodiments and the advantages of the present invention. The components in the drawings are not drawn to scale, and similar component symbols are usually used to represent similar components.

[0042] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0043] Embodiment 1:

[0044] Figure 1 The following is a schematic flowchart of a method for video rendering according to an embodiment of the present invention. In this embodiment, D3D and Qt are used for hybrid video rendering, including the following steps:

[0045] S1: In the Qt framework, create a window for D3D to perform video drawing, block the rendering messages of Qt on the window, and create a rendering facility of D3D on the window, so that D3D takes over the drawing of the window;

[0046] In a specific embodiment, the creation of a window in the Qt framework that allows D3D to perform video drawing specifically includes:

[0047] Create a window through the class QWindow provided by Qt, and use the method provided by QWidget to encapsulate the window into a QWidget control, and the QWidget control can be embedded into the UI implemented by QWidget.

[0048] In a specific embodiment, the blocking of the rendering messages of Qt on the window and the creation of a rendering facility of D3D on the window, so that D3D takes over the drawing of the window, specifically includes:

[0049] Block the rendering events of Qt by overloading the event method of QWindow, create a rendering facility of D3D on the window, and create a SwapChain on the window to convert the rendering method from Qt to D3D;

[0050] Among them, the video part is all rendered by D3D, and the non-video part is drawn by Qt.

[0051] In a specific embodiment, the video part is all rendered by D3D, and the non-video part is drawn by Qt, specifically includes:

[0052] Qt implements UI drawing, and D3D uses shaders to implement the generation and rendering of video, OSD, and video effects.

[0053] S2: When creating a D3D texture for decoding, specify it as D3D11_BIND_SHADER_RESOURCE so that the decoded D3D texture can be bound to a D3D shader for rendering;

[0054] In a specific embodiment, since DXVA requires a texture array to store the decoded video data (in YUV format) during the decoding process, the texture array created by default cannot be directly bound to a D3D shader for rendering. Therefore, it is necessary to specify it as D3D11_BIND_SHADER_RESOURCE when creating the texture, so that the generated YUV texture after decoding can be bound to the shader. It should be noted that since the shader can only receive three components of RGB, the YUV texture data needs to be mapped.

[0055] Figure 2 The flowchart of mapping YUV format texture data according to an embodiment of the present invention is as follows. The process of enabling the decoded D3D texture to be bound to a D3D shader for rendering is specifically as follows:

[0056] Map the texture data in YUV format generated after DXVA decoding, so as to implement the conversion from YUV format to RGB format for the texture data in the D3D shader:

[0057] Map the Y component of the texture data into a first texture containing only the R component, and map the UV components of the texture data into a second texture containing RG components; and, in the shader, access the Y data through the R component of the first texture, and access the UV data through the RG components of the second texture.

[0058] S3: Establish a rendering intermediate layer, which is used to parse an OpenGL shader into a D3D shader and encapsulate D3D functions into the form of OpenGL, so that users can create and compile D3D shaders in the OpenGL way.

[0059] In a specific embodiment, the parsing of the OpenGL shader into a D3D shader specifically includes:

[0060] Translate OpenGL into the SPIR-V format, and then convert SPIR-V into a D3D shader. Among them, the steps of translating OpenGL into the SPIR-V format and converting SPIR-V into a D3D shader are encapsulated into the glCompileShader function to implement the creation and compilation of D3D shaders in the OpenGL way.

[0061] In a specific embodiment, as described in S3 above, the rendering intermediate layer is used to parse OpenGL shaders into D3D shaders and encapsulate D3D functions into the form of OpenGL. The specific process is as follows:

[0062] If the OpenGL module provided by Qt is required in the UI rendering thread, the D3D functions are encapsulated into a complete OpenGL framework to implement the functions of the complete OpenGL module, and the OpenGL framework is compiled into a library file identical to OpenGL for dynamic loading by Qt during runtime;

[0063] If the OpenGL module provided by Qt is not used in the UI rendering thread, that is, the rendering method called by the user is used, the D3D functions are encapsulated into an OpenGL subset to implement partial functions of the OpenGL module. The above OpenGL subset is used by the user to parse OpenGL shaders into D3D shaders through the rendering intermediate layer, so as to create and compile D3D shaders in the OpenGL manner.

[0064] When the UI rendering thread is executed, the rendering intermediate layer executes the process as Figure 3 shown. First, it is judged whether the OpenGL module provided by Qt needs to be used, that is, whether it is rendered by Qt:

[0065] If so, the above library file is loaded, and the OpenGL module provided by Qt performs the rendering;

[0066] If not, the user parses the OpenGL shaders into D3D shaders through the rendering intermediate layer, so as to create and compile D3D shaders in the OpenGL manner to achieve the rendering.

[0067] Embodiment 2:

[0068] The present invention also provides a video rendering device, as Figure 4 shown. The device includes a processor 401, a memory 402, a bus 403, and a computer program stored in the memory 402 and operable on the processor 401. The processor 401 includes one or more processing cores. The memory 402 is connected to the processor 401 through the bus 403. The memory 402 is used to store program instructions. When the processor executes the computer program, the steps in the above method embodiment of Embodiment 1 of the present invention are implemented.

[0069] Further, as an executable solution, the video rendering device may be a computer unit, which may be a computing device such as a desktop computer, a notebook, a palm computer, and a cloud server. The computer unit may include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the above-described composition structure of the computer unit is only an example of the computer unit and does not constitute a limitation on the computer unit. It may include more or fewer components than the above, or combine some components, or different components. For example, the computer unit may further include input / output devices, network access devices, buses, etc., and the embodiments of the present invention do not limit this.

[0070] Further, as an executable solution, the so-called processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor is the control center of the computer unit and connects various parts of the entire computer unit using various interfaces and lines.

[0071] The memory may be used to store the computer program and / or module. The processor realizes various functions of the computer unit by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the mobile phone, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0072] Embodiment 3:

[0073] The present invention also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the method in the above embodiments of the present invention are implemented.

[0074] If the modules / units integrated in the computer unit are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such understanding, all or part of the processes in the method of the above embodiments of the present invention can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0075] Although the present invention is specifically shown and described in conjunction with the preferred embodiments, those skilled in the art should understand that various changes can be made to the present invention in form and detail without departing from the spirit and scope of the present invention defined by the appended claims, and all such changes are within the protection scope of the present invention.

Claims

1. A video rendering method, characterized in that: Using D3D and Qt for hybrid video rendering involves the following steps: S1: In the Qt framework, a window for D3D video rendering is created, Qt rendering messages on the window are shielded, and a D3D rendering facility is created on the window so that D3D takes over the rendering of the window; S2: When creating a D3D texture for decoding, specify it as D3D11_BIND_SHADER_RESOURCE so that the decoded D3D texture can be bound to the D3D shader for rendering; S3: Establishing a rendering intermediate layer, which is used to parse OpenGL shaders into D3D shaders and encapsulate D3D functions into OpenGL format, so that users can create and compile D3D shaders in OpenGL format; The step of binding the decoded D3D texture to a D3D shader for rendering specifically includes: Map the YUV format texture data generated after DXVA decoding, so as to realize the conversion of the YUV format to RGB format in the D3D shader: Mapping the Y component of the texture data to a first texture containing only the R component, and mapping the UV component of the texture data to a second texture containing the RG component; and, in a shader, accessing the Y data through the R component of the first texture and accessing the UV data through the RG component of the second texture; The step of parsing the OpenGL shader into a D3D shader specifically includes: Translate OpenGL into SPIR-V format, and then convert SPIR-V into D3D shader, wherein the steps of translating OpenGL into SPIR-V format and converting SPIR-V into D3D shader are encapsulated into glCompileShader function to realize creation and compilation of D3D shader through OpenGL; The method of encapsulating the D3D function into OpenGL specifically includes: In the UI rendering thread, determine whether you need to use the OpenGL module provided by Qt: If so, encapsulate the D3D function into a complete OpenGL framework, and compile the OpenGL framework into a library file identical to OpenGL so that it can be dynamically loaded by Qt at runtime; If not, the D3D function is encapsulated into an OpenGL subset, so that the user can parse the OpenGL shader into a D3D shader through the rendering intermediate layer, thereby creating and compiling the D3D shader in an OpenGL manner.

2. The method according to claim 1, characterized in that In the Qt framework, a form is created that allows D3D to perform video drawing, specifically including: A form is created by using the QWindow class provided by Qt, and the form is encapsulated into a QWidget control using the method provided by QWidget. The QWidget control can be embedded in a UI implemented by QWidget.

3. The method according to claim 1, characterized in that The shielding of Qt rendering messages on the form and the creation of D3D rendering facilities on the form, so that D3D takes over the drawing of the form, specifically includes: Shield Qt's rendering events by overloading QWindow's event method, create D3D rendering facilities on the window, and create a SwapChain on the window to convert the rendering mode from Qt to D3D; Among them, the video part is all rendered by D3D, and the non-video part is drawn by Qt.

4. The method according to claim 3, characterized in that The video part is all rendered by D3D, and the non-video part is drawn by Qt, specifically including: Qt implements UI drawing, and D3D uses shaders to generate and render videos, OSD, and video effects.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer processor, the video rendering method according to any one of claims 1 to 4 is implemented.

6. A video rendering system, characterized in that: include: D3D rendering window establishment module: configured to create a window for D3D video rendering in the Qt framework, shield Qt rendering messages on the window, and create D3D rendering facilities on the window so that D3D takes over the rendering of the window; D3D shader binding module: Configuration is used to specify it as D3D11_BIND_SHADER_RESOURCE when creating a D3D texture for decoding, so that the decoded D3D texture can be bound to the D3D shader for rendering; A rendering intermediate layer establishment module is configured to establish a rendering intermediate layer, which is used to parse OpenGL shaders into D3D shaders and encapsulate D3D functions into OpenGL format, so that users can create and compile D3D shaders in OpenGL format. The step of binding the decoded D3D texture to a D3D shader for rendering specifically includes: Map the YUV format texture data generated after DXVA decoding, so as to realize the conversion of the YUV format to RGB format in the D3D shader: Mapping the Y component of the texture data to a first texture containing only the R component, and mapping the UV component of the texture data to a second texture containing the RG component; and, in a shader, accessing the Y data through the R component of the first texture and accessing the UV data through the RG component of the second texture; The step of parsing the OpenGL shader into a D3D shader specifically includes: Translate OpenGL into SPIR-V format, and then convert SPIR-V into D3D shader, wherein the steps of translating OpenGL into SPIR-V format and converting SPIR-V into D3D shader are encapsulated into glCompileShader function to realize creation and compilation of D3D shader through OpenGL; The method of encapsulating the D3D function into OpenGL specifically includes: In the UI rendering thread, determine whether you need to use the OpenGL module provided by Qt: If so, encapsulate the D3D function into a complete OpenGL framework, and compile the OpenGL framework into a library file identical to OpenGL so that it can be dynamically loaded by Qt at runtime; If not, the D3D function is encapsulated into an OpenGL subset, so that the user can parse the OpenGL shader into a D3D shader through the rendering intermediate layer, thereby creating and compiling the D3D shader in an OpenGL manner.

7. A video rendering device, characterized in that: The method comprises a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the video rendering method according to any one of claims 1 to 4.

Images