Data zero-copy rendering optimization method and device based on hard decoding

Abstract

The invention provides a data zero-copy rendering optimization method and device based on hard decoding, and the method comprises the steps: creating a client display window, and initializing OpenGL ES equipment and EGL interface context; aVC video data to be decoded are acquired, and the AVC video data are decoded by using a video hard decoder, so that Mppframe information can be obtained; according to the Mppframe information, EGLImageKHR image data is created, according to the EGLImageKHR image data, and according to the EGLImageKHR image data, texture binding is carried out in a glEGLImageTargetTexture2DOES mode, and updating of texture data is completed. And executing texture drawing according to the texture data, a preset texture coordinate and a rendering state, and completing display rendering through eglSwap Buffer. The CPU consumption can be obviously reduced, and the high-quality and high-frame-rate rendering capability is supported.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a data zero-copy rendering optimization method and device based on hard decoding. Background Art

[0002] As cloud desktop office solutions and products are widely used, low-cost thin client terminals have become a preferred solution for customers. For example, using a thin client terminal based on Rockchip (Rock, RK) chips, the terminal uses hardware-supported Advanced Video Coding (AVC), and displays the output data through the central processing unit (Central Processing Unit, CPU) / graphics processing unit (Graphics Processing Unit, GPU) rendering.

[0003] When decoding and rendering video data, it is often necessary to convert the format of a large amount of display data and copy it to memory, which increases the display cost, reduces the display efficiency, and cannot meet the display requirements of high frame rate and high image quality. The hard decoding + zero-copy hard rendering method can avoid the memory copy of data and effectively improve the rendering efficiency, but it has certain requirements for the terminal device and the implementation method is relatively complicated. Summary of the invention

[0004] The main purpose of the embodiments of the present invention is to provide a data zero-copy rendering optimization method and device based on hard decoding, which can significantly reduce CPU consumption and support high-quality and high-frame-rate rendering capabilities.

[0005] In the first aspect, a data zero-copy rendering optimization method based on hard decoding is provided, the method comprising: creating a client display window, and initializing an OpenGL ES device and an EGL interface context; obtaining AVC video data to be decoded, and applying a video hard decoder to decode the AVC video data to obtain Mppframe information; creating EGLImageKHR image data according to the Mppframe information, and binding a texture according to the EGLImageKHR image data in a glEGLImageTargetTexture2DOES manner to complete the update of the texture data; performing texture drawing according to the texture data, preset texture coordinates and rendering status, and completing display rendering through eglSwapBuffer.

[0006] In a possible implementation, a client display window is created, and an OpenGL ES device and an EGL interface context are initialized, which also includes: creating texture data for display, creating a shader, the shader including: a vertex shader and a fragment shader; and binding the window to the EGL interface.

[0007] In another possible implementation, the video hard decoder is used to decode the AVC video data, and before obtaining the Mppframe information, it includes: creating a video hard decoder supported by the device, creating an MppBufferGroup according to the buf_size of the MppFrame returned by get_frame, and configuring it to the video hard decoder through the MPP_DEC_SET_EXT_BUF_GROUP of the control interface.

[0008] In another possible implementation, a video hard decoder is used to decode the AVC video data to obtain Mppframe information, including: decoding each frame of the AVC video data in turn; detecting whether the decoded data information has changed; if so, setting the video hard decoder memory mode and status configuration, and then continuing decoding; otherwise, continuing data decoding and outputting Mppframe information.

[0009] In another possible implementation, EGLImageKHR image data is created according to Mppframe information, including: if the image data to be rendered is complete Mppframe information, EGLImageKHR parameters are directly configured according to the Mppframe information, and EGLImageKHR image data is created through eglCreateImageKHR extension support.

[0010] In another possible implementation, EGLImageKHR parameters are configured according to the Mppframe information, including: if the image data to be rendered is the Mppframe information of a preset area, extracting the Mppframe information of the preset area according to the Mppframe information and storing it in the device memory; configuring EGLImageKHR parameters for the Mppframe information of the preset area stored in the device memory, and creating EGLImageKHR image data through the eglCreateImageKHR extension support.

[0011] In another possible implementation, the Mppframe information includes: frame format, frame width, frame height, and step length.

[0012] In the second aspect, a data zero-copy rendering optimization device based on hard decoding is provided, and the device includes: an initialization unit, used to create a client display window, and initialize the OpenGL ES device and the EGL interface context; a decoding unit, used to obtain the AVC video data to be decoded, and apply the video hard decoder to decode the AVC video data to obtain the Mppframe information; a texture update unit, used to create EGLImageKHR image data according to the Mppframe information, and bind the texture according to the EGLImageKHR image data in a glEGLImageTargetTexture2DOES manner to complete the update of the texture data; a texture drawing unit, used to perform texture drawing according to the texture data, preset texture coordinates and rendering status, and complete the display rendering through eglSwapBuffer.

[0013] In a third aspect, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a data zero-copy rendering optimization method based on hard decoding as provided in the first aspect is implemented.

[0014] In a fourth aspect, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, a data zero-copy rendering optimization method based on hard decoding as provided in the first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in describing the embodiments of the present application.

[0016] Figure 1 A flowchart of a data zero-copy rendering optimization method based on hard decoding provided by an embodiment of the present invention;

[0017] Figure 2 It is a complete implementation diagram of the data zero-copy rendering optimization method based on hard decoding according to an embodiment of the present invention;

[0018] Figure 3 A schematic diagram of texture updating of a data zero-copy rendering optimization method based on hard decoding according to an embodiment of the present invention;

[0019] Figure 4 A structural diagram of a data zero-copy rendering optimization device based on hard decoding provided by an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of the physical structure of an electronic device provided by the present invention.

[0021] Specific implementation method

[0022] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar modules or modules with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present invention.

[0023] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, modules and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, modules, components and / or groups thereof. It should be understood that when we refer to a module as being "connected" or "coupled" to another module, it may be directly connected or coupled to the other modules, or there may be intermediate modules. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any modules and all combinations of one or more associated listed items.

[0024] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation method of the present application will be further described in detail below with reference to the accompanying drawings.

[0025] The technical solution of the present application and how to solve the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0026] like Figure 1 The flowchart of a data zero-copy rendering optimization method based on hard decoding provided by an embodiment of the present invention is shown, which is applied to a client, and the method includes:

[0027] Step S11, creating a client display window, and initializing the OpenGL ES device and the EGL interface context.

[0028] OpenGL ES (OpenGL for Embedded Systems) is a subset of the Open Graphics Library (OpenGL) 3D graphics API, designed for embedded devices such as mobile phones, PDAs, and game consoles. EGL is a bridge interface between OpenGL and the native window system.

[0029] Step S12, obtaining the AVC video data to be decoded, and applying the video hardware decoder to decode the AVC video data to obtain Mppframe information.

[0030] Step S13, creating EGLImageKHR image data according to the Mppframe information, and binding the texture according to the EGLImageKHR image data in the glEGLImageTargetTexture2DOES mode, thereby completing the update of the texture data.

[0031] Step S14, performing texture rendering according to texture data, preset texture coordinates and rendering state, and completing display rendering through eglSwapBuffer.

[0032] The embodiment of the present invention applies a video hardware decoder to decode AVC video data to obtain Mppframe information; creates EGLImageKHR image data according to the Mppframe information, and binds the texture in the glEGLImageTargetTexture2DOES manner according to the EGLImageKHR image data to complete the update of the texture data; performs texture drawing according to the texture data, preset texture coordinates and rendering state, and completes display rendering through eglSwapBuffer, that is, using the hard-decoded data to generate a zero-copy texture data update mechanism of EGLImageKHR, which can significantly reduce CPU consumption and support high-quality and high-frame-rate rendering capabilities.

[0033] In step S11, texture data for display is created, a shader is created, and the shader includes: a vertex shader and a fragment shader; and the window is bound to the EGL interface.

[0034] Before step S12, a video hardware decoder (Media Process Platform, Mpp) supported by the device is created, an MppBufferGroup is created according to the buf_size of the MppFrame returned by get_frame, and is configured to the video hardware decoder through MPP_DEC_SET_EXT_BUF_GROUP of the control interface.

[0035] In step S12, the video hard decoder is used to decode the AVC video data to obtain Mppframe information. Optionally, each frame of the AVC video data is decoded in turn; it is detected whether the decoded data information has changed; if so, the video hard decoder memory mode is set and the state is configured, and then decoding is continued; otherwise, data decoding is continued and Mppframe information is output. The AVC video data is preferably H264 data.

[0036] In step S13, if the image data to be rendered is complete Mppframe information, EGLImageKHR parameters are configured directly according to the Mppframe information, and EGLImageKHR image data is created through eglCreateImageKHR extension support. The Mppframe information includes: frame format, frame width, frame height and step length.

[0037] If the image data to be rendered is the Mppframe information of the preset area, the Mppframe information of the preset area is extracted according to the Mppframe information and stored in the device memory; the Mppframe information of the preset area stored in the device memory is configured with EGLImageKHR parameters, and the EGLImageKHR image data is created through the eglCreateImageKHR extension support. That is, after the video data is hard-decoded, the output frame provides the memory fd, and EGL generates the corresponding texture data using the frame fd through the supported EGLImageKHR extension method.

[0038] The embodiment of the present invention performs texture binding in glEGLImageTargetTexture2DOES mode according to EGLImageKHR image data to complete the update of texture data. The way of updating texture data is the key to achieving zero data copy from decoding to rendering. The traditional glTexImage2D / glTexSubImage2D mode of updating data texture will result in a copy of the device data memory. However, through the extended mode supported by EGLImageKHR, the device memory fd address in the hard-decoded output frame is used to create EGLImageKHR, and the data of the current texture is set through glEGLImageTargetTexture2DOES, thereby completing the update of texture data in a zero-copy mode, which can significantly reduce CPU consumption and support high-quality and high-frame-rate rendering capabilities.

[0039] After the texture data is updated, texture rendering is performed according to the updated texture data, preset texture coordinates and rendering state, and display rendering is completed through eglSwapBuffer. The embodiment of the present invention uses OPenGL ES to control the rendering of data, and controls the display position and effect, as well as the drawing operation of the updated texture through self-written vertex shader (Shader) and fragment shader. Display coordinates are processed in the vertex shader, and pixel colors are processed in the fragment shader to complete the update of texture data.

[0040] In an embodiment of the present invention, when using the MPP decoder to hard-decode AVC video data, the memory application method of DRM can be configured, and the output of the decoded data exists in the device memory. When the device memory is dumped to the CPU using a mapping method, it needs to go through the CPU cache, and during the rendering process, it needs to go through operations such as format conversion or memory copying, which takes a long time to process. Textures rendered using OpenGL ES can directly use device memory, and the data results of the decoded output can be used directly as textures. With the same display effect, the embodiment of the present invention can significantly reduce CPU consumption, enable the client to support high-quality and high-frame rate (1080p / 60fps) rendering capabilities, and enable the client to friendly support the rendering display of scenes such as dual-screen cloud desktops.

[0041] The complete implementation process of the data zero-copy rendering optimization method based on hard decoding in the embodiment of the present invention is as follows: Figure 2 As shown, first create a local window for display, create EGL and initialize it. Create a video hardware decoder for AVC decoding of video compression data. Get compressed data, insert the data into the video hardware decoder, and decode the data. When the decoded data information is detected to change, set the decoder memory mode and configure the state. Otherwise, continue to decode the data and output the Frame. Create a new EGLImageKHR image data based on the extracted frame information, bind the texture to the EGLImageKHR image data, and update the texture. Finally, complete the texture coordinate and rendering state settings, and execute drawing and rendering display. For the detailed process of texture update, see Figure 3 , parse the complete frame data output by the video hard decoder MPP decoding. Create EGLImageKHR image data according to the frame information. The frame information includes the memory descriptor fd, the frame format format, the frame width width, and the frame height height. Finally, perform texture target binding and drawing to complete the texture update. According to the embodiment of the present invention, the video hard decoder is set to apply for memory in MPP_BUFFER_TYPE_DRM mode based on the frame information output by the video hard decoding. The memory usage of the video hard decoder can be limited based on experience, and at the same time, it can also ensure that the system can provide such a large memory space.

[0042] The present invention provides a client solution for a thin client based on an RK chip, which performs hardware decoding of AVC video data through device support and uses OpenGL ES+EGL to render and display data. During rendering, the frame data output by hard decoding can be directly used as texture for drawing and display without the need for intermediate processes of data conversion, data mapping and data memory copying, thereby solving the display problems of long rendering time, high CPU occupancy and inability to meet high frame rate rendering requirements during data client rendering.

[0043] The embodiment of the present invention creates a client display window and initializes an OpenGL ES device and an EGL interface context; obtains AVC video data to be decoded, and applies a video hardware decoder to decode the AVC video data to obtain Mppframe information; creates EGLImageKHR image data according to the Mppframe information, and binds the texture according to the EGLImageKHR image data in a glEGLImageTargetTexture2DOES manner to complete the update of the texture data; performs texture drawing according to the texture data, preset texture coordinates and rendering status, and completes display rendering through eglSwapBuffer, which can significantly reduce CPU consumption and support high-quality and high-frame-rate rendering capabilities.

[0044] like Figure 4 FIG. 1 is a structural diagram of a data zero-copy rendering optimization device based on hard decoding provided by an embodiment of the present invention, the device comprising:

[0045] Initialization unit 401, used to create a client display window and initialize the OpenGL ES device and EGL interface context;

[0046] The decoding unit 402 is used to obtain the AVC video data to be decoded, and apply the video hardware decoder to decode the AVC video data to obtain Mppframe information;

[0047] The texture update unit 403 is used to create EGLImageKHR image data according to the Mppframe information, and bind the texture according to the EGLImageKHR image data in the glEGLImageTargetTexture2DOES mode to complete the update of the texture data;

[0048] The texture rendering unit 404 is used to perform texture rendering according to texture data, preset texture coordinates and rendering state, and complete display rendering through eglSwapBuffer.

[0049] The device of the above embodiment is applied to the corresponding method in the above embodiment, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0050] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor (processor) 501, a communication interface (Communications Interface) 502, a memory (memory) 503 and a communication bus 504, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus. The processor may call the logic instructions in the memory to execute a data zero-copy rendering optimization method based on hard decoding, the method comprising: creating a client display window, and initializing the OpenGL ES device and the EGL interface context; obtaining the AVC video data to be decoded, and applying the video hard decoder to decode the AVC video data to obtain the Mppframe information; creating EGLImageKHR image data according to the Mppframe information, and binding the texture according to the EGLImageKHR image data in the glEGLImageTargetTexture2DOES mode to complete the update of the texture data; performing texture drawing according to the texture data, the preset texture coordinates and the rendering state, and completing the display rendering through eglSwapBuffer.

[0051] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0052] On the other hand, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute a data zero-copy rendering optimization method based on hard decoding provided by the above-mentioned method embodiments, the method including: creating a client display window, and initializing an OpenGL ES device and an EGL interface context; obtaining AVC video data to be decoded, and applying a video hard decoder to decode the AVC video data to obtain Mppframe information; creating EGLImageKHR image data according to the Mppframe information, and binding the texture according to the EGLImageKHR image data in a glEGLImageTargetTexture2DOES manner to complete the update of the texture data; performing texture drawing according to the texture data, preset texture coordinates and rendering state, and completing display rendering through eglSwapBuffer.

[0053] On the other hand, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute a data zero-copy rendering optimization method based on hard decoding provided in the above-mentioned embodiments, the method comprising: creating a client display window, and initializing an OpenGL ES device and an EGL interface context; obtaining AVC video data to be decoded, and applying a video hard decoder to decode the AVC video data to obtain Mppframe information; creating EGLImageKHR image data according to the Mppframe information, and binding the texture according to the EGLImageKHR image data in a glEGLImageTargetTexture2DOES manner to complete the update of the texture data; performing texture drawing according to the texture data, preset texture coordinates and rendering status, and completing display rendering through eglSwapBuffer.

[0054] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0055] The above is only a partial implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data zero-copy rendering optimization method based on hard decoding, characterized in that: The method comprises: Create a client display window and initialize the OpenGL ES device and EGL interface context; Obtaining AVC video data to be decoded, and applying a video hardware decoder to decode the AVC video data to obtain Mppframe information; Create EGLImageKHR image data according to the Mppframe information, and perform texture binding in the glEGLImageTargetTexture2DOES mode according to the EGLImageKHR image data to complete the update of texture data; Texture rendering is performed according to the texture data, preset texture coordinates and rendering state, and display rendering is completed through eglSwapBuffer.

2. The method according to claim 1, characterized in that The step of creating a client display window and initializing the OpenGLES device and the EGL interface context also includes: Creating texture data for display, creating a shader, the shader comprising: a vertex shader and a fragment shader; Bind the window to the EGL interface.

3. The method according to claim 2, characterized in that The application of the video hardware decoder to decode the AVC video data to obtain the Mppframe information includes: Create a video hardware decoder supported by the device, create an MppBufferGroup according to the buf_size of the MppFrame returned by get_frame, and configure it to the video hardware decoder through MPP_DEC_SET_EXT_BUF_GROUP of the control interface.

4. The method according to claim 1, characterized in that The application video hardware decoder decodes the AVC video data to obtain Mppframe information, including: Decoding each frame of the AVC video data in sequence; Detect whether the decoded data information has changed; If yes, then the memory mode setting and state configuration of the video hard decoder are performed, and then decoding is continued; Otherwise, continue to decode the data and output the Mppframe information.

5. The method according to claim 1, characterized in that The step of creating EGLImageKHR image data according to the Mppframe information includes: If the image data to be rendered is the complete Mppframe information, the EGLImageKHR parameters are configured directly according to the Mppframe information, and the EGLImageKHR image data is created through the eglCreateImageKHR extension support.

6. The method according to claim 1, characterized in that The configuring of EGLImageKHR parameters according to the Mppframe information includes: If the image data to be rendered is Mppframe information of a preset area, extracting the Mppframe information of the preset area according to the Mppframe information and storing it in a device memory; The Mppframe information of the preset area stored in the device memory is configured with EGLImageKHR parameters, and EGLImageKHR image data is created through the eglCreateImageKHR extension support.

7. The method according to claim 5 or 6, characterized in that The Mppframe information includes: frame format, frame width, frame height and step length.

8. A data zero-copy rendering optimization device based on hard decoding, characterized in that: The device comprises: Initialization unit, used to create a client display window and initialize the OpenGL ES device and EGL interface context; A decoding unit, used for acquiring AVC video data to be decoded, and applying a video hardware decoder to decode the AVC video data to obtain Mppframe information; A texture update unit, used for creating EGLImageKHR image data according to the Mppframe information, and binding the texture in a glEGLImageTargetTexture2DOES manner according to the EGLImageKHR image data to complete the update of the texture data; The texture rendering unit is used to perform texture rendering according to the texture data, preset texture coordinates and rendering state, and complete display rendering through eglSwapBuffer.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, a data zero-copy rendering optimization method based on hard decoding is implemented as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a data zero-copy rendering optimization method based on hard decoding is implemented as described in any one of claims 1 to 7.

Images