Xwindow-based video playing hardware acceleration method

Abstract

The application is suitable for the field of GPU technology, and provides a video playing hardware acceleration method based on XWindow, which comprises the following steps: obtaining compressed video stream data, decoding and saving the video stream data; obtaining the physical address of a pixel map by using Xserver self-defined extension; converting and scaling the decoded video original data, and outputting the obtained to-be-displayed data to the physical address corresponding to the pixel map; displaying the pixel map to a window and outputting to a display device for display. The method directly saves the step of data transmission by using XServer self-defined extension to obtain the physical address of the pixel map and outputting the to-be-displayed data after data conversion and scaling to the physical address corresponding to the pixel map. The application is a video playing hardware acceleration method with a decoding process and a display process separated, the decoding process is used for decoding task only, and the data conversion and scaling operation is placed in the display process, so that the problems of frequent data exchange, lag and other video playing problems can be solved.

Description

Technical Field

[0001] This invention belongs to the field of GPU technology, and in particular relates to a hardware acceleration method for video playback based on X Window. Background Technology

[0002] Currently, most Linux desktop operating systems use the X Window system for their graphics display. Furthermore, mainstream GPUs, in addition to supporting graphics processing and display, generally also support video processing. X Window-based video playback applications typically use the CPU or GPU for video decoding. After decoding, the X Window application captures mouse dragging and stretching operations through the X Server to obtain the coordinates and size information of the video to be displayed. Based on this information, the application uses the CPU or GPU to perform color conversion and scaling on the decoded video data. After processing the decoded video, the application also needs to send the data back to the X Server for display on the display device. However, this method of X Window video playback involves frequent data exchange between the CPU and GPU, increasing CPU or GPU utilization and resulting in low video delivery efficiency.

[0003] To address this issue, one existing solution is to use a shared memory mechanism. This involves creating a globally shared memory segment accessible to both the XServer and the application to store the raster data of the XImage object, thus eliminating the need for extensive raster data copying during video playback. However, this solution has the drawback that the shared memory segment resides on the CPU, making it inconvenient for the GPU to access, perform data conversion, scaling, and display; data transfer is still required. Another solution is to use XVideo for video playback. The drawback is that the video data to be displayed still needs to be passed to the XServer, although a dedicated interface exists, making the transfer more efficient. Additionally, existing technologies utilize OpenGL for image display. The drawback is the need for more complex hardware, such as 3D hardware, to implement an efficient OpenGL interface.

[0004] Furthermore, patent number 202110228958.6 discloses a graphics processor and a decoding and display method. This graphics processor includes a hardware decoding module, a double-rate synchronous dynamic random access memory (DRAM), an output module, and a 2D hardware module. Video decoding, data copying, and output display are performed by the graphics processor, effectively reducing the utilization of the central processing unit (CPU) and the time spent on data processing and data copying, thus improving the efficiency of video decoding and display. However, this patented method is not based on X Window video playback applications and is typically used in embedded systems. Even if this method were applied to X Window video playback, many technical problems would arise.

[0005] Specifically, this patented method requires a hardware decoding module to first obtain the parameters of the video window used to display the video data, and then decode the video data according to the parameters of the video window so that the decoded video data matches the video window. Data conversion and scaling operations are all completed by the decoding module. The problems with this approach are: 1. The decoding process is lengthy, requiring parameter acquisition, decoding based on the parameters, and finally data conversion and scaling. This lengthy process leads to greater display latency and increases the total processing time for each frame, resulting in a lower frame rate. 2. According to video compression principles, if bidirectional inter-frame prediction exists, the decoded frame will not be displayed immediately. There is a significant time interval between determining the window parameters for the current frame and its final display. If the application changes the window during this time, it cannot meet the application requirement of real-time window response. 3. It is not conducive to generating multiple display buffer frames. If multiple display buffer frames are generated, the time interval between determining the window parameters, decoding, and final display will also be long, failing to meet the application requirement of real-time window response. Summary of the Invention

[0006] In view of the above problems, the purpose of this invention is to provide a solution to the technical problems of low display efficiency, such as frequent data exchange, high CPU or GPU utilization, and even video playback stuttering, in video playback applications based on X Window under the Linux desktop operating system.

[0007] The present invention adopts the following technical solution:

[0008] The XWindow-based video playback hardware acceleration method includes the following steps:

[0009] Step S1: Obtain compressed video stream data, decode and save the video stream data;

[0010] Step S2: Obtain the physical address of the pixel image using the Xserver custom extension;

[0011] Step S3: Convert and scale the decoded raw video data, and output the resulting data to be displayed to the physical address corresponding to the pixel map;

[0012] Step S4: Display the pixel image in the window and output it to the display device for display.

[0013] Furthermore, the specific process of step S1 is as follows:

[0014] S11. Read the encapsulated video data from the hard drive or network;

[0015] S12. Decapsulate the encapsulated video data to obtain compressed video stream data, and parse the frame header information of the video stream data.

[0016] S13. Based on the width, height, and compressed video format information in the frame header, check whether hardware decoding is supported;

[0017] S14. If hardware decoding is not supported, request main memory space;

[0018] S15. Read in the compressed video stream data and call the decoding library to perform software decoding. Store the decoded original video data in the main memory space.

[0019] S16. If hardware decoding is supported, request the first and second storage spaces.

[0020] S17. Read the compressed video stream data and store it in the first storage space;

[0021] S18. Read the compressed video stream data from the first storage space and perform hardware decoding. Store the decoded original video data in the second storage space.

[0022] Furthermore, the specific process of step S2 is as follows:

[0023] S21. Obtain the parameters for displaying the video window;

[0024] S22. Create a new pixel image and request a third storage space, then associate and bind the pixel image with the third storage space;

[0025] S23. Use the Xserver custom extension to obtain the physical address of the pixel image, that is, the physical address of the third storage space;

[0026] S24. Obtain the data address of the original decoded video data to be displayed. When software decoding is used, the data address is the physical address of the main memory space. When hardware decoding is used, the data address is the physical address of the second memory space.

[0027] Furthermore, the specific process of step S3 is as follows:

[0028] The original video data is read from the data address and converted. At the same time, the original video data is scaled to the size that matches the window, and the result is output to the third storage space.

[0029] The beneficial effects of this invention are:

[0030] First, in X Window-based video playback applications, the method of this invention utilizes a custom extension of XServer to obtain the physical address of the pixel image and outputs the data to be displayed after data conversion and scaling to the physical address corresponding to the pixel image. This eliminates the step where the application needs to send the data to be displayed back to XServer for display on the display device, reducing the data copying process. Furthermore, it uses dedicated hardware as much as possible for the corresponding operations, reducing CPU and GPU usage, freeing up hardware performance, and making video playback smoother.

[0031] Secondly, the method of this invention separates decoding and display. The decoding process only performs decoding tasks, while the data conversion and scaling operations are performed by the display process. This allows for a pipelined operation, resulting in lower decoding latency, shorter average total processing time per frame, and faster window response when applications change. Furthermore, since the decoding process only performs decoding, it better aligns with video compression and decoding principles. It can decode multiple frames at once, fully utilizing inter-frame prediction, and then rearranges and displays them in the correct order, resulting in more accurate frame display. Moreover, because the decoding process only performs decoding, it can continuously decode multiple frames before displaying them. Therefore, this method has more display buffer frames, reducing the probability of stuttering and resulting in a smoother display.

[0032] Finally, the method of this invention does not use interfaces such as OpenGL that require complex 3D hardware, and can use hardware with simpler structures or be applicable to a wider range of GPUs. Attached Figure Description

[0033] Figure 1 This is a structural block diagram of the video playback system provided in an embodiment of the present invention;

[0034] Figure 2 This is a flowchart of a video playback hardware acceleration method based on X Window provided in an embodiment of the present invention.

[0035] Figure 3 This is a detailed flowchart of step S1 provided in an embodiment of the present invention;

[0036] Figure 4 This is a structural block diagram of the GPU execution unit provided in an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0038] To illustrate the technical solution described in this invention, specific embodiments are described below.

[0039] Example 1:

[0040] The method of this invention is implemented based on Figure 1 The video playback system shown is implemented as follows. The system includes a host computer and a GPU. The GPU is inserted into the host computer's PCIe slot and connected to the host via the PCIe bus. The X Window-based video playback application runs on the host computer's CPU and memory. The CPU can access and operate the GPU via the PCIe bus, including accessing the GPU's video memory. The GPU can also access the host computer's memory via the PCIe bus. The GPU's execution unit can perform hardware decoding of compressed video stream data, as well as data conversion, scaling, and output operations. In this video playback system, the GPU can retrieve compressed video stream data from memory through the CPU. After retrieving the compressed video stream data, the GPU can perform hardware decoding. Based on the above system, and combined with... Figure 2 As shown, the video playback hardware acceleration method based on X Window provided in this embodiment includes the following steps:

[0041] Step S1: Obtain compressed video stream data, decode and save the video stream data.

[0042] A typical X Window-based video playback application includes at least one decoding process and one display process. This step is the decoding process, and steps S2-S4 are the display process. The storage location of the video stream data differs between hardware decoding and software decoding. (Combined...) Figure 3 As shown, the specific process of this step is as follows:

[0043] S11. Read encapsulated video data from hard drive or network.

[0044] After capturing video data, the camera compresses and encapsulates the raw video data into encapsulated video data according to the settings. The X Window-based application, i.e. the host CPU, reads the encapsulated video data from the local hard drive or online (such as a network server).

[0045] S12. Decapsulate the encapsulated video data to obtain compressed video stream data, and parse the frame header information of the video stream data.

[0046] S13. Based on the width, height, and compressed video format information in the frame header, check whether hardware decoding is supported.

[0047] The CPU will encapsulate the video data according to the corresponding protocol and further parse the frame header information of the compressed video stream data. The frame header information contains various parameters of the video, including video width and height, compressed video format, etc., which can be used to determine whether the GPU currently connected to the host supports video hardware decoding.

[0048] S14. If hardware decoding is not supported, request main memory space.

[0049] If hardware decoding is not supported, the data needs to be stored locally. Therefore, the CPU needs to request main memory space H1 locally, that is, request a memory range to store the original video data after decoding.

[0050] S15. Read in the compressed video stream data and call the decoding library to perform software decoding. The decoded original video data is stored in the main memory space.

[0051] The CPU reads the compressed video stream data, calls the CPU decoding library to decode it, and stores the decoded raw video data in main memory space H1. Since the CPU reads the compressed video stream data in segments, it performs corresponding software decoding and saves the decoded data after each segment is read. After each software decoding and data saving process, it checks whether the compressed video stream data has ended. If the compressed video stream data is not finished, it continues reading; if the video stream data has ended, the process terminates.

[0052] S16. If hardware decoding is supported, request the first and second storage spaces.

[0053] If GPU hardware decoding is supported, the CPU needs to request two memory spaces from the GPU, which are called video memory. The first memory space T1 is used to store the compressed video stream data, and the second memory space T2 is used to store the original decoded video data.

[0054] S17. Read the compressed video stream data and store it in the first storage space.

[0055] S18. Read the compressed video stream data from the first storage space and perform hardware decoding. Store the decoded original video data in the second storage space.

[0056] The CPU reads the compressed video stream data and stores it in the GPU's first storage space T1. The GPU's execution unit reads the compressed video stream data from the first storage space T1 and performs hardware decoding. After hardware decoding, the original video data is stored in the second storage space T2.

[0057] Similarly, during hardware decoding, the CPU reads the compressed video stream data in segments. After each unit performs hardware decoding and saves the data, it checks whether the video stream data has ended. If the video stream data has not ended, it continues to read the next segment for hardware decoding. If the video stream data has ended, the process ends.

[0058] Step S2: Use the Xserver custom extension to obtain the physical address of the pixel image.

[0059] Steps S2-S4 constitute the display process. Specifically, the process for this step is as follows:

[0060] S21. Obtain the parameters for displaying the video window.

[0061] The parameters of the window can be the display position of the video window, the size of the video window, and the output format of the video window, etc.

[0062] S22. Create a new pixel image and request a third storage space, then associate and bind the pixel image with the third storage space.

[0063] S23. Use the Xserver custom extension to obtain the physical address of the pixel image, that is, the physical address of the third storage space.

[0064] In this step, a new pixel map (P_rgb) is created, and a third storage space (T3) is allocated and bound to it. The physical address of the pixel map (P_rgb), i.e., the physical address of the third storage space (T3), is obtained using the Xserver custom extension (ljw_ext). This step requires implementing the custom extension (ljw_ext) in the GPU's XServer driver to obtain the physical address of the pixel map, and loading the custom extension (ljw_ext) along with the XServer initialization during driver loading.

[0065] S24. Obtain the data address of the original decoded video data to be displayed. When software decoding is used, the data address is the physical address of the main memory space. When hardware decoding is used, the data address is the physical address of the second memory space.

[0066] Software decoding is CPU decoding, and the original video data is stored in the main memory space H1. Hardware decoding is GPU decoding, and the original video data is stored in the second storage space T2.

[0067] Step S3: Convert and scale the decoded raw video data, and output the resulting data to be displayed to the physical address corresponding to the pixel map.

[0068] The GPU reads the raw video data from the data address and performs data conversion. Simultaneously, it scales the raw video data to a size matching the window size and outputs the result to the third storage space T3. The data conversion and scaling operations can be performed by the 2D module of the GPU execution unit or by the post-processing module of the execution unit's decoder. Data conversion generally refers to converting the raw video data in YUV format to RGB format data so that the GPU can output and display it.

[0069] Step S4: Display the pixel image P_rgb in the draw window and output it to the display device for display.

[0070] From the method process of this embodiment, it can be seen that this method is based on the X Window graphics display system and is used in the desktop computer field. This method uses a custom extension of XServer to obtain the physical address of the pixel image and outputs the data to be displayed after data conversion and scaling to the physical address corresponding to the pixel image, without the need for data transfer; this step of data transfer can be directly eliminated. Moreover, this invention is a video playback hardware acceleration method that separates the decoding process from the display process. The decoding process is dedicated to decoding tasks, while the data conversion and scaling operations are placed in the display process, which can solve the problems described in the background art. This invention enables more efficient implementation of X Window-based video playback applications, making video playback smoother and more fluid, minimizing CPU or GPU utilization, and using simpler hardware structures or a wider range of applicable GPUs.

[0071] Example 2:

[0072] This embodiment provides a video playback system with hardware acceleration for video playback, the structure of which is as follows: Figure 1 As shown. It includes a host and a GPU. The GPU is inserted into the host's PCIe slot and connected to the host via the PCIe bus. The GPU includes an execution unit and video memory, while the host includes a CPU, memory, and a PCIe slot. The execution units of the CPU and GPU can implement the steps described in Embodiment 1.

[0073] Specifically, the execution units of the GPU include:

[0074] The hardware decoding module is used to decode compressed video stream data into raw video data and store the decoded raw video data in video memory or host memory.

[0075] The data conversion and scaling module is used to read the decoded raw video data from the video memory or host memory, perform data conversion and scaling, generate RGB data to be displayed that conforms to the window size, and output it to the physical address corresponding to the pixel map in the video memory.

[0076] The output module is used to display the RGB data to be shown in the window and output it to the display device for display.

[0077] The host's CPU is used to load X Window-based applications. First, the encapsulated video data is decapsulated. Then, it is analyzed to determine if hardware decoding is supported. If hardware decoding is not supported, a decoding library is called for software decoding, and the decoded data is saved in memory. If hardware decoding is supported, the GPU's first and second storage spaces are allocated. The video stream data is compressed and stored in the first storage space, where it is hardware decoded by the GPU's hardware decoding module. The decoded original video data is then stored in the second storage space.

[0078] The GPU uses the Xserver custom extension to obtain the physical address of the pixel image, namely the third storage space T3. The decoded raw video data is then converted and scaled by the data conversion and scaling module. The resulting data to be displayed is output to the physical address corresponding to the pixel image. Finally, the output module displays the pixel image P_rgb in the window draw and outputs it to the display device for display.

[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hardware acceleration method for video playback based on X Window, characterized in that, The method includes the following steps: Step S1: Obtain compressed video stream data, decode and save the video stream data; Step S2: Obtain the physical address of the pixel image using the Xserver custom extension; Step S3: Convert and scale the decoded raw video data, and output the resulting data to be displayed to the physical address corresponding to the pixel map; Step S4: Display the pixel image in the window and output it to the display device for display. The specific process of step S1 is as follows: S11. Read the encapsulated video data from the hard drive or network; S12. Decapsulate the encapsulated video data to obtain compressed video stream data, and parse the frame header information of the video stream data. S13. Based on the width, height, and compressed video format information in the frame header, check whether hardware decoding is supported; S14. If hardware decoding is not supported, request main memory space; S15. Read in the compressed video stream data and call the decoding library to perform software decoding. Store the decoded original video data in the main memory space. S16. If hardware decoding is supported, request the first and second storage spaces. S17. Read the compressed video stream data and store it in the first storage space; S18. Read the compressed video stream data from the first storage space and perform hardware decoding. Store the decoded original video data in the second storage space. The specific process of step S2 is as follows: S21. Obtain the parameters for displaying the video window; S22. Create a new pixel image and request a third storage space, then associate and bind the pixel image with the third storage space; S23. Use the Xserver custom extension to obtain the physical address of the pixel image, that is, the physical address of the third storage space; S24. Obtain the data address of the original decoded video data to be displayed. When software decoding is used, the data address is the physical address of the main memory space. When hardware decoding is used, the data address is the physical address of the second memory space.

2. The video playback hardware acceleration method based on X Window as described in claim 1, characterized in that, The specific process of step S3 is as follows: The original video data is read from the data address and converted. At the same time, the original video data is scaled to the size that matches the window, and the result is output to the third storage space.

Images