A video hardware acceleration apparatus and method suitable for virtual display cards

Abstract

The application relates to a video hardware acceleration device and method suitable for a virtual display card, the device comprising a virtual display card video driver, a video processing module of the virtual display card, a virtual display card video protocol, a virtual display card video service program, and the virtual display card being a Virtio GPU; the method is assisted by the above acceleration device, a video processing request of a virtual machine is forwarded to a physical display card on a host side, and a video processing module in the physical display card is used for hardware video coding and decoding. The application realizes the video hardware acceleration function of the virtual display card, can support hardware decoding, coding and processing of the video, and improves the video processing performance of the virtual machine while significantly reducing the occupation of CPU resources.

Description

Technical Field

[0001] This patent application belongs to the field of virtualization technology, and more specifically, relates to a video hardware acceleration device and method suitable for virtual graphics cards. Background Technology

[0002] Currently, there are two main categories of technical solutions for display virtualization in virtualization technology:

[0003] One type is hardware virtualization solutions, represented by technologies such as NVIDIA GRID vGPU. These solutions divide a physical graphics card into several virtual graphics cards at the hardware level and then allocate them to virtual machines through pass-through. Because they are mainly implemented in hardware, their performance is excellent and they are often used in professional scenarios such as deep learning and scientific computing. However, they also have problems such as high cost, need for licensing, and lack of flexibility in partitioning.

[0004] Another type is software virtualization solutions, represented by virtual graphics cards such as QXL and Virtio GPU. These solutions simulate graphics card devices through software, so they are low-cost, highly compatible, and easy to deploy, and are widely used in scenarios such as cloud desktops and remote offices.

[0005] While virtual graphics cards are widely used, their drawbacks are also quite obvious. For purely software-simulated virtual graphics cards like QXL, which rely entirely on the CPU for image rendering and video processing, performance is extremely low. Compared to QXL, Virtio GPU currently supports 3D hardware acceleration, forwarding the virtual machine's graphics rendering requests to the physical graphics card on the host machine for processing, thus greatly enhancing its graphics performance. However, in video processing, Virtio GPU, like QXL, still uses software encoding and decoding, resulting in severely insufficient video processing performance and significant CPU resource consumption.

[0006] The disadvantages of existing technology are:

[0007] Chinese invention patent “A hardware decoding method, decoding device and storage medium based on virtualization” (patent number: CN 107911700 B). This invention discloses a hardware decoding method, decoding device and storage medium based on virtualization. The invention obtains the video data to be decoded in a first operating system, which is the operating system of a virtual machine. In the first operating system, the video data to be decoded is intercepted by a preset hook program and sent to a second operating system, which is the operating system of the host machine. In the second operating system, the video data to be decoded is decoded to obtain the corresponding decoded video data. By forwarding the video data to be decoded in the first operating system to the second operating system to call the GPU for decoding, this invention improves the problem of excessive CPU load caused by decoding by the CPU, and thus solves the technical problem of excessive physical device operation burden when the virtual machine performs video decoding in the prior art. However, this invention also has obvious shortcomings: (1) This invention only supports hardware decoding function and does not support hardware encoding and video processing functions; (2) Because this invention adopts API hijacking technology based on hook program, it can only support a specific decoding API at the same time and cannot support other APIs. In practice, the API hijacked by this invention is Microsoft's DXVA (DirectX Video Acceleration) interface, so it is only applicable to the Windows operating system; (3) after the version of the hijacked API is updated, the invention needs to be modified and re-adapted, which can easily cause compatibility problems.

[0008] Currently, some virtual graphics cards (such as Virtio GPU) already support 3D hardware acceleration, which can transfer the graphics rendering requests of the virtual machine to the physical graphics card on the host machine (i.e., the physical machine) for processing. As a result, its graphics performance is greatly improved, significantly enhancing the user experience in scenarios such as games, 3D graphics, and modeling.

[0009] While virtual graphics cards offer significant improvements in graphics rendering, they still have a noticeable weakness in video processing. Because virtual graphics cards lack hardware acceleration for video, users playing high-definition videos with media players or editing videos with video editing software must rely on the CPU for software decoding or encoding. This obviously leads to insufficient performance and consumes a large amount of CPU resources, thus affecting the normal operation of other applications and even the operating system. Tests showed that with a Kunpeng 920 processor and an AMD Radeon RX550 graphics card, when playing 4K / 60fps H.264 video in a virtual machine (with Virtio GPU), the player's CPU usage could reach approximately 321%, severely impacting the user experience.

[0010] On the other hand, mainstream physical graphics cards currently include not only graphics rendering modules but also built-in video processing modules specifically for hardware video encoding and decoding. Currently, virtual graphics cards still use software encoding and decoding, which consumes significant CPU resources while simultaneously wasting the powerful computing capabilities of the host machine's physical graphics card.

[0011] In summary, there is an urgent need to invent a video hardware acceleration mechanism to add video hardware acceleration functionality to virtual graphics cards. This would allow virtual graphics cards to utilize the physical graphics card on the host machine for hardware encoding and decoding of video, thereby improving the video processing performance of virtual machines, reducing CPU resource consumption, and enhancing the user experience. Summary of the Invention

[0012] To address the aforementioned technical challenges, this invention creates a new video hardware acceleration device and method suitable for virtual graphics cards. This device and method can realize the video hardware acceleration function of virtual graphics cards, improve the video performance of virtual machines, reduce CPU resource consumption, and enhance the user experience.

[0013] To solve the above problems, the technical solution adopted by the present invention is as follows:

[0014] A video hardware acceleration device for virtual graphics cards is located in a host machine or a virtual machine. The virtual machine includes an application program and a first video acceleration library connected by information. The host machine includes a second video acceleration library, a physical graphics card video driver, and a physical graphics card connected by information in sequence. The host machine allocates a virtual graphics card to the virtual machine. The functional implementation modules of the virtual graphics card are distributed in the virtual machine and the host machine. The functional implementation modules of the virtual graphics card include a virtual graphics card video driver distributed in the virtual machine, a video processing module of the virtual graphics card distributed in the host machine, and a virtual graphics card video service program. The virtual graphics card video driver and the video processing module of the virtual graphics card are connected by information through a virtual graphics card video protocol.

[0015] The virtual graphics card video driver is connected to the first video acceleration library and the video processing module of the virtual graphics card. It is responsible for managing and controlling the video processing module of the virtual graphics card and providing access interfaces for the upper-level first video acceleration library or applications.

[0016] The virtual graphics card video protocol defines a communication protocol between the virtual graphics card video driver and the video processing module of the virtual graphics card by defining multiple commands and their associated data structures.

[0017] The video processing module of the virtual graphics card is connected to the input terminal information of the virtual graphics card video service program. It is responsible for receiving and parsing the "virtual graphics card video protocol" commands, extracting the request data, and forwarding it to the "virtual graphics card video service program" for processing.

[0018] The virtual graphics card video service program connects to the input terminal of the second video acceleration library, is responsible for receiving and parsing request data, and then calls the interface of the second video acceleration library on the host side to process it using the physical graphics card.

[0019] A further improvement of the technical solution of the present invention lies in the following: multiple commands and their associated data structures defined by the virtual graphics card video protocol, including one or more of the following: querying video function characteristics, creating a codec, destroying a codec, creating a video buffer, destroying a video buffer, starting a codec frame, decoding a macroblock, decoding a bitstream, encoding a bitstream, and ending a codec frame.

[0020] A further improvement of the technical solution of the present invention is that: the virtual graphics card is a Virtio GPU, and correspondingly, the virtual graphics card video driver is a Virtio GPU video driver; the virtual graphics card video protocol is a Virtio GPU video protocol; the virtual graphics card video service program is a Virtio GPU video service program; and the first video acceleration library and the second video acceleration library are both libva video acceleration libraries.

[0021] A video hardware acceleration method for virtual graphics cards forwards video processing requests from the virtual machine to the physical graphics card on the host machine, and utilizes the video processing module in the physical graphics card for hardware video encoding and decoding. This method leverages the aforementioned acceleration device and includes the following steps:

[0022] S1. Create a virtual machine within the host machine and allocate a virtual graphics card to the virtual machine;

[0023] S2. Start the virtual machine and simultaneously start the "Virtual Graphics Card Video Service Program" on the host machine;

[0024] S3. The application inside the virtual machine calls the first video acceleration library to encode, decode, or process the video.

[0025] S4. The first video acceleration library calls the interface of the "virtual graphics card video driver" to encode, decode or process the video.

[0026] S5. The "Virtual Graphics Card Video Driver" converts the call from the first video acceleration library into a "Virtual Graphics Card Video Protocol" command and sends it to the "Video Processing Module" of the virtual graphics card.

[0027] S6. The "video processing module" of the virtual graphics card receives and parses the "virtual graphics card video protocol" command, extracts the request data, and forwards it to the "virtual graphics card video service program".

[0028] S7. The "Virtual Graphics Card Video Service Program" receives and parses the request data, and then calls the interface of the second video acceleration library on the host side, relying on the physical graphics card video driver and related modules of the physical graphics card to complete the request processing.

[0029] S8. The "virtual graphics card video service program" in the host machine encapsulates the processing results into response data and returns it to the application on the virtual machine side along the same path.

[0030] S9. The application within the virtual machine receives and processes the response data;

[0031] S10: Video encoding, decoding, or processing is complete.

[0032] A further improvement of the technical solution of the present invention is that: in step S7, the request is processed by relying on the physical graphics card video driver, as well as the video processing module and graphics rendering module of the physical graphics card.

[0033] A video hardware acceleration method for virtual graphics cards, which forwards video processing requests from the virtual machine to the physical graphics card on the host machine, includes the following steps:

[0034] V1. Create a virtual machine within the host machine and allocate a Virtio GPU to the virtual machine;

[0035] V2. Start the virtual machine and simultaneously start the "Virtio GPU Video Service Program" on the host machine;

[0036] V3: Applications within the virtual machine call the first libva video acceleration library to encode, decode, or process video.

[0037] V4, the first libva video acceleration library calls the "Virtio GPU video driver" interface to encode, decode or process video;

[0038] V5, the "Virtio GPU Video Driver" converts calls from the first libva video acceleration library into "Virtio GPU Video Protocol" commands and sends them to the "Virtio GPU Video Processing Module" of the virtual graphics card;

[0039] V6. The "Virtio GPU Video Processing Module" of the virtual graphics card receives and parses the "Virtio GPU Video Protocol" commands, extracts the requested data, and forwards it to the "Virtio GPU Video Service Program".

[0040] V7, the "Virtio GPU Video Service Program" receives and parses the request data, and then calls the interface of the second libva video acceleration library on the host side, relying on the physical graphics card video driver and related modules of the physical graphics card to complete the request processing;

[0041] V8, the "Virtio GPU Video Service" program in the host machine encapsulates the processing results into response data and returns it to the application on the virtual machine side along the same path;

[0042] V9: Applications within the virtual machine receive and process response data;

[0043] V10, video encoding, decoding, or processing is complete.

[0044] Due to the adoption of the above technical solution, the beneficial effects achieved by this invention are:

[0045] 1. By inventing a novel video hardware acceleration mechanism, the video hardware acceleration function of a virtual graphics card is realized, supporting hardware decoding, encoding, and processing of video. This significantly reduces CPU resource consumption while improving the video processing performance of the virtual machine. Testing showed that, with a Kunpeng 920 processor and an AMD Radeon RX550 graphics card, when playing 4K / 60fps H.264 video in a virtual machine (with Virtio GPU), the player's CPU usage was greatly reduced from 321% to approximately 8.4%, significantly improving the user experience.

[0046] 2. This invention not only supports hardware decoding of video, but also hardware encoding, hardware video processing, etc.

[0047] 3. This invention implements the video hardware acceleration function of the virtual graphics card at the device layer and driver layer. Therefore, it is not bound to the upper layer or specific video acceleration library or codec API. It can support multiple codec interfaces such as VA-API, VDPAU, NVDEC, and NVENC at the same time, and is not affected by their version changes.

[0048] 4. This invention supports a variety of encoding and decoding standards, including but not limited to H.264 / AVC, H.265 / HEVC, MPEG-2, VC1, VP9, ​​etc. The specific support depends on the hardware encoding and decoding environment on the host side, and it has the characteristics of strong applicability and flexible use. Attached Figure Description

[0049] Figure 1 This is a block diagram of the video hardware acceleration function of the virtual graphics card in this invention;

[0050] Figure 2 This is a block diagram of the video hardware acceleration function of the Virtio GPU in this invention. Detailed Implementation

[0051] The present invention will be further described in detail below with reference to the embodiments.

[0052] First, let's introduce the definitions of the abbreviations and key terms involved in this invention:

[0053] Cloud desktop: Also known as desktop virtualization, cloud desktop is a typical application of the virtualization and cloud computing era. Cloud desktops utilize cloud computing technology to build server clusters in cloud data centers, creating multiple virtual machines to provide remote services such as remote computing, storage, and applications, offering personalized content. Users connect to the remote cloud desktop via various cloud terminal devices over a network to obtain their own computing, storage, personalized applications, and data content, achieving a user experience consistent with a local PC.

[0054] QXL: A software-simulated virtual graphics card.

[0055] Virtio GPU, short for Virtual I / O GPU, also written as virtio-gpu, is a virtual graphics card based on Virtio technology. Virtio and related device specifications are maintained by the OASIS organization, aiming to provide a universal and efficient virtual device mechanism for virtual environments. Currently, both QEMU and the Linux kernel support Virtio GPU devices. Compared to purely software-emulated virtual graphics cards like QXL, Virtio GPU can utilize tools such as VirGL to use the host machine's physical graphics card for 3D hardware acceleration, thereby significantly improving the graphics performance of virtual machines.

[0056] libva: A video acceleration library, it is an open-source implementation of the VA-API (Video Acceleration API) video acceleration interface specification, developed primarily by Intel Corporation.

[0057] libdpau: A video decoding library, an open-source implementation of the VDPAU (Video Decode and Presentation API for Unix) video acceleration interface specification, primarily developed by NVIDIA.

[0058] NVDEC: A video decoding interface specification and library developed by NVIDIA.

[0059] NVENC: A video coding interface specification and library developed by NVIDIA.

[0060] In the above embodiment, although the video acceleration library used is the libva library, this embodiment is not limited to the libva library and can also support various video acceleration libraries such as libvdpau, NVENC, and NVDEC.

[0061] This invention first provides a video hardware acceleration device suitable for virtual graphics cards, located in a host machine or a virtual machine. The virtual machine includes an application program and a first video acceleration library connected by information. The host machine includes a second video acceleration library, a physical graphics card video driver, and a physical graphics card connected by information in sequence. The host machine allocates a virtual graphics card to the virtual machine. The functional implementation module of the virtual graphics card is distributed in the virtual machine and the host machine. The functional implementation module of the virtual graphics card includes a virtual graphics card video driver distributed in the virtual machine, a virtual graphics card video processing module distributed in the host machine, and a virtual graphics card video service program. The virtual graphics card video driver and the virtual graphics card video processing module are connected by information through a virtual graphics card video protocol.

[0062] The virtual graphics card video driver is connected to the first video acceleration library and the video processing module of the virtual graphics card. It is responsible for managing and controlling the video processing module of the virtual graphics card and providing access interfaces for the upper-level first video acceleration library or applications.

[0063] The virtual graphics card video protocol defines a communication protocol between the virtual graphics card video driver and the video processing module of the virtual graphics card by defining multiple commands and their associated data structures.

[0064] The video processing module of the virtual graphics card is connected to the input terminal information of the virtual graphics card video service program. It is responsible for receiving and parsing the "virtual graphics card video protocol" commands, extracting the request data, and forwarding it to the "virtual graphics card video service program" for processing.

[0065] The virtual graphics card video service program connects to the input terminal of the second video acceleration library, is responsible for receiving and parsing request data, and then calls the interface of the second video acceleration library on the host side to process it using the physical graphics card.

[0066] The virtual graphics card video protocol defines multiple commands and their associated data structures, including: querying video feature characteristics, creating codecs, destroying codecs, creating video buffers, destroying video buffers, starting encoding / decoding frames, decoding macroblocks, decoding bitstreams, encoding bitstreams, and ending encoding / decoding frames—one or more of these. For example, if the virtual graphics card is Virtio GPU, then the corresponding virtual graphics card video driver is the Virtio GPU video driver; the virtual graphics card video protocol is the Virtio GPU video protocol; the virtual graphics card video service program is the Virtio GPU video service program; and both the first and second video acceleration libraries are the libva video acceleration library.

[0067] In addition, this invention also provides a video hardware acceleration method suitable for virtual graphics cards, such as... Figure 1 By forwarding the virtual machine's video processing requests to the physical graphics card on the host machine, and utilizing the video processing module in the physical graphics card for hardware video encoding and decoding, the aforementioned acceleration device is employed, including the following steps:

[0068] 1) Create a virtual machine within the host machine and allocate a virtual graphics card to the virtual machine;

[0069] 2) Start the virtual machine, and simultaneously start the "Virtual Graphics Card Video Service Program" on the host machine;

[0070] 3) Applications within the virtual machine call the first video acceleration library to encode, decode, or process video;

[0071] 4) The first video acceleration library calls the interface of the "virtual graphics card video driver" to encode, decode, or process video.

[0072] 5) The "virtual graphics card video driver" converts the calls from the first video acceleration library into "virtual graphics card video protocol" commands and sends them to the "video processing module" of the virtual graphics card;

[0073] 6) The "video processing module" of the virtual graphics card receives and parses the "virtual graphics card video protocol" command, extracts the request data, and forwards it to the "virtual graphics card video service program";

[0074] 7) The "Virtual Graphics Card Video Service Program" receives and parses the request data, and then calls the interface of the second video acceleration library on the host side, relying on the physical graphics card video driver and related modules of the physical graphics card to complete the request processing;

[0075] 8) The "virtual graphics card video service program" in the host machine encapsulates the processing results into response data and returns it to the application on the virtual machine side along the same path;

[0076] 9) The application within the virtual machine receives and processes the response data;

[0077] 10) Video encoding, decoding, or processing is complete.

[0078] In step 7) above, the request is processed by relying on the physical graphics card video driver, as well as the video processing module and graphics rendering module of the physical graphics card.

[0079] In addition to the video hardware acceleration methods mentioned above, such as Figure 2As shown, when the virtual graphics card is a Virtio GPU, this invention also provides a video hardware acceleration method suitable for virtual graphics cards. This method utilizes the aforementioned acceleration device by forwarding the virtual machine's video processing requests to the physical graphics card on the host machine, and includes the following steps:

[0080] 11) Create a virtual machine on the host machine and allocate a Virtio GPU to the virtual machine;

[0081] 12) Start the virtual machine and simultaneously start the "Virtio GPU Video Service" program on the host machine;

[0082] 13) Applications within the virtual machine call the first libva video acceleration library to encode, decode, or process video;

[0083] 14) The first libva video acceleration library calls the interface of "Virtio GPU video driver" to encode, decode or process video;

[0084] 15) The "Virtio GPU Video Driver" converts calls from the first libva video acceleration library into "Virtio GPU Video Protocol" commands and sends them to the "Virtio GPU Video Processing Module" of the virtual graphics card.

[0085] 16) The "Virtio GPU Video Processing Module" of the virtual graphics card receives and parses the "Virtio GPU Video Protocol" commands, extracts the request data, and forwards it to the "Virtio GPU Video Service Program".

[0086] 7) The "Virtio GPU video service program" receives and parses the request data, and then calls the interface of the second libva video acceleration library on the host side, relying on the physical graphics card video driver and related modules of the physical graphics card to complete the request processing;

[0087] 18) The "Virtio GPU Video Service" program in the host machine encapsulates the processing results into response data and returns it to the application on the virtual machine side along the same path;

[0088] 19) The application within the virtual machine receives and processes the response data;

[0089] 10) Video encoding, decoding, or processing is complete.

[0090] Let's elaborate further below.

[0091] For example, in the above embodiments, the commands defined by the "Virtio GPU Video Protocol" include:

[0092]

[0093] Taking the encoding and decoding process as an example, the interaction flow of the "Virtio GPU Video Protocol" is as follows:

[0094] 1. The "Virtio GPU Video Driver" sends the "GET_CAPSET" command to query the video function features of the "Virtio GPU Virtual Graphics Card" and records and saves them;

[0095] 2. The "Virtio GPU video driver" sends the "CREATE_VIDEO_BUFFER" command to create a video buffer for storing image data to be encoded or decoded;

[0096] 3. The "Virtio GPU video driver" sends the "CREATE_VIDEO_CODEC" command to create a codec corresponding to the codec specification and bind the video buffer to the codec;

[0097] 4. The "Virtio GPU video driver" sends the "BEGIN_FRAME" command to indicate that frame encoding / decoding has begun;

[0098] 5. The "Virtio GPU Video Driver" sends the "DECODE_MACROBLOCK", "DECODE_BITSTREAM", or "ENCODE_BITSTREAM" commands to encode and decode the image;

[0099] 6. The "Virtio GPU video driver" sends the "END_FRAME" command to indicate the end of the encoding / decoding frame;

[0100] 7. The "Virtio GPU Video Driver" reads and processes the encoded or decoded data to complete the processing of one frame of image;

[0101] 8. Proceed to step 4 again until all image frames have been processed.

[0102] 9. The "Virtio GPU video driver" sends the "DESTROY_VIDEO_CODEC" command to destroy the video codec;

[0103] 10. The "Virtio GPU video driver" sends the "DESTROY_VIDEO_BUFFER" command to destroy the video buffer;

[0104] 11. Encoding and decoding complete.

[0105] The protocol essentially encapsulates the entire process, hence it will be described in detail. The "Virtio GPU Video Driver" generally acts as the requester (front end) of the protocol, the "Virtio GPU Video Processing Module" acts as a relay agent for the protocol messages, and the "Virtio GPU Video Service Program" acts as the processor and responder of the protocol messages (back end).

[0106] Since each module is an important component of the entire process and forms a whole, the interfaces and functions of other modules are listed below, and then described within the overall encoding and decoding process:

[0107] The "Virtio GPU video driver" provides an interface for the libva video acceleration library on the virtual machine side:

[0108]

[0109]

[0110] The "Virtio GPU Video Service Program" provides the following interface to the "Virtio GPU Video Processing Module":

[0111] Interface Name illustrate vrend_video_init Initialize the "Virtio GPU Video Service Program" vrend_video_fini Deinitialize "Virtio GPU Video Service Program" vrend_video_fill_caps Fill in the supported video features vrend_video_create_codec Create a video codec vrend_video_destroy_codec Destroy video codec vrend_video_create_buffer Create video buffer vrend_video_destroy_buffer Destroy the video buffer vrend_video_begin_frame Start encoding / decoding frames vrend_video_decode_macroblock Decoding macroblocks vrend_video_decode_bitstream Decoding bitstream vrend_video_encode_bitstream Encoded bitstream vrend_video_end_frame End of encoding / decoding frame

[0112] The interface provided by the "Virtio GPU Video Processing Module" to the "Virtio GPU Video Driver" is the "Virtio GPU Video Protocol," which will not be elaborated here.

[0113] The libva video acceleration library is an existing module and not an invention of this invention. However, for the sake of completeness, its key interfaces are briefly described here:

[0114] Interface Name illustrate vaInitialize Initialize the libva video acceleration library vaTerminate Destroy the libva video acceleration library vaQueryConfigProfiles Query Support Specifications vaQueryConfigEntrypoints Query supported entry points vaCreateConfig Create configuration vaDestroyConfig Destroy configuration vaCreateContext Create codec context vaDestroyContext Destroy the codec context vaCreateSurfaces Create a canvas vaDestroySurfaces Destroy the canvas vaCreateBuffer Create a buffer vaDestroyBuffer Destroy the buffer vaBeginPicture Start encoding / decoding frames vaRenderPicture Encoder / decoder frame vaEndPicture End of encoding / decoding frame

[0115] Taking the encoding and decoding process as an example, the specific steps in this embodiment are as follows (for ease of description, the complex details inside each module are omitted):

[0116] 1. Create a virtual machine within the host machine and allocate a Virtio GPU virtual graphics card to the virtual machine;

[0117] 2. Start the virtual machine;

[0118] 3. The "Virtio GPU Video Processing Module" and "Virtio GPU Video Service Program" on the host machine are initialized, specifically including the following sub-steps:

[0119] (1) The “Virtio GPU video processing module” calls the “vrend_video_init” interface to start and initialize the “Virtio GPU video service program”;

[0120] (2) The “Virtio GPU Video Service Program” calls the “vaInitialize” interface of the libva video acceleration library on the host side to initialize the libva video acceleration library;

[0121] (3) The libva video acceleration library on the host side calls the interface of the physical graphics card driver to initialize the video function.

[0122] 4. Initialize the "Virtio GPU Video Driver" on the virtual machine side, which includes the following sub-steps:

[0123] (1) The “Virtio GPU Video Driver” sends the “GET_CAPSET” command to the “Virtio GPU Video Processing Module” on the host side to query the supported video features;

[0124] (2) The “Virtio GPU video processing module” receives the command and performs validity verification, extracts the command parameters, and calls the “vrend_video_fill_caps” interface of the “Virtio GPU video service program” to query the supported video function features;

[0125] (3) The “Virtio GPU Video Service Program” calls the “vaQueryConfigProfiles” and “vaQueryConfigEntrypoints” interfaces of the libva video acceleration library on the host side to query the video features supported by the physical graphics card, and then returns them to the “Virtio GPU Video Processing Module”.

[0126] (4) The “Virtio GPU Video Processing Module” fills in the response message based on the return information from the “Virtio GPU Video Service Program” and sends it to the “Virtio GPU Video Driver”.

[0127] (5) The “Virtio GPU Video Driver” parses the response message, records and saves the video feature information.

[0128] 5. The application on the virtual machine side calls the "vaInitialize" interface of the libva video acceleration library on the virtual machine side to initialize the libva video acceleration library on the virtual machine side;

[0129] 6. The application on the virtual machine side queries the video characteristics of the "Virtio GPU virtual graphics card," specifically including the following sub-steps:

[0130] (1) The application on the virtual machine side calls the interfaces such as “vaQueryConfigProfiles” and “vaQueryConfigEntrypoints” of the libva video acceleration library on the virtual machine side to query video features;

[0131] (2) The libva video acceleration library on the virtual machine side calls the "get_video_param" and "is_video_format_supported" interfaces of "Virtio GPU video driver" to query the video features and formats supported by "Virtio GPU virtual graphics card";

[0132] (3) "Virtio GPU Video Driver" returns the query results based on the saved video feature information.

[0133] (4) The application determines whether the "Virtio GPU virtual graphics card" supports the required encoding and decoding specifications based on the query results. If it supports them, it proceeds to step 7; otherwise, it reports an error and exits.

[0134] 7. Create a video buffer, which includes the following sub-steps:

[0135] (1) The application on the virtual machine side calls the "vaCreateSurfaces" interface of the libva video acceleration library to create a canvas;

[0136] (2) The libva video acceleration library on the virtual machine side calls the "create_video_buffer" interface of "Virtio GPU video driver" to create a video buffer;

[0137] (3) The “Virtio GPU video driver” constructs a “CREATE_VIDEO_BUFFER” command message according to the request and forwards it to the “Virtio GPU video processing module”.

[0138] (4) The “Virtio GPU video processing module” receives the command and performs validity verification, extracts the command parameters, and calls the “vrend_video_create_buffer” interface of the “Virtio GPU video service program” to create a video buffer;

[0139] (5) The “Virtio GPU Video Service Program” calls the “vaCreateSurfaces” and other interfaces of the libva video acceleration library on the host machine to create a canvas;

[0140] (6) The libva video acceleration library on the host side calls the interface of the physical graphics card driver to complete the creation of the video buffer;

[0141] 8. Create a video codec, which includes the following sub-steps:

[0142] (1) The application on the virtual machine side calls the “vaCreateConfig” and “vaCreateContext” interfaces of the libva video acceleration library to create a video codec context;

[0143] (2) The libva video acceleration library on the virtual machine side calls the "create_video_codec" interface of "Virtio GPU video driver" to create a video codec;

[0144] (3) The “Virtio GPU video driver” constructs a “CREATE_VIDEO_CODEC” command message according to the request and forwards it to the “Virtio GPU video processing module”.

[0145] (4) The “Virtio GPU video processing module” receives the command and performs validity verification, extracts the command parameters, and calls the “vrend_video_create_codec” interface of the “Virtio GPU video service program” to create a video codec;

[0146] (5) The “Virtio GPU Video Service Program” calls the “vaCreateConfig” and “vaCreateContext” interfaces of the libva video acceleration library on the host machine to create a video encoding and decoding context;

[0147] (6) The libva video acceleration library on the host side calls the interface of the physical graphics card driver to complete the creation of the video codec;

[0148] 9. Begin encoding and decoding frames, which includes the following sub-steps:

[0149] (1) The application on the virtual machine side calls the "vaBeginPicture" interface of the libva video acceleration library to start encoding and decoding frames and binds the canvas to the video encoding and decoding context;

[0150] (2) The libva video acceleration library on the virtual machine side calls the "codec.begin_frame" interface of "Virtio GPU video driver" to start encoding and decoding frames and binds the video buffer to the codec;

[0151] (3) The “Virtio GPU video driver” constructs a “BEGIN_FRAME” command message according to the request and forwards it to the “Virtio GPU video processing module”.

[0152] (4) The “Virtio GPU video processing module” receives the command and performs validity verification, extracts the command parameters, and calls the “vrend_video_begin_frame” of the “Virtio GPU video service program” to start encoding and decoding the frame;

[0153] (5) The “Virtio GPU Video Service Program” calls the “vaBeginPicture” and other interfaces of the libva video acceleration library on the host machine to start encoding and decoding frames and binds the canvas to the video encoding and decoding context;

[0154] (6) The libva video acceleration library on the host side calls the interface of the physical graphics card driver to start encoding and decoding frames;

[0155] 10. Encoding and decoding frames, specifically including the following sub-steps:

[0156] (1) The application on the virtual machine side calls the "vaRenderPicture" interface of the libva video acceleration library to encode and decode frames;

[0157] (2) The libva video acceleration library on the virtual machine side calls the corresponding "codec.decode_macrobloc" of the "Virtio GPU video driver" based on the currently created encoding and decoding context.

[0158] Use the “codec.decode_bitstream” or “codec.encode_bitstream” interface to decode macroblocks, decode bitstreams, or encode bitstreams;

[0159] (3) The “Virtio GPU video driver” constructs the corresponding “DECODE_MACROBLOCK”, “DECODE_BITSTREAM” or “ENCODE_BITSTREAM” command message according to the request and forwards it to the “Virtio GPU video processing module”.

[0160] (4) The “Virtio GPU video processing module” receives the command and performs validity verification, extracts the command parameters, and calls the corresponding “vrend_video_begin_frame”, “vrend_video_decode_bitstream” or “vrend_video_encode_bitstream” interface of the “Virtio GPU video service program” to decode macroblocks, decode bitstreams or encode bitstreams.

[0161] (5) The “Virtio GPU Video Service Program” calls the “vaCreateBuffer” and “vaRenderPicture” interfaces of the libva video acceleration library on the host machine to encode and decode frames;

[0162] (6) The libva video acceleration library on the host machine side calls the physical graphics card driver interface to encode and decode frames;

[0163] 11. End the encoding / decoding frame, which includes the following sub-steps:

[0164] (1) The application on the virtual machine side calls the "vaEndPicture" interface of the libva video acceleration library to end the encoding and decoding frame;

[0165] (2) The libva video acceleration library on the virtual machine side calls the "codec.end_frame" interface of "Virtio GPU video driver" to end the encoding and decoding frame;

[0166] (3) The “Virtio GPU Video Driver” constructs an “END_FRAME” command message based on the request and forwards it to the “Virtio GPU Video Processing Module”;

[0167] (4) The “Virtio GPU video processing module” receives the command and performs validity verification, extracts the command parameters, and calls the “vrend_video_end_frame” of the “Virtio GPU video service program” to end the encoding and decoding frame;

[0168] (5) The “Virtio GPU Video Service Program” calls the “vaEndPicture” and other interfaces of the libva video acceleration library on the host machine to end the encoding and decoding frame;

[0169] (6) The libva video acceleration library on the host side calls the physical graphics card driver interface to end the encoding and decoding frame, and stores the encoded and decoded data into the video buffer or other specified buffer.

[0170] 12. The virtual machine application reads the encoded and decoded data from the video buffer or other specified buffers, and forwards, displays or saves it according to user needs to complete the processing of one frame of image;

[0171] 13. Proceed to step 9 again until all image frames have been processed.

[0172] 14. Destroy the video buffer, video codec, etc., and the encoding / decoding process ends.

[0173] As can be seen from the above process, the novel video hardware acceleration mechanism proposed in this invention realizes the video hardware acceleration function of the virtual graphics card, supporting hardware decoding, encoding, and processing of video. While improving the video processing performance of the virtual machine, it significantly reduces CPU resource consumption. Testing showed that, with a Kunpeng 920 processor and an AMD Radeon RX550 graphics card, when playing 4K / 60fps H.264 video in a virtual machine (with a Virtio GPU), the player's CPU usage was significantly reduced from 321% to approximately 8.4%, significantly improving the user experience and demonstrating high potential for widespread application.

Claims

1. A video hardware acceleration device suitable for virtual graphics cards, located in a host machine or a virtual machine, wherein the virtual machine includes an application program and a first video acceleration library connected by information, and the host machine includes a second video acceleration library, a physical graphics card video driver, and a physical graphics card connected by information in sequence, wherein the host machine allocates a virtual graphics card to the virtual machine, and the functional implementation modules of the virtual graphics card are distributed in the virtual machine and the host machine, characterized in that, The virtual graphics card's functional implementation modules include a virtual graphics card video driver distributed in the virtual machine, a virtual graphics card video processing module distributed in the host machine, and a virtual graphics card video service program. The virtual graphics card video driver and the virtual graphics card video processing module are connected through the virtual graphics card video protocol. The virtual graphics card video driver is connected to the first video acceleration library and the video processing module of the virtual graphics card. It is responsible for managing and controlling the video processing module of the virtual graphics card and providing access interfaces for the upper-level first video acceleration library or applications. The virtual graphics card video protocol defines a communication protocol between the virtual graphics card video driver and the video processing module of the virtual graphics card by defining multiple commands and their associated data structures. The video processing module of the virtual graphics card is connected to the input terminal information of the virtual graphics card video service program. It is responsible for receiving and parsing the "virtual graphics card video protocol" commands, extracting the request data, and forwarding it to the "virtual graphics card video service program" for processing. The virtual graphics card video service program connects to the input terminal of the second video acceleration library, is responsible for receiving and parsing request data, and then calls the interface of the second video acceleration library on the host side to process it using the physical graphics card.

2. The video hardware acceleration device and method suitable for virtual graphics cards according to claim 1, characterized in that, The virtual graphics card video protocol defines multiple commands and their associated data structures, including one or more of the following: querying video feature characteristics, creating a codec, destroying a codec, creating a video buffer, destroying a video buffer, starting a codec frame, decoding a macroblock, decoding a bitstream, encoding a bitstream, and ending a codec frame.

3. A video hardware acceleration device suitable for virtual graphics cards according to claim 2, characterized in that, The virtual graphics card is Virtio GPU, and correspondingly, the virtual graphics card video driver is the Virtio GPU video driver; the virtual graphics card video protocol is the Virtio GPU video protocol; the virtual graphics card video service program is the Virtio GPU video service program; the first video acceleration library and the second video acceleration library are both libva video acceleration libraries.

4. A video hardware acceleration method suitable for virtual graphics cards, which forwards video processing requests from the virtual machine to the physical graphics card on the host machine, and utilizes the video processing module in the physical graphics card for hardware video encoding and decoding, characterized in that... The acceleration device according to claim 1 includes the following steps: S1. Create a virtual machine within the host machine and allocate a virtual graphics card to the virtual machine; S2. Start the virtual machine and simultaneously start the "virtual graphics card video service program" on the host machine; S3. The application within the virtual machine calls the first video acceleration library to encode, decode, or process the video. S4. The first video acceleration library calls the interface of the "virtual graphics card video driver" to encode, decode or process the video. S5. The "Virtual Graphics Card Video Driver" converts the calls from the first video acceleration library into "Virtual Graphics Card Video Protocol" commands and sends them to the "Video Processing Module" of the virtual graphics card. S6. The "video processing module" of the virtual graphics card receives and parses the "virtual graphics card video protocol" command, extracts the requested data, and forwards it to the "virtual graphics card video service program". S7. The "Virtual Graphics Card Video Service Program" receives and parses the request data, and then calls the interface of the second video acceleration library on the host side, relying on the physical graphics card video driver and related modules of the physical graphics card to complete the request processing. S8. The "virtual graphics card video service program" in the host machine encapsulates the processing results into response data and returns it to the application on the virtual machine side along the same path. S9. The application within the virtual machine receives and processes the response data; S10: Video encoding, decoding, or processing is complete.

5. A video hardware acceleration method suitable for virtual graphics cards according to claim 4, characterized in that, In step S7, the request is processed by relying on the physical graphics card video driver, as well as the video processing module and graphics rendering module of the physical graphics card.

6. A video hardware acceleration method for virtual graphics cards, characterized in that, by forwarding video processing requests from the virtual machine to the physical graphics card on the host machine, The acceleration device according to claim 3 is used in the following steps: V1. Create a virtual machine within the host machine and allocate a Virtio GPU to the virtual machine; V2. Start the virtual machine and simultaneously start the "Virtio GPU Video Service" program on the host machine; V3: Applications within the virtual machine call the first libva video acceleration library to encode, decode, or process video. V4, the first libva video acceleration library calls the "Virtio GPU video driver" interface to encode, decode or process video; V5, "Virtio GPU Video Driver" converts calls from the first libva video acceleration library into "Virtio GPU Video Protocol" commands and sends them to the "Virtio GPU Video Processing Module" of the virtual graphics card; V6. The "Virtio GPU Video Processing Module" of the virtual graphics card receives and parses the "Virtio GPU Video Protocol" commands, extracts the request data, and forwards it to the "Virtio GPU Video Service Program". V7, "Virtio GPU Video Service Program" receives and parses request data, then calls the interface of the second libva video acceleration library on the host side, and relies on the physical graphics card video driver and related modules of the physical graphics card to complete the request processing; V8, the "Virtio GPU Video Service" program in the host machine encapsulates the processing results into response data and returns it to the application on the virtual machine side along the same path; V9: Applications within the virtual machine receive and process response data; V10, video encoding, decoding, or processing is complete.

Images