Video codec card virtualization method, device, storage medium and terminal

Abstract

This invention provides a video codec card virtualization method, apparatus, storage medium, and terminal. The method includes: receiving a video codec instruction, the video codec instruction including priority information; creating a virtual video codec card, the virtual video codec card generating a video codec request based on the video codec instruction; and sending the video codec request to a physical video codec card, so that the physical video codec card can perform video codec operations based on the priority information. In this invention, the response granularity of the video codec request is smaller than the switching granularity of the virtual device; it eliminates the need to refresh all registers, saving bus interface bandwidth; and it eliminates the need to implement a scheduling mechanism for the virtual video codec card, simplifying the design of the host-side driver.

Description

Technical Field

[0001] This invention relates to the field of video encoding and decoding, and in particular to a method, apparatus, storage medium, and terminal for virtualizing video encoding and decoding cards. Background Technology

[0002] With the rapid development of the internet, there is an increasing demand for virtualization of various peripherals in the server field. Peripheral virtualization is of great significance for improving the utilization of hardware resources and meeting the growing market demand. Among these applications, video encoding and decoding is a crucial area for virtualization technology.

[0003] Existing video codec card virtualization technology primarily uses time-sharing multiplexing to share hardware devices. A corresponding virtual video codec card is created for each virtual machine, and each virtual video codec card includes registers that simulate the physical video codec card. When a virtual machine sends commands and data to the physical video codec card, it first writes the addresses of the commands and data to the virtual registers of the virtual video codec card. Then, the virtual video codec card scheduler selects a virtual video codec card and refreshes all the virtual registers of the virtual video codec card onto the real hardware device, i.e., the physical video codec card. The physical video codec card then performs the corresponding video encoding and decoding operations.

[0004] Existing video codec card virtualization technology has the following drawbacks: First, the scheduling algorithm needs to store multiple copies of the virtual device on the host side. When there are many registers (the VPU (Video Processing Unit) has thousands of registers), it requires a lot of memory, and the register refresh efficiency is low. Second, the design and implementation cost of the virtual device scheduling mechanism is high. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a video codec card virtualization method, apparatus, storage medium and terminal to solve the problems of high memory consumption, low register refresh efficiency and high cost in the prior art for video codec card virtualization.

[0006] To achieve the above and other related objectives, a first aspect of the present invention provides a video codec card virtualization method, comprising: receiving a video codec instruction, the video codec instruction including priority information; creating a virtual video codec card, the virtual video codec card generating a video codec request based on the video codec instruction; and sending the video codec request to a physical video codec card, so that the physical video codec card can perform video codec operations based on the priority information.

[0007] In some embodiments of the first aspect of the present invention, the physical video codec card includes a video codec unit, a microprocessor unit, a memory unit, and a bus interface unit. The video codec card virtualization method includes: the physical video codec card receiving the video codec request through the bus interface unit and storing it in the memory unit; firmware on the microprocessor unit parsing the video codec request; and the video codec unit performing video codec operations based on the parsed video codec request.

[0008] In some embodiments of the first aspect of the present invention, the generation of the video encoding / decoding request includes: the host creating a virtual video codec card corresponding to the virtual machine based on a device virtualization infrastructure, the virtual video codec card not retaining all registers of the physical video codec card; the device virtualization infrastructure driving the virtual video codec card to generate the video encoding / decoding request based on received video encoding / decoding instructions.

[0009] In some embodiments of the first aspect of the present invention, the video codec card virtualization method includes: using the VFIO-MDEV (Virtual Function I / O Mediated devices) device virtualization infrastructure to create a virtual video codec card corresponding to the virtual machine through the host's driver layer; after the virtual video codec card receives the video codec instructions sent by the virtual machine, the VFIO-MDEV device virtualization infrastructure drives the generation of the video codec request.

[0010] In some embodiments of the first aspect of the present invention, the video codec card virtualization method includes: designing a quality of service strategy based on the priority information to provide different user services.

[0011] To achieve the above and other related objectives, a second aspect of the present invention provides a video codec card virtualization device, comprising: an instruction receiving module for receiving video codec instructions, the video codec instructions including priority information; a codec request generation module for creating a virtual video codec card, the virtual video codec card generating video codec requests based on the video codec instructions; and a codec module for sending the video codec requests to a physical video codec card, so that the physical video codec card can perform video codec operations based on the priority information.

[0012] To achieve the above and other related objectives, a third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the video codec card virtualization method.

[0013] To achieve the above and other related objectives, a fourth aspect of the present invention provides an electronic terminal, comprising: a processor and a memory; the memory for storing a computer program, and the processor for executing the computer program stored in the memory to enable the terminal to execute the video codec card virtualization method.

[0014] As described above, this invention proposes a video codec card virtualization method, apparatus, storage medium, and terminal, which has the following beneficial effects: It overcomes the shortcomings of traditional time-sharing shared hardware device video codec virtualization methods, proposing a new approach that can more efficiently utilize hardware resources. This is achieved by packaging video codec instructions into requests containing priority information, thus optimizing the time-sharing sharing method into a priority request queue method, thereby realizing the virtualization of the video codec card. In this invention, the response granularity of the video codec request (v-request) is smaller than the switching granularity of the virtual device; it eliminates the need to refresh all registers, saving bus interface bandwidth; it eliminates the need to implement a virtual video codec card scheduling mechanism, simplifying the design of the host-side driver; furthermore, it designs a Quality of Service (QoS) policy to provide different user services, improving user experience; it uses the firmware of the physical video codec card to parse video codec requests, better protecting the security of enterprise information (such as VPU IP information); and it fully utilizes the microprocessor resources on the physical video codec card. Attached Figure Description

[0015] Figure 1 The diagram shown is a flowchart illustrating a video codec card virtualization method according to an embodiment of the present invention.

[0016] Figure 2 The diagram shown is a structural block diagram of a video codec card according to an embodiment of the present invention.

[0017] Figure 3 The diagram shown illustrates the working principle of a video codec card virtualization method based on time-division multiplexing in one embodiment of the present invention.

[0018] Figure 4 The diagram shown illustrates the working principle of a video codec card virtualization method based on a priority request queue, according to an embodiment of the present invention.

[0019] Figure 5 The diagram shown is a structural schematic of a video codec card virtualization device according to an embodiment of the present invention.

[0020] Figure 6 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of the present invention. Detailed Implementation

[0021] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0022] It should be noted that in the following description, reference is made to the accompanying drawings, which illustrate several embodiments of the present invention. It should be understood that other embodiments may also be used, and changes in mechanical composition, structure, electrical system, and operation may be made without departing from the spirit and scope of the invention. The following detailed description should not be considered limiting, and the scope of the embodiments of the invention is defined only by the claims of the published patents. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. Spatially related terms, such as “upper,” “lower,” “left,” “right,” “below,” “below,” “lower part,” “above,” “upper part,” etc., may be used in the text to illustrate the relationship between one element or feature shown in the figures and another element or feature.

[0023] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," "fixing," and "holding" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0024] Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition arise only when combinations of elements, functions, or operations are inherently mutually exclusive in some manner.

[0025] The purpose of this invention is to provide a video codec card virtualization method, apparatus, storage medium, and terminal to solve the problems of high memory consumption, low register refresh efficiency, and high cost in the prior art for video codec card virtualization.

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this invention are further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining this invention and are not intended to limit this invention. The following embodiments mainly use the Linux system and its device virtualization infrastructure VFIO-MDEV as examples to illustrate this invention, but this method is not limited to the Linux system or the device virtualization infrastructure VFIO-MDEV.

[0027] Example 1

[0028] like Figure 1 As shown in the figure, this invention proposes a flowchart of a video codec card virtualization method, which specifically includes the following steps:

[0029] Step S11. Receive video encoding / decoding instructions, which include priority information. These instructions can be sent by the virtual machine client and may simultaneously send the data to be encoded / decoded or data address information to the server. The priority information is assigned to the client based on preset priority rules.

[0030] Step S12. Create a virtual video codec card. The virtual video codec card packages and generates video codec requests based on the video codec instructions. A mapping relationship exists between the virtual video codec card and the physical video codec card (the actual hardware device). The virtual machine uses the corresponding hardware device by operating the virtual video codec card. Furthermore, the packaged video codec requests include priority information.

[0031] In a preferred embodiment of this invention, the video encoding / decoding request is generated as follows: the host creates a virtual video codec card corresponding to the virtual machine based on the device virtualization infrastructure, and the virtual video codec card does not retain all the registers of the physical video codec card; the device virtualization infrastructure drives the virtual video codec card to generate the video encoding / decoding request based on the received video encoding / decoding instructions.

[0032] Taking the Linux system as an example, it includes the VFIO-MDEV device virtualization infrastructure; the video codec card virtualization method includes: the VFIO-MDEV device virtualization infrastructure creates a virtual video codec card corresponding to the virtual machine through the host's driver layer; after the virtual video codec card receives the video codec instructions sent by the virtual machine, the VFIO-MDEV device virtualization infrastructure driver generates the video codec request.

[0033] Step S13. The video encoding / decoding request is sent to the physical video codec card, so that the physical video codec card can perform video encoding / decoding operations based on the priority information. The physical video codec card generates a request queue, i.e., a priority request queue, based on the priority information in the request.

[0034] In a preferred embodiment of this invention, the physical video codec card includes a video codec unit, a microprocessor unit, a memory unit, and a bus interface unit. The video codec card virtualization method includes: the physical video codec card receiving the video codec request through the bus interface unit and storing it in the memory unit; the firmware on the microprocessor unit parsing the video codec request; and the video codec unit performing video codec operations based on the parsed video codec request.

[0035] like Figure 2 As shown, as an optional implementation, this embodiment of the invention proposes a structural block diagram of a video codec card, which includes a video codec unit 21 (preferably VPU), a memory unit 22 (preferably DDR (Double Data Rate Synchronous Dynamic Random Access Memory)), a microprocessor unit 23, and a bus interface unit 24 (preferably PCIE bus (peripheral component interconnect express high-speed serial computer expansion bus)).

[0036] The video codec card communicates with the server host via its PCIe slot. The server host runs the video codec card's driver to operate it. Its working principle is as follows: The server host sends the commands and data to be encoded / decoded to the registers and memory unit 22 of the video codec unit 21 via the bus interface unit 24. The video codec unit 21 performs encoding / decoding operations to obtain the processed data, and the host retrieves the processed data from the video codec card. The microprocessor unit 23 on the video codec card is mainly used to detect the card's operating status.

[0037] In a preferred embodiment of this invention, the video codec card virtualization method includes: designing a Quality of Service (QoS) policy based on the priority information to provide different user services, i.e., providing corresponding services for different users. QoS is a qualitative agreement between the network and users, as well as between users communicating with each other on the network, regarding the transmission and sharing of information. When the network is overloaded or congested, QoS can ensure that important traffic is not delayed or dropped, while guaranteeing the efficient operation of the network.

[0038] This embodiment provides schematic diagrams illustrating the working principle of a video codec card virtualization method based on time-division multiplexing. Figure 3 A schematic diagram illustrating the working principle of a video codec card virtualization method based on a priority request queue. Figure 4 ).

[0039] like Figure 3 As shown, in the time-division multiplexing-based video codec card virtualization method, there is a one-to-one correspondence between the virtual machine (VM) and the virtual video codec card in the physical host (Host), and the virtual video codec card includes virtual registers (V-registers, which simulate the registers of the physical video codec card). Its workflow is as follows: VFIO-MDEV creates a virtual video codec card through the host's driver layer. The VM writes the video codec instructions and the addresses of the data to be processed to the virtual registers of the virtual video codec card. The virtual codec card scheduler selects a virtual codec card and refreshes all the virtual registers of the selected virtual codec card to the registers of the physical video codec card (HW Video Card), enabling the physical video codec card to perform the corresponding video codec operations.

[0040] like Figure 4 As shown, in the video codec card virtualization method based on priority request queues, there is a one-to-one correspondence between the virtual machine (VM) and the virtual video codec card in the physical host, and the virtual video codec card does not include virtual registers. When the virtual machine operates the virtual video codec card, the VFIO-MDEV driver on the host generates a video codec request (v-request) and sends it to the physical video codec card; the physical video codec card generates a video codec request queue (Request-queues), and the firmware on it responds to the v-requests according to priority and operates the video codec unit (VPU) to perform the corresponding video codec operations.

[0041] In some embodiments, the video codec card virtualization method can be applied to a controller, such as an ARM (Advanced RISC Machines) controller, an FPGA (Field Programmable Gate Array) controller, a SoC (System on Chip) controller, a DSP (Digital Signal Processing) controller, or an MCU (Microcontroller Unit) controller, etc. In some embodiments, the video codec card virtualization method can also be applied to a computer including components such as memory, a memory controller, one or more processing units (CPUs), peripheral interfaces, RF circuits, audio circuits, speakers, microphones, input / output (I / O) subsystems, a display screen, other output or control devices, and external ports; the computer includes, but is not limited to, personal computers such as desktop computers, laptops, tablets, smartphones, smart TVs, and personal digital assistants (PDAs). In other embodiments, the video codec card virtualization method can also be applied to a server, which can be deployed on one or more physical servers according to various factors such as function and load, or can be composed of distributed or centralized server clusters.

[0042] Example 2

[0043] like Figure 5 As shown in the diagram, this embodiment of the invention proposes a structural schematic of a video codec card virtualization device, which includes: an instruction receiving module 51, used to receive video codec instructions, the video codec instructions including priority information; a codec request generation module 52, used to create a virtual video codec card, the virtual video codec card packaging and generating video codec requests based on the video codec instructions; and a codec module 53, used to send the video codec requests to a physical video codec card, so that the physical video codec card can perform video codec operations based on the priority information.

[0044] It should be noted that the modules provided in this embodiment are similar to the methods and implementation methods provided above, and therefore will not be repeated. It should also be understood that the division of the various modules in the above device is merely a logical functional division; in actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can all be implemented in software through processing element calls; they can all be implemented in hardware; or some modules can be implemented in software through processing element calls, while others are implemented in hardware. For example, the encoding / decoding module 53 can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and its functions can be called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element mentioned here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0045] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0046] Example 3

[0047] like Figure 6As shown in the diagram, this embodiment of the invention provides a structural schematic of an electronic terminal. The electronic terminal provided in this embodiment includes: a processor 61, a memory 62, and a communicator 63; the memory 62 is connected to the processor 61 and the communicator 63 via a system bus and completes communication between them; the memory 62 is used to store computer programs; the communicator 63 is used to communicate with other devices; and the processor 61 is used to run the computer programs, enabling the electronic terminal to execute the various steps of the video codec card virtualization method described above.

[0048] The system bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include Random Access Memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0049] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0050] Example 4

[0051] This invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the video codec card virtualization method described above.

[0052] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0053] In summary, this invention provides a video codec card virtualization method, apparatus, storage medium, and terminal. By packaging video codec instructions into requests containing priority information, it optimizes the time-sharing method into a priority request queue method, thereby achieving video codec card virtualization. In this invention, the response granularity of video codec requests is smaller than the switching granularity of virtual devices; it eliminates the need to simulate or refresh all registers, saving bus interface bandwidth and host storage space; it eliminates the need to implement a virtual video codec card scheduling mechanism, simplifying host-side driver design and saving manpower; furthermore, it designs a quality of service strategy to provide different services to different users, improving user experience; parsing video codec requests through the firmware of the physical video codec card better protects enterprise information security; and it fully utilizes the microprocessor resources on the physical video codec card. Therefore, this invention effectively overcomes the problems of excessive memory consumption, low register refresh efficiency, and high cost in existing video codec card virtualization technologies, and has high industrial application value.

[0054] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for virtualizing a video codec card, characterized in that, include: Receive video encoding / decoding instructions, the video encoding / decoding instructions including priority information; A virtual video codec card is created, which packages and generates video codec requests based on the video codec instructions. The video encoding / decoding request is sent to the physical video codec card so that the physical video codec card can perform video encoding / decoding operations based on the priority information; The physical video codec card includes a video codec unit, a microprocessor unit, a memory unit, and a bus interface unit. The physical video codec card receives the video codec request through the bus interface unit and stores it in the memory unit; The firmware on the microprocessor unit parses the video encoding / decoding request; The video encoding / decoding unit performs video encoding / decoding operations based on the parsed video encoding / decoding request; The methods for generating the video encoding / decoding request include: The host creates a virtual video codec card corresponding to the virtual machine based on the device virtualization infrastructure. This virtual video codec card does not retain all the registers of the physical video codec card. The device virtualization infrastructure drives the virtual video codec card to generate the video codec request based on the received video codec instructions. The response granularity of video encoding / decoding requests is smaller than the switching granularity of virtual devices.

2. The video codec card virtualization method according to claim 1, characterized in that, include: The VFIO-MDEV device virtualization infrastructure is used to create a virtual video codec card corresponding to the virtual machine through the host's driver layer; After receiving the video encoding / decoding instructions sent by the virtual machine, the virtual video codec card generates the video encoding / decoding request driven by the VFIO-MDEV device virtualization infrastructure.

3. The video codec card virtualization method according to claim 1, characterized in that, include: Based on the priority information, a service quality strategy is designed to provide different user services.

4. A video codec card virtualization device, characterized in that, include: The instruction receiving module is used to receive video encoding / decoding instructions, which include priority information. The encoding / decoding request generation module is used to create a virtual video codec card, which generates video encoding / decoding requests based on the video encoding / decoding instructions. The encoding / decoding module is used to send the video encoding / decoding request to the physical video encoding / decoding card, so that the physical video encoding / decoding card can perform video encoding / decoding operations based on the priority information; The physical video codec card includes a video codec unit, a microprocessor unit, a memory unit, and a bus interface unit. The physical video codec card receives the video codec request through the bus interface unit and stores it in the memory unit; The firmware on the microprocessor unit parses the video encoding / decoding request; The video encoding / decoding unit performs video encoding / decoding operations based on the parsed video encoding / decoding request; The methods for generating the video encoding / decoding request include: The host creates a virtual video codec card corresponding to the virtual machine based on the device virtualization infrastructure. This virtual video codec card does not retain all the registers of the physical video codec card. The device virtualization infrastructure drives the virtual video codec card to generate the video codec request based on the received video codec instructions. The response granularity of video encoding / decoding requests is smaller than the switching granularity of virtual devices.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the video codec card virtualization method according to any one of claims 1 to 3.

6. An electronic terminal, characterized in that, include: Processor and memory; The memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to cause the terminal to perform the video codec card virtualization method as described in any one of claims 1 to 3.

Images