Virtualization system based on video codec VPU

Abstract

The invention provides a virtualization system based on a video codec VPU, which comprises a plurality of hosts, a VPU module and an arbiter, the VPU module comprises a plurality of controllers and a VPU kernel, and the hosts send use right request signals of the VPU kernel to the arbiter through the corresponding controllers in a software driving layer; when a use right endowing signal returned by the arbiter is received, accessing or configuring a VPU kernel register through a corresponding controller, and controlling the coding and decoding operation of the VPU kernel on the code stream data; and after the encoding and decoding operations are completed, the host sends a use right ending signal to the arbiter through the controller, so that the arbiter takes back the use right of the host and waits for use right request signals of other hosts to the VPU kernel. According to the method, all VPU virtualized channels can obtain the coding and decoding use right of the VPU timely and reasonably through a software blocking method, and the probability of jamming or blurred screen phenomena is reduced within the coding and decoding capacity range of the VPU.

Description

Technical Field

[0001] The present invention relates to the technical field of VPU virtualization, and more specifically, to a virtualization system based on a video codec VPU. Background Art

[0002] VPU (Video Processing Unit) is the core engine of a video processing platform. It has video encoding and decoding functions, which can reduce the video processing load of the CPU, thereby reducing the control bandwidth of the CPU. In recent years, with the growing demand for multimedia applications, the demand for video encoding and decoding in electronic devices has also increased year by year. In order to cope with this situation, various chip manufacturers have begun to integrate VPU encoding and decoding modules inside the chip, and support VPU virtualization, so that multiple hosts can share the same VPU encoding and decoding module, thereby realizing multi-channel video encoding and decoding requirements.

[0003] With the increasing encoding and decoding capabilities of VPU, the introduction of VPU virtualization solutions can enable VPU to complete the encoding and decoding requirements of multiple channels with relatively small losses, which can quickly reduce the cost of chip design and integration. This is similar to GPU (Graphics Processing Unit), which also supports different virtualization solutions, allowing multiple hosts to share a GPU. However, unlike the GPU virtualization solution, the VPU virtualization solution is still in its infancy. The VPU virtualization solutions currently on the market have problems such as weak targeting, low utilization, and unclear purpose.

[0004] In order to facilitate integration and calling, the VPU modules currently on the market usually use the APB interface to encapsulate their configuration interface and the AXI interface to encapsulate their codec input and output interfaces. The host configures the registers inside the VPU module or controls it to start or end decoding through the APB interface. The virtualization solution of the VPU is that multiple hosts configure the same VPU module through the APB interface. Therefore, the virtualization solution of the VPU must add an arbitrator inside it, arbitrate the APB configuration signals output by different hosts according to the arbitration algorithm, and output the arbitration results to the codec core module inside the VPU.

[0005] Currently, there is a VPU virtualization method based on bandwidth utilization switching, that is, the arbitration algorithm of the VPU internal arbiter will adjust the priority according to the bandwidth utilization of the VPU and different host codec channels (AXI interface). The greater the bandwidth utilization, the lower its priority, and ignore the request for access to the VPU by the host with a bandwidth utilization of 100% and no bandwidth overflow capability, thereby ensuring that the bandwidth utilization of all host codec channels of the VPU is in a similar floating range.

[0006] The purpose of the existing VPU virtualization technology based on bandwidth utilization switching is actually to reasonably allocate the requests of each virtualized channel of the VPU, reduce the priority of the host that occupies the VPU for a long time (i.e., the host with relatively high bandwidth utilization), and increase the priority of the host that occupies the VPU for a short time (i.e., the host with relatively low bandwidth utilization), so as to achieve a state in which the access requests of all host channels of the VPU are basically balanced. When the number of virtualized channels of the VPU increases to 16 or 32, this VPU virtualization method needs to calculate the bandwidth occupancy rate of the APB channel of each VPU host channel accessing the VPU codec in the current time period in real time, and needs to be updated and iterated quickly to meet the decoding requirements of the VPU multi-channel. However, this method of balancing the overall utilization rate also causes the area of ​​the VPU to become very large after integration, and the calculation delay increases. When the bandwidth utilization rate of a virtualized channel of the VPU changes, its large calculation delay will cause the arbitrator inside the VPU to be unable to quickly change its priority, resulting in the VPU codec being stuck (other hosts occupying the VPU for a long time) due to unreasonable priority allocation. Summary of the invention

[0007] The present invention aims at the technical problems existing in the prior art and provides a virtualization system based on a video codec VPU, which can overcome the problem of easy freezing or screen distortion of the prior VPU virtualization.

[0008] The present invention provides a virtualization system based on a video codec VPU, comprising a plurality of hosts, a VPU module and an arbitrator, wherein the VPU module comprises a plurality of controllers and a VPU core, each of the hosts is connected to a corresponding controller via an APB interface, each of the controllers is connected to the arbitrator, the arbitrator is connected to the VPU core, and the VPU core is connected to an external storage area via the Cache module;

[0009] Host i sends a VPU core usage right request signal to the arbitrator through a corresponding controller in the software driver layer, wherein i is the index of the host, 1≤i≤N, and N is the number of hosts;

[0010] When the host i receives the VPU core usage right granting signal returned by the arbitrator, it accesses or configures the register inside the VPU core through the corresponding controller, and controls the VPU core to encode and decode the code stream data; wherein the arbitrator adopts a polling arbitration algorithm to determine the order of each host's usage right to the VPU core;

[0011] After the encoding and decoding operation is completed, host i sends a VPU core usage right end signal to the arbitrator through the controller, so that the arbitrator takes back the VPU core usage right of host i and waits for VPU core usage right request signals from other hosts.

[0012] Based on the above technical solution, the present invention can also make the following improvements.

[0013] Optionally, the VPU module also includes a Cache module, and the VPU module encapsulates multiple APB Slave interfaces and multiple APB Master interfaces. Each controller in the VPU module communicates with the corresponding host through the APB Slave interface, and each controller communicates with the VPU core and the Cache module respectively through the APB Master interface. The Cache module communicates with the external storage area through the AXI interface.

[0014] Optionally, each of the controllers includes a plurality of address registers, and the host i sends a VPU core usage right request signal to the arbitrator through the corresponding controller in the software driver layer, including:

[0015] Host i writes 0x_01 data to the 0x_0001 address register of the corresponding controller, so that the controller corresponding to the host pulls up its req request signal and starts to request the right to use the VPU Core from the arbitrator;

[0016] After the encoding and decoding operation is completed, the host i sends a VPU core usage right end signal to the arbitrator through the controller, including:

[0017] The host i writes data 0x_00 to the 0x_0001 address register of the corresponding controller, so that the controller corresponding to the host pulls down its req request signal, informing the arbitrator that the right to use the VPU Core has ended.

[0018] Optionally, the system also includes a data selector, which is connected to the arbitrator, and is respectively connected to the VPU core and each external storage area through an AXI interface, wherein a plurality of the external storage areas correspond to a plurality of the hosts, and each of the external storage areas is used to store code stream data after the encoding and decoding operation of the VPU core on the corresponding host request.

[0019] Optionally, the data selector receives the host currently having the right to use the VPU core transmitted by the arbitrator, and stores the code stream data after the VPU core performs encoding and decoding operations in an external storage area corresponding to the host currently having the right to use the VPU core.

[0020] Optionally, the arbitrator uses a round-robin arbitration algorithm to determine the order of use rights of multiple host channels to the VPU core, including:

[0021] Set the initial priority order of multiple host channels;

[0022] In the first arbitration cycle, when multiple host channels receive a request signal for the right to use the VPU core, the host channel with the highest priority among the multiple host channels requesting the right to use is selected, and the right to use the VPU core is granted to the host channel with the highest priority;

[0023] In the second arbitration cycle, the priority of the host channel selected in the first arbitration cycle is reduced to the lowest, and the priority of each host channel is updated in turn to obtain the priority order of multiple host channels in the second arbitration cycle, and in the second arbitration cycle, the host channel with the right to use the VPU core is arbitrated;

[0024] The same method is used to dynamically update the arbitration and priority order of multiple host channels in each arbitration cycle.

[0025] Optionally, the arbitrator adopts a round-robin arbitration algorithm to determine the order of use rights of multiple host channels to the VPU core, including:

[0026] Set the initial priority order of multiple host channels to P1<P2<P3<....P h <....P g <...<P i <...<P n , P i represents the i-th host channel;

[0027] In the first arbitration cycle, the priority order of multiple host channels is the initial priority order P1<P2<P3<....P h <....P g <...<P i <...<P n , if in the first arbitration cycle, the host channel P h and P g Generate a usage right request, host channel P g The priority is greater than the host channel P h The priority of the VPU core is given to the host channel P during the arbitration period. g ;

[0028] In the second arbitration cycle, the host channel P selected in the first arbitration cycle is g The priority of each host channel is reduced to the lowest, and the priority of each host channel is updated in turn: P g<...<P i <...<P n <P1<P2<P3<....P h <...., obtain the priority order of multiple host channels in the second arbitration cycle, and in the second arbitration cycle, arbitrate the selected host channel with the right to use the VPU core;

[0029] The same method is used to dynamically update the arbitration and priority order of multiple host channels in each arbitration cycle.

[0030] The present invention provides a virtualization system based on a video codec VPU, in which a host i sends a VPU kernel usage right request signal to an arbitrator through a corresponding controller in a software driver layer; upon receiving a usage right grant signal returned by the arbitrator, the register inside the VPU kernel is accessed or configured through the corresponding controller, and the VPU kernel is controlled to perform encoding and decoding operations on code stream data; after the encoding and decoding operations are completed, the host i sends a usage right end signal to the arbitrator through the controller, so that the arbitrator takes back the usage right of the host i and waits for other hosts to request the usage right of the VPU kernel. The present invention ensures that all VPU virtualization channels can obtain the VPU's encoding and decoding usage right in a timely and reasonable manner through a software blocking method, and reduces the probability of freezing or screen distortion within the VPU's encoding and decoding capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a single-channel VPU independent module system diagram;

[0032] Figure 2 It is a schematic diagram of the internal structure of a single-channel VPU independent module;

[0033] Figure 3 This is a 16-channel VPU virtualization structure diagram;

[0034] Figure 4 A schematic diagram of the hardware structure and software driver of a four-channel VPU virtualization system provided by the present invention;

[0035] Figure 5 for Figure 4 The access flow chart of the 4 virtualization channels to realize VPU virtualization;

[0036] Figure 6 It is a schematic diagram of the non-contention response waveform during the virtualization process of the four-channel VPU;

[0037] Figure 7 Schematic diagram of weight changes during the virtualization process of a four-channel VPU;

[0038] Figure 8A schematic diagram of the response waveform of the VPU virtualization process using the polling arbitration algorithm;

[0039] Fig. 9 Access flow chart of VPU virtualization for requesting usage rights for four channels simultaneously. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be arbitrarily combined with each other to form a feasible technical solution. This combination is not subject to the constraints of the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0041] The virtualization system based on the video codec VPU proposed by the present invention is a combination of hardware and driver software. The APB access request signal of the VPU host channel is no longer controlled by the VPU hardware itself, but is handed over to the software driver layer at the top level of the host for control. When a host channel of the VPU occupies the VPU module for encoding and decoding, the driver software of other hosts cannot obtain the PREADY signal returned by the arbitrator, so that other hosts of the VPU suspend the APB bus and start waiting for the next arbitration result of the arbitrator. The arbitrator uses a polling arbitration algorithm to change the arbitration priority of each VPU host in real time. When a host channel is selected by the arbitrator, its priority will be reduced to the lowest in the next arbitration selection, so that each VPU host channel will be accessed once within a short period of time, so that each channel in the VPU virtualization process cannot occupy the VPU for a long time, thereby not causing the encoding and decoding of other channels to be stuck.

[0042] See also Figure 1 , which is the basic block diagram of a single-channel VPU independent module, mainly including the host, VPU module and external storage area. The top layer of the VPU module IP is generally encapsulated into an APB Slave interface to receive the configuration information of the VPU host or return the VPU encoding and decoding status to the host. The VPU module's code stream interaction interface will be encapsulated into an AXIMaster interface to interact with the external storage area to meet the VPU module's higher bandwidth requirements for decoding.

[0043] See also Figure 2 , is a schematic diagram of the internal structure of a single-channel VPU independent module. The VPU module generally includes a controller, a VPU core, and other storage conversion cache modules. All modules inside the VPU module complete register configuration and decoding interaction operations under the control of the controller. The functions of each module are as follows:

[0044] (1) Controller: The controller is the only module in the encoder that is directly connected to the VPU host. It communicates with the VPU host through the APBSlave interface. The controller also has an APB Master interface for controlling the cache module and the internal registers of the VPU Core. During the encoding and decoding process of the VPU module, the controller receives the signal from the host, completes its own register configuration, and then starts to use the APB interface to control the cache module and the VPU core to start the encoding and decoding process.

[0045] (2) Cache module: The purpose of the cache module in the commonly used codec module is generally to save the bandwidth of the transmission bus. That is, the AXI code stream signal output by the VPU core will be converted and output inside the cache module, making the code stream data bandwidth utilization rate of the entire VPU module higher. During the encoding and decoding process, the use or state change of the cache module is controlled by the controller.

[0046] (3) VPU core: In the VPU module, the VPU core is the core codec module. In the decoder, the VPU core uses the AXI interface to read the compressed original bitstream data in formats such as H.264 or H.265 from the external storage area through the Cache module, and writes the restored video data to the external storage area after decoding. The function of the encoder's VPU core is opposite to that of the decoder. Its role is to convert the video data into compressed bitstream data after encoding and write it to the external storage area.

[0047] Most of the existing VPU codecs have relatively high codec efficiency and speed, which are sufficient to meet the video codec requirements of most electronic devices. When facing the codec of conventional resolution, the codec capability of VPU is often redundant, which makes the virtualization technology of VPU popularized and applied. VPU virtualization technology allows multiple host devices to share a VPU module, so that the performance of VPU module can be maximized, while reducing the chip tape-out cost.

[0048] like Figure 3 As shown, Figure 3The structure diagram of a 16-host channel VPU virtualization system is described in the specification, which can support up to 16 hosts to share the VPU module for encoding and decoding operations. Each VPU host can be configured with its corresponding controller separately, and the 16 host channels correspond to the 16 controllers one by one. The 16 controllers will compete for the right to use the VPU core, and the arbitrator will complete the arbitration through the internal arbitration algorithm, so that each host has the opportunity to access the VPU core.

[0049] The core of VPU virtualization is to allow as many hosts as possible to share the same VPU core. However, as the number of host channels increases, the output of the arbitrator will often be unbalanced, causing some hosts to occupy the VPU core for a long time, while other channels cannot get the right to use the VPU core for a long time and become stuck. Therefore, the arbitration algorithm and weight switching mechanism in the VPU virtualization scheme are very important. The present invention is to realize the virtualization of VPU by software blocking and polling arbitration around these two points, reducing the probability of freezing or screen distortion when multiple host channels share VPU.

[0050] Based on this, a virtualization system based on a video codec VPU provided by the present invention mainly includes multiple hosts and VPU modules, a VPU codec core and an arbitrator. The VPU module includes multiple controllers and an independent VPU core. Each host is connected to the corresponding controller through an APB interface, each controller is connected to the arbitrator, the arbitrator is connected to the VPU core, and the VPU core is connected to an external storage area through a Cache module.

[0051] When multiple VPU host channels access the VPU module simultaneously, the working principle is as follows:

[0052] Host i sends a VPU core usage right request signal to the arbitrator through a corresponding controller in the software driver layer, wherein i is the index of the host, 1≤i≤N, and N is the number of hosts;

[0053] When host i receives the VPU core usage right granting signal returned by the arbitrator, it accesses or configures the registers inside the VPU core through the corresponding controller, and controls the VPU core's encoding and decoding operations on the bitstream data; wherein the arbitrator determines the order of multiple hosts' usage rights to the VPU core through a polling arbitration algorithm.

[0054] After the encoding and decoding operation is completed, host i sends a VPU core usage right end signal to the arbitrator through the controller, so that the arbitrator takes back the VPU core usage right of host i and waits for VPU core usage right request signals from other hosts.

[0055] For details, see Figure 4, is a diagram of the internal structure of a four-channel VPU virtualization system, which solves the problem of a single channel occupying the VPU core for a long time through software blocking and polling arbitration. Under this virtualization condition, each host can configure the registers inside its corresponding controller through the APB signal, and can also access or configure the VPU core through the controller. The host writes 0x_01 data to the 0x_0001 address register of the controller, which will cause the controller corresponding to the host to pull up its req request signal and start requesting the use of the VPU core from the arbitration. If the data 0x_00 is written to the corresponding register address, the controller will pull down its req request signal and no longer compete for the use of the VPU core.

[0056] like Figure 4 As shown in the figure, while the four VPU host channels write data to registers at addresses 0x_10ca, 0x_10ec, 0x_107d, etc., they actually access the registers of the VPU core through the controller corresponding to the host. That is, after receiving the host command, the controller will send the APB signal of the corresponding address to the arbitrator, but the arbitrator may not respond to this request in time, because the arbitrator has not granted the right to use the VPU core to the channel at this time. The APB signal sent by the controller to the arbitrator cannot receive the PREADY signal returned by the arbitrator, causing the APB bus to suspend and wait for the arbitrator to grant the right to use the VPU core before it resumes.

[0057] Therefore, in the VPU virtualization method described in the present invention, when the host 1 in the VPU virtualization channel wants to access or configure the VPU kernel, it must write data 0x_01 to the register corresponding to the 0x_01 address in the controller in advance in the software driver layer, so that the req signal output by the controller to the arbitrator is pulled high, and the VPU kernel is requested to use the right. When the arbitrator grants the right to use the VPU kernel to the corresponding host under the deployment of the arbitration algorithm, the APB bus of the host 1 accessing the VPU kernel through the controller will not be blocked. The host can access or configure the register of the VPU kernel at this time, and control the VPU kernel to start encoding and decoding operations. After the operation of the VPU kernel by the host 1 is completed, it is necessary to write data 0x_00 to the register corresponding to the 0x_01 address in the controller, so that the req signal output by the controller to the arbitrator is pulled low and released, and no longer competes for the right to use the VPU kernel. The arbitrator will take back the right to use the host 1 at this time, and re-grant the right to use the VPU kernel through the arbitration algorithm and the req signal of other host channels.

[0058] exist Figure 5 The access process of implementing VPU virtualization for four virtualized host channels is described in the figure, which mainly includes the following steps:

[0059] Step S1: Host 1 controls its corresponding controller to pull up the req1 request signal, requesting the arbitrator to grant host 1 the right to use the VPU core of the corresponding channel.

[0060] In the virtualization solution described in the present invention, multiple VPU hosts can share the same VPU core, and the arbitrator is responsible for scheduling the VPU core. Therefore, before obtaining the right to use the VPU core, each host must drive the controller to send a req request signal to the arbitrator, and each host has its own unique corresponding req signal. The arbitrator will grant the right to use the VPU core to a host that generates the req signal according to the arbitration algorithm.

[0061] Step S2: After VPU host 1 pulls up the req signal corresponding to channel 1, it starts to send APB data for accessing the VPU core to the arbitrator. However, before the arbitrator assigns weight, this access will be suspended by the arbitrator in a software blocking manner until host 1 obtains the arbitrator weight.

[0062] After the VPU host 1 channel pulls up the req signal requesting to occupy the VPU core, it will not wait for the weight signal returned by the arbitrator, but will directly access the data inside the VPU core through the APB bus. However, since channel 1 does not receive the weight signal returned by the arbitrator at this time, the arbitrator will keep the PREADY signal of this APB transmission at a low level, and suspend the access of host 1 through software blocking (APB transmission cannot handshake) until the arbitrator re-arbitrates and assigns the weight to host channel 1. Among them, before accessing the VPU core, each host will assign 0x_01 to the 0x_0001 address register inside the controller (pull up the req signal), and then directly send an APB data request to access the VPU core to the arbitrator, but this request will be suspended by the arbitrator through software blocking (because the current VPU core is assigned to other channels by the arbitrator) until the arbitrator assigns the output weight signal to the channel.

[0063] Step S3: VPU host 1 waits for the arbitrator to assign a weight signal to channel 1.

[0064] During the waiting process, the APB request of host 1 to access the VPU core will continue to be suspended, and the software program executed by host 1 will be blocked in the current state and cannot continue to execute. The arbitrator will re-assign the weight according to the usage of the VPU core, the req signals of other channels and the dynamic priority of each channel.

[0065] Step S4: the arbitrator grants the VPU core usage right to host 1, the APB access of its channel will be restored, the software is no longer blocked, and host 1 starts to control the VPU core to perform encoding and decoding operations.

[0066] After receiving the arbitration signal returned by the arbitrator, host 1 obtains the right to use the VPU core. At the same time, the APB access data sent by the host channel to the VPU core through the arbitrator will be released from the suspended state and the right to access the VPU core will be restored. The driver software executed inside host 1 is no longer blocked at this time and can access the registers of the VPU core sequentially. After configuring the registers, it will also control it to perform encoding and decoding operations.

[0067] Step S5: When host 1 controls the VPU core to perform encoding and decoding, the weight signal output by the arbitrator is not only returned to each controller, but also output to the data selector, which exchanges data for encoding and decoding operations between the VPU core and the storage area corresponding to the channel of host 1.

[0068] Step S6: After the encoding and decoding operation using the VPU core is completed, host 1 will drive the controller to pull the req signal low, that is, the req1 signal input to the arbitrator is released. At this time, the arbitrator will release the use right of host 1, and reallocate the output weight of the arbitrator according to the req requests and channel priorities of other channels, and grant the use right of the VPU core to other channels that need to be used. Figure 4 In the situation described in, the driver software of each host channel will assign 0x_00 to the controller's 0x_0001 address register after finishing using the VPU core, so that the req signal corresponding to the channel is pulled low and no longer competes for the right to use the VPU core.

[0069] Step S7: After the host 1 releases the right to use the VPU core, if other channels do not pull up the req signal to request the right to use the VPU core, the arbitrator will keep the VPU core in an idle state. If there are one or more other channels that pull up the req request, the arbitrator will reassign the weight according to the arbitration algorithm and start scheduling the VPU to complete the decoding requirements of other channels.

[0070] By blocking the software, it can be ensured that the driver running on each host of the VPU virtualization channel is only suspended, and will not be stuck. The arbitrator will solve the encoding and decoding requirements of each VPU channel according to the occupancy of the VPU module and the internal arbitration algorithm. This means that the arbitration algorithm basically determines the fairness of the arbitrator's allocation of VPU core usage rights. An unreasonable arbitration algorithm will cause the VPU module to drop frames or black screen during the virtualization process.

[0071] In order to ensure that each virtualized host channel of the VPU can obtain the right to use the VPU core within a certain period of time, the present invention designs an arbitration algorithm based on polling arbitration for the arbitrator, which can ensure that a single channel will not occupy the VPU core for a long time during the arbitration process, and each arbitration request req signal has a dynamic priority under the condition of satisfying the first-come-first-served principle.

[0072] The arbiter uses a round-robin arbitration algorithm to determine the order in which multiple hosts use the VPU core, including:

[0073] Set the initial priority order of multiple host channels;

[0074] In the first arbitration cycle, when multiple host channels receive a request signal for the right to use the VPU core, the host channel with the highest priority among the multiple host channels requesting the right to use is selected, and the right to use the VPU core is granted to the host channel with the highest priority;

[0075] In the second arbitration cycle, the priority of the host channel selected in the first arbitration cycle is reduced to the lowest, and the priority of each host channel is updated in turn to obtain the priority order of multiple host channels in the second arbitration cycle, and in the second arbitration cycle, the host with the right to use the VPU core is arbitrated;

[0076] The same method is used to dynamically update the arbitration and priority order of multiple host channels in each arbitration cycle.

[0077] For example, if there are n host channels, the initial priority order of multiple host channels is set to P1<P2<P3<....P h <....P g <...<P i <...<P n , Pi represents the i-th host channel;

[0078] In the first arbitration cycle, the priority order of multiple host channels is the initial priority order P1<P2<P3<....<P i <...<P n , if in the first arbitration cycle, the host channel P h and P g Generate a usage right request, host channel P g The priority is greater than the host channel P h The priority of the VPU core is given to the host channel P during the arbitration period. g ;

[0079] In the second arbitration cycle, the host channel P selected in the first arbitration cycle is gThe priority is reduced to the lowest level, and the priority of each host is cyclically updated to: P g <... < P i <... < P n < P1 < P2 < P3 <....P h <...., obtain the priority order of multiple hosts in the second arbitration cycle, and in the second arbitration cycle, arbitrate the host that has the right to use the VPU core

[0080] Arbitration of multiple hosts and dynamic update of the priority order in each arbitration cycle are achieved by the same method.

[0081] Specifically, the design of the polling arbiter can ensure that each VPU host channel can obtain the right to use the VPU core for encoding and decoding operations. When the arbiter arbitrates the req requests of the VPU multi-host channels, the arbiter will continuously adjust the priority of each VPU host channel as the number of arbitrations changes. Table 1 shows the algorithm of a four-channel polling arbiter. In the first cycle of the four VPU virtualization channels D, C, B, and A, the priority rankings of the four VPU virtualization channels by the arbiter are 3, 2, 1, and 0 respectively. The lower the priority ranking value, the higher its corresponding priority. In the first cycle, the priority order of the four channels is D < C < B < A. Among them, channels D and B generate requests in the first cycle. Since the priority of channel B is higher, the arbiter assigns the weight to channel B. When the arbiter assigns the weight to channel B, in the arbitration process of the second cycle, since channel B has been responded to, its priority is reduced to the lowest. The priority order of the four channels becomes B < A < D < C. Therefore, in the second arbitration cycle, since only channel A generates a request and is responded to, in the process of the third arbitration cycle, the priority of the arbiter becomes A < D < C < B. And at this time, since requests are generated by all four channels, channel B is responded to again. This polling arbiter algorithm with dynamic priority changes makes the priority of each channel not fixed, so that there is a possibility for each channel to obtain the right to use the VPU core.

[0082] Table 1 Four-channel polling arbitration algorithm

[0083]

[0084] The polling arbiter scheme can not only ensure that each host channel can obtain the right to use the VPU core, but also ensure that a single VPU host channel will not occupy the VPU core for a long time. As shown in Table 2, when requests exist in all four channels in each cycle, the priorities of the four channels will continuously change, so that each virtualization channel can obtain the right to use the VPU core once, ensuring the reasonable distribution of the weights output by the arbiter.

[0085] Table 2 Changes in priority of the four channels when all four channels have requests in each cycle

[0086]

[0087] The VPU virtualization method based on software blocking and polling arbitration designed by the present invention meets the first-come-first-served characteristic when responding to multiple VPU virtualization requests. That is, when the VPU core is idle, there is no situation where multiple channels compete for the right to use the VPU core at the same time, so the arbitrator will give the weight to the virtualization channel whose req signal arrives first. The non-contention response waveform during the four-channel VPU virtualization process is shown in Figure 1. Figure 6 As shown, for the usage right request signals of different channels, since there is no other channel competing with it for the usage right of the VPU core, the arbitrator will promptly give the weight to the virtualized channel that needs to use the VPU core.

[0088] When multiple host channels compete for the right to use the VPU core, the arbitrator will grant the right to use the VPU core according to the priority of different host channels. After the host channels respond, the arbitrator will dynamically change the priority of different host channels. Figure 7 The figure shows the weight change during the four-channel VPU virtualization process. Assuming that the four host channels will send requests to the arbitrator in each arbitration cycle, the default priority of the four host channels A, B, C, and D is A>B>C>D. In different arbitration cycles, the priority of each host channel will be dynamically changed according to the result of responding to the host channel. Once the channel request is responded to, its priority will be reduced to the lowest. The closer the waiting sequence is to the bottom, the higher the priority.

[0089] The channel jump situation of the present invention when multiple virtualization channels simultaneously request the right to use the VPU core is as follows: Figure 7 and Figure 8 As shown. After any channel is assigned a weight by the arbitrator, its priority will be reduced to the lowest in the next arbitration cycle, and the priority of other channels that are not assigned a weight will be increased. Therefore, when the four channels A, B, C, and D are arbitrating, if there are request req signals from the four channels in each arbitration cycle, their priority waiting sequence will be constantly refreshed, thereby modifying the channel priority, resulting in all virtualized channels being able to obtain the right to use the VPU core within the specified time period.

[0090] Another feature of the VPU virtualization method based on software blocking and polling arbitration described in the present invention is that when multiple VPU virtualization channels compete for the right to use the VPU core at the same time, the channel not selected by the arbitrator is suspended because the driver is suspended by the PREADY signal in the APB bus, so the req request of the unselected virtualization channel will continue to maintain a high level to request the arbitrator for the right to use the VPU core, until the channel obtains the right to use the VPU core and completes the decoding operation, the driver will release the req signal. Figure 8 As shown in the figure, due to the arbitration principle of the polling arbiter, channels A, B, C, and D can occupy the VPU core in turn. When channel A occupies the VPU core, channel B keeps req_chB at a high level until channel B obtains the right to use the VPU core and completes encoding and decoding. When the VPU core is occupied by channels A, B, and C, channel D also keeps req_chD at a high level until channel D occupies the VPU core and completes the use, and then pulls it down.

[0091] The four-channel VPU virtualization steps based on software blocking and polling arbitration described in the present invention are as follows: Fig. 9 As shown. In the first arbitration cycle, the four channels A, B, C, and D simultaneously generate req signals to the arbitrator. Since channel A has a default high priority and obtains the right to use the VPU core first, the other channels will also be accessed by the arbitrator in turn through the polling algorithm. In each arbitration cycle, the four channels simultaneously compete for the right to use the VPU core. The steps of the VPU virtualization method can be found in Fig. 9 , including the following steps:

[0092] Step S1 ′: the four host channels A, B, C, and D of the VPU simultaneously request the right to use the VPU core from the arbitrator, and the requests req of the four channels are all pulled high.

[0093] Step S2': Since the default priorities of the four VPU host channels A, B, C, and D are A>B>C>D, in the first arbitration cycle, channel A obtains the right to use the VPU core due to its high priority. The req requests generated by other channels (B, C, and D) do not receive a response from the arbitrator.

[0094] Step S3': The req requests generated by channels B, C, and D cannot get a response from the arbitrator, so that the driver software running in these channels is blocked and suspended because it cannot obtain the PREADY signal of the APB bus. The driver software cannot be executed further.

[0095] Step S4': After channel A finishes using the VPU core, the req signal corresponding to the channel is pulled low to release the right to use the VPU core. The arbitrator starts to reallocate the right to use the VPU core.

[0096] Step S5': After channel A releases the VPU core, the polling arbitration algorithm inside the arbitrator will reorder the priorities of the four channels. Since channel A has just obtained the right to use the VPU core, its priority is reduced to the lowest in the next arbitration cycle, and the priorities of the four channels become B>C>D>A. Channel B obtains the right to use the VPU core due to its higher priority.

[0097] Step S6': Channel B can directly access the VPU core because it obtains the weight output by the arbitrator. Therefore, the PREADY signal of the APB bus in channel B is released, and the host driver software in channel B is no longer blocked. However, since channels C and D have not yet obtained the right to use the VPU core, the driver software of their corresponding channels remains blocked.

[0098] Step S7': by reallocating the VPU core usage rights through the polling arbitration algorithm, channels C and D will also obtain the VPU core usage rights in turn according to the previous steps, that is, through software blocking and polling arbitration methods, it is ensured that each channel applying for access to the VPU core can access the VPU core in a relatively reasonable manner.

[0099] The present invention provides a virtualization system based on a video codec VPU. During the virtualization process of the VPU, multiple host channels need to share the encoding and decoding capabilities of the VPU module, and each host channel has the ability to access the VPU kernel. Compared with the traditional virtualization scheme based on simple arbitration, its fixed priority and untimely request will cause a part of the host channels to occupy the VPU kernel for a long time, while the other part of the host channels cannot obtain the right to use the VPU kernel, causing the encoding and decoding to be stuck. The present invention uses a method combining software and hardware to give the permission to request the use of the VPU kernel to the software, and to give the permission to arbitrate and output multi-channel requests to the hardware. The polling arbitration scheme enables all channel priorities to be dynamically changed, and there is no channel with a fixed priority, making the allocation of the VPU module more reasonable and flexible. The software blocking scheme allows the channel that cannot obtain the arbitrator weight to enter the waiting queue, and there will be no "queue jumping" phenomenon, ensuring that all VPU virtualization channels can obtain the right to use the VPU module in a timely and reasonable manner, and reduce the probability of stuck or flower screen within the encoding and decoding capability of the VPU module.

[0100] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0101] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0105] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0106] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A virtualization system based on a video codec VPU, characterized in that: It includes multiple hosts, VPU modules and arbitrators, the VPU modules include multiple controllers and VPU cores, each of the hosts is connected to a corresponding controller via an APB interface, each of the controllers is connected to the arbitrator, the arbitrator is connected to the VPU core, and the VPU core is connected to an external storage area via the Cache module; Host i sends a VPU core usage right request signal to the arbitrator through a corresponding controller in the software driver layer, wherein i is the index of the host, 1≤i≤N, and N is the number of hosts; When the host i receives the VPU core usage right granting signal returned by the arbitrator, it accesses or configures the register inside the VPU core through the corresponding controller, and controls the VPU core to encode and decode the code stream data; wherein the arbitrator adopts a polling arbitration algorithm to determine the order of multiple hosts' usage rights to the VPU core; After the encoding and decoding operation is completed, host i sends a VPU core usage right end signal to the arbitrator through the controller, so that the arbitrator takes back the VPU core usage right of host i and waits for VPU core usage right request signals from other hosts.

2. The virtualization system based on the video codec VPU according to claim 1, characterized in that: The VPU module also includes a Cache module, and the VPU module is encapsulated with multiple APB Slave interfaces and multiple APB Master interfaces. Each controller in the VPU module communicates with the corresponding host through the APB Slave interface, and each controller communicates with the VPU core and the Cache module respectively through the APB Master interface. The Cache module communicates with the external storage area through the AXI interface.

3. The virtualization system based on the video codec VPU according to claim 1, characterized in that: Each of the controllers includes a plurality of address registers, and the host i sends a VPU core usage right request signal to the arbitrator through the corresponding controller in the software driver layer, including: Host i writes 0x_01 data to the 0x_0001 address register of the corresponding controller, so that the controller corresponding to the host pulls up its req request signal and starts to request the right to use the VPU Core from the arbitrator; After the encoding and decoding operation is completed, the host i sends a VPU core usage right end signal to the arbitrator through the controller, including: The host i writes data 0x_00 to the 0x_0001 address register of the corresponding controller, so that the controller corresponding to the host pulls down its req request signal, informing the arbitrator that the right to use the VPU Core has ended.

4. The virtualization system based on the video codec VPU according to claim 1, characterized in that: The system also includes a data selector, which is connected to the arbitrator and is respectively connected to the VPU core and each external storage area through an AXI interface, wherein a plurality of the external storage areas correspond to a plurality of the hosts, and each of the external storage areas is used to store code stream data after the VPU core performs encoding and decoding operations on a corresponding host request.

5. The virtualization system based on the video codec VPU according to claim 4, characterized in that: The data selector receives the host currently having the right to use the VPU core transmitted by the arbitrator, and stores the code stream data after the VPU core performs encoding and decoding operations in an external storage area corresponding to the host currently having the right to use the VPU core.

6. The virtualization system based on the video codec VPU according to claim 1, characterized in that: The arbiter uses a polling arbitration algorithm to determine the order of use rights of multiple hosts to the VPU core, including: Set the initial priority order of multiple host channels; In the first arbitration cycle, when multiple host channels receive a request signal for the right to use the VPU core, the host channel with the highest priority among the multiple host channels requesting the right to use is selected, and the right to use the VPU core is granted to the host channel with the highest priority; In the second arbitration cycle, the priority of the host channel selected in the first arbitration cycle is reduced to the lowest, and the priority of each host channel is updated in turn to obtain the priority order of multiple host channels in the second arbitration cycle, and in the second arbitration cycle, the host channel with the right to use the VPU core is arbitrated; The same method is used to dynamically update the arbitration and priority order of multiple host channels in each arbitration cycle.

7. The virtualization system based on the video codec VPU according to claim 6, characterized in that: Among them, The arbiter uses a round-robin arbitration algorithm to determine the order of use rights of multiple host channels to the VPU core, including: Set the initial priority order of multiple host channels to P1<P2<P3<....P h <....P g <...<P i <...<P n , P i represents the i-th host channel; In the first arbitration cycle, the priority order of multiple host channels is the initial priority order P1<P2<P3<....P h <....P g <...<P i <...<P n , if in the first arbitration cycle, the host channel P h and P g Generate a usage right request, host channel P g The priority is greater than the host channel P h The arbiter grants the use right of the VPU core in the first arbitration cycle to the host channel P. g ; In the second arbitration cycle, the host channel P selected in the first arbitration cycle is g The priority of each host channel is reduced to the lowest, and the priority of each host channel is updated in turn: P g <...<P i <...<P n <P1<P2<P3<....P h <...., obtain the priority order of multiple host channels in the second arbitration cycle, and in the second arbitration cycle, arbitrate the selected host channel with the right to use the VPU core; The same method is used to dynamically update the arbitration and priority order of multiple host channels in each arbitration cycle.

Images