Video encoding method and system, device, and storage medium

Abstract

Provided in the embodiments of the present disclosure are a video encoding method and system, a device, and a storage medium. In the embodiments of the present disclosure, hardware encoders implemented on different hardware unloading devices form an encoding resource pool, a resource management and control node uniformly schedules the hardware encoders in the encoding resource pool, and computing nodes can present virtual encoders for virtualization instances so as to implement the function of decoupling front and rear encoding resource ends. Thus, without a virtualization instance's awareness, a computing node may mount a hardware encoder on a local hardware unloading device in proximity, and may also mount a hardware encoder on a far-end hardware unloading device.

Description

Video encoding method, device, system and storage medium

[0001] Cross-reference

[0002] This disclosure claims priority to the Chinese patent application filed with the China Patent Office on August 3, 2023, application number 202310975826.9, application name “Video Coding Method, Device, System and Storage Medium,” the entire contents of which are incorporated by reference into this disclosure. Technical Field

[0003] The present disclosure relates to the field of cloud computing technology, and in particular to a video encoding method, device, system, and storage medium. Background Art

[0004] In current cloud computing infrastructure, Data Processing Unit (DPU) chips are becoming standard components, similar to traditional network interface cards. DPU-based acceleration technology enables hardware acceleration by offloading computing, storage, and network virtualization functions from the host to the DPU chip. For example, offloading video encoders to the DPU chip enables true "plug-and-play acceleration" capabilities for the host, enabling video encoding services while maintaining the speed advantage of hardware encoding.

[0005] In actual applications, if hardware acceleration of the video encoder is implemented based on the DPU, but the client virtual machine on the host where the DPU is located does not need to use the video encoding service, the video encoder on the DPU will become an idle resource, resulting in a waste of encoding resources.

[0006] Summary of the Invention

[0007] Various aspects of the present disclosure provide a video encoding method, device, system, and storage medium to implement multiplexing of encoding resources, improve utilization of encoding resources, and reduce waste of encoding resources.

[0008] An embodiment of the present disclosure provides a video encoding system, comprising: a resource management and control node and an encoding resource pool, wherein the encoding resource pool comprises a plurality of hardware offloading devices distributed on a plurality of computing nodes, and a plurality of hardware encoders are implemented on the plurality of hardware offloading devices; the resource management and control node is configured to allocate a target hardware encoder to a target virtualization instance from the encoding resource pool, and control the target computing node carrying the target virtualization instance to mount the target hardware encoder; the target hardware encoder is a hardware encoder on a local hardware offloading device distributed on the target computing node, or a hardware encoder on a remote hardware offloading device distributed on other computing nodes; the target computing node is configured to present a virtual encoder to the target virtualization instance, and control the target hardware encoder to provide video encoding services for the target virtualization instance in the form of the virtual encoder.

[0009] An embodiment of the present disclosure also provides a computing node, which carries a target virtualization instance and includes a first processor and a local hardware offloading device, wherein a hardware encoder is implemented on the local hardware offloading device; the first processor is configured to mount a target hardware encoder allocated to the target virtualization instance from a coding resource pool, present a virtual encoder to the target virtualization instance, and control the target hardware encoder to provide video encoding services for the target virtualization instance in the form of the virtual encoder; wherein the coding resource pool includes hardware encoders on the local hardware offloading device and hardware encoders on remote hardware offloading devices distributed on other computing nodes; the target hardware encoder is a hardware encoder on the local hardware offloading device, or a hardware encoder on a remote hardware offloading device.

[0010] An embodiment of the present disclosure also provides a hardware offloading device, comprising: a hardware encoder implemented on the hardware offloading device, the hardware encoder belonging to a hardware encoder in an encoding resource pool, the encoding resource pool including multiple hardware encoders implemented on multiple hardware offloading devices; the hardware encoder is configured to provide video encoding services for the target virtualization instance in the form of a virtual encoder when assigned to a target virtualization instance and mounted to a target computing node that hosts the target virtualization instance, the virtual encoder being a virtualization of the hardware encoder; wherein the target computing node is a local computing node where the hardware offloading device is located, or a remote computing node interconnected with the hardware offloading device through a network.

[0011] An embodiment of the present disclosure provides a video encoding method, which is applied to a resource management and control node. The method includes: allocating a target hardware encoder to a target virtualization instance from an encoding resource pool, wherein the encoding resource pool includes multiple hardware encoders implemented on multiple hardware offloading devices, and the multiple hardware offloading devices are distributed on multiple computing nodes; controlling the target computing node that hosts the target virtualization instance to mount the target hardware encoder, so that the target virtualization instance virtualizes the target hardware encoder into a recognizable virtual encoder, and uses the video encoding service provided by the target hardware encoder in the form of the virtual encoder; wherein the target hardware encoder is a hardware encoder on a local hardware offloading device distributed on the target computing node, or a hardware encoder on a remote hardware offloading device distributed on other computing nodes.

[0012] An embodiment of the present disclosure also provides a video encoding method, which is applied to a computing node, wherein the computing node carries a target virtualization instance, and the computing node includes a local hardware offloading device, and a hardware encoder is implemented on the local hardware offloading device; the method includes: mounting a target hardware encoder allocated to the target virtualization instance from a coding resource pool, wherein the coding resource pool includes a hardware encoder on the local hardware offloading device and a hardware encoder on a remote hardware offloading device distributed on other computing nodes, and the target hardware encoder is a hardware encoder on the local hardware offloading device, or a hardware encoder on a remote hardware offloading device; presenting a virtual encoder to the target virtualization instance, and controlling the target hardware encoder to provide video encoding services for the target virtualization instance in the form of the virtual encoder.

[0013] An embodiment of the present disclosure provides a hardware acceleration system, comprising: a resource management and control node and a resource pool, wherein the resource pool includes multiple hardware offload devices distributed on multiple computing nodes, and multiple hardware accelerators are implemented on the multiple hardware offload devices; the resource management and control node is configured to allocate a target hardware accelerator from the resource pool to a target virtualization instance, and control a target computing node hosting the target virtualization instance to mount the target hardware accelerator; the target hardware accelerator is a hardware accelerator on a local hardware offload device distributed on the target computing node, or a hardware accelerator on a remote hardware offload device distributed on other computing nodes; the target computing node is configured to present a virtual accelerator to the target virtualization instance, and control the target hardware accelerator to provide a target service for the target virtualization instance in the form of the virtual accelerator.

[0014] An embodiment of the present disclosure provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement the steps of the video encoding method provided by the embodiment of the present disclosure.

[0015] In the embodiment of the present disclosure, the hardware encoders implemented on different hardware offloading devices form an encoding resource pool, and the hardware encoders in the encoding resource pool are uniformly scheduled through the resource management node. Combined with the computing node's ability to present the virtual encoder to the virtualized instance, the front-end and back-end decoupling of encoding resources is realized, so that the computing node can mount the hardware encoder on the local hardware offloading device nearby without the perception of the virtualized instance, and can also mount the hardware encoder on the remote hardware offloading device, thereby realizing the reuse of the hardware encoder, improving the utilization rate of hardware encoding resources, and solving the problem of waste of encoding resources.

[0016] Furthermore, if the encoding capacity of the virtualized instance on a single computing node needs to be expanded, by reusing the hardware encoder, the hardware encoder that meets its encoding needs can be flexibly allocated to the virtualized instance without the need for hot migration of the virtualized instance, which can effectively solve the encoder expansion problem.

[0017] Furthermore, by reusing hardware encoders, multiple hardware encoders can be assigned to the same virtualization instance to achieve aggregation of encoding capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation of the present disclosure. In the drawings:

[0019] FIG1 is a schematic structural diagram of a video encoding system provided by an embodiment of the present disclosure;

[0020] FIG2 is a schematic diagram of a state in which a target computing node according to an embodiment of the present disclosure performs encoding on a mounted local or remote hardware encoder with front-end and back-end decoupling;

[0021] FIG3 is a schematic diagram of the internal structure of a DPU and its interconnection with computing nodes provided by an embodiment of the present disclosure;

[0022] FIG4 is a schematic diagram illustrating an RDMA process in a scenario where a remote hardware encoder is mounted, in combination with the internal architecture of a VM and a DPU, according to an embodiment of the present disclosure;

[0023] FIG5 is an interactive flow chart of a computing node mounting a remote hardware encoder for video encoding according to an embodiment of the present disclosure;

[0024] FIG6 is a schematic diagram of a flow chart of a video encoding method provided by an embodiment of the present disclosure;

[0025] FIG7 is a schematic diagram of a flow chart of another video encoding method provided by an embodiment of the present disclosure;

[0026] FIG8 is a schematic diagram of the structure of another hardware acceleration system provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] To make the objectives, technical solutions, and advantages of the present disclosure more clear, the technical solutions of the present disclosure will be clearly and completely described below in conjunction with the specific embodiments of the present disclosure and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present disclosure.

[0028] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0029] In the scenario of implementing encoding hardware acceleration based on DPU, there may be idle encoding resources, resulting in a technical problem of wasted encoding resources. To this end, in the embodiment of the present disclosure, the hardware encoders implemented on different hardware offloading devices are formed into an encoding resource pool, and the hardware encoders in the encoding resource pool are uniformly scheduled through the resource management node. Combined with the function of the computing node being able to present the virtual encoder to the virtualized instance to achieve the front-end and back-end decoupling of encoding resources, the computing node can mount the hardware encoder on the local hardware offloading device nearby without being aware of the virtualized instance, and can also mount the hardware encoder on the remote hardware offloading device, thereby realizing the reuse of the hardware encoder, improving the utilization rate of hardware encoding resources, and solving the problem of waste of encoding resources.

[0030] Furthermore, if the encoding capacity of the virtualized instance on a single computing node needs to be expanded, by reusing the hardware encoder, the hardware encoder that meets its encoding needs can be flexibly allocated to the virtualized instance without the need for hot migration of the virtualized instance, which can effectively solve the encoder expansion problem.

[0031] Furthermore, by reusing hardware encoders, multiple hardware encoders can be assigned to the same virtualization instance to achieve aggregation of encoding capabilities.

[0032] The technical solutions provided by various embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0033] FIG1 is a schematic diagram of the structure of a video encoding system provided by an embodiment of the present disclosure. As shown in FIG1 , the system 100 includes a resource management node 101 and an encoding resource pool 102 . The encoding resource pool 102 includes multiple hardware offload devices 20 distributed across multiple computing nodes 10 , and multiple hardware encoders 21 are implemented or integrated on the multiple hardware offload devices 20 .

[0034] In this embodiment, the implementation form of the resource management and control node 101 is not limited and can be various physical devices, hardware modules, systems, software programs, functional modules, application programming interfaces (APIs), or software development kits (SDKs) with computing, communication, and processing capabilities. For example, the management and control node can be a mobile phone, a laptop, a wearable device, a traditional server, a cloud server, a cloud host, a virtual machine, or a container.

[0035] In this embodiment, the computing node 10 can be any physical device that can provide hardware resources, such as a mobile phone, a notebook, a storage device, a traditional server, or a cloud server, a cloud host, or other cloud devices with hardware resources. In this embodiment, the computing node 10 is configured to carry a virtualization instance and is responsible for providing the hardware resources required for the operation of the virtualization instance, such as but not limited to: a central processing unit (CPU), a memory (such as memory), a network card, a persistent memory, an input and output device, and other devices with specific processing functions, such as an input / output memory management unit (IOMMU), etc., wherein the IOMMU can be configured to convert the physical address of the virtualization instance and the physical address of the computing node. In this embodiment, a virtualization instance generally refers to an application instance implemented based on virtualization technology, for example, a virtualization instance can be a virtual machine (VM), a container instance, etc.

[0036] In this embodiment, in addition to the above-mentioned hardware resources, the computing node 10 also includes a hardware offloading device 20. The hardware offloading device 20 refers to a device that can operate independently, has virtualization capabilities, network interconnection functions, and supports hardware offloading of various software functions in the cloud computing infrastructure. In terms of implementation form, the hardware offloading device 20 can be a module or a chip. In an optional embodiment, the hardware offloading device 20 can be implemented as a subsystem of the DPU in the field of cloud computing. The subsystem can reuse some existing functions in the DPU, such as the agent function of management and control, the virtualization framework, and the processor. In another optional embodiment, the hardware offloading device 20 can be a programmable device implemented based on programmable hardware, wherein the programmable hardware can be an application specific integrated circuit (ASIC), a system on chip (SOC), a field programmable gate array (FPGA), or a complex programmable logic device (CPLD). In addition, the hardware offloading device 20 in the embodiment of the present disclosure can be implemented as a pluggable structure and plugged into the computing node 10. The pluggable structure has advantages such as flexibility and scalability, but is not limited to this. For example, the hardware offloading device 20 can also be directly integrated into the computing node 10. Optionally, the hardware offloading device 20 and the computing node 10 can be interconnected via various buses, including but not limited to the high-speed serial computer expansion bus standard (Peripheral Component Interconnect Express, PCIE).

[0037] In this embodiment, a hardware encoder 21 is implemented or integrated on the hardware offload device 20. The hardware encoder 21 can be a hardware module implemented using programmable hardware such as an ASIC, SOC, FPGA, or CPLD on the hardware offload device 20, and is configured to provide video encoding services. Offloading the video encoding service to the hardware encoder 21 on the hardware offload device 20 not only completes the video encoding task, but also maintains the speed and efficiency advantages of hardware encoding, thereby achieving hardware acceleration of video encoding.

[0038] In this embodiment, hardware encoders 21 are integrated into different hardware offload devices 20. These hardware encoders 21, or the hardware offload devices 20 where these hardware encoders 21 are located, are interconnected via a network to form an encoding resource pool 102. In this embodiment, the resource management and control node 101 centrally schedules and manages the hardware encoders 21 in the encoding resource pool 102, enabling reuse of hardware encoder resources.

[0039] In this embodiment, the hardware encoder 21 can provide video encoding services for virtualized instances with encoding requirements. Based on this, the resource management node 101 can allocate a suitable hardware encoder 21 from the encoding resource pool 102 to any virtualized instance with encoding requirements, and mount the allocated hardware encoder 21 to the computing node 10 that hosts the virtualized instance, so that the hardware encoder 21 can provide video encoding services for the virtualized instance. For the sake of ease of description and distinction, the virtualized instance that has encoding requirements and requires a hardware encoder is referred to as a target virtualized instance. The target virtualized instance can be any newly deployed virtualized instance or any deployed virtualized instance that needs to expand its encoding capabilities. There is no limitation on this.

[0040] In actual applications, the resource management and control node 101 can collect the basic capability information of each hardware encoder 21 and maintain the usage status information of each hardware encoder 21. The basic capability information of the hardware encoder 21 refers to information that can characterize the encoding capability that the hardware encoder 21 can provide, including but not limited to: the picture resolution and picture smoothness that the hardware encoder 21 can provide, such as 1080p / 30fps, 1080p / 60fps, etc. Among them, 1080p / 30fps means that the picture resolution is 1920×1080 and the picture smoothness is 30 frames per second; 1080p / 60fps means that the picture resolution is 1920×1080 and the picture smoothness is 60 frames per second. Optionally, a way to collect basic capability information includes: when the hardware unloading device 20 is started, it actively reports its own basic capability information to the resource management and control node 101. Another way to collect basic capability information includes: the resource management node 101 regularly sends an information collection request to the hardware offloading device 20, and the hardware offloading device 20 regularly reports its own basic capability information to the resource management node 101 according to the information collection request.

[0041] The usage status information of the hardware encoder 21 includes whether the hardware encoder 21 is in an idle state and, if not in an idle state, how much of the hardware encoder's basic capability information has been used. It may also include information such as which virtualization instances are using it. In this embodiment, the resource management node 101 may record and maintain the usage status information of the hardware encoder during the process of allocating the hardware encoder to the target virtualization instance.

[0042] In an optional embodiment, when a user needs to deploy a target virtualization instance, the user can send a video encoding requirement to the resource management node 101 to request the resource management node 101 to allocate a hardware encoder for the target virtualization instance; the resource management node 101 can respond to the user's video encoding requirement and determine the encoding resource requirement information of the target virtualization instance, which describes how much encoding capacity the target virtualization instance requires, such as 1080p / 30fps or 1080p / 60fps, etc.; based on the basic capability information and usage status information of multiple hardware encoders, a hardware encoder that meets the encoding resource requirement information is selected as the target hardware encoder. In this embodiment, for ease of description and distinction, the hardware encoder allocated to the target virtualization instance is referred to as the target hardware encoder.

[0043] In this embodiment, the method by which a user submits a video encoding request to the resource management node 101 is not limited. In an optional embodiment, the resource management node 101 can provide a resource management page to the user in the form of a cloud service. The resource management page can be a web page, an application page, or a mini-program page, without limitation. The user submits the video encoding request to the resource management node 101 through the resource management page, and submits the identification information and encoding resource requirement information of the target virtualization instance associated with the video encoding request. In another optional embodiment, the resource management node 101 interacts with the user in the form of a command window. The user can enter a relevant command in the command window to submit the video encoding request to the resource management node 101. The command includes the identification information and encoding resource requirement information of the target virtualization instance. Of course, the resource management node 101 can also provide an API to the user, and the user can also submit the video encoding request to the resource management node 101 through the API, and submit the identification information and encoding resource requirement information of the target virtualization instance associated with the video encoding request.

[0044] In this embodiment, the computing node that carries the target virtualization instance is referred to as the target computing node, wherein the target computing node can be selected from multiple computing nodes according to the user's demand for deploying the target virtualization instance. In an optional embodiment, the resource management and control node can be responsible for selecting a suitable target computing node for the target virtualization instance. Of course, the selection of the target computing node can also be the responsibility of other devices or nodes, and there is no limitation on this. When the target computing node is selected, the target virtualization instance can be deployed on the target computing node according to the image file or memory snapshot corresponding to the target virtualization instance. The memory snapshot is the memory data of the target virtualization instance or a virtualization instance of the same type in a certain running state. The target virtualization instance can be quickly restored based on the memory snapshot.

[0045] When a target computing node is selected to host a target virtualization instance and a target hardware encoder has been assigned to the target virtualization instance, the resource control node 101 can also control the target computing node to mount the target hardware encoder so that the target hardware encoder can provide video encoding services for the target virtualization instance. Optionally, the resource control node 101 maintains identification information of each hardware encoder, which can be the name, ID, or IP address of the hardware encoder; the resource control node 101 can transmit the identification information of the target hardware encoder to the target computing node via the network, and the target computing node determines the assigned target hardware encoder based on the identification information of the target hardware encoder and mounts the target hardware encoder.

[0046] For the purpose of explanation, the target hardware encoder mounted on the target computing node may be a local hardware encoder or a remote hardware encoder. Therefore, the video encoding system of this embodiment can also be referred to as a distributed video encoding system. The terms "local" and "remote" are relative to the target computing node. A local hardware encoder refers to a hardware encoder on a local hardware offload device, and a local hardware offload device refers to a hardware offload device distributed on the target computing node. Correspondingly, a remote hardware encoder refers to a hardware encoder on a remote hardware offload device, and a remote hardware offload device refers to a hardware offload device distributed on other computing nodes.

[0047] When the target computing node mounts the target hardware encoder, the target hardware encoder can be used to provide video encoding services for the target virtualized instance. To achieve distributed video encoding, the target computing node can decouple the front-end and back-end of encoding resources through virtualization technology. That is, the target computing node can present a virtual encoder to the target virtualized instance, maintaining a unified front-end for the target virtualized instance, thereby ensuring the consistency of the encoding front-end driver of the virtualized instance. At the same time, for the encoding back-end, a local hardware encoder or a remote hardware encoder can be mounted on demand. Based on the decoupling of the front-end and back-end of encoding resources, the target hardware encoder can be controlled to provide video encoding services to the target virtualized instance in the form of a virtual encoder, thereby achieving distributed video encoding.

[0048] In this embodiment, the virtualization technology used by the target computing node is not limited. For example, it can be used but is not limited to: Single Root I / O Virtualization (SR-IOV). The target computing node can perform device virtualization on the target hardware encoder based on SR-IOV technology, and virtualize it into a standard virtio-video (virtualized input and output - video) device. The virtio-video device is a specific implementation form of the virtual encoder under SR-IOV technology. In this embodiment, the use of SR-IOV technology allows the target hardware encoder to be efficiently shared between virtualization instances. That is, whether the target hardware encoder is local or remote, it can be virtualized into multiple virtio-video devices (i.e., virtual encoders). The virtio-video device can also be called a virtual function (VF) of SR-IOV. These SR-IOV VFs can share one or more target hardware encoders.

[0049] The target computing node includes an operating system on top of the hardware resources, and a virtual machine manager runs on top of the operating system. The virtual machine manager is configured to create an isolated operating environment on the computing node where it is located, and to create and manage virtualized instances in the isolated operating environment. The virtual machine manager can be any software program, component or module that can create an isolated operating environment on the computing node where it is located, and to create and manage virtualized instances in the isolated operating environment, without limitation. Optionally, the virtual machine manager can be implemented as a virtual operating system simulator (Quick EMUlator, QEMU) or a kernel-based virtual machine (KVM) or a hypervisor, etc., without limitation. Among them, the hypervisor is also called a virtual machine monitor (VMM).

[0050] In an optional embodiment, as shown in FIG2 , a state diagram of the decoupling of the front-end and back-end of encoding of a mounted local or remote hardware encoder using virtualization technology is provided for a target computing node. In FIG2 , taking the target computing node using QEMU or KVM as an example, QEMU or KVM uses SR-IOV technology to virtualize the target hardware encoder mounted on the target computing node into one or more SR-IOV VFs to provide services for the target virtualization instance. In addition, in FIG2 , the target virtualization instance is a VM, specifically VM0-VM3, as an example, and VM0 and VM1 are used to mount a local hardware encoder, and VM2 and VM3 are used to mount a remote hardware encoder as an example, wherein the remote hardware encoder is interconnected with the target computing node through a network.

[0051] Furthermore, based on the front-end and back-end decoupling of the target hardware encoder, the target computing node can control the target hardware encoder to provide video encoding services for the target virtualized instance in the form of a virtual encoder. In this embodiment, the implementation method of the target computing node controlling the target hardware encoder to provide video encoding services for the target virtualized instance in the form of a virtual encoder is not limited. In an optional embodiment, the target computing node can, with the cooperation of the virtual encoder, control the target hardware encoder to provide video encoding services for the target virtualized instance through a local hardware offload device.

[0052] Specifically, as shown in FIG3 , the hardware offload device 20 of this embodiment further includes a processor (CPU) 22. The processor 22 is the control center of the hardware offload device 20 and is responsible for running various software programs required for the normal operation of the hardware offload device 20, including but not limited to the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol stack configured for network transmission, the firmware program of the hardware encoder 21, and the drivers for other hardware modules implemented on the hardware offload device 20, such as the driver for the Remote Direct Memory Access (RDMA) module mentioned in subsequent embodiments. In this embodiment, the processor 22 represents the hardware offload device 20, has an independent IP address, and an independent TCP / IP protocol stack, enabling network interconnection based on the TCP / IP protocol.

[0053] Based on the above, the target computing node can provide the encoding request and the video frame to be encoded of the target virtualized instance to the local hardware offloading device through the virtual encoder, specifically to the processor 22 of the local hardware offloading device; the processor 22 on the local hardware offloading device is configured to control the target hardware encoder to encode the video frame to be encoded according to the encoding request to obtain the code stream data required by the target virtualized instance. Among them, the target computing node provides the encoding request and the video frame to be encoded of the target virtualized instance to the local hardware offloading device through the virtual encoder, including: the target virtualized instance can write the encoding request and the video frame to be encoded into the memory of the target virtualized instance, specifically into the memory space allocated to the virtual encoder in the memory, and the target virtualized instance communicates with the processor 22 of the local hardware offloading device through the memory space.

[0054] Further optionally, the memory space allocated to the virtual encoder includes a first cache space and a second cache space, the first cache space being configured to cache the video frames to be encoded and the encoded bitstream data, and the second cache space being configured to cache various control plane messages required for the video encoding process, such as instructions for turning on or off the target hardware encoder, instructions for obtaining attribute information of the target hardware encoder, and encoding requests. Optionally, the first cache space can be implemented as a frame cache and an encoded stream (ES) cache, the frame cache being configured to cache the video frames to be encoded, the ES cache being configured to cache the encoded bitstream data, and the second cache space being implemented as a Virtio ring cache, configured to cache control plane messages.

[0055] Furthermore, as shown in FIG3 , the hardware offloading device 20 of this embodiment further includes: an RDMA module 23. In the process of controlling the target hardware encoder to encode the video frame to be encoded, the processor 22 on the local hardware offloading device can transmit data between the memory of the target virtualization instance and the memory of the target hardware encoder through the RDMA module 23 on the local hardware offloading device and / or the RDMA module 23 on the remote hardware offloading device, depending on whether the target hardware encoder is local or remote. Specifically, the processor 22 on the local hardware offloading device can read the encoding request from the second cache space, such as the Virtio Ring cache, and according to the encoding request, control the RDMA module 23 on the local hardware offloading device to DMA the video frame to be encoded from the memory of the target virtualization instance to the memory of the target hardware encoder, so that the target hardware encoder can encode the video frame to be encoded to obtain the code stream data.

[0056] Furthermore, the processor 22 on the hardware offload device where the target hardware encoder is located can control the RDMA module 23 on the hardware offload device where the target hardware encoder is located to DMA the code stream data from the memory of the target hardware encoder to the memory of the target virtualized instance. Specifically, if the target hardware encoder is a local hardware encoder, then the processor 22 on the hardware offload device where the target hardware encoder is located is the processor 22 on the local hardware offload device. The processor 22 on the local hardware offload device can control the RDMA module 23 on the local hardware offload device to DMA the code stream data from the memory of the local hardware encoder (i.e., the target hardware encoder) to the memory of the target virtualized instance. If the target hardware encoder is a remote hardware encoder, the processor 22 on the local hardware offloading device is also configured to forward the encoding request to the processor 22 on the remote hardware offloading device, and send the first memory address set as the cached code stream data in the memory of the target virtualization instance to the processor 22 on the remote hardware offloading device via the TCP / IP protocol; accordingly, the processor 22 on the remote hardware offloading device is also configured to receive the encoding request and the first memory address sent by the processor 22 on the local hardware offloading device, and upon receiving the encoding request, apply for the second memory address set as the cached video frame to be encoded and the third memory address set as the cached code stream data from the memory of the remote hardware encoder (i.e., the target hardware encoder), and send the second memory address to the processor 22 on the local hardware offloading device. The processor 22 on the local hardware offloading device will also receive the second memory address sent by the processor 22 on the remote hardware offloading device. For ease of description, the memory address set as the cached video frame to be encoded in the memory of the target virtualization instance is referred to as the fourth memory address. Furthermore, the processor 22 on the local hardware offload device can control the RDMA module 23 on the local hardware offload device to DMA the to-be-encoded video frame from the memory of the target virtualized instance to the memory of the remote hardware encoder (i.e., the target hardware encoder) based on the fourth memory address and the second memory address. Correspondingly, the processor 22 on the remote hardware offload device can control the RDMA module 23 on the remote hardware offload device to DMA the code stream data from the memory of the remote hardware encoder (i.e., the target hardware encoder) to the memory of the target virtualized instance based on the third memory address and the first memory address.

[0057] Based on the above, taking the virtualization instance as a virtual machine and the hardware offload device as a DPU subsystem as an example, the resources in the video encoding system provided by this embodiment are classified according to their functions. The system includes the following functional subsystems:

[0058] (1) Virtual machine subsystem: It includes virtual machines on each computing node and is responsible for providing an isolated operating environment. In the video encoding scenario, encoding applications can be run in the isolated operating environment provided by the virtual machine. The encoding application can cache the video frames to be encoded and the encoded bitstream data in the memory of the virtual machine through the virtual encoder. At the same time, it can also cache the control plane messages of the virtual encoder (such as the virtio-video device), including but not limited to: hardware encoder opening or closing instructions, encoding requests, etc. In Figure 3, the virtualization instance on the computing node is a VM as an example. The VM includes a CPU and memory. In Figure 3, vCPU represents the CPU of the VM, vMemory represents the memory of the VM, and it is interconnected with the RDMA module on the local DPU through the IOMMU.

[0059] (2) DPU processor subsystem: includes the processor on each DPU, which is mainly configured to support the operation of various software programs required for the normal operation of the DPU. In particular, in the video encoding scenario, it is responsible for receiving control plane messages from the encoding application in the VM and forwarding the control plane messages to the local or remote hardware encoder as needed. In the embodiment of the present disclosure, the processor on the DPU has its own memory, such as but not limited to: Dynamic Random Access Memory (DRAM), Static Random-Access Memory (SRAM), etc. In Figure 3, DRAM is used as an example, and in order to distinguish it from the memory of the hardware encoder, the CPU's DRAM is recorded as DRAM0 and the hardware encoder's DRAM is recorded as DRAM1.

[0060] (3) DPU encoding subsystem: includes a hardware encoder on the DPU and encoding firmware running on the processor of the DPU. The processor runs the encoding firmware and can receive control plane messages from the local VM, such as encoding requests, and can also receive control plane messages from the network (i.e., remote VM), such as encoding requests, and control the corresponding hardware encoder to perform encoding. In the embodiment of the present disclosure, the hardware encoder also has a memory for caching the video frames to be encoded and the encoded bitstream data, and can further be used to store the encoding context, such as reconstructed frames. Optionally, an external DRAM can be used as the memory of the hardware encoder. The use of an external DRAM can provide high-bandwidth data access while reducing processing delays. In Figure 3, DRAM1 represents the external DRAM of the hardware encoder. In order to make the external DRAM of the encoder visible to the RDMA module, the DRAM can be exposed to the CPU memory space of the computing node, which can be achieved through the base address register (BAR) space of the PCIE.

[0061] 3 shows both a DRAM controller (DRAM Controller) 24 and a Media Access Control (MAC) / Physical (PHY) module 25. The DRAM controller 24 can be used to control access to the external DRAM, while the MAC / PHY module 25 can be used for network transmission.

[0062] (4) RDMA subsystem: This includes the RDMA module on the DPU, which is configured to perform zero-copy of the encoded video frames and the encoded bitstream data between the VM's memory and the hardware encoder's memory. It should be noted that the initiator of the RDMA module's data DMA behavior is the processor on the DPU. The processor is responsible for creating, using, and destroying RDMA-related resources, including but not limited to queue pairs (QPs), including send queues (SQs) and receive queues (RQs).

[0063] In this embodiment, the RDMA module may copy data in the following ways:

[0064] Scenario 1: Mounting a local hardware encoder

[0065] DMA the video frames to be encoded from the virtual machine memory to the external memory of the local hardware encoder;

[0066] DMA the encoded bitstream data from the local hardware encoder's external memory to the virtual machine's memory.

[0067] Scenario 2: Mounting a remote hardware encoder

[0068] Remote DMA of the video frames to be encoded from the virtual machine memory to the external memory of the remote hardware encoder;

[0069] Remote DMA encoded bitstream data from the external memory of the remote hardware encoder to the virtual machine memory.

[0070] The following is an exemplary description of the process of mounting a remote hardware encoder on a computing node to provide video encoding services for a virtual machine, in conjunction with the RDMA process schematic diagram shown in FIG4 and the interaction flow chart shown in FIG5 .

[0071] In Figure 4, a VM is hosted on the computing node, and an encoding application is running in the VM. Virtualization technology is used to virtualize the remote hardware encoder into a virtual encoder with a unified front-end driver, such as a Virtio-video device. The virtual encoder applies for a first cache space in the VM's memory to cache the video frames to be encoded and the encoded bitstream data, such as a local frame cache and a local ES cache, and a second cache space to cache control plane messages (such as encoding requests), such as a Virtio ring cache.

[0072] The encoding application sends the address of the frame buffer (denoted as the fourth memory address), the address of the ES buffer (denoted as the first memory address), and the address of the Virtio ring buffer to the processor on the local DPU through the Virtio-video device. The encoding application in the VM communicates with the processor on the local DPU through the Virtio ring buffer, delivering control plane messages required for encoding, such as encoding requests, to the processor. It also delivers the video frames to be encoded and the encoded bitstream data through the local frame buffer and local ES buffer.

[0073] In Figure 4, the processor on the local DPU runs a virtual encoder, such as the backend driver corresponding to the Virtio-video device. It obtains encoding requests from the Virtio ring buffer according to the address of the Virtio ring buffer, forwards the encoding requests to the processor on the remote DPU via the TCP / IP protocol, and informs the processor on the remote DPU of the address of the local ES cache in the VM memory set to cache the encoded bitstream data (i.e., the first memory address).

[0074] After receiving the encoding request, the processor on the remote DPU requests DRAM space external to the remote DPU's hardware encoder as a cache for the video frames to be encoded and the encoded bitstream data. This cache, for example, may include a remote frame cache and a remote ES cache. The processor then returns the address of the remote frame cache (recorded as the second memory address) to the processor on the local DPU via TCP / IP. The address of the remote ES cache in the external DRAM space of the remote hardware encoder (i.e., the hardware encoder on the remote DPU) is recorded as the third memory address.

[0075] At this point, the processor on the local DPU has obtained the fourth memory address in the VM memory set to cache the video frame to be encoded and the first memory address set to cache the encoded stream data, as well as the second memory address in the external DRAM space of the remote hardware encoder set to cache the video frame to be encoded and the third memory address set to cache the encoded stream data. Based on this, as shown in Figure 4, the processor on the local DPU can control the RDMA module on the local DPU to initiate an RDMA write operation through the RDMA verbs interface, that is, control the RDMA module to directly DMA the video frame to be encoded from the local frame cache corresponding to the fourth memory address in the VM memory to the remote frame cache corresponding to the second memory address in the external DRAM of the remote hardware encoder. Among them, the RDMA verbs interface is a software transport interface between the RDMA chip and the upper-layer application (such as the encoding application).

[0076] When a certain number of video frames to be encoded are cached in the external DRAM of the remote hardware encoder, the processor on the remote DPU runs the encoding firmware (Encoder Firmware), controls the hardware encoder to encode the video frames to be encoded in the external DRAM, and writes the encoded bitstream data to the remote ES cache corresponding to the third memory address in the external DRAM.

[0077] At a certain moment, the hardware encoder completes the encoding process of the video frame to be encoded. Prior to this, the processor on the remote DPU has obtained the first memory address in the VM memory set as the cache for the encoded bitstream data and the third memory address in the external DRAM of the remote hardware encoder set as the cache for the encoded bitstream data. Based on this, as shown in Figure 4, the processor on the remote DPU can control the RDMA module on the remote DPU through the RDMA verbs interface to initiate an RDMA write operation, that is, to control the RDMA module to DMA the bitstream data directly from the remote ES cache corresponding to the third memory address in the external DRAM of the hardware encoder to the local ES cache corresponding to the first memory address in the VM memory.

[0078] Furthermore, as shown in Figure 5, the simple interactive process of mounting a remote hardware encoder for video encoding includes the following steps:

[0079] 50. The encoding application in the VM sends an encoding request to the processor on the local DPU. The encoding request includes the address of the local frame buffer (i.e., the fourth memory address) and the address of the local ES buffer (i.e., the first memory address).

[0080] 51. The processor on the local DPU forwards the encoding request to the processor on the remote DPU. The encoding request includes the address of the local ES cache.

[0081] 52. The processor on the remote DPU exposes the remote frame buffer and remote ES buffer from the external memory of the remote hardware encoder.

[0082] 53. The processor on the remote DPU returns a coded response message to the processor on the local DPU. The coded response message includes the address of the remote frame buffer (ie, the second memory address).

[0083] 54. The processor on the local DPU controls the RDMA module on the local DPU to perform RDMA operations.

[0084] 55. The RDMA module on the local DPU reads the video frame to be encoded from the local frame buffer and writes it to the remote frame buffer.

[0085] 56. The hardware encoder on the remote DPU encodes the video frame to be encoded and writes the encoded stream data to the remote ES cache.

[0086] 57. The hardware encoder on the remote DPU notifies the processor on the remote DPU that encoding is complete.

[0087] 58. The processor on the remote DPU controls the RDMA module on the remote DPU to perform RDMA operations.

[0088] 59. The RDMA module on the remote DPU reads the stream data from the remote ES cache and writes it to the local ES cache.

[0089] 60. The processor on the local DPU notifies the encoding application in the VM that the encoding operation is complete.

[0090] For detailed description of the above steps, please refer to the above text and will not be repeated here.

[0091] The embodiment of the present disclosure proposes a distributed video encoding solution, in which the hardware encoder is uniformly scheduled and managed by the resource management node, and further combined with the front-end and back-end decoupling function of the encoding resources provided by the computing node, the virtualized instance (such as VM) can use a unified or standard encoder front-end driver, and can mount local or remote hardware encoders on the back end as needed, so as to realize the reuse of hardware encoding resources, greatly alleviate the problem of idle encoding resources, simplify the expansion of encoding capabilities of virtualized instances, and realize the aggregation of encoding capabilities. Furthermore, in the solution provided by the embodiment of the present disclosure, if a remote hardware encoder is mounted, the RDMA module in the hardware offloading device is used to transmit the video frames to be encoded and the encoded code stream data in the data plane, which can greatly reduce the data transmission load and improve the transmission efficiency.

[0092] The disclosed embodiment also provides a hardware offloading device, which has the structure described in the embodiment shown in Figure 3, which is not shown in the figure here. A hardware encoder is implemented on the hardware offloading device of this embodiment, and the hardware encoder belongs to the hardware encoder in the encoding resource pool. The encoding resource pool includes multiple hardware encoders implemented on multiple hardware offloading devices. Each hardware encoder is configured to provide video encoding services to the target virtualization instance in the form of a virtual encoder when it is assigned to the target virtualization instance and mounted on the target computing node that carries the target virtualization instance. The virtual encoder is a virtualization of the hardware encoder; wherein the target computing node is the local computing node where the hardware offloading device of this embodiment is located, or a remote computing node that is network-connected to the hardware offloading device of this embodiment.

[0093] In an optional embodiment, as shown in FIG3 , the hardware offloading device of this embodiment further includes a processor configured to obtain an encoding request and a to-be-encoded video frame from a target virtualized instance, and control the hardware encoder to encode the to-be-encoded video frame to obtain the bitstream data required by the target virtualized instance.

[0094] In an optional embodiment, as shown in FIG3 , the hardware offload device of this embodiment further includes an RDMA module. Based on the RDMA module, when controlling the hardware encoder to encode the video frames to be encoded, the processor is specifically configured to: control the video frames to be encoded to be DMA-ed from the memory of the target virtualized instance to the memory of the hardware encoder of this embodiment, so that the hardware encoder can encode the video frames to obtain bitstream data; and control the RDMA module on the hardware offload device of this embodiment to DMA the bitstream data from the memory of the hardware encoder to the memory of the target virtualized instance.

[0095] Further optionally, in the case where the target computing node is a local computing node, when the processor obtains the encoding request and the video frame to be encoded of the target virtualization instance from the target computing node, it is specifically configured as follows: receiving the encoding request and the video frame to be encoded of the target virtualization instance through the virtual encoder. Specifically, the target virtual machine virtualization instance writes the encoding request and the video frame to be encoded into the memory of the target virtualization instance, the processor reads the encoding request from the memory, and controls the video frame to be encoded from the memory of the target virtualization instance to be DMAed to the memory of the hardware encoder. Further, in the case where the target computing node is a local computing node, when the processor controls the video frame to be encoded from the memory of the target virtualization instance to be DMAed to the memory of the hardware encoder, it is specifically configured as follows: according to the encoding request, controlling the RDMA module of the hardware offloading device of this embodiment to DMA the video frame to be encoded from the memory of the target virtualization instance to the memory of the hardware encoder. Or,

[0096] Further optionally, in the case where the target computing node is a remote computing node, when the processor obtains the encoding request of the target virtualization instance and the video frame to be encoded from the target computing node, the processor is specifically configured to: receive the encoding request forwarded by the processor of the hardware offloading device on the remote computing node, and control the video frame to be encoded to be DMAed from the memory of the target virtualization instance to the memory of the hardware encoder. Further, in the case where the target computing node is a remote computing node, when the processor controls the video frame to be encoded to be DMAed from the memory of the target virtualization instance to the memory of the hardware encoder, the processor is specifically configured to: apply for a second memory address set as a cache for the video frame to be encoded from the memory of the hardware encoder, and send it to the processor of the hardware offloading device on the remote computing node, so that the processor of the hardware offloading device on the remote computing node controls the RDMA module of the hardware offloading device on the remote computing node to DMA the video frame to be encoded from the memory of the target virtualization instance to the memory of the hardware encoder of this embodiment according to the second memory address. Further optionally, the processor can also receive the first memory address set as the cache code stream data in the memory of the target virtualization instance forwarded by the processor of the hardware offload device on the remote computing node, so as to control the RDMA module on the hardware offload device of this embodiment to DMA the code stream data from the memory of the hardware encoder to the memory of the target virtualization instance after the encoding is completed.

[0097] For a detailed description of the functional modules and hardware modules in the hardware offloading device provided in this embodiment and their beneficial effects, please refer to the aforementioned system embodiment and will not be elaborated on in detail here.

[0098] The embodiment of the present disclosure also provides a computing node, which has the structure described in the embodiment shown in Figures 1-4, which is not illustrated here. The computing node of this embodiment carries a target virtualization instance, and the computing node of this embodiment includes a first processor and a local hardware offloading device, and a hardware encoder is implemented on the local hardware offloading device. The first processor is the processor of the computing node. The first processor can be configured to execute a computer program stored in a memory, so as to be configured to implement the following operations: mounting a target hardware encoder allocated to the target virtualization instance from the encoding resource pool, presenting a virtual encoder to the target virtualization instance, and controlling the target hardware encoder to provide video encoding services for the target virtualization instance in the form of a virtual encoder. Among them, the encoding resource pool includes a hardware encoder on the local hardware offloading device and a hardware encoder on a remote hardware offloading device distributed on other computing nodes; the target hardware encoder is a hardware encoder on the local hardware offloading device, or a hardware encoder on a remote hardware offloading device.

[0099] In an optional embodiment, the local hardware offloading device further includes: a second processor. The second processor is a processor on the hardware offloading device, i.e., the processor in the hardware offloading device shown in FIG3 , and is the control center of the hardware offloading device. Based on this, the first processor is specifically configured to provide the encoding request and the video frame to be encoded of the target virtualization instance to the local hardware offloading device through the virtual encoder, so as to control the target hardware encoder through the local hardware offloading device to provide video encoding services for the target virtualization instance. The second processor on the local hardware offloading device runs the encoding backend program, so as to be configured to: according to the encoding request, control the target hardware encoder to encode the video frame to be encoded to obtain the code stream data required by the target virtualization instance.

[0100] Optionally, the local hardware offload device further includes an RDMA module. Based on this, the second processor on the local hardware offload device is specifically configured to control the RDMA module on the local hardware offload device to transfer the to-be-encoded video frames from the memory of the target virtualized instance to the memory of the target hardware encoder via DMA, based on the encoding request, so that the target hardware encoder can encode the to-be-encoded video frames to generate bitstream data.

[0101] Further optionally, the second processor on the local hardware offload device is also configured to control the RDMA module on the local hardware offload device to DMA code stream data from the memory of the target hardware encoder to the memory of the target virtualization instance when the target hardware encoder is the hardware encoder on the local hardware offload device.

[0102] Further optionally, the second processor on the local hardware offloading device is also configured to forward the encoding request to the second processor on the remote hardware offloading device when the target hardware encoder is a hardware encoder on the remote hardware offloading device, and send the first memory address in the memory of the target virtualization instance for caching the code stream data to the second processor on the remote hardware offloading device; and receive the second memory address in the memory of the target hardware encoder for caching the video frame to be encoded, which is sent by the second processor on the remote hardware offloading device.

[0103] For a detailed description and beneficial effects of the functional modules and hardware modules in the computing node provided in this embodiment, please refer to the aforementioned system embodiment and will not be elaborated here.

[0104] FIG6 is a flow chart of a video encoding method provided by an embodiment of the present disclosure. The method is described from the perspective of a resource control node. As shown in FIG6 , the method includes:

[0105] 601. Allocate a target hardware encoder for a target virtualized instance from an encoding resource pool, where the encoding resource pool includes multiple hardware encoders implemented on multiple hardware offload devices, and the multiple hardware offload devices are distributed on multiple computing nodes.

[0106] 602. Control the target computing node hosting the target virtualization instance to mount the target hardware encoder, so that the target virtualization instance virtualizes the target hardware encoder into a recognizable virtual encoder and uses the video encoding service provided by the target hardware encoder in the form of the virtual encoder.

[0107] The target hardware encoder is a hardware encoder on a local hardware offloading device distributed on the target computing node, or a hardware encoder on a remote hardware offloading device distributed on other computing nodes.

[0108] FIG7 is a flow chart illustrating another video encoding method provided by an embodiment of the present disclosure. This method is described from the perspective of a computing node, which hosts a target virtualization instance and includes a local hardware offload device, on which a hardware encoder is implemented. As shown in FIG7 , the method includes:

[0109] 701. Mount a target hardware encoder allocated to a target virtualized instance from an encoding resource pool. The encoding resource pool includes hardware encoders on a local hardware offload device and hardware encoders on remote hardware offload devices distributed on other computing nodes. The target hardware encoder is a hardware encoder on a local hardware offload device or a hardware encoder on a remote hardware offload device.

[0110] 702. Present a virtual encoder to a target virtualized instance;

[0111] 703. Control the target hardware encoder to provide video encoding services for the target virtualized instance in the form of a virtual encoder.

[0112] In an optional embodiment, the target hardware encoder is controlled to provide video encoding services for the target virtualization instance in the form of a virtual encoder, including: providing the encoding request and the video frame to be encoded of the target virtualization instance to the local hardware offload device through the virtual encoder, so that the processor of the local hardware offload device controls the target hardware encoder to encode the video frame to be encoded according to the encoding request to obtain the code stream data required by the target virtualization instance.

[0113] In an optional embodiment, the processor of the local hardware offload device controls the target hardware encoder to encode the video frame to be encoded according to the encoding request to obtain the code stream data required by the target virtualization instance, including: according to the encoding request, controlling the remote direct data access (RDMA) module on the local hardware offload device to DMA the video frame to be encoded from the memory of the target virtualization instance to the memory of the target hardware encoder, so that the target hardware encoder can encode the video frame to be encoded to obtain the code stream data.

[0114] In an optional embodiment, the method further includes: when the target hardware encoder is a hardware encoder on a local hardware offload device, controlling the RDMA module on the local hardware offload device to DMA code stream data from the memory of the target hardware encoder to the memory of the target virtualization instance; or, when the target hardware encoder is a hardware encoder on a remote hardware offload device, forwarding the encoding request to the processor on the remote hardware offload device, and sending a first memory address in the memory of the target virtualization instance for caching code stream data to the processor on the remote hardware offload device; and receiving a second memory address in the memory of the target hardware encoder set as a cache of the video frame to be encoded sent by the processor on the remote hardware offload device.

[0115] The detailed implementation and beneficial effects of each step in the video encoding method provided by the above embodiments of the present disclosure have been described in detail in the aforementioned embodiments and will not be elaborated on here.

[0116] In addition to being applicable to video encoding scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to other application scenarios requiring hardware acceleration, such as hardware acceleration scenarios for virtual network cards, hardware acceleration scenarios for virtual switches, hardware acceleration for serial communications, hardware acceleration scenarios for message encapsulation, hardware acceleration scenarios for Network Address Translation (NAT), etc. Based on this, the embodiments of the present disclosure also provide a universal hardware acceleration system 800, as shown in FIG8 , the system 800 includes: a resource management node 801 and a resource pool 802, the resource pool 802 including a plurality of hardware offload devices 90 distributed on a plurality of computing nodes 80, and a plurality of hardware accelerators 91 implemented or integrated on the plurality of hardware offload devices 90.

[0117] The resource management and control node 801 is configured to allocate a target hardware accelerator to the target virtualization instance from the resource pool 802 and control the target computing node hosting the target virtualization instance to mount the target hardware accelerator; the target hardware accelerator is a hardware accelerator on a local hardware offload device distributed on the target computing node, or a hardware accelerator on a remote hardware offload device distributed on other computing nodes.

[0118] The target computing node is configured to present a virtual accelerator to the target virtualized instance and control the target hardware accelerator to provide a target service for the target virtualized instance in the form of the virtual accelerator.

[0119] The system 800 provided in this embodiment differs from the system 100 shown in FIG. 1 in that the resource pool and the hardware accelerator contained therein have broader implications and are no longer limited to encoding resource pools and encoding accelerators in video encoding scenarios. The hardware accelerators in system 800 can provide different target services based on the application scenario, such as, but not limited to, network card services provided by a virtual network card, data forwarding services provided by a virtual switch, packet encapsulation services, and network address translation services. All other aspects and beneficial effects of system 800 are the same or similar to those of system 100 shown in FIG. For details, please refer to the description in the preceding embodiment and will not be repeated here.

[0120] Corresponding to system 800 shown in FIG8 , the disclosed embodiment further provides a computing node and a hardware offloading device, which are substantially identical in structure and function to the computing node and the hardware offloading device in the aforementioned embodiment, with the difference that the resource pool and the hardware accelerator contained therein have a broader meaning and are no longer limited to the encoding resource pool and encoding accelerator in the video encoding scenario. Regarding the other contents and beneficial effects of the computing node and the hardware offloading device corresponding to system 800 provided in this embodiment, which are identical or similar to those of the computing node and the hardware offloading device provided in the aforementioned embodiment, please refer to the description in the aforementioned embodiment and will not be repeated here.

[0121] An embodiment of the present disclosure also provides a hardware acceleration method, comprising: allocating a target hardware accelerator to a target virtualization instance from a resource pool, the resource pool including multiple hardware accelerators implemented on multiple hardware offload devices, the multiple hardware offload devices being distributed on multiple computing nodes; controlling a target computing node hosting the target virtualization instance to mount the target hardware accelerator, so that the target virtualization instance virtualizes the target hardware accelerator into a recognizable virtual accelerator and uses a target service provided by the target hardware accelerator in the form of the virtual accelerator; wherein the target hardware accelerator is a hardware accelerator on a local hardware offload device distributed on the target computing node, or a hardware accelerator on a remote hardware offload device distributed on other computing nodes.

[0122] An embodiment of the present disclosure also provides a hardware acceleration method, which is applied to a computing node, wherein the computing node carries a target virtualization instance and includes a local hardware offload device, and a hardware accelerator is implemented on the local hardware offload device; the method includes: mounting a target hardware accelerator allocated to the target virtualization instance from a resource pool, wherein the resource pool includes hardware accelerators on the local hardware offload device and hardware accelerators on remote hardware offload devices distributed on other computing nodes, and the target hardware accelerator is a hardware accelerator on the local hardware offload device or a hardware accelerator on the remote hardware offload device; presenting a virtual accelerator to the target virtualization instance, and controlling the target hardware accelerator to provide a target service for the target virtualization instance in the form of the virtual accelerator.

[0123] The hardware acceleration method provided in the embodiments of the present disclosure is basically the same as the video encoding method in the aforementioned embodiments, except that the resource pool involved and the hardware accelerator contained in the resource pool have a broader meaning and are no longer limited to the encoding resource pool and encoding accelerator in the video encoding scenario. Therefore, the detailed implementation methods and beneficial effects of each step in the hardware acceleration method have been described in detail in the aforementioned embodiments and will not be elaborated here.

[0124] Accordingly, an embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor is enabled to implement each step in the above method embodiment.

[0125] Exemplary embodiments of the present disclosure further provide a computer program product, which includes a computer program and / or instructions. When the computer program and / or instructions are executed by a processor, the processor is enabled to perform the steps in the above method embodiments.

[0126] The memory in the above embodiments can be implemented by any type of volatile or non-volatile memory device or a combination thereof, such as SRAM, Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0127] Optionally, the computing node in the above embodiment further includes a communication component, which is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as wireless fidelity (WIreless Fidelity), WiFi, 2G, 3G, 4G / LTE (Long Term Evolution), 5G and other mobile communication networks, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0128] Optionally, the computing node in the above embodiment may further include a display, the display including a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0129] The computing nodes in the above embodiments also include a power supply component configured to provide power to various components of the device in which the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0130] Optionally, the computing node in the above embodiment may further include an audio component that can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC), and when the device where the audio component is located is in an operating mode, such as call mode, recording mode, and voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal can be further stored in a memory or sent via a communication component. In some embodiments, the audio component also includes a speaker that is configured to output audio signals.

[0131] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to magnetic disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program code.

[0132] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device configured to implement the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0133] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0134] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps configured to implement the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0135] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interfaces, network interfaces, and memory.

[0136] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0137] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), SRAM, DRAM, other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be configured to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0138] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a list of elements does not include those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0139] The above are merely examples of the present disclosure and are not intended to limit the present disclosure. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure are intended to be included within the scope of the claims of the present disclosure. Industrial Applicability

[0140] The disclosed embodiment proposes a video encoding method, which can form an encoding resource pool with hardware encoders implemented on different hardware offloading devices, and uniformly schedule the hardware encoders in the encoding resource pool through a resource management node. In combination with the computing node being able to present a virtual encoder to a virtualized instance, the front-end and back-end decoupling of encoding resources is realized, so that the computing node can mount the hardware encoder on the local hardware offloading device nearby without the virtualized instance being aware of it, or mount the hardware encoder on the remote hardware offloading device, thereby realizing the reuse of the hardware encoder, improving the utilization rate of hardware encoding resources, and solving the problem of waste of encoding resources. Furthermore, if the virtualized instance on a single computing node needs to expand its encoding capacity, by reusing the hardware encoder, the hardware encoder that meets its encoding needs can be flexibly allocated to the virtualized instance without the need for hot migration of the virtualized instance, which can well solve the encoder expansion problem. Furthermore, by reusing the hardware encoder, multiple hardware encoders can be allocated to the same virtualized instance to achieve the aggregation of encoding capacity.

Claims

1. A video encoding system, comprising: A resource management and control node and an encoding resource pool, wherein the encoding resource pool includes a plurality of hardware offloading devices distributed on a plurality of computing nodes, and a plurality of hardware encoders are implemented on the plurality of hardware offloading devices; The resource management and control node is configured to allocate a target hardware encoder to a target virtualization instance from the encoding resource pool, and control a target computing node carrying the target virtualization instance to mount the target hardware encoder; The target hardware encoder is a hardware encoder on a local hardware offloading device distributed on the target computing node, or a hardware encoder on a remote hardware offloading device distributed on other computing nodes; The target computing node is configured to present a virtual encoder to the target virtualized instance, and control the target hardware encoder to provide video encoding services for the target virtualized instance in the form of the virtual encoder.

2. The system according to claim 1, wherein: The resource management and control node is specifically configured to: respond to the user's video encoding requirements and determine the encoding resource requirement information of the target virtualization instance; and select a target hardware encoder that meets the encoding resource requirement information based on the basic capability information and usage status information of the multiple hardware encoders.

3. The system according to claim 1, wherein: The plurality of hardware offloading devices further include: a processor; The target computing node is specifically configured to: provide the encoding request of the target virtualized instance and the to-be-encoded video frame to the local hardware offloading device through the virtual encoder, so as to control the target hardware encoder to provide video encoding service for the target virtualized instance through the local hardware offloading device; The processor on the local hardware offloading device is configured to control the target hardware encoder to encode the to-be-encoded video frame according to the encoding request, so as to obtain the code stream data required by the target virtualized instance.

4. The system according to claim 3, wherein: The plurality of hardware offloading devices further include: a remote direct data access RDMA module; The processor on the local hardware offloading device is specifically configured to: according to the encoding request, control the RDMA module on the local hardware offloading device to DMA the video frame to be encoded from the memory of the target virtualization instance to the memory of the target hardware encoder, so that the target hardware encoder encodes the video frame to be encoded to obtain the code stream data.

5. The system according to claim 4, wherein: The processor on the hardware offloading device where the target hardware encoder is located is configured to control the RDMA module on the hardware offloading device where the target hardware encoder is located to DMA the code stream data from the memory of the target hardware encoder to the memory of the target virtualized instance.

6. The system according to claim 5, wherein: The processor on the local hardware offloading device is further configured to forward the encoding request to the processor on the remote hardware offloading device when the target hardware encoder is the hardware encoder on the remote hardware offloading device, and send a first memory address in the memory of the target virtualized instance for caching the bitstream data to the processor on the remote hardware offloading device; and receive a second memory address in the memory of the target hardware encoder for caching the video frame to be encoded, which is sent by the processor on the remote hardware offloading device; The processor on the remote hardware offloading device is further configured to, upon receiving the encoding request, The target hardware encoder's memory applies for a second memory address set to cache the to-be-encoded video frame and sends the address to the processor on the local hardware offloading device.

7. A computing node, wherein: The computing node carries a target virtualization instance, and the computing node includes a first processor and a local hardware offloading device, and a hardware encoder is implemented on the local hardware offloading device; The first processor is configured to mount a target hardware encoder allocated to the target virtualized instance from an encoding resource pool, present a virtual encoder to the target virtualized instance, and control the target hardware encoder to provide video encoding services for the target virtualized instance in the form of the virtual encoder; Among them, the encoding resource pool includes hardware encoders on the local hardware offloading device and hardware encoders on remote hardware offloading devices distributed on other computing nodes; the target hardware encoder is the hardware encoder on the local hardware offloading device, or the hardware encoder on the remote hardware offloading device.

8. The computing node according to claim 7, wherein: The local hardware offloading device also includes: a second processor; The first processor is specifically configured to: provide the encoding request and the to-be-encoded video frame of the target virtualized instance to the local hardware offloading device through the virtual encoder, so as to control the target hardware encoder to provide video encoding service for the target virtualized instance through the local hardware offloading device; The second processor on the local hardware offloading device is configured to control the target hardware encoder to encode the to-be-encoded video frame according to the encoding request, so as to obtain the bitstream data required by the target virtualized instance.

9. The computing node according to claim 8, wherein: The local hardware offloading device also includes: a remote direct data access RDMA module; The second processor on the local hardware offloading device is specifically configured to: according to the encoding request, control the RDMA module on the local hardware offloading device to DMA the video frame to be encoded from the memory of the target virtualization instance to the memory of the target hardware encoder, so that the target hardware encoder can encode the video frame to be encoded to obtain the code stream data.

10. A hardware offloading device, wherein: A hardware encoder is implemented on the hardware offloading device, and the hardware encoder belongs to a hardware encoder in an encoding resource pool, and the encoding resource pool includes multiple hardware encoders implemented on multiple hardware offloading devices; The hardware encoder is configured to provide video encoding services for the target virtualized instance in the form of a virtual encoder when the hardware encoder is assigned to the target virtualized instance and mounted on the target computing node hosting the target virtualized instance, wherein the virtual encoder is a virtualization of the hardware encoder; The target computing node is a local computing node where the hardware offloading device is located, or a remote computing node interconnected with the hardware offloading device through a network.

11. The hardware offloading device according to claim 10, further comprising: The processor is configured to obtain an encoding request of the target virtualization instance and a video frame to be encoded, and control the hardware encoder to encode the video frame to be encoded to obtain code stream data required by the target virtualization instance.

12. The hardware offloading device according to claim 11, wherein: Also includes: Remote direct data access RDMA module; The processor is specifically configured to: control the to-be-encoded video frame to be read from the memory of the target virtualized instance; DMA to the memory of the hardware encoder so that the hardware encoder encodes the video frame to be encoded to obtain the code stream data, and controls the RDMA module of the hardware offload device to DMA the code stream data from the memory of the hardware encoder to the memory of the target virtualization instance.

13. A video encoding method, wherein: Applied to a resource management and control node, the method includes: Allocating a target hardware encoder for a target virtualized instance from an encoding resource pool, wherein the encoding resource pool includes a plurality of hardware encoders implemented on a plurality of hardware offloading devices, and the plurality of hardware offloading devices are distributed on a plurality of computing nodes; Controlling the target computing node carrying the target virtualization instance to mount the target hardware encoder, so that the target virtualization instance virtualizes the target hardware encoder into a recognizable virtual encoder, and uses the video encoding service provided by the target hardware encoder in the form of the virtual encoder; The target hardware encoder is a hardware encoder on a local hardware offloading device distributed on the target computing node, or a hardware encoder on a remote hardware offloading device distributed on other computing nodes.

14. A video encoding method, wherein: Applied to a computing node, the computing node carries a target virtualization instance, and the computing node includes a local hardware offloading device, and a hardware encoder is implemented on the local hardware offloading device; the method includes: Mounting a target hardware encoder allocated to the target virtualized instance from an encoding resource pool, wherein the encoding resource pool includes hardware encoders on a local hardware offloading device and hardware encoders on remote hardware offloading devices distributed on other computing nodes, and the target hardware encoder is a hardware encoder on the local hardware offloading device or a hardware encoder on a remote hardware offloading device; A virtual encoder is presented to the target virtualized instance, and the target hardware encoder is controlled to provide video encoding services for the target virtualized instance in the form of the virtual encoder.

15. The method according to claim 14, wherein: Controlling the target hardware encoder to provide video encoding services for the target virtualized instance in the form of the virtual encoder includes: The encoding request of the target virtualization instance and the video frame to be encoded are provided to the local hardware offloading device through the virtual encoder, so that the processor of the local hardware offloading device controls the target hardware encoder to encode the video frame to be encoded according to the encoding request to obtain the code stream data required by the target virtualization instance.

16. A hardware acceleration system, comprising: A resource management and control node and a resource pool, wherein the resource pool includes a plurality of hardware offloading devices distributed on a plurality of computing nodes, and a plurality of hardware accelerators are implemented on the plurality of hardware offloading devices; The resource management and control node is configured to allocate a target hardware accelerator to a target virtualization instance from the resource pool, and control a target computing node that hosts the target virtualization instance to mount the target hardware accelerator; the target hardware accelerator is a hardware accelerator on a local hardware offloading device distributed on the target computing node, or a hardware accelerator on a remote hardware offloading device distributed on other computing nodes; The target computing node is configured to present a virtual accelerator to the target virtualized instance, and to control the target hardware accelerator to provide a target service for the target virtualized instance in the form of the virtual accelerator.

17. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor is enabled to implement the steps of the method according to any one of claims 13 to 15.

Images