A method for developing a video encoding platform

Abstract

This invention discloses an open video coding platform and method. The open video coding platform includes: a SW control layer, which is the overall platform control layer, including instruction generation, data flow control, and video encoding / decoding packet header processing; an HW / SW main control unit layer, including an HW main control unit and an SW main control unit, used to control the underlying software and hardware IPs; underlying module information, providing the IDs, working statuses, and independent information of all underlying IPs; a general data interface layer, which is a general data interface for both software and hardware; and a module layer, including: a basic coding unit, used to provide basic coding functions; a central accelerator, including the main computational hardware acceleration units abstracted from the encoding / decoding algorithms, which can be called by upper-layer software through an instruction set to accelerate software computation; an encoding processing unit, which provides additional encoding information to the basic coding unit to improve encoding performance; and a video encoding preprocessing unit, including enhancement, noise reduction, and SR video processing IPs.

Description

Technical Field

[0001] This invention relates to the field of video coding technology, and more specifically, to an open video coding platform and method. Background Technology

[0002] With technological advancements, the application of video continues to expand, from long-form to short-form videos, from video conferencing to video e-commerce, and from processing existing video content to real-time video processing. This widespread use of video has also driven the continuous development of video processing technologies. Simultaneously, issues such as how to efficiently test and verify algorithms, how to ensure the correct integration of hardware and software, how to rapidly iterate hardware designs, and how to quickly integrate with mature systems have become urgent problems to be solved. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides an open video encoding platform and method.

[0004] According to one aspect of the present invention, an open video encoding platform is provided, comprising: a Switch control layer, a Wi-Fi / Switch main control unit layer, underlying module information, a general data interface layer, and a module layer.

[0005] The SW control layer is the overall platform control layer, including instruction generation, data flow control, and video encoding / decoding packet header processing;

[0006] The HW / SW main control unit layer includes the HW main control unit and the SW main control unit, which are used to control the underlying software and hardware IP.

[0007] The underlying module information provides the ID, working status, and individual information of all underlying IPs;

[0008] The general data interface layer is a general data interface for both hardware and software, supports a high-efficiency arbiter, supports dual channels for data and commands, and supports out-of-order transmission of data and commands.

[0009] The module layer includes: basic coding units, which provide basic coding functions;

[0010] The central accelerator includes the main computing hardware acceleration units abstracted from the encoding and decoding algorithms, which can be called by upper-layer software through instruction sets to accelerate software computing functions;

[0011] The encoding processing unit provides additional encoding information to the basic encoding unit to improve encoding performance;

[0012] The video encoding preprocessing unit includes enhancement, noise reduction, and SR video processing IP.

[0013] Optionally, the HW / SW main control unit layer completes the control of the underlying software and hardware IP, including the control of general data interfaces, top-level command parsing, data preparation and transfer;

[0014] The SW main control unit provides the SW software with a control path to the underlying HW IP.

[0015] Optionally, the basic coding unit provides one or more of the following information channels: QP, QP delta, intra-block partitioning, intra-prediction module, inter-block partitioning, inter-prediction mode, reference frame selection, MV, search range selection, and ROI region.

[0016] Optionally, the central processing unit includes: a SAD computing array, an SSE computing array, a SATD computing array, a configurable two-dimensional filter computing array, a MAC computing array, a Huffman coding unit, and a CABAC computing unit.

[0017] According to another aspect of the present invention, an open video coding method is provided, comprising:

[0018] The SW control layer sends video encoding-related instructions to the HW / SW main control unit based on the customer's video encoding requirements for video data.

[0019] The software and hardware in the HW / SW main control unit control module layer begin video encoding of the video data;

[0020] The processed video stream data is transmitted to external devices via a universal data interface.

[0021] Optionally, the software and hardware in the HW / SW main control unit control module layer begin the video encoding operation of the video data, including:

[0022] The HW / SW main control unit controls the transmission of video data to the module layer through the hardware interface portion of the general data interface;

[0023] The module layer processes the video data according to the processing unit corresponding to the video encoding.

[0024] Optionally, the operation of transmitting the processed video stream data to an external device via a general data interface includes:

[0025] The video data processed by the HW / SW main control unit is transmitted to the basic encoder for encoding processing through a general data interface to obtain video stream data, which is then transmitted to an external device, such as a memory or display device, through the general data interface.

[0026] Therefore, this invention provides an open video encoding platform that offers a unified platform for verification, testing, and application development, enabling customers to quickly access mature systems. Attached Figure Description

[0027] Exemplary embodiments of the present invention can be more fully understood by referring to the following figures:

[0028] Figure 1 This is a schematic diagram of an open video coding platform provided in the first aspect of an embodiment of the present invention;

[0029] Figure 2 This is a flowchart illustrating the open video coding method provided in the second aspect of the present invention;

[0030] Figure 3 This is a schematic diagram of a basic encoder provided in the second aspect of an embodiment of the present invention;

[0031] Figure 4 This is a schematic diagram of block partitioning based on AI algorithm provided in the second aspect of the present invention;

[0032] Figure 5 This is a schematic diagram of the dynamic mv search range based on the basic encoder provided in the second aspect of the present invention. Detailed Implementation

[0033] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0034] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0035] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of the present invention are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0036] It should also be understood that in the embodiments of the present invention, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0037] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more unless explicitly defined or given contrary instructions in the context.

[0038] Furthermore, the term "and / or" in this invention is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this invention generally indicates that the preceding and following related objects have an "or" relationship.

[0039] It should also be understood that the description of the various embodiments in this invention emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0040] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0041] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0042] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0043] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0044] The embodiments of this invention can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Well-known examples of terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0045] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0046] Exemplary methods

[0047] Figure 1 This is a schematic diagram of an open video coding platform provided in the first aspect of an embodiment of the present invention.

[0048] like Figure 1 As shown, open video coding platforms include:

[0049] SW control layer, HW / SW main control unit layer, underlying module information, general data interface layer, module layer.

[0050] The SW control layer is the overall platform control layer, including instruction generation, data flow control, and video encoding / decoding packet header processing;

[0051] The HW / SW main control unit layer includes the HW main control unit and the SW main control unit, which are used to control the underlying software and hardware IP.

[0052] The underlying module information provides the ID, working status, and individual information of all underlying IPs;

[0053] The general data interface layer is a general data interface for both hardware and software, supports a high-efficiency arbiter, supports dual channels for data and commands, and supports out-of-order transmission of data and commands.

[0054] The module layer includes: basic coding units, which provide basic coding functions;

[0055] The central accelerator includes the main computing hardware acceleration units abstracted from the encoding and decoding algorithms, which can be called by upper-layer software through instruction sets to accelerate software computing functions;

[0056] The encoding processing unit provides additional encoding information to the basic encoding unit to improve encoding performance;

[0057] The video encoding preprocessing unit includes enhancement, noise reduction, and SR video processing IP.

[0058] Optionally, the HW / SW main control unit layer completes the control of the underlying software and hardware IP, including the control of general data interfaces, top-level command parsing, data preparation and transfer;

[0059] The SW main control unit provides the SW software with a control path to the underlying HW IP.

[0060] Optionally, the basic coding unit provides one or more of the following information channels: QP, QP delta, intra-block partitioning, intra-prediction module, inter-block partitioning, inter-prediction mode, reference frame selection, MV, search range selection, and ROI region.

[0061] Specifically, the basic encoder is a high-performance, multi-standard encoder independently developed by our company, supporting multiple additional information channels. The encoder supports multiple encoding standards including AVC, HEVC, and AV1. It is also specifically optimized for desktop scenarios, video content scenarios, and gaming scenarios. Furthermore, based on the OpenVideo development platform, it provides the following additional information channels:

[0062] QP, QP delta, intra-block partitioning, intra-prediction mode, inter-block partitioning, inter-prediction mode, reference frame selection, MV, search range selection, ROI region.

[0063] Optionally, the central processing unit includes: a SAD computing array, an SSE computing array, a SATD computing array, a configurable two-dimensional filter computing array, a MAC computing array, a Huffman coding unit, and a CABAC computing unit.

[0064] Specifically, CAU (Centre Accelerator Unit) includes the main computational hardware acceleration unit abstracted from the encoding and decoding algorithm, which can be called by upper-layer software through instruction set to accelerate certain software operations.

[0065] SAD (Sum of absolute differences) computation array, SE (Sum of squared differences) computation array, SATD (Sum of absolute transformed differences) computation array, configurable two-dimensional filter computation array, MAC (multiply and accumulate operations) computation array, Huffman coding unit, CABAC computation unit.

[0066] Specifically, video codec IP, as the most crucial link in the video industry chain, is a vital link from video source to customer, and from ISP to display. Various pre-processing and post-processing algorithms can be seamlessly integrated around video codec IP. Therefore, a unified verification, testing, and application development platform becomes extremely important. As the entire chip design centered around video codec becomes increasingly complex and functional requirements grow, this platform is essential to address both internal testing, integration, and upgrade needs, and to reduce the need for customers to repeatedly build their own platforms.

[0067] The Open Video Encoding Platform System is a general-purpose hardware and software platform based on a unified data interface. It features versatility, modularity, replaceability, and upgradeability.

[0068] Open video coding platforms can leverage FPGAs to quickly build and validate different video IPs and codec module-level optimizations.

[0069] refer to Figure 1 As shown, the SW control layer is the overall platform control layer, which includes functions such as instruction generation, data flow control, and video encoding / decoding packet header processing.

[0070] HW / SW Master Control Unit Layer: Includes the HW master control unit, SW master control unit, and information sharing with the underlying module 321. The two master control units mainly complete the control of the underlying software and hardware IP, including the control of general data interfaces, top-level command parsing, data preparation and migration, etc. In addition, the SW master control unit also provides the SW with an additional control path to the underlying HW IP.

[0071] The underlying module information mainly provides the IDs, working statuses, and individual information of all underlying IPs.

[0072] General Data Interface Layer: A general data interface for both hardware and software, supporting efficient arbiters, dual channels for data and commands, and out-of-order transmission of data and commands.

[0073] The underlying module layer mainly includes four types of modules. The base encoder provides basic encoding functions. The CAU (centre accelerator unit) includes the main hardware acceleration units abstracted from the encoding / decoding algorithms, which can be called by upper-layer software via instruction sets to accelerate certain software operations. The encoder processor provides additional encoding information to the base encoder to improve encoding performance. The pre-processor is the video encoding pre-processing module, including video processing IPs such as enhancement, noise reduction, and SR. These underlying modules can be either hardware or software versions.

[0074] Therefore, this invention provides an open video encoding platform that offers a unified platform for verification, testing, and application development, enabling customers to quickly access mature systems.

[0075] Furthermore, according to a second aspect of this application, an open video coding method is provided, with reference to... Figure 2 As shown, the Open Video Coding Method 200 includes the following steps:

[0076] Step 201: The SW control layer sends video encoding-related instructions to the HW / SW main control unit based on the customer's video encoding requirements for video data.

[0077] Step 202: The software and hardware in the HW / SW main control unit control module layer begin video encoding of the video data;

[0078] Step 203: Transmit the processed video stream data to an external device via a general data interface.

[0079] Optionally, the software and hardware in the HW / SW main control unit control module layer begin the video encoding operation of the video data, including:

[0080] The HW / SW main control unit controls the transmission of video data to the module layer through the hardware interface portion of the general data interface;

[0081] The module layer processes the video data according to the processing unit corresponding to the video encoding.

[0082] Optionally, the operation of transmitting the processed video stream data to an external device via a general data interface includes:

[0083] The video data processed by the HW / SW main control unit is transmitted to the basic encoder for encoding processing through a general data interface to obtain video stream data, which is then transmitted to an external device, such as a memory or display device, through the general data interface.

[0084] Specifically, the SW top-level control activates the platform based on customer processing needs. It issues commands to the SW main control unit and the HW main control unit, respectively controlling the software and hardware modules in the underlying modules to begin operation. For example, processing a frame of encoding with pre-filtering involves controlling the video data (YUV) to pass through the hardware interface of the general data interface to the underlying pre-filter hardware processing unit MCTF. After processing by the MCTF, the video data (YUV) is then controlled by the main control unit to enter the basic encoder for encoding processing through the general data interface. The encoded video stream data (bit stream) is then transmitted to the DDR or display device via the general data interface.

[0085] In addition, refer to Figure 3 As shown, the encoderprocessor can provide various additional information to the base encoder. Different customers can choose to use either the modules provided by our company or their own modules to complete the corresponding functions, according to their own capabilities and needs.

[0086] The OVCP platform provides multiple additional information channels based on the basic encoder, allowing customers to select several for replacement and testing according to different application scenarios.

[0087] Furthermore, examples of block partitioning based on AI algorithms include... Figure 4 As shown, an example of the dynamic MV search range based on the base encoder is as follows: Figure 5 As shown.

[0088] The basic principles of the present invention have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in the present invention are merely examples and not limitations, and should not be considered as essential features of each embodiment of the present invention. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the present invention to the necessity of employing the aforementioned specific details.

[0089] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0090] The block diagrams of devices, systems, devices, and systems involved in this invention are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, systems, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0091] The methods and systems of the present invention may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of the present invention are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present invention may also be implemented as a program recorded on a recording medium, the program comprising machine-readable instructions for implementing the methods according to the present invention. Thus, the present invention also covers recording media storing programs for performing the methods according to the present invention.

[0092] It should also be noted that in the systems, apparatus, and methods of the present invention, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered equivalents of the present invention. The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the invention. Therefore, the invention is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0093] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the invention to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A video encoding development platform, characterized in that, include: The system comprises the SW control layer, HW / SW main control unit layer, general data interface layer, module layer, central accelerator, encoding processing unit, and video encoding preprocessing unit. The SW control layer is the overall platform control layer, including instruction generation, data flow control, and video encoding / decoding packet header processing; The HW / SW master control unit layer includes an HW master control unit and an SW master control unit, which are used to control the underlying software and hardware IP. The control of the underlying software and hardware IP by the HW / SW master control unit layer includes control of the general data interface, top-level command parsing, data preparation and migration. The SW master control unit provides the control path for the SW software to the underlying HW IP. The general data interface layer is a general-purpose hardware and software data interface that supports a high-efficiency arbiter, supports dual channels for data and commands, and supports out-of-order transmission of data and commands. The module layer includes: a basic coding unit, which provides basic coding functions; the basic coding unit provides one or more of the following information channels: QP, QP delta, intra-frame block partitioning, intra-frame prediction module, inter-frame block partitioning, inter-frame prediction mode, reference frame selection, MV, search range selection, and ROI region. The central accelerator includes the main computing hardware acceleration units abstracted from the encoding and decoding algorithms, which can be called by upper-layer software through instruction sets to accelerate software computing functions; The encoding processing unit provides additional encoding information to the basic encoding unit to improve encoding performance; The video encoding preprocessing unit includes video processing modules for enhancement, noise reduction, and super-resolution techniques; The central accelerator includes: a SAD computing array, an SSE computing array, a SATD computing array, a configurable two-dimensional filter computing array, a MAC computing array, a Huffman coding unit, and a CABAC computing unit.

2. The video encoding method implemented by the video encoding platform according to claim 1, characterized in that, include: The SW control layer sends video encoding-related instructions to the HW / SW main control unit based on the customer's video encoding requirements for video data. The software and hardware in the HW / SW main control unit control module layer begin the video encoding process for the video data; The processed video stream data is transmitted to an external device via a general data interface; The software and hardware in the HW / SW main control unit control module layer begin the operation of video encoding of the video data, including: The HW / SW main control unit controls the transmission of the video data to the module layer through the hardware interface portion of the general data interface; The module layer processes the video data according to the processing unit corresponding to the video encoding; The operation of transmitting processed video stream data to external devices via a common data interface includes: The video data processed by the HW / SW main control unit is transmitted to the basic encoder for encoding processing through the general data interface to obtain the video stream data, which is then transmitted to the external device through the general data interface, wherein the external device is a memory or a display device.

Images