High-efficient and low-delay data buffering and processing method for hevc encoder

Abstract

The application provides a high-efficiency low-delay HEVC encoder data buffering and processing method, aiming at reducing the encoding delay of an H.265 / HEVC hard encoder as much as possible, reducing the increasing storage space and data bandwidth pressure caused by increasingly high video quality, and providing a high-efficiency encoding data access scheme. The adaptive input video resolution is realized, so as to control the data buffering and the edge padding width of the reconstructed image. The sliding window reading of the search frame reconstructed pixels is realized, so as to reduce the DDR4 bandwidth demand and data reading delay. The burst length of each reading and writing of the DDR4 is reasonably controlled, the data is stored in units of CTU blocks, and the DDR4 line change or bank switching is reduced, so as to further improve the efficiency of the storage system.

Description

Technical Field

[0001] This invention belongs to the field of video encoding and decoding technology, and particularly relates to a high-efficiency, low-latency HEVC encoder data caching and processing method, especially a high-efficiency storage and retrieval scheme for HEVC video encoding data for heterogeneous FPGAs. Background Technology

[0002] The high-efficiency video coding standard H.265 / HEVC improves coding efficiency by approximately 50% compared to its predecessor, H.264, while maintaining the same video compression quality. However, H.265 also increases complexity compared to H.264. [1] For H.265 hardware encoders to compress and encode ultra-high-definition video, a crucial issue needs to be addressed: how to handle the exponentially increased data throughput while minimizing access latency and reducing storage bandwidth pressure. This is because the H.265 standard allows for dynamic CTU partitioning within the range of 8×8 to 64×64 and supports 33 intra-frame prediction modes. [3] This further increases the demand for memory space access.

[0003] To minimize encoding latency and storage bandwidth, existing technologies have proposed various solutions, such as: (1) combining novel non-volatile memories, such as ReRAM and STT-RAM, with DRAM to form a layered memory structure, which effectively saves overall memory area and reduces power consumption for encoding data access. However, in this solution, DRAM and non-volatile memories require different memory controllers to manage memory access, which undoubtedly increases hardware costs. [4] This scheme divides the data required for deblocking filtering and luminance / chrominance prediction in the encoding process into independent data access events, which increases the number of DDR accesses and cannot further improve the efficiency of memory access. (2) The video data to be encoded is received by the FPGA-extended PCIe interface, then buffered in DDR, and then the FPGA preprocessed data is sent to the DSP chip using the SRIO (Serial RapidIO) interface, and HEVC compression encoding is completed on the DSP chip. [5] The hardware cost of this scheme is high. Using multiple interfaces to connect different hardware circuits increases the instability of the system and also increases the video encoding delay. The transmission bandwidth utilization is not high. (3) Design a three-level physical storage structure of DDR-BRAM-LUT, using DDR for frame buffering, BRAM for line buffering, and LUT for reference pixel buffering. [6] This alleviates the bandwidth pressure on off-chip memory to some extent, but it does not take into account the relationship between the CTU's storage arrangement in memory and DDR access efficiency, resulting in low data access efficiency.

[0004] References

[0005] [1] J.Vanne, M.Viitanen, T.D.Hamalainen, A.Hallapuro, “Comparative Rate-Distortion-Complexity Analysis of HEVC and AVC Video Codecs”, IEEE Trans.on CAS for Video Technology, vol.22, no.12, pp.1885-1898, Dec.2012.

[0006] [2] Bouaafia, S., Khemiri, R., Messaoud, S.et al. Deep CNN Co-design for HEVC CU Partition Prediction on FPGA-SoC. Neural Process Lett(2022). https: / / doi.org / 10.1007 / s11063-022-10765-1

[0007] [3] Bouaafia, S., Khemiri, R., Messaoud, S.et al. Deep CNN Co-design for HEVC CU Partition Prediction on FPGA-SoC. Neural Process Lett(2022).

[0008] https: / / doi.org / 10.1007 / s11063-022-10765-1

[0009] [4] D.S.Silveira, A.Mativi, M.S.Porto and S.Bampi, "Energy Savings with Non-Volatile Memory System for High Definition Video Encoders," 201917th IEEE International New Circuits and Systems Conference(NEWCAS), 2019, pp.1-4, doi:10.1109 / NEWCAS44328.2019.8961238.

[0010] [5] Chen Feiyang. Design of video processing system based on H.265 encoding [D]. Chengdu University of Technology, 2017.

[0011] [6] Li Shen, Chai Zhilei, Yan Wei, Xia Jun, Zhao Jianbin. Research and implementation of parallel computing method for H.265 intra-frame mode decision [J]. Small and Microcomputer Systems, 2018, 39(11):2523-2527. Summary of the Invention

[0012] To address the gaps and shortcomings in existing technologies, and to minimize encoding latency in H.265 / HEVC hardware encoders while reducing the ever-increasing storage space and data bandwidth pressure caused by increasingly higher video quality, this invention proposes a high-efficiency, low-latency HEVC encoder data caching and processing method. This method represents a highly efficient encoded data access scheme, achieving adaptive input video resolution to control data caching and reconstructed image edge padding width. It also implements sliding window reading of reconstructed pixels from the search box, reducing DDR4 bandwidth requirements and data read latency. By rationally controlling the burst length of each DDR4 read / write operation, data is stored in 64×64 CTU blocks, reducing DDR4 line breaks or bank switching, thereby further improving the efficiency of the storage system.

[0013] The present invention specifically adopts the following technical solution:

[0014] A high-efficiency, low-latency HEVC encoder data caching and processing method, characterized by the following steps:

[0015] Step S1: Receive video data, convert the RGB888 data format to YUV444, and then downsample it to YUV420 format;

[0016] Step S2: Calculate the resolution of the current input video and output the result for use in subsequent steps;

[0017] Step S3: Store the YUV420 format video data into the address space given by DDR4 in 64×64 CTB blocks, respectively, for the luminance Y component, chrominance U and V components;

[0018] Step S4: When the input video data reaches 64 lines of valid pixels, a start signal is sent to the H.265_Encoder / HEVC encoder kernel to read the 64×64 raw data into the H.265_Encoder in a pipeline manner;

[0019] Step S5: The reconstructed data output by the encoder is stored in the address space given by DDR4 in 64×64 CTU blocks via DMA transfer;

[0020] Step S6: During inter-frame coding, reconstructed image data is read in units of CTU, and edge padding is performed on the image according to the current resolution. Then, the search box pixels are read by sliding.

[0021] Step S7: Store the encoded bitstream in DDR4 in real time. An interrupt is generated every time the stored data is 1KB, which notifies the CPU to read the bitstream data, pack it into RTP format, and send it to the destination IP via UDP.

[0022] Furthermore, step S1 specifically includes the following steps:

[0023] Step S11: After receiving video data, output it in native video signal format;

[0024] Step S12: After converting RGB888 data to YUV444 using the RGB_to_YUV420 color space conversion module, it is then downsampled sequentially to YUV422 and YUV420. The color conversion formula is:

[0025]

[0026] Where Y represents the luminance component; U and V represent the chrominance components; R represents the red channel; G represents the green channel; and B represents the blue channel. Color conversion operations are implemented in the FPGA, converting floating-point operations into fixed-point operations.

[0027] Furthermore, in step S2, resolution calculation is initiated based on the horizontal and vertical synchronization signals of the received video, and the effective pixel width and height of the video are statistically analyzed to obtain the resolution value; when two consecutive frames of video signal are lost, the previous statistical result is maintained to avoid misjudgment of video signal loss caused by input interface jitter.

[0028] Furthermore, step S3 specifically includes the following steps:

[0029] Step S31: Distribute and store the luminance component Y in units of 64×64;

[0030] Step S32: The two 32×32 chromaticity components U and V are distributed and stored as a single data unit;

[0031] Step S33: Using address bit control, write data from the AXI_HP port to DDR4 in DMA mode.

[0032] Furthermore, in step S4, two 64×64 ping-pong RAMs are instantiated on the FPGA to store CU and PU data belonging to the same CTU read from DDR4, so as to reduce the bandwidth pressure of off-chip memory and reduce encoder encoding latency by using a small amount of logic resources.

[0033] Furthermore, step S6 specifically includes the following steps:

[0034] Step S61: Receive the coordinates of the current predicted block in the image frame;

[0035] Step S62: Initiate an AXI4 burst read signal and read the corresponding reconstructed image data from DDR4 according to the block coordinates;

[0036] Step S63: Store the read data in 80 rows of RAM;

[0037] Step S64: Fill the edges of the image;

[0038] Step S65: Output the filled 64 rows of search box data;

[0039] Step S66: Continue reading data down to fill the 16 empty rows.

[0040] Compared with existing technologies, this invention and its preferred solutions achieve adaptive input video resolution, thereby controlling data caching and the width of reconstructed image edge filling; realize sliding window reading of search box reconstructed pixels, reducing DDR4 bandwidth requirements and data reading latency; and by reasonably controlling the burst length of each DDR4 read / write operation, data is stored in units of 64×64 CTU blocks, reducing DDR4 line feeds or bank switching, thereby further improving the efficiency of the storage system. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of DDR4 memory space partitioning in an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of the storage arrangement of the Y component in an embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of the AXI_HP interface with the memory in an embodiment of the present invention.

[0044] Figure 4 This is a schematic diagram of image filling in an embodiment of the present invention.

[0045] Figure 5 This is a schematic diagram of the movement of search box data in an embodiment of the present invention.

[0046] Figure 6 This is a schematic diagram of the Y component sliding reading in an embodiment of the present invention.

[0047] Figure 7 This is a timing diagram of the encoder reading data blocks in an embodiment of the present invention.

[0048] Figure 8 This is a diagram illustrating the encoding and decoding effects in an embodiment of the present invention.

[0049] Figure 9 This is a schematic diagram of the workflow in an embodiment of the present invention. Detailed Implementation

[0050] To make the features and advantages of this patent more apparent and understandable, specific embodiments are provided below for detailed explanation:

[0051] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0052] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0053] like Figures 1-9 As shown, in the implementation of this embodiment, the encoding system used can utilize a heterogeneous FPGA platform containing a PL (Programmable Logic) and a PS (Processing System). The highly parallel computing of the PL hardware accelerates the encoding operation, while the PS controls the encoder encoding parameters. [2] This allows the encoder to both accelerate encoding operations and provide a control interface to the CPU on the PS side without sacrificing flexibility.

[0054] This leads to a hardware architecture that effectively improves the data access efficiency of the H.265 encoder. It reduces the number of DDR accesses, and the use of the same memory for multiple modules lowers hardware costs. Taking into account the characteristics of DDR4 hardware structure and H.265 encoding, data access is performed in 64×64 CTU data blocks, data storage locations in memory are rationally arranged, and an adaptive input resolution, sliding read-fill reconstruction scheme for the encoded image is proposed. This allows the proposed architecture to better meet the high-efficiency and low-latency requirements of the H.265 encoder.

[0055] The high-efficiency, low-latency HEVC encoder data caching and processing method designed specifically includes the following steps:

[0056] Step S1: Receive video data through the HDMI_RX module. The RGB_to_YUV420 module converts the RGB888 data format to YUV444 and then downsamples it to YUV420 format.

[0057] Step S2: The Resolution_Detection module calculates the resolution of the current input video and outputs the result to subsequent modules for use;

[0058] Step S3: The Original_Data_Write module uses DMA to transfer the YUV420 format video data into a specific address space of DDR4, storing the luminance Y component and chrominance UV component into 64×64 CTB blocks respectively.

[0059] Step S4: When the input video data reaches 64 lines of valid pixels, a start signal is sent to the H.265_Encoder kernel. The Original_Data_read module reads the 64×64 raw data in a pipeline manner and sends it to the H.265_Encoder.

[0060] Step S5: The Recovery_Data_Write module stores the reconstructed data output by the encoder into a 64×64 CTU block in a specific address space of DDR4 via DMA transfer;

[0061] Step S6: During inter-frame encoding, the Recovery_Data_Read module reads the reconstructed image data in units of CTU, performs edge padding on the image according to the current resolution, and then slides to read the search box pixels;

[0062] Step S7: The Bit_Stream_Write module stores the encoded bitstream into DDR4 in real time. An interrupt is generated every time the stored data is 1KB, which notifies the CPU to read the bitstream data and pack it into RTP format, and send it to the destination IP via UDP.

[0063] As a preferred embodiment, step S1 specifically includes the following steps:

[0064] Step S11: The HDMI_RX module receives video data and outputs it in native video signal format;

[0065] Step S12: After converting RGB888 data to YUV444 using the RGB_to_YUV420 color space conversion module, it is then downsampled sequentially to YUV422 and YUV420. The color conversion formula is:

[0066]

[0067] Where Y represents the luminance component; U and V represent the chrominance components; R represents the red channel; G represents the green channel; and B represents the blue channel. Color conversion operations are implemented in the FPGA, requiring the conversion of floating-point operations to fixed-point operations.

[0068] Step S2 specifically includes the following features:

[0069] (1) The Resolution_Detection module starts resolution calculation based on the line and field synchronization signals of the received video, and calculates the effective pixel width and height of the video to obtain the resolution value;

[0070] (2) When two consecutive frames of video signal are lost, the previous statistical results are kept output to avoid misjudgment of video signal loss caused by input interface jitter.

[0071] The data storage scheme in this embodiment has the following characteristics:

[0072] The 64-bit AXI4 data bus reads and writes DDR4 with a burst length of 32, which improves the efficiency of the AXI4 bus.

[0073] Each module of the system reads and writes DDR4 memory in a continuous memory access manner, which improves access efficiency;

[0074] Each module reads and writes data in units of a continuous 2KB memory page. Each time data is read, a page can be read at once, reducing memory access latency caused by memory page jumps.

[0075] As a preferred option, step S3 of the Original_Data_Write module specifically includes the following steps:

[0076] Step S31: Distribute and store the luminance component Y in units of 64×64;

[0077] Step S32: The two 32×32 chromaticity components U and V are distributed and stored as a single data unit;

[0078] Step S33: Using address bit control, write the data from the AXI_HP port to the DDR4 in DMA mode.

[0079] The Recovery_Data_Write module in step S4 has the following characteristics:

[0080] Two 64×64 ping-pong RAMs are instantiated on the FPGA to store CU (Coding Unit) and PU (Prediction Unit) data from the same CTU read from DDR4. This achieves the effect of reducing off-chip memory bandwidth pressure and reducing encoder encoding latency by using a small amount of logic resources.

[0081] In step S6, the Recovery_Data_Read module has the following steps:

[0082] Step S61: Receive the coordinates of the current predicted block in the image frame;

[0083] Step S62: Initiate an AXI4 burst read signal and read the corresponding reconstructed image data from DDR4 according to the block coordinates;

[0084] Step S63: Store the read data in 80 rows of RAM;

[0085] Step S64: Fill the edges of the image;

[0086] Step S65: Output the filled 64 rows of search box data;

[0087] Step S66: Continue reading data down to fill the 16 empty rows.

[0088] With the above design, a 64×64 data block only requires 64 clock cycles to be sent to the encoder. The timing waveform of the data read from the storage system to the encoder is shown below. Figure 7 As shown in the image. A test video was captured using a high-definition webcam and played on a laptop. The webcam output 1080P 30fps video to the encoder for encoding, and the stream was decoded and played back using self-developed software on the PC. The encoding / decoding results are shown in the image. Figure 8 As shown.

[0089] The above-mentioned algorithm-related program design scheme provided in this embodiment can be stored in a computer-readable storage medium in the form of code, and implemented in the form of a computer program. The basic parameter information required for calculation is input through computer hardware, and the calculation result is output.

[0090] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This invention is described with reference to flowchart illustrations of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each step in the flowchart, and combinations of steps in the flowchart, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, generate instructions for implementing the steps in the flowchart. Figure 1 A device for a function specified in one or more processes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A function specified in one or more flowcharts.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 Steps of a specified function in one or more processes.

[0094] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0095] This patent is not limited to the above-described preferred embodiments. Anyone can derive other forms of efficient and low-latency HEVC encoder data caching and processing methods based on the inspiration of this patent. All equivalent changes and modifications made within the scope of this patent application shall fall within the scope of this patent.

Claims

1. A high-efficiency, low-latency HEVC encoder data buffering and processing method, characterized in that, Includes the following steps: Step S1: Receive video data, convert the RGB888 data format to YUV444, and then downsample it to YUV420 format; Step S2: Calculate the resolution of the current input video and output the result for use in subsequent steps; Step S3: Store the YUV420 format video data into the address space given by DDR4 in 64×64 CTB blocks, respectively, for the luminance Y component, chrominance U and V components. Step S4: When the input video data reaches 64 lines of valid pixels, a start signal is sent to the H.265_Encoder / HEVC encoder kernel to read the 64×64 raw data into the H.265_Encoder in a pipeline manner; Step S5: The reconstructed data output by the encoder is stored in the address space given by DDR4 in 64×64 CTU blocks via DMA transfer; Step S6: During inter-frame coding, reconstructed image data is read in units of CTU, and edge filling is performed on the image according to the current resolution. Then, the search box pixels are read by sliding. Step S7: Store the encoded bitstream in DDR4 in real time. An interrupt is generated every time the stored data is 1KB, which notifies the CPU to read the bitstream data and pack it into RTP format, and send it to the destination IP via UDP.

2. The high-efficiency, low-latency HEVC encoder data buffering and processing method according to claim 1, characterized in that: Step S1 specifically includes the following steps: Step S11: After receiving video data, output it in native video signal format; Step S12: After converting RGB888 data to YUV444 using the RGB_to_YUV420 color space conversion module, it is then downsampled sequentially to YUV422 and YUV420. The color conversion formula is: Where Y represents the luminance component; U and V represent the chrominance components; R represents the red channel; G represents the green channel; and B represents the blue channel. Color conversion operations are implemented in the FPGA, converting floating-point operations into fixed-point operations.

3. The high-efficiency, low-latency HEVC encoder data buffering and processing method according to claim 1, characterized in that: In step S2, resolution calculation is initiated based on the line and field synchronization signals of the received video, and the effective pixel width and height of the video are counted to obtain the resolution value; When two consecutive frames of video signal are lost, the previous statistical result is retained to avoid misjudgment of video signal loss caused by input interface jitter.

4. The high-efficiency, low-latency HEVC encoder data buffering and processing method according to claim 1, characterized in that: Step S3 specifically includes the following steps: Step S31: Distribute and store the luminance component Y in units of 64×64; Step S32: The two 32×32 chromaticity components U and V are distributed and stored as a single data unit; Step S33: Using address bit control, write data from the AXI_HP port to DDR4 in DMA mode.

5. The high-efficiency, low-latency HEVC encoder data buffering and processing method according to claim 1, characterized in that: In step S4, two 64×64 ping-pong RAMs are instantiated on the FPGA to store CU and PU data belonging to the same CTU read from DDR4, so as to reduce the bandwidth pressure of off-chip memory and reduce encoder encoding latency by using a small amount of logic resources.

6. The high-efficiency, low-latency HEVC encoder data buffering and processing method according to claim 1, characterized in that: Step S6 specifically includes the following steps: Step S61: Receive the coordinates of the current predicted block in the image frame; Step S62: Initiate an AXI4 burst read signal and read the corresponding reconstructed image data from DDR4 according to the block coordinates; Step S63: Store the read data in 80 rows of RAM; Step S64: Fill the edges of the image; Step S65: Output the filled 64 rows of search box data; Step S66: Continue reading data down to fill the 16 empty rows.

Images