A method and system for RGB image coding based on GPU acceleration

Abstract

This invention belongs to the field of image processing and video coding technology, specifically a GPU-accelerated RGB image coding method and system. The method includes performing one of three processing steps on the RGB image using the GPU: color space conversion using bilinear interpolation, data rearrangement based on channel separation and stacking, or data rearrangement based on tile block rearrangement, to generate input data suitable for a YUV420 format hardware encoder. The bilinear interpolation method combines color space conversion and chroma downsampling into a single-step GPU calculation, reducing integer truncation times, eliminating color darkening, and improving line clarity. The channel stacking and tile rearrangement methods losslessly store RGB information in the Y channel, utilizing the encoder's full-resolution characteristics to achieve lossless chroma encoding. This invention is suitable for high-fidelity, low-latency video acquisition and transmission scenarios, achieving completely lossless chroma encoding.

Description

Technical Field

[0001] This invention relates to the field of image processing and video encoding technology, and in particular to a GPU-accelerated RGB image encoding method and system. Background Technology

[0002] In embedded video acquisition and transmission systems, input signals from interfaces such as HDMI are typically output as raw image data in RGB888 format. To achieve efficient compressed transmission, the RGB image needs to be encoded using a hardware video encoder (such as an H.264 / H.265 encoder). However, the hardware video encoders integrated into current mainstream embedded SoCs (such as Rockchip RK3588, HiSilicon Hi3559, and NVIDIA Jetson series) are limited by their design and typically only support YUV420SP (NV12 / NV21) formats as input, and do not support direct encoding of RGB888 images.

[0003] To adapt to hardware encoders, traditional processing steps require converting RGB888 to YUV420. This process has the following technical drawbacks: (1) Loss of chromaticity information and darkening of color: The traditional two-step method of "conversion first, averaging later" performs RGB to YUV conversion on each pixel independently (resulting in an integer truncation), and then averages the UV values ​​within the 2×2 block (truncating again). The cumulative systematic negative bias from multiple truncations leads to a lower chromaticity value and a darker overall color of the image.

[0004] (2) Decreased detail sharpness: Some simplified implementations use "single-point sampling" for chroma downsampling, that is, only the RGB value of one pixel in a 2×2 block is used to calculate the shared UV, resulting in the complete loss of chroma information of the remaining 3 / 4 pixels. For high-frequency details such as single-pixel-width colored lines and text edges, this method will cause color blurring or even disappearance, which will seriously reduce image sharpness.

[0005] CPU and bandwidth bottlenecks: In traditional pipelines, the acquisition, processing, and encoding modules rely on the CPU to perform multiple memory copies (memcpy). For 1920×1080@60fps video, this consumes a large amount of CPU resources, consumes approximately 1.09GB / s of DDR bandwidth, introduces additional latency, and disrupts the parallelism between modules. Summary of the Invention

[0006] This invention proposes a GPU-accelerated RGB image encoding method and system, aiming to solve the technical problems of color distortion, detail loss and system performance bottleneck in the prior art.

[0007] In a first aspect, the present invention provides a GPU-accelerated RGB image encoding method, comprising: Obtain an RGB888 image; The image is processed by the GPU using any one of the first, second, or third processing methods to generate input data suitable for the YUV420 format. The input data is then encoded using a hardware encoder. The first process is a color space conversion process that integrates bilinear interpolation, the second process is a data rearrangement process based on channel separation and stacking, and the third process is a data rearrangement process based on pixel block rearrangement.

[0008] The technical advantages of the GPU-accelerated RGB image encoding method disclosed in this invention are as follows: The GPU performs one of three different processing steps on the RGB image: color space conversion with fused bilinear interpolation, data rearrangement based on channel separation and stacking, or data rearrangement based on tile partitioning, thereby generating input data suitable for a YUV420 format hardware encoder. Specifically, the fused interpolation process combines color space conversion and chroma downsampling into a single parallel computation step on the GPU. By first averaging the RGB values ​​of a 2×2 pixel block and then calculating the shared UV, the number of integer truncations is reduced from 5 in traditional methods to 2, mathematically eliminating the color darkening phenomenon caused by the accumulation of multiple truncations. Furthermore, bilinear interpolation significantly improves the color reproduction of single-pixel lines. The channel stacking and tile rearrangement processes store the full RGB color information losslessly in the Y channel of the YUV420 frame, utilizing the full-resolution encoding capability of the Y channel by the hardware encoder to achieve completely lossless chroma encoding. This method provides multiple flexible options for high-fidelity encoding of RGB video.

[0009] Further, the first process includes: Divide the input RGB888 image into multiple 2×2 pixel blocks; For each pixel block, calculate the luminance component Y of four pixels and write the calculation result to the Y plane of the YUV format output buffer; For each 2×2 pixel block, the R, G, and B components of its four pixels are summed and then shifted two bits to the right to obtain the average R value avg_R, average G value avg_G, and average B value avg_B of the pixel block. Based on avg_R, avg_G, and avg_B, calculate the chromaticity components U and V shared by the 2×2 pixel block, and write the calculation results into the UV plane of the output buffer.

[0010] Furthermore, the luminance component and chrominance component are calculated using the following formulas: Y=((66×R+129×G+25×B+128)\ 8)+16; U=((-38×avg_R-74×avg_G+112×avg_B+128)\ 8)+128; V=((112×avg_R-94×avg_G-18×avg_B+128)\ 8)+128; in, This is a right shift operation, where Y is the luminance component.

[0011] Further, the second process includes: Channel separation step: The input W×H RGB888 image is separated into three independent single-channel planes (B, G, R) by the GPU, where W represents the image width and H represents the image height. Channel stacking steps: Stack the three single-channel planes B, G, and R along a specified direction to form a single-channel grayscale image; Virtual frame construction steps: Construct a YUV420SP virtual frame, fill the Y plane of the virtual frame with grayscale image data, and fill the entire UV plane of the virtual frame with preset neutral chromaticity values.

[0012] Furthermore, in the channel stacking step, the specified direction is the vertical direction, and the B, G, and R planes are stacked sequentially from top to bottom, and the size of the output grayscale image is W×(H'×3), where H' is the height after aligning H upwards to an integer multiple of the block size required by the encoder.

[0013] Furthermore, in the channel stacking step, the specified direction is horizontal, and the B, G, and R planes are stacked sequentially from left to right, and the size of the output grayscale image is (W'×3)×H, where W' is the width after aligning W upwards to an integer multiple of the block size required by the encoder.

[0014] Furthermore, the second process also includes a decoding and restoration step: The encoded video stream is decoded to obtain YUV420SP frames; Extract the Y-plane data of the YUV420SP frame to obtain the grayscale image; Based on the stacking direction during encoding, the grayscale image is divided into three independent planes of the same size, corresponding to the B, G, and R channels respectively; The data from the B, G, and R channels are interleaved to restore the image to RGB888 format.

[0015] Furthermore, the third process includes: Pixel block rearrangement steps: The input W×H RGB888 image is processed by the GPU to rearrange the pixel blocks. The BGR interleaved data in each T×T Tile block is separated and rearranged into three independent T×T planes: B, G, and R. These three planes are then arranged in the horizontal direction to obtain rearranged data with a size of (W'×3)×H', where T is the side length of the pixel block, and W' and H' are the values ​​of W and H after being aligned upwards to integer multiples of T, respectively. Virtual frame construction steps: Construct a YUV420SP virtual frame, fill the rearranged data into the Y plane of the virtual frame, and fill the entire UV plane of the virtual frame with preset neutral chromaticity values.

[0016] Furthermore, T is 16, and the alignment method is rounded up to an integer multiple of 16.

[0017] In a second aspect, the present invention provides a GPU-accelerated RGB image encoding system, comprising: a video acquisition module for acquiring video frames in RGB888 format and storing the frame data in a first DMA-BUF shared memory, wherein DMA-BUF represents a direct memory access buffer; The GPU processing module is used to directly access and process the video frame data by importing the file descriptor of the first DMA-BUF to execute the method and write the processing result into the second DMA-BUF shared memory; The hardware video encoding module is used to directly read the processing result and perform hardware encoding by importing the file descriptor of the second DMA-BUF.

[0018] The technical advantages of the system disclosed in this invention are as follows: By introducing a DMA-BUF zero-copy mechanism, the video acquisition module, GPU processing module, and hardware video encoding module can directly share memory through file descriptors, and synchronization primitives are used to ensure data consistency. This completely eliminates the CPU's involvement in data transmission, significantly reduces CPU utilization and DDR bandwidth consumption, and enables each module to work in parallel in a deeply pipelined manner, reducing end-to-end encoding latency to approximately 60ms, thus meeting the stringent requirements of real-time video processing for high performance and low latency. Attached Figure Description

[0019] Figure 1 This is a diagram illustrating the architecture and DMA-BUF zero-copy data flow of a GPU-accelerated RGB image encoding system in an embodiment of the present invention. Figure 2 This is a flowchart illustrating the color space conversion method incorporating bilinear interpolation in an embodiment of the present invention. Figure 3 This is a schematic diagram of the channel separation vertical stacking method in an embodiment of the present invention; Figure 4 This is a schematic diagram of the Tile block rearrangement method in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] This invention addresses the technical problems mentioned in the background section, such as color distortion, loss of detail, and system performance bottlenecks when adapting RGB images to a YUV420 hardware encoder. This embodiment uses the Rockchip RK3588 SoC platform as an example to describe in detail the complete implementation process of the method and system described in this invention. Figure 1 As shown, the system mainly consists of three modules: a video acquisition module (VI), a GPU processing module, and a hardware video encoding module (VENC). Data transfer between these three modules is achieved through a DMA-BUF mechanism, enabling zero-copy data transfer. The entire processing flow unfolds in the following steps.

[0022] Step 100: System Initialization and DMA-BUF Memory Pool Pre-allocation. Upon system startup, the video capture module drives the HDMI-RX receiver based on the V4L2 framework and pre-allocates two DMA-BUF memory pools from the driver: an input pool for storing raw captured frames and an output pool for storing frames processed by the GPU. Each DMA-BUF buffer is associated with a unique file descriptor (fd) for cross-module sharing.

[0023] Step 200: Video Acquisition and Direct DMA Storage. The video acquisition module obtains an idle DMA-BUF (denoted as fd_in) from the input pool and directly writes the RGB888 format video frames input from the HDMI input into this buffer via the DMA controller. After acquisition is complete, the driver passes fd_in along with frame information (timestamp, size, etc.) to the GPU processing module via a message queue. This process does not require CPU involvement in data transfer.

[0024] Step 300: GPU zero-copy import and preprocessing synchronization. Upon receiving `fd_in`, the GPU processing module (running on an ARMMali-G610 GPU based on OpenCL) calls the `cl_import_memory_arm()` extension function to import `fd_in` as an OpenCL input memory object `input_mem`. Simultaneously, it obtains an idle DMA-BUF (`fd_out`) from the output pool and imports it as an output memory object `output_mem`.

[0025] To ensure data consistency, before starting the GPU kernel, the CPU performs DMA_BUF_IOCTL_SYNC on fd_in via the ioctl system call, sets the flags to DMA_BUF_SYNC_END|DMA_BUF_SYNC_WRITE, and notifies the system VI that the DMA engine has completed the write, so the GPU will see the complete data in subsequent reads.

[0026] Step 400: GPU Parallel Processing (Mode-Based Branching). This system supports three processing modes, selected via the runtime parameter -t. The GPU kernel execution logic in each mode is described below.

[0027] Mode 1: Summing the R, G, and B components of each of the four pixels and shifting the sum two places to the right (i.e., dividing by 4 and rounding down) yields the average color value of the 2×2 block. The processing flow for this mode is as follows: Figure 2 As shown.

[0028] Sub-step 410: Divide into 2×2 pixel blocks. Divide the input RGB888 image into multiple non-overlapping 2×2 pixel blocks. The global workspace of the GPUOpenCL kernel is configured as (width / 2, height / 2), with each work item processing one 2×2 block. The local workspace is configured as (8, 8), allowing each work item to process a 16×16 pixel region to optimize locality.

[0029] Sub-step 420: Read the BGR values ​​of four pixels. For the 2×2 block corresponding to the current work item, with its top-left corner coordinates at (x, y), read the BGR888 values ​​of four pixels from the input buffer: P_{00} is located at (y, x), with components (R_{00}, G_{00}, B_{00}); P_{01} is located at (y, x+1), with components (R_{01}, G_{01}, B_{01}); P_{10} is located at (y+1, x), with components (R_{10}, G_{10}, B_{10}); P_{11} is located at (y+1, x+1), with components (R_{11}, G_{11}, B_{11}).

[0030] Sub-step 430: Calculate the luminance component Y and store it. Calculate the luminance values for four pixels respectively and write them to the corresponding positions in the output Y plane. The calculation formula (based on the BT.601 standard) is: Y = ((66 × R + 129 × G + 25 × B + 128) >> 8) + 16; The four Y values are respectively written to the output buffer positions: (y, x), (y, x + 1), (y + 1, x), (y + 1, x + 1).

[0031] Sub-step 440: Bilinear interpolation (area averaging). Sum the R, G, and B components of the four pixels respectively and then shift them two bits to the right (i.e., divide by 4) to obtain the average color value of this 2×2 block.

[0032] Sub-step 450: Calculate the shared chrominance components U and V. Use the average RGB value to calculate the U and V values shared by this 2×2 block.

[0033] Sub-step 460: Write to the UV plane. Write the U and V values to the output UV buffer (the interleaved area with a size of (width / 2) × (height / 2) after the Y plane) in the NV21 format. Usually, the UV order is V first and U later, and the write positions are: (width × height) + (uv_y × uv_width + uv_x) × 2; Write V at this position and write U in the next byte.

[0034] Mode 2 (Second processing): Rearrange the data stacked based on channel separation.

[0035] The following describes it in detail with vertical stacking (-t2) as the preferred solution and briefly describes horizontal stacking (-t3) as a variant. The processing flow is as Figure 3 shown.

[0036] Vertical stacking: Channel separation and vertical stacking. The input BGR888 image has a size of W × H and is aligned in height. The GPU kernel processes each pixel (x, y) (0 ≤ x < W, 0 ≤ y < H'): If y < H, read the original BGR values (B, G, R); otherwise, fill with 0.

[0037] Output positions: The B channel is at row y and column x of the output grayscale image; the G channel is at row y + H' and column x; the R channel is at row y + 2H' and column x.

[0038] Write the B, G, and R values to the corresponding positions respectively.

[0039] Horizontal stacking variant: If horizontal stacking (-t3) is selected, the output grayscale image size is (W'×3)×H. The B, G, and R channels occupy columns 0 to W'-1, W' to 2W'-1, and 2W' to 3W'-1 in sequence, and the writing logic is similar. W' is the alignment width.

[0040] Construct a virtual YUV420SP frame. Calculate the size of the Y plane Y_{size}=W×(H'×3), and the size of the UV plane UV_{size}=Y_{size} / 2. Write the grayscale image data generated in step 470 to the first Y_{size} bytes (Y plane) of the output DMA-BUF, and fill the last UV_{size} bytes with neutral gray.

[0041] When constructing a virtual YUV420SP frame, the hardware encoder's resolution limitations and alignment requirements must be considered: Most hardware encoders require the width and height of the image to be integer multiples of 16 (or 32, 64, etc., depending on the encoder specifications). Therefore, align the height H up to an integer multiple of 16 to get H', ensuring that the total height H' after vertical stacking is also an integer multiple of 16. Thus, the height after vertical stacking meets the 16-alignment. When horizontally stacking, the width W' is aligned to 16, and the stacked width W'×3 also meets the 16-alignment.

[0042] In addition, some encoders may have limitations on the maximum width or height they support (e.g., some platforms support a maximum of 4096 pixels). If the stacked size exceeds the limit, downgrading or selecting another mode is required. For example, if the height H'×3 after vertical stacking exceeds the encoder's maximum height, the system should automatically switch to the horizontal stacking mode (if the width is not exceeded), or use the fusion interpolation mode. The system will query the encoder capabilities during initialization and automatically select the appropriate mode or prompt the user.

[0043] Mode 3: Data rearrangement processing based on Tile block rearrangement (parameter -t1). The processing flow is as Figure 4 shown.

[0044] Tile block rearrangement. The input BGR888 image size is W×H, the Tile size T = 16, and the aligned width is W' and height is H'. The output rearranged data size is (W'×3)×H'. The GPU kernel processes each pixel (x,y) (0≤x<W', 0≤y<H').

[0045] Read BGR values: If x<W and y<H, read (B,G,R) from the original image; otherwise, fill with 0. Output position: Each Tile occupies a 48×16 area (16×3 columns, 16 rows) in the output.

[0046] Construct a virtual YUV420SP frame, use the rearranged data as the Y plane, and fill the UV plane with 0x80.

[0047] The width of the UV plane after tile rearrangement is (W' × 3) / 2, which may not be a multiple of 16. However, the encoder usually allows the width of the UV plane to be the same as half of the width of the Y plane without additional alignment because the UV plane data is stored continuously and the encoder will handle it internally.

[0048] Step 500: GPU processing is completed and synchronized. Call clFinish() to block until the GPU kernel execution is complete. After that, the CPU performs DMA_BUF_IOCTL_SYNC on fd_out, setting the flags to DMA_BUF_SYNC_START|DMA_BUF_SYNC_READ, notifying the system that the subsequent VENC module can prepare to read the output data. Finally, the GPU module passes fd_out to the hardware video encoding module.

[0049] Step 600: Zero-copy reading and encoding by the hardware encoder. After receiving fd_out, the hardware video encoding module configures it as the input source of the encoder. According to the selected mode, configure the encoder resolution: Mode 1: width × height (YUV420SP).

[0050] Mode 2 vertical stacking: W × (H' × 3).

[0051] Mode 2 horizontal stacking: (W' × 3) × H.

[0052] Mode 3: (W' × 3) × H'.

[0053] The H.264 / H.265 hardware encoder directly performs DMA reading of the YUV420SP data in this DMA-BUF through the system MMU for encoding to generate a video bitstream. The entire process the CPU is only responsible for control and synchronization, and the data path is zero-copy.

[0054] Step 700: Decoding and restoration. If Mode 2 or Mode 3 is used during encoding, the decoding end needs to perform an inverse transformation to restore the RGB888 image.

[0055] For Mode 2 vertical stacking: After decoding the YUV420SP frame, extract the Y plane data (size W × (H' × 3)). Divide it into three equal parts by height: the top H' rows are the B channel, the middle H' rows are the G channel, and the bottom H' rows are the R channel. For each pixel (x, y) (y < H), read the B, G, and R values from the three planes and interleave the output.

[0056] For Mode 2 horizontal stacking: Restore it by dividing it into three equal parts by width.

[0057] For mode 3: Perform inverse Tile transformation. For each pixel (x, y) in the output image, calculate its corresponding Tile index and local coordinates, and parse the B, G, and R values ​​from the Y-plane data.

[0058] Based on the same inventive concept, embodiments of the present invention also provide a GPU-accelerated RGB image encoding system, characterized in that it includes: The video acquisition module is used to acquire video frames in RGB888 format and store the frame data in the first DMA-BUF shared memory, where DMA-BUF stands for Direct Memory Access Buffer. The GPU processing module is used to directly access and process the video frame data by importing the file descriptor of the first DMA-BUF to execute the method and write the processing result into the second DMA-BUF shared memory; The hardware video encoding module is used to directly read the processing result and perform hardware encoding by importing the file descriptor of the second DMA-BUF.

[0059] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of the invention as set forth in the appended claims.

Claims

1. A GPU-accelerated RGB image encoding method, characterized in that, include: Obtain an RGB888 image; The image is processed by the GPU using any one of the first, second, or third processing methods to generate input data suitable for the YUV420 format. The input data is then encoded using a hardware encoder. The first process is a color space conversion process that integrates bilinear interpolation, the second process is a data rearrangement process based on channel separation and stacking, and the third process is a data rearrangement process based on pixel block rearrangement.

2. The method according to claim 1, characterized in that, The first process includes: Divide the input RGB888 image into multiple 2×2 pixel blocks; For each pixel block, calculate the luminance component Y of four pixels and write the calculation result to the Y plane of the YUV format output buffer; For each 2×2 pixel block, the R, G, and B components of its four pixels are summed and then shifted two bits to the right to obtain the average R value avg_R, average G value avg_G, and average B value avg_B of the pixel block. Based on avg_R, avg_G, and avg_B, calculate the chromaticity components U and V shared by the 2×2 pixel block, and write the calculation results into the UV plane of the output buffer.

3. The method according to claim 2, characterized in that, The luminance and chrominance components are calculated using the following formulas: Y=((66×R+129×G+25×B+128)\ 8)+16; U=((-38×avg_R-74×avg_G+112×avg_B+128)\ 8)+128; V=((112×avg_R-94×avg_G-18×avg_B+128)\ 8)+128; in, This is a right shift operation, where Y is the luminance component.

4. The method according to claim 1, characterized in that, The second process includes: Channel separation step: The input W×H RGB888 image is separated into three independent single-channel planes (B, G, R) by the GPU, where W represents the image width and H represents the image height. Channel stacking steps: Stack the three single-channel planes B, G, and R along a specified direction to form a single-channel grayscale image; Virtual frame construction steps: Construct a YUV420SP virtual frame, fill the Y plane of the virtual frame with grayscale image data, and fill the entire UV plane of the virtual frame with preset neutral chromaticity values.

5. The method according to claim 4, characterized in that, In the channel stacking step, the specified direction is the vertical direction. The B, G, and R planes are stacked sequentially from top to bottom, and the size of the output grayscale image is W×(H'×3), where H' is the height after aligning H upwards to an integer multiple of the block size required by the encoder.

6. The method according to claim 4, characterized in that, In the channel stacking step, the specified direction is horizontal, and the B, G, and R planes are stacked sequentially from left to right. The size of the output grayscale image is (W'×3)×H, where W' is the width after aligning W upwards to an integer multiple of the block size required by the encoder.

7. The method according to claim 4, characterized in that, The second process also includes a decoding and restoration step: The encoded video stream is decoded to obtain YUV420SP frames; Extract the Y-plane data of the YUV420SP frame to obtain the grayscale image; Based on the stacking direction during encoding, the grayscale image is divided into three independent planes of the same size, corresponding to the B, G, and R channels respectively; The data from the B, G, and R channels are interleaved to restore the image to RGB888 format.

8. The method according to claim 1, characterized in that, The third process includes: Pixel block rearrangement steps: The input W×H RGB888 image is processed by the GPU to rearrange the pixel blocks. The BGR interleaved data in each T×T Tile block is separated and rearranged into three independent T×T planes: B, G, and R. These three planes are then arranged in the horizontal direction to obtain rearranged data with a size of (W'×3)×H', where T is the side length of the pixel block, and W' and H' are the values ​​of W and H after being aligned upwards to integer multiples of T, respectively. Virtual frame construction steps: Construct a YUV420SP virtual frame, fill the rearranged data into the Y plane of the virtual frame, and fill the entire UV plane of the virtual frame with preset neutral chromaticity values.

9. The method according to claim 8, characterized in that, T is 16, and the alignment method is rounded up to an integer multiple of 16.

10. A GPU-accelerated RGB image encoding system, characterized in that, include: The video acquisition module is used to acquire video frames in RGB888 format and store the frame data in the first DMA-BUF shared memory, where DMA-BUF stands for Direct Memory Access Buffer. The GPU processing module is configured to directly access and process the video frame data by importing the file descriptor of the first DMA-BUF, so as to execute the method as described in any one of claims 1 to 9, and write the processing result into the second DMA-BUF shared memory; The hardware video encoding module is used to directly read the processing result and perform hardware encoding by importing the file descriptor of the second DMA-BUF.

Images