An image pre-processing device suitable for video coding

Abstract

The present invention relates to the field of image processing technology, and proposes an image pre-processing device suitable for video encoding. The device uses an online method to realize functions such as macroblock scaling, image layer overlay, and thumbnail output of the source image, and includes: a data source management module, a two-dimensional scaling module, an image layer overlay module, an output module, etc. The data source management module pre-generates the source data read instructions and caches the source data required for the current target macroblock row output; the image layer overlay module reads and pre-reads the required corresponding overlay layer macroblock data according to the coordinate position of each image layer; the output module obtains the corresponding size image data according to the fixed reduction multiple, and then outputs it to the video encoding module and writes it into the off-chip memory. The device uses block segmentation and ping-pong storage of the data source to coordinate the pipeline processing of each module, while reducing bandwidth consumption and meeting the real-time requirements of high-definition video encoding.

Description

Technical Field

[0001] The invention relates to the field of video encoding, and in particular to a device for pre-processing a video encoding source. Background Art

[0002] As users' demand for high-definition video increases, the bandwidth resources and real-time performance of image processing have become system bottlenecks.

[0003] In the prior art, a two-dimensional image scaling filter can scale the original image in both horizontal and vertical directions. The traditional scaling method is to first scale horizontally, write the intermediate result to the off-chip memory, and then read the horizontal scaling result from the off-chip memory to perform the vertical scaling process. This method is relatively simple to implement, but it requires multiple read and write operations of the off-chip memory, which consumes more system bandwidth and reduces the system processing performance. One is to implement the two-dimensional scaling operation in one step. This method requires a lot of line buffers to store the original image pixel information required by the filter, which will use a lot of on-chip memory. Especially in the field of video coding, image processing is output in macroblock format. In order to meet the fast output of data in a macroblock, if the traditional method is adopted, such as a second-order vertical filter, 32 lines need to be cached in the worst case, which requires a lot of hardware resources. In addition, this whole-line scaling method cannot meet the real-time requirements of video coding. Summary of the invention

[0004] In the field of video coding, in order to solve the problems existing in the existing image processing technology, the present invention provides a pre-processing device suitable for video coding, which can overcome the problems existing in the existing technology.

[0005] The technical solution adopted by the present invention to solve its technical problems is: a pre-processing device suitable for video encoding is proposed, including a data source management module, which is used to generate the source image read instructions required for the current encoding output macroblock row and then read out the data and cache it; a two-dimensional scaling module, which is used for the two-dimensional scaling output of the current target macroblock 16x16 pixels; an image mixing layer module, which reads the current and pre-reads the next required image layer data according to the macroblock stacking coordinate relationship of each image layer, and then stacks and outputs it; an output module, which completes the output of the encoding target macroblock and thumbnail according to the video encoding macroblock data output format and thumbnail size; these modules all use an online pipeline method to complete the processing. Among them, the read instruction generator in the data source management module will calculate the read data instructions for the source data required for the 16 lines in the current macroblock row in advance according to the scaling factor and store them in the read instruction buffer until the source data required for the current macroblock row scaling is read, and the read instructions in the read instruction buffer are updated for the next macroblock row scaling; the data cache module cuts the source image horizontally by every 32 pixels, and then uses 3 SRAM rotation mode. According to the SRAM empty and full status, the hardware dynamically reads the corresponding block data from left to right (data volume = total number of rows x 32). When scaling, the two-dimensional scaling module divides the macroblock into 4 16x4 blocks, the order of the blocks is from left to right, and the blocks are scaled from top to bottom. The horizontal 2-point parallel scaling is adopted, and then the vertical scaling is performed. Only a small number of internal registers are required to store the intermediate results. The image layer overlay module can realize 4-image layer overlay. On the one hand, it determines whether the next macroblock has layer overlay according to the current macroblock coordinates to decide whether to pre-read the corresponding image overlay layer source and store it in SRAM. On the other hand, it reads the image to be overlaid by the current macroblock from SRAM and overlays it one by one with the result of the previous level scaling output. The output module outputs the pixels output by the previous pipeline level at 444 to 420 to the video encoding level, and scales down the length and width in equal proportion according to the scaling ratio (4, 8, 16) to output the thumbnail. The scaling process uses a register and an accumulator to accumulate the pixel values ​​until the valid scaling value is output and reset.

[0006] The beneficial effects of the present invention are as follows: an image pre-processing device suitable for video encoding is proposed, which can complete online real-time filtering operations on the encoding input source, and the intermediate results do not need to access the off-chip memory, but are directly sent to the video encoding input stage in an online manner, which can effectively reduce bandwidth consumption, and compared with the traditional scaling method, the device reduces cache resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 The invention is a block diagram of an image pre-processing device for video coding.

[0008] Figure 2 It is a schematic diagram of the video source block reading sequence and storage method adopted by the device.

[0009] Figure 3 It is a two-dimensional scaling output sequence diagram of macroblock internal data.

[0010] Figure 4 This is a schematic diagram of the layer overlay module. DETAILED DESCRIPTION

[0011] The specific implementation of the present invention is further described in detail below in conjunction with the drawings and examples.

[0012] like Figure 1 As shown, it is a block diagram of a pre-processed image according to an embodiment of the present invention, in which a data source management module 101 completes the reading and storage operation of a source image, a two-dimensional scaling module 102 completes the scaling output of the internal data of a 16x16 macroblock, an image layer overlay module 103 completes the overlay of four layers with the scaling output video layer of the previous level, and an output module 104 completes the 444-to-420 conversion of the previous level overlay result and outputs it to the video encoding level and performs thumbnail output with a scaled size of (2, 4, 8).

[0013] The specific process of the embodiment of the present invention is as follows: take 1920x1080 reduced to 1280x720, use 2nd order vertical filtering, the scaling factor scale is 384, 4 image layers are superimposed, and the thumbnail is reduced by 4 times as an example. The data source management module 101 first obtains the coordinate position Yoi=MBy+i(y0, y1, y2...y15) of 16 lines in the target image according to the current output macroblock coordinates, and then rounds according to the formula Ysrci=Yoi*scale to obtain the source image line position coordinates required for each line, and summarizes all the source image line coordinates required for the entire macroblock row (maximum 32 lines); converts the source image coordinates into control instructions for reading data from the external memory DDR, each instruction includes the DDR address of the corresponding line and the data burst length, and stores these instructions in the instruction memory. Figure 2 The source image block format and the data SRAM storage format are given in FIG. The data block diagram 201 is to cut the source image into fixed 32 pixels in the horizontal direction (expandable), and to obtain the required source data interval for each target macroblock row in the vertical direction according to the scaling factor, namely (Ysrc0, Ysrc15+1). These data blocks are read in order from left to right. The SRAM storage diagram 202 uses 3 SRAMs for storage rotation, which can effectively resist the data delay of DDR. The cache controller in the data source management module 101 sends continuous read instructions to the off-chip memory controller according to the empty and full status of the cache to quickly obtain the required source data until the entire target macroblock row is output and the instruction memory is updated again.

[0014] In one embodiment, the two-dimensional scaling module 102 scales and outputs a macroblock according to Figure 3The macroblock is output in a manner that divides the macroblock into 4 16x4 blocks, the order of the blocks is from left to right, and every 4 points in the block are from top to bottom. This division method can ensure that less source data can be partially scaled and processed, improve processing performance, and can also cooperate well with the data processing of the next-level image layer overlay module 103 and the output module 104. This module scales each line in parallel by 2 points horizontally, and then uses 8 registers to store 2 lines (one line corresponds to 4 points output scaled in the horizontal direction), and finally scales vertically and outputs 4 points in parallel. The maximum time required for the entire macroblock scaling is 264 clock cycles, which can meet the requirements of high-definition video real-time encoding.

[0015] In one embodiment, the image layer overlay module 103 completes Figure 4 The mixed superposition of the four image layers (non-overlapping) shown in the figure. The pre-fetch module retrieves whether superposition is required based on the macroblock coordinate position processed by the current device, and on the other hand, determines whether the next macroblock position is superimposed based on the macroblock coordinate position of the four image layers in the video layer, and reads the corresponding superimposed layer source. The whole process uses a ping-pong operation between the current superimposed block and the next superimposed block.

[0016] In one embodiment, the output module 104 reduces the result of the previous level output by 4 times. The entire process uses a register and an accumulator, and adopts a pipeline method. Four clock cycles can complete the output of a valid thumbnail point, without storing all 4x4 data before starting the calculation.

[0017] The above description is made in conjunction with the embodiments, which do not constitute a limitation of the present invention. For those skilled in the art, the technical solutions recorded in the above embodiments can be modified, or some of the technical features therein can be replaced by equivalents; and all these modifications and replacements are deemed to be within the protection scope of the present invention.

Claims

1. An image pre-processing device suitable for video coding, characterized in that: include: A data source management module is used to pre-generate the source image read instructions required for the current encoding output macroblock row and then read out the corresponding data and cache it. The source image read instruction generator of the data source management module will calculate the source data read instructions required for 16 lines in the current target macroblock row in advance according to the scaling factor, and store them in the read instruction buffer until the source data required for scaling the current macroblock row is read. The read instructions in the read instruction buffer are updated for data reading for scaling the next macroblock row. The source image read instructions required for the current macroblock row stored in the instruction buffer can continuously send command requests for reading source data to the off-chip memory controller to reduce the delay of data returning from the off-chip memory and improve processor performance. Two-dimensional scaling module, used for scaling output of 16x16 pixels of the current target macroblock; The output module completes the output of the encoded target macroblock and thumbnail according to the video encoding macroblock data output format and thumbnail size.

2. The image pre-processing device according to claim 1, characterized in that: The data cache module in the data source management module cuts the source image horizontally into fixed sections of 32 pixels each, and uses a three-sRAM rotation method. The cache controller reads the corresponding block data (data volume = total number of rows x 32) from left to right according to the SRAM empty or full status.

3. The image pre-processing device according to claim 1, characterized in that: The scaling coefficient in the two-dimensional scaling module can be flexibly configured by the processor. When scaling, the macroblock is divided into four 16x4 blocks. The order of the blocks is from left to right, and the scaling is performed from top to bottom within the block. Only a small number of registers are required to store the intermediate results.

4. The image pre-processing device according to claim 1, characterized in that: The output module outputs the result of the previous pipeline stage in equal proportion thumbnail according to the scaling ratio (2, 4, 8). The scaling process only requires a small number of registers and accumulators to perform pipeline accumulation on pixel values ​​until a valid scaling value is output and then cleared.

Images