3D nr motion estimation hardware implementation system
By designing a hardware implementation system for 3DNR motion estimation, and utilizing frame buffers, line buffers, register matrices, and filtering units to dynamically adjust the search window mode, the hardware complexity and matching accuracy issues in 3DNR motion estimation are resolved, achieving real-time performance and consistency, and avoiding artifacts and edge breaks.
Patent Information
- Application Number
- CN202511343742.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-19
- Publication Date
- 2025-11-25
- Estimated Expiration
- 2045-09-19
Smart Images
Figure CN120856876B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of video image processing, and particularly relates to a 3DNR motion estimation hardware implementation system. BACKGROUND
[0002] 3DNR (3-Dimensional Noise Reduction) technology is an image processing technology for realizing efficient noise reduction by analyzing noise characteristics in the time domain (multiple frames) and the space domain (single frame) of a video image. When processing in the time domain, the static region of adjacent frames is compared, and noise is reduced by weighted average of multiple frames.
[0003] In order to better prevent the occurrence of artifacts, the technology of motion estimation is introduced in the process of 3DNR implementation. Motion estimation includes block matching method, optical flow method, pixel recursion method, etc., and the block matching method is commonly used. The block matching algorithm has many implementation methods in the software layer, such as three-step method, four-step method, diamond method, hexagon method, full search method, etc. Although some algorithms have been implemented in hardware, there are still problems of complex hardware implementation logic, limited real-time performance or poor matching accuracy. In addition, when the block matching search window is at the image edge, improper implementation will affect the complexity and image artifact problem. SUMMARY
[0004] The purpose of the present application is to provide a 3DNR motion estimation hardware implementation system, which can guarantee the real-time performance of 3DNR processing, better solve the consistency of the maximum radiation radius of the block matching search window at the image edge, and further guarantee the smooth transition of the motion field, avoid the generation of artifacts and edge break phenomenon; in addition, maintaining the consistency of the maximum radiation radius of the search window is conducive to supporting regular data flow and parallel computing, reducing the design complexity, and solving the problems proposed in the background technology.
[0005] To achieve the above purpose, the present application provides the following technical scheme: a 3DNR motion estimation hardware implementation system, which comprises a frame buffer unit, a line buffer unit, a register matrix unit, a block matching unit and a filtering unit;
[0006] The frame buffer unit is used for buffering two frames of video images; the line buffer unit is used for buffering multiple lines of data; the register matrix unit is used for storing a pixel array to construct a block matching search window; and the filtering unit is used for filtering the image pixel block.
[0007] The block matching unit comprises a search window mode processing unit and a current frame block unit; the search window mode processing unit determines the reference frame block matching search window mode, and the current frame block unit is used for obtaining a matrix pixel block of the current frame.
[0008] The 3DNR motion estimation hardware implementation system specifically executes the following process steps:
[0009] a. When the frame synchronization signal is valid, the two frames of video images are cached to the frame cache unit;
[0010] b. When the frame cache unit stores a fixed number of frames, the line cache unit initiates a read request to the frame cache unit, and the frame cache unit writes to the line cache unit in a line-by-line manner;
[0011] c. After the line cache unit is filled with pixels, multiple lines are read out at the same time, the register matrix unit writes to the register matrix, and the block matching search window is constructed, and at the same time, the frame cache unit continues to write to the line cache unit;
[0012] d. After the search window is formed, enter the block matching unit; according to the position of the current frame pixel block in the image obtained by the current frame block unit, the search window mode processing unit determines the horizontal and vertical search range of the search window in the reference frame, and then finds the optimal similar block in the search window through block comparison, and saves the motion vector value and residual value of the block;
[0013] e. After finding the optimal similar block in the search window of the reference frame, the filtering unit performs filtering processing according to the optimal block matching residual threshold and motion vector threshold setting.
[0014] Preferably, in step d, the search window mode processing unit also dynamically adjusts the search window according to the position of the current frame pixel block in the image and the edge type of the current frame block;
[0015] If the size of the current matching block is NxN and the search radius is R, the ideal situation is that the search radius of the horizontal and vertical four directions is R; but at the image edge, it is impossible to satisfy that the search radius of the horizontal and vertical four directions is R, at this time, only the search radius of at most three directions of the horizontal and vertical is R.
[0016] Preferably, in step e, the filtering unit performs filtering processing according to the block matching residual threshold and motion vector threshold setting, specifically including:
[0017] The filtering unit filters according to the optimal block matching residual threshold and motion vector threshold, and the filtering unit performs filtering operation in units of blocks;
[0018] If the motion vector threshold is MV_TH and the matching residual threshold is SAD_TH, the MV_TH and SAD_TH are configured by registers; when the residual value SAD saved by the block matching unit is greater than the set SAD_TH or the recorded motion vector value MV is greater than the set MV_TH, the time domain noise reduction is not performed, only the current frame pixel block is output, otherwise, the current frame pixel block and the reference frame matching pixel block are subjected to the weighted operation and then output.
[0019] Preferably, the frame buffer unit includes but is not limited to DDR and SRAM.
[0020] Compared with the prior art, the 3DNR motion estimation hardware implementation system has the following beneficial effects:
[0021] The 3DNR motion estimation hardware implementation system can not only ensure the real-time performance of 3DNR processing and the block matching accuracy in motion estimation, but also solve the consistency of the maximum radiation radius of the block matching search window in the image edge, reduce the hardware design complexity and facilitate the hardware implementation. BRIEF DESCRIPTION OF DRAWINGS
[0022] Figure 1 The figure is a structural block diagram of the 3DNR motion estimation hardware implementation system.
[0023] Figure 2 The figure is a principle diagram of the frame buffer unit for video image frame buffer operation.
[0024] Figure 3 The figure is a motion estimation block matching scheme.
[0025] Figure 4 The figure is a search window mode of the 3DNR motion estimation hardware implementation system. DETAILED DESCRIPTION
[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.
[0027] 3DNR is a video processing technology aiming to reduce the noise in video images. In 3DNR, the main target of motion estimation is to identify the moving objects, so as to align the inter-frame information and prevent the occurrence of artifacts in the superposition of multiple frames.
[0028] As shown in Figure 1 The embodiments of the present application provide a 3DNR motion estimation hardware implementation system, which comprises a frame buffer unit, a line buffer unit, a register matrix unit, a block matching unit and a filter unit.
[0029] The frame buffer unit is used for buffering two frame video images; the line buffer unit is used for buffering multiple lines of data; the register matrix unit is used for storing a pixel array to construct a block matching search window; and the filter unit is used for filtering an image pixel block.
[0030] The block matching unit comprises a search window mode processing unit and a current frame block unit; the search window mode processing unit determines a reference frame block matching search window mode, and the current frame block unit is used for obtaining a matrix pixel block of a current frame.
[0031] The present application is described according to a block diagram structure as shown in the figure, and the 3D NR motion estimation hardware implementation system specifically executes the following steps: Figure 1
[0032] a. When a frame synchronization signal is valid, two frame video images are buffered to the frame buffer unit;
[0033] b. When the frame buffer unit stores a fixed number of frames, the line buffer unit initiates a read request to the frame buffer unit, and the frame buffer unit writes into the line buffer unit in a line mode;
[0034] c. After the line buffer unit is filled with pixels, multiple lines are read out at the same time, the register matrix unit writes into the register matrix to construct a block matching search window, and meanwhile the frame buffer unit continues to write into the line buffer unit;
[0035] d. After the search window is formed, the block matching unit is entered; according to the position of the current frame pixel block in the image obtained by the current frame block unit, the search window mode processing unit determines the horizontal and vertical search ranges in the search window of the reference frame, and then finds the optimal similar block in the search window through block comparison, and saves the motion vector value and residual value of the block;
[0036] e. After the optimal similar block is found in the search window of the reference frame, the filter unit performs filtering processing according to the optimal block matching residual threshold value and the motion vector threshold value setting.
[0037] As an embodiment of the present application, in step d, the search window mode processing unit further dynamically adjusts the search window according to the position of the current frame pixel block in the image and the edge type of the current frame block.
[0038] If the size of the current matching block is NxN and the search radius is R, the ideal case is that the search radius of the horizontal and vertical four directions is R; but at the image edge, the search radius of at least two directions is R, and only the horizontal and vertical at most three directions are retained.
[0039] In an embodiment of the present invention, in step e, the filtering unit performs filtering processing based on the block matching residual threshold and the motion vector threshold, specifically including:
[0040] The filtering unit performs filtering based on the optimal block matching residual threshold and the motion vector threshold, and the filtering unit performs filtering operations on a block-by-block basis.
[0041] If the motion vector threshold is MV_TH and the matching residual threshold is SAD_TH, MV_TH and SAD_TH are configured through registers. When the residual value SAD stored in the block matching unit is greater than the set SAD_TH or the recorded motion vector value MV is greater than the set MV_TH, no temporal noise reduction is performed, and only the current frame pixel block is output. Otherwise, the current frame pixel block and the reference frame matching pixel block are weighted and output.
[0042] As an embodiment of the present invention, the frame buffer unit includes, but is not limited to, DDR and SRAM.
[0043] Furthermore, embodiments of the present invention also provide a specific application scenario, as follows:
[0044] Taking 720P video format as an example, the size of the matching block in the current frame pixel block and the search window of the reference frame is 4x4, the size of the search window of the reference frame is 20x20, and the maximum search radius in the horizontal and vertical directions is 6.
[0045] The frame buffer unit in the 3DNR motion estimation hardware implementation system stores two video frames; here, two SRAMs are used for frame buffering. 3DNR noise reduction is not performed on the first frame of the video image; it only takes effect from the second frame onwards. The principle is as follows... Figure 2 As shown. Frm_flg is the parity flag signal for video frames; Frm_flg = 1 represents odd-numbered frames, and 0 represents even-numbered frames. The two single-port SRAMs implement a ping-pong operation for the data stream. When there are more than two video frames, Frm_flg is 0, and the video image input data is written to SRAM1 while simultaneously being read from SRAM2, writing the data to the line buffer unit; when Frm_flg is 1, the video image input data is written to SRAM2 while simultaneously being read from SRAM1, writing the data to the line buffer unit.
[0046] The line buffer unit uses 24 lines. Each line buffer is a single-port SRAM, which can be named Line_buffer1 to line_buffer24. These 24 SRAMs are used to construct a 20x20 search window, i.e., a register matrix. When the first 20 lines of buffers (line_buffer1 to line_buffer20) are writing data to the 20x20 register matrix, the frame buffer module needs to store the next four lines (21 to 24) immediately following the first 20 lines into the line buffers (line_buffer5 to line_buffer24). This ensures that after processing the first 20 lines, processing can continue with lines 5 to 24. Simultaneously, when writing data from lines 5 to 24 to the 20x20 register matrix, the frame buffer module needs to write the next four lines (25 to 28) into line_buffer1 to line_buffer4, overwriting the processed first four lines, and so on for subsequent operations.
[0047] Matching schemes for motion estimation blocks, such as Figure 3 As shown, it is actually a 20x20 register matrix. The first step involves searching all blocks within the 20x20 search window with a fixed step size of 2, starting from the matching block Xc. The SAD (Sum of Absolute Differences) is calculated, and the block X0 with the highest similarity is found. The coordinates of X0 are recorded. The second step involves calculating the SAD values of the eight surrounding blocks centered on X0, and finding the block Y0 with the highest similarity. The motion vector value and residual value of Y0 are written to the register for recording. This completes the block matching operation between the current frame pixel block and the reference frame search window.
[0048] In a single image frame, pixel blocks may be located at image edges. This necessitates a search window mode processing unit to ensure consistency in the maximum radiation radius of block matching. In a 720P data format image with a resolution of 1280x720, the starting coordinates of the top-left corner are (0,0), and the maximum coordinates of the bottom-right corner are (1279,719). The search window dynamically adjusts based on the edge type, with a maximum search radius of 6. The search window mode can be divided into 25 modes based on the image position of the pixel blocks in the current frame, such as... Figure 4As shown. The black square represents the starting point of the current 4x4 block, and the gray square represents the radius of the search window. In A11, the current 4x4 block is located at the top left corner of the image, and the search radius of 6 is only in the down and right directions. In A12, the current 4x4 block is located at position (4,0) of the image, and the search radius to the right and down is 6, while the search radius to the left is 4. In A13, the current 4x4 block is located at coordinate (n,0) of the image, where n is divisible by 4 and satisfies the conditions n≥8 and n<1272, the search radius to the left, right, and down directions is 6. In A14, the current 4x4 block is located at coordinates (1272,0) on the image, with a search radius of 6 to the left and bottom, and a search radius of 4 to the right; in A15, the current 4x4 block is located at coordinates (1276,0) on the image, with a search radius of 6 to the left and bottom; in A21, the current 4x4 block is located at position (0,4) on the image, with a search radius of 6 to the right and bottom, and a search radius of 4 to the top; in A22, the current 4x4 block is located at position (4,4) on the image, with a search radius of 6 to the right and bottom, and a search radius of 4 to the top and left; in A23, the current 4x4 block is located at coordinates (n,4) on the image, where n is divisible by 4, and the conditions n≥8 and n<1272 are met, with a search radius of 6 to the left, right, and bottom, and a search radius of 4 to the top. In A24, the current 4x4 block is located at coordinates (1272, 4) in the image, with a search radius of 6 to the left and below, and a search radius of 4 to the top and right. In A25, the current 4x4 block is located at coordinates (1276, 4) in the image, with a search radius of 6 to the left and below, and a search radius of 4 to the top. In A31~A35, the current 4x4 block is located at the vertical coordinate m in the image, where m is divisible by 4, and the conditions n≥8 and n<712 are met. In A31, the current 4x4 block is located at coordinates (0, m) in the image, with a search radius of 6 to the top, bottom, and right. In A32, the current 4x4 block is located at coordinates (4, m) in the image, with a search radius of 6 to the top, bottom, and right, and a search radius of 4 to the left.In A33, the current 4x4 block is located at coordinates (n, m) in the image, where n is divisible by 4, and the conditions n ≥ 8 and n < 1272 are met. The search radii to the top, bottom left, and right are 6. In A34, the current 4x4 block is located at coordinates (1272, m) in the image. The search radii to the top, bottom, and left are 6, and the search radius to the right is 4. The current 4x4 block is located at coordinates (1276, m) in the image. The search radii to the top and bottom are 6, and the search radius to the right is 4. In A41, The current 4x4 block is located at position (0, 712) in the image. The search radius to the right and above is 6, and the search radius to the bottom is 4. In A42, the current 4x4 block is located at position (4, 712) in the image. The search radius to the right and above is 6, and the search radius to the bottom and left is 4. In A43, the current 4x4 block is located at position (n, 712) in the image. When n is divisible by 4 and satisfies the condition n ≥ 8 and n < 1272, the search radius to the left, right, and top is 6, and the search radius to the bottom is 4. In A44, the current 4x4 block is located at coordinates (1272, 712) on the image. The search radius to the left and top is 6, and the search radius to the bottom and right is 4. In A45, the current 4x4 block is located at coordinates (1276, 712) on the image. The search radius to the left and top is 6, and the search radius to the bottom is 4. In A51, the current 4x4 block is located at the bottom left corner of the image. The search radius is only in the top and right directions, which are 6. In A52, the current 4x4 block is located at (4, 716) on the image. The search radius to the right and top is 6, and the search radius to the left is 4. In A53, the current 4x4 block is located at coordinates (n, 716) on the image. When n is divisible by 4 and satisfies the condition n ≥ 8 and n < 1272, the search radius to the left, right, and top is 6. In A54, the current 4x4 block is located at coordinates (1272, 716) in the image, with a search radius of 6 to the left and top, and a search radius of 4 to the right; in A55, the current 4x4 block is located at coordinates (1276, 716) in the image, with a search radius of 6 to the left and top.
[0049] After finding the optimal matching block in the search window of the reference frame, filtering is performed based on the motion estimation threshold and the optimal block matching residual threshold. 3DNR filtering is a block-by-block filtering operation. The motion vector threshold is MV_TH, and the matching residual threshold is SAD_TH. MV_TH and SAD_TH can be configured via registers. When the residual value SAD stored in the block matching unit is greater than the set SAD_TH or the recorded motion vector value MV is greater than the set MV_TH, no temporal denoising is performed, and only the current frame pixel block is output. Otherwise, the current frame pixel block and the reference frame matching pixel block are weighted and then output.
[0050] The hardware implementation system for 3DNR motion estimation of the present invention not only ensures the real-time performance of 3DNR processing and the block matching accuracy in motion estimation, but also solves the problem of consistency of the maximum radiation radius searched by the block matching search window at the image edge, reducing the complexity of hardware design and facilitating hardware implementation.
[0051] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0052] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A hardware implementation system for three-dimensional noise reduction and 3DNR motion estimation, characterized in that, The hardware implementation system for 3D noise reduction and 3DNR motion estimation includes a frame buffer unit, a line buffer unit, a register matrix unit, a block matching unit, and a filtering unit. The frame buffer unit is used to buffer two frames of video images; the line buffer unit is used to buffer multiple lines of data; the register matrix unit is used to store the pixel array to construct the block matching search window; and the filtering unit is used to filter the image pixel blocks. The block matching unit includes a search window mode processing unit and a current frame block unit; the search window mode processing unit determines the reference frame block matching search window mode, while the current frame block unit is used to obtain a matrix pixel block of the current frame; The specific execution flow steps of this 3D noise reduction and 3DNR motion estimation hardware implementation system are as follows: a. When the frame synchronization signal is valid, two frames of video images are buffered into the frame buffer unit; b. When the frame buffer unit has stored a fixed number of frames, the line buffer unit sends a read request to the frame buffer unit, and the frame buffer unit writes to the line buffer unit in a line-by-line manner; c. After the row buffer unit is filled with pixels, multiple rows are read out simultaneously, the register matrix unit writes to the register matrix to build a block matching search window, and the frame buffer unit continues to write to the row buffer unit. d. After the search window is formed, it enters the block matching unit; based on the position of the current frame pixel block in the image obtained by the current frame block unit, the search window mode processing unit determines the horizontal and vertical search range within the reference frame search window, and then finds the optimal similar block within the search window through block comparison, and saves the block's motion vector value and residual value. e. After finding the optimal similar block in the search window of the reference frame, the filtering unit performs filtering processing according to the optimal block matching residual threshold and motion vector threshold settings; In step d, the search window mode processing unit also dynamically adjusts the search window based on the position of the current frame pixel block in the image and the edge type of the current frame block. If the current matching block size is NxN and the search radius is R, then the search radius in all four directions (horizontal and vertical) is R. At the edge of the image, at most three directions (horizontal and vertical) must be retained, and the search radius in at least two directions must be R.
2. The hardware implementation system for three-dimensional noise reduction and 3DNR motion estimation according to claim 1, characterized in that, In step e, the filtering unit performs filtering processing based on the block matching residual threshold and motion vector threshold settings, specifically including: The filtering unit performs filtering based on the optimal block matching residual threshold and the motion vector threshold, and the filtering unit performs filtering operations on a block-by-block basis. If the motion vector threshold is MV_TH and the matching residual threshold is SAD_TH, MV_TH and SAD_TH are configured through registers. When the residual value SAD stored in the block matching unit is greater than the set SAD_TH or the recorded motion vector value MV is greater than the set MV_TH, no temporal noise reduction is performed, and only the current frame pixel block is output. Otherwise, the current frame pixel block and the reference frame matching pixel block are weighted and output.
3. The hardware implementation system for three-dimensional noise reduction and 3DNR motion estimation according to claim 1, characterized in that, The frame buffer unit includes DDR and SRAM.
Citation Information
Patent Citations
Adaptive movement estimation method
CN101389023A
Motion estimation aided noise reduction
US9131073B1