Hardware implementation method of time-space domain filtering system combined with image information
By implementing a hardware method for a spatiotemporal domain filtering system, the problems of image denoising delay and poor effect in existing technologies are solved. This method achieves fast and effective image denoising while preserving detailed information, and is applicable to various noise types and motion scenarios.
Patent Information
- Application Number
- CN202511880158.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-12
- Publication Date
- 2026-05-05
AI Technical Summary
Existing image denoising methods suffer from excessive latency or poor algorithm performance on different platforms, especially in motion scenarios where they struggle to effectively suppress noise and preserve image details.
A spatiotemporal filtering system is employed, which achieves fast and effective image denoising while preserving detail information through image downsampling layering, edge filtering convolution kernel processing, high-speed interaction channel between FPGA and DDR3, motion compensation denoising, and bilinear interpolation.
It effectively suppresses noise and reduces image detail loss in motion scenes, has a fast processing time, is applicable to various noise types, and has the function of edge information protection.
Smart Images
Figure CN121981906A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a hardware implementation method for a spatiotemporal filtering system that incorporates image information, belonging to the field of image denoising and enhancement technology. Background Technology
[0002] Noise is a significant cause of image interference. Both spatial and temporal denoising methods offer some degree of noise reduction. However, current denoising methods are implemented on different platforms and exhibit time delays, ranging from delays of a few lines of signal to delays of several frames. Specifically, this can be described in the following aspects: (1) Denoising algorithm based on computer platform: This platform mainly uses CPU to run denoising algorithm to process images. Compared with other processing platforms, the computer platform has excellent denoising effect, but the denoising time is measured in seconds. (2) Denoising Counting Based on Embedded Platforms: Denoising algorithms process data by frame counting. Currently, the mainstream platforms include ARM and DSP platforms. The internal characteristic of DSP platforms is that their floating-point calculation speed is faster than other platforms, while ARM platforms are more focused on the input and output of control algorithm data streams.
[0003] (3) GPU-based methods: This platform mainly uses deep learning networks for image denoising, which has achieved more promising denoising results compared to filtering, model-based and traditional learning methods. GPU platforms, such as the NVIDIA A100 platform, are currently one of the popular platforms.
[0004] However, the problems with the existing technology platforms mentioned above can be summarized as follows: excessive latency or near-real-time processing, but the algorithm processing effect is not as good as that of computer platforms. Summary of the Invention
[0005] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a hardware implementation method for a spatiotemporal filtering system that combines image information. This method can suppress image noise while preserving as much detail information and target contours as possible in the image. Furthermore, it can effectively suppress the "trailing" phenomenon caused by moving targets in the image scene when there is a large change in the scene. The technical solution of this invention is: A hardware implementation method for a spatiotemporal filtering system incorporating image information includes the following steps: Step 1: Apply image downsampling and upsampling phase difference to each frame of the image to obtain multi-layer images at different resolutions, and obtain the preprocessed image of the original image. Step 2: Use edge filtering convolution kernels to process the data stream of each layer of the original image preprocessing map in a pipeline manner to obtain four-directional edge-filtered image video stream data; Step 3: Utilize the high-speed interaction channel between FPGA and DDR3 to obtain multi-frame image buffer data by reading and writing the filtered image video stream data of each layer; Step 4: Utilize the high-speed interaction channel between FPGA and DDR3 and the denoising formula between two frames, and combine the differential filtered image of the current frame with the motion-compensated result of the previous frame to perform motion compensation denoising on each layer of filtered image in the time domain. Step 5: The motion-compensated and denoised video stream is processed in a pipeline manner through a bilinear interpolation formula convolution kernel to obtain image data restored to the original resolution, and the video stream data of each layer is fused. Step 6: Use the edge filtering convolution kernel from Step 2 to process the fused image data in a pipeline manner to obtain spatially processed image and video stream data; Step 7: Using the clock parameters and image resolution of the spatially processed image video stream data, output it to the screen via the HDMI interface.
[0006] Furthermore, in step 1, each frame of the image is downsampled and layered, and then upsampled to obtain multi-layered images at different resolutions, thus obtaining a preprocessed image of the original image. Specifically: Step 1-1: Perform downsampling on each frame of image 6 times at different magnifications, namely 2x, 4x, 8x, 16x, 32x and 64x. For each downsampling layer, the original image input clock is downsampled by frequency division. The 6 sets of clocks after frequency division are used as the control clock for downsampling. The sampling of the video stream is triggered by the rising edge of the control clock, thus completing the overall data downsampling process. Steps 1-2: The downsampled image data enters a 2x2 upsampling window constructed using the Shiftram Generator IP core in Vivado. The upsampling algorithm is triggered by the rising edge of the clock, and the downsampled image data stream undergoes 2x bilinear interpolation upsampling within the 2x2 upsampling window. After output, the data is buffered in the FIFO Generator IP core. When 64x sampled data arrives, all six layers of data are output simultaneously for difference analysis. Subtracting two images with different blur levels at the same scale space yields difference maps D1, D2, D3, D4, D5, and D6 at different resolutions, specifically represented as follows:
[0007] In the formula, Indicates the first Layer differential pyramid downsampling image, Indicates the first Layered pyramid downsampling image, Indicates the first Layer-scale image restoration.
[0008] Furthermore, step 2 utilizes edge-filtering convolution kernels to perform pipelined processing on the data stream of each layer of the original image preprocessing map, obtaining four-directional edge-filtered image video stream data, specifically as follows: Step 2-1: Using the shiftram Generator IP core in Vivado, construct a 16×16 filtering algorithm window. The buffer size of each IP core is set to the number of row pixels of the current layer image. The original image preprocessing image video stream data is used as input data. Step 2-2: Calculate the time required for image and video stream processing in pipeline mode. Use the Shiftram Generator IP core in Vivado to build an image and video stream field signal buffer. Store the video stream field signal in the buffer. Steps 2-3: The video stream enters the 16*16 filtering algorithm window. The four-way edge filtering operator is used to perform calculations in the convolution window. The calculation process is carried out in a pipeline form inside the FPGA. After the calculation is completed, the filtered video data stream is output.
[0009] Furthermore, step 3 utilizes the high-speed interaction channel between the FPGA and DDR3 to obtain multi-frame image buffer data through reading and writing of the filtered image video stream data of each layer, specifically as follows: Step 3-1: Configure the FDMA_control module and FDMA in the FPGA. The FDMA_control module includes a FIFO. The FDMA_control module controls the DDR3 interface timing, writing FIFO data from the FPGA to DDR3 or reading data from DDR3 to the FPGA. The FDMA manages the FIFO and detects its storage status. When the FIFO meets the conditions for writing to DDR3, it actively initiates a write request to the FDMA. Step 3-2: In the initial state, the video stream data line signal, field signal, and video stream data first enter the FIFO. When the FIFO is full of a burst, a write request state will be triggered. The FDMA_control module sends a write DDR request signal areq to the FDMA and waits for the FDMA to reply with a write enable signal wr_en to complete the handshake response; once the FDMA sends back the write enable signal wr_en, it enters the write state. During write operation, the write DDR request signal areq and the write enable signal wr_en are both active. Image data is read from the FIFO in the FDMAcontrol module and then written to DDR3 memory via the AXI4 bus. Step 3-2: When the initial state is idle, it means that no image data data_img has been written to the FIFO; when the valid signal rises, it means that the image data in the FIFO is buffered, and at the same time, it enters the read request state; the valid signal is a data validity identification signal, which is used to tell the FIFO that the data now transmitted is valid data that can be stored. In read request state, the write enable signal wr_en is set to valid && ready, indicating that in read request state, the image data in the FIFO buffer is written to DDR3 memory via the AXI bus; when the last image data in a group of FIFO buffers is transferred, the system returns to the read initial state and the data stream stops.
[0010] Furthermore, step 4 utilizes the high-speed interaction channel between the FPGA and DDR3, as well as the denoising formula between two frames, and combines the differential filtered image of the current frame with the motion-compensated result of the previous frame to perform motion-compensated denoising on each layer of filtered image in the time domain, specifically as follows: The differential filter image direction parameters of the current frame processed in steps 4-1 and 2 are buffered in the FIFO. The read enable terminal of the FIFO is controlled by the row enable of the previous frame motion-compensated video stream data obtained by the FPGA reading DDR3 in step 3-2. After the read enable signal becomes high bit, the differential filter image is read from the FIFO terminal and processed simultaneously with the motion-compensated video stream data of one frame. Step 4-2, Current Frame Parameter Processing: The denoising formula is described using Verilog. In Step 4-1, the two video streams are denoised using temporal motion compensation based on the denoising formula, which is as follows:
[0011] in, This represents the result of the i-th layer of DOG motion compensation denoising for the n-th frame image. Indicates the first Image data after layer differential pyramid downsampling filtering. Represents the normalization parameters of the motion signal. Differential signal normalization parameters.
[0012] Furthermore, step 5 involves pipelined processing of the motion-compensated and denoised video stream through a bilinear interpolation kernel to obtain image data restored to its original resolution, and then fusing the video stream data from each layer. Specifically: Step 5-1: Construct five 2*2 upsampling windows using Shiftram. The depth of each window is the row effective pixel value of the current layer image resolution. These windows correspond to the 6 layers of video stream image data and are used for subsequent upsampling processing. Step 5-2: Calculate the time required for processing each layer of image and video streams, and use the ShiftRamGenerator IP core in Vivado to build a buffer for the image and video stream field signals, and store the video stream field signals in the buffer. Step 5-3: Before each layer of video stream enters the buffer, a 2*2 upsampling window is constructed. Inside the window, the upsampling calculation is performed using a bilinear interpolation algorithm. The upsampling calculation process is carried out in a pipelined manner inside the FPGA. After the calculation is completed, the filtered video data stream is output. Step 5-4: After each layer of filtered video data stream is restored to its original resolution, it enters the FIFO for data buffering. Write enable is the line valid signal of the video data stream, and read enable is the line valid signal of the lowest layer data after it has been restored to its original resolution. When the line valid signal of the lowest layer video in the current frame is pulled high to restore the original video resolution, the data of the other 5 layers are output from the FIFO together, and the 6 layers of data are added at the same time to obtain the fused video data stream.
[0013] Furthermore, step 7 utilizes the clock parameters and image resolution of the spatially processed image / video stream data to output it to the screen via the HDMI interface, specifically: Step 7-1: Based on the resolution of the image / video stream data processed in the spatial domain, construct the output interface for horizontal and vertical signals and output them. The output has six states, namely: S0 state: Idle state, waiting for a new output task; S1 state: Output FIFO data status; S2 state: Counting judgment state, used to determine whether the counter has reached the row count. If the row count has been reached, directly switch to S5 state; otherwise switch to S3 state. The counter is a logic counting module designed inside the FPGA to control the timing of HDMI row output, used to determine whether a row of image data has been output. S3 state: Horizontal blanking state. According to the timer, the horizontal blanking period is counted in one frame of the image. After the horizontal blanking period is counted, it is checked whether the amount of data read in the FIFO is sufficient for more than one row of data. If it is not sufficient, it transitions to the S4 state. S4 state: waiting state. When the amount of data read in the FIFO is sufficient to read more than one row of data, it transitions to S1 state and prepares to start outputting data. S5 state: End state. In this state, the next rising edge of the clock will cause the data output to re-enter the S0 state, ready to process the next output task. In state S0, the data stream buffering function is implemented. When the number of buffered data in the FIFO is greater than or equal to COL, it will enter state S1; COL represents the number of pixels in one row. In states S1 and S2, the new format data is output; after each line of data is output, the system will enter state S3. In state S3, a blanking period is generated between each row of image data; when the blanking period ends and the FIFO has not cached enough data for more than one row, it will enter state S4 and wait for the FIFO to cache the image data. After reading the ROW row data, it indicates that the format conversion of one frame of image data with a resolution of COL×ROW is complete, and the system enters state S5; at the beginning of the next clock cycle, the system will enter state S0 to prepare for the format conversion of a new frame of image data. Step 7-2: According to the HMDI interface protocol, set the relevant clock parameters, and connect the horizontal and vertical output signals of the output interface according to the video stream output in Step 7-1 to complete the processing and output of the video from inside the FPGA to the display screen.
[0014] The advantages of this invention compared to the prior art are: 1. Significant noise reduction effect: This invention can simultaneously handle multiple types of noise, with a significant noise reduction effect.
[0015] 2. For motion scenes: Traditional algorithms typically only process single-frame images, and their performance is poor in continuous-frame images. In contrast, this invention calculates the displacement of multiple frames through phase correlation to estimate the camera's motion parameters, thus ensuring denoising performance while remaining unaffected by camera motion.
[0016] 3. Minimal information loss: Traditional denoising algorithms may lose some edge information while reducing noise, especially when the noise is strong. This invention, however, has a certain degree of edge information protection, resulting in minimal loss of detail.
[0017] 4. Fast algorithm processing time: Compared with traditional CPU-based serial processing, FPGA, as a hardware device for parallel processing, can process image data simultaneously in a pipeline manner, thus enabling fast near real-time image processing operations.
[0018] 5. Providing a foundation for future chip integration: As a hardware circuit, FPGA is essentially a logic gate circuit, and can also be called an algorithm verification board. Therefore, algorithms implemented on FPGA boards can provide a foundation for future chip integration, enabling the algorithm to be further productized. Attached Figure Description
[0019] Figure 1 A flowchart illustrating the hardware implementation method of a spatiotemporal filtering system that incorporates image information.
[0020] Figure 2 This is a general block diagram of the data flow of the overall hardware system.
[0021] Figure 3 The flowchart shows the hardware implementation of downsampling differential pyramid filtering.
[0022] Figure 4 This is a schematic diagram of a convolution kernel for image FPGA hardware processing.
[0023] Figure 5 This is a flowchart of FPGA and DDR3 high-speed read / write process.
[0024] Figure 6 Hardware processing flowchart for parameter update module.
[0025] Figure 7 The hardware processing flowchart for the image fusion module.
[0026] Figure 8 Diagram of HDMI output interface.
[0027] Figure 9 This is a schematic diagram of a four-directional spatial domain edge filtering convolution kernel.
[0028] Figure 10 The result of the processing is shown in Figure (1).
[0029] Figure 11 The result of the processing is shown in Figure (2). Detailed Implementation
[0030] The specific embodiments of the present invention will now be described in further detail with reference to the accompanying drawings.
[0031] This invention provides a hardware implementation method for a spatiotemporal filtering system that incorporates image information. like Figure 1 , Figure 2 As shown, this invention proposes a hardware implementation method for a spatiotemporal filtering system that combines image information. Figure 3 As shown, this section illustrates the specific operation of the image pyramid in the hardware. The hardware implementation method of this invention includes the following steps: Step 1: Apply image downsampling and upsampling phase difference to each frame of the image to obtain multi-layer images at different resolutions, thus obtaining the preprocessed image of the original image; Step 1-1: Perform image downsampling at six different magnifications on each frame simultaneously, namely 2x, 4x, 8x, 16x, 32x and 64x. Each downsampling layer uses a frequency division module to downsample the original image input clock by a factor. The six sets of clocks after frequency division are used as the control clocks for the downsampling algorithm module. The sampling of the video stream is triggered by the rising edge of the control clock, thus completing the overall data downsampling process. Steps 1-2: The downsampled image data enters the 2x2 upsampling algorithm window built using the Shiftram Generator IP core in Vivado. The algorithm processing control section in the upsampling algorithm module is triggered by the rising edge of the clock to start the algorithm processing. The downsampled image data stream undergoes a 2x bilinear interpolation upsampling processing algorithm within the 2x2 upsampling window. After output, the data enters the FIFO Generator IP core for buffering. When 64x sampled data arrives, all six layers of data are simultaneously output to the difference section. The difference section subtracts two images with different blur levels in the same scale space to obtain difference images D1, D2, D3, D4, D5, and D6 at different resolutions, specifically represented as follows:
[0032] In the formula, Indicates the first Layer differential pyramid downsampling image, Indicates the first Layered pyramid downsampling image, Indicates the first Layer-scale image restoration; The operation process in the FPGA is as follows: The layer pyramid downsampled image is cached in the FIFO, waiting for the next layer. When the enable signal for layer-scale image restoration arrives, the two signals are output synchronously and subtracted to obtain the result. .
[0033] Step 2: Use edge filtering convolution kernels to process the data stream of each layer of the original image preprocessing map in a pipeline manner to obtain four-directional edge-filtered image video stream data.
[0034] Step 2-1: Using the Shiftram Generator IP core in Vivado, construct a 16*16 filtering algorithm window. The buffer size of each IP core is set to the number of pixels in the current layer image row. The original image preprocessing image video stream data is used as the input data for this module. Step 2-2: Calculate the time required for image and video stream processing in pipeline mode. Use the Shiftram Generator IP core in Vivado to build an image and video stream field signal buffer. Store the video stream field signal in the buffer. Steps 2-3: The video stream enters the 16*16 filtering algorithm window. The four-way edge filtering operator is used to perform calculations in the convolution window. The calculation process is carried out in a pipeline form inside the FPGA. After the calculation is completed, the filtered video data stream is output.
[0035] like Figure 4The diagram illustrates the principle of window spatial filtering. First, image data from one frame enters the on-chip buffer pixel by pixel. This buffer, located within the FPGA, consists of a shift register (shiftram IP) connected end-to-end. Simultaneously, data from the three RAMs is output into a set of reg-type registers of the same size as the convolution window. The data in these registers is then convolved with the filtering template to obtain the four-directional edge filtering result.
[0036] Step 3 utilizes the high-speed interaction channel between the FPGA and DDR3 to obtain multi-frame image buffer data by reading and writing the filtered image video stream data of each layer.
[0037] like Figure 5 As shown, the overall process of FPGA storing data into DDR includes the following steps: First, the VGA timing video stream data needs to be buffered in the FIFO of the FDMA control module. When a certain amount of data is buffered, the FDMA control sends a write request signal areq to the FDMA. The write request signal areq and the write enable signal wren sent by the DDR to the FDMA control via the FDMA form a handshake signal. When this handshake signal goes high simultaneously, the FDMA module starts working and writes the image data and data address into the DDR through the data channel. At the same time, the FDMA sends control signals such as brust type, lock type, and cache type to the DDR through the control signal channel.
[0038] The specific steps include: Step 3-1: Configure the FDMA_control module and FDMA in the FPGA. The FDMA_control module includes a FIFO. The FDMA_control module controls the DDR3 interface timing, writing FIFO data from the FPGA to DDR3 or reading data from DDR3 to the FPGA. The FDMA manages the FIFO and detects its storage status. When the FIFO meets the conditions for writing to DDR3, it actively initiates a write request to the FDMA. Step 3-2: In the initial state, the video stream data line signal, field signal, and video stream data first enter the FIFO. When the FIFO is full of a burst, a write request state is triggered; The FDMA_control module sends a write DDR request signal areq to the FDMA and waits for the FDMA to reply with a write enable signal wr_en, completing the handshake response. Once the FDMA sends back the write enable signal wr_en, the system enters the write state. During write operation, the write request signal areq and the write enable signal wr_en are both active. Image data is read from the FIFO in the FDMAcontrol module and then written to the DDR memory via the AXI4 bus. Step 3-3: Initially, the FIFO is in an idle state, indicating that no image data (data_img) has been written to it. When the valid signal rises, it indicates that the image data in the FIFO is buffered, and the FIFO enters a read request state. The valid signal is a data validity indicator signal, used to tell the FIFO that the data being transmitted is valid and can be stored. In read request mode, the write DDR enable signal wr_en is set to valid && ready, indicating that in read request mode, image data in the FIFO buffer is written to DDR3 memory via the AXI bus. When the last image data in a FIFO buffer is transferred, the system returns to the read initial state, and data transmission stops.
[0039] Step 4 utilizes the high-speed interaction channel between FPGA and DDR3, as well as the denoising formula between two frames, to perform motion compensation denoising on each layer of filtered image in the time domain, combining the differential filtered image of the current frame with the motion-compensated result of the previous frame.
[0040] The differentially filtered image direction parameters of the current frame processed in steps 4-1 and 2 are buffered in the FIFO. The read enable pin of the FIFO is controlled by the row enable signal from the motion-compensated video stream data of the previous frame obtained by the FPGA reading DDR3 in step 3-2. After the read enable signal becomes high-order, the differentially filtered image is read from the FIFO and processed simultaneously with the motion-compensated video stream data of the previous frame, such as... Figure 6 As shown; Step 4-2, Current Frame Parameter Processing: The denoising formula is described using Verilog. In Step 4-1, the two video streams are denoised using temporal motion compensation based on the denoising formula, which is as follows:
[0041] in, This represents the result of the i-th layer of DOG motion compensation denoising for the n-th frame image. Indicates the first Image data after layer differential pyramid downsampling filtering. Represents the normalization parameters of the motion signal. Differential signal normalization parameters.
[0042] Step 5 involves pipelined convolution of the motion-compensated and denoised video stream using bilinear interpolation kernels to restore the image data to its original resolution, and then fusing the video stream data from each layer, as shown below. Figure 7 As shown.
[0043] Step 5-1: First, five 2x2 upsampling windows were constructed using Shift+ram. The depth of each window was the row-level effective pixel value of the current layer's image resolution. These windows corresponded to six layers of video stream image data and were used for subsequent upsampling processing. Step 5-2: The time required for processing each layer of image and video streams was calculated, and a buffer for the image and video streaming field signals was constructed using the ShiftRamGenerator IP core in Vivado. The video streaming field signals were stored in the buffer to ensure the validity and stability of the data. Step 5-3: Each layer of video stream enters the previously constructed 2*2 upsampling window, and the upsampling calculation is completed within the window using a bilinear interpolation algorithm. This calculation process is performed in a pipelined manner within the FPGA, and the filtered video data stream is output after the calculation is completed. Taking the lowest layer data as an example, the original resolution is 640*512. The resolution of the lowest layer unsampled image is 20*16. After passing through a bilinear interpolation upsampling module, the same operation needs to be performed 4 more times to restore the original resolution. The number of times each layer passes through the upsampling module decreases accordingly. Step 5-4: After each layer of video stream is restored to its original resolution, it enters the FIFO for data buffering. The write enable is the line valid signal of the video stream data, and the read enable is the line valid signal of the lowest layer data after it has been restored to its original resolution. When the line valid signal of the lowest layer video in the current frame that has restored its original video resolution goes high, the data of the other 5 layers are output from the FIFO together, and the 6 layers of data are added together to obtain the fused video data stream.
[0044] like Figure 8 As shown in the diagram, this is the HDMI interface IP core. The VID_IN of this interface is the input image data stream, including IMG_DATA, IMG_FVAL, and IMG_LVAL. PXICLK_I is the pixel clock, which needs to be synchronized with the camera clock. PXLCLK_5X_I is the camera's five-times clock signal, which needs to be divided by the FPGA before transmission. After setting these port signals, the algorithm image processed by the hardware FPGA can be output through programming and power-on.
[0045] like Figure 9The diagram shows the convolution kernel of the four-way edge filtering algorithm. This diagram represents how a one-dimensional data flow in an FPGA constructs a two-dimensional image filtering window through the Shift Ram IP core. The main reason is that the data flow of this IP core is interconnected from beginning to end. After the data is output from the Shift Ram, part of the data flows to the register, and the other part flows to the next Shift Ram, thus serving as the data input for the second row of data in the window.
[0046] Step 6: Input the fused video stream data into the same spatial domain filtering module as in Step 2 for filtering to remove low-frequency noise and obtain the output video. Step 7 uses the clock parameters and image resolution of the spatially processed image video stream data to output to the screen via the HDMI interface; Step 7-1: Based on the processed video resolution, construct the output interface horizontal and vertical signals and build the output module. The output has six states, namely: S0 state: Idle state. The system is in an idle state, waiting for a new output task; S1 state: Output FIFO data state. Video stream data is output according to the new line and field technology; S2 state: Counting judgment state. Used to determine whether the overall counter has reached the row count. If the row count has been reached, proceed directly to S5 state; otherwise, proceed to S3 state. S3 state: Horizontal blanking state. The horizontal blanking period is timed within a frame of the image according to the internal timer. After the horizontal blanking period is completed, it checks whether the amount of data read in the FIFO is sufficient for more than one row of data. If not, it enters the S4 state. S4 state: Waiting state. When the amount of data to be read in the FIFO is sufficient for more than one row of data, the system enters the S1 state, ready to start outputting data; S5 state: End state. In this state, the next rising edge of the clock will cause the system to re-enter the S0 state, ready to process the next output task; In state S0, the data stream buffering function is implemented, and the data input from the previous module is placed into the FIFO of this module for image data buffering. When the count of buffered data in the FIFO is greater than or equal to COL (representing the number of pixels in one row), the system will enter state S1; In states S1 and S2, data in the new format is output. After each line of data is output, the system enters state S3. In state S3, the system generates a blanking period between each row of image data. When the blanking period ends and the FIFO does not have enough data for more than one row, the system will enter state S4, waiting for the FIFO to cache the image data. After reading the ROW row data, it indicates that the format conversion of one frame of image data with a resolution of COL×ROW is complete, and the system enters state S5. At the beginning of the next clock cycle, the system will enter state S0, ready to start the format conversion of a new frame of image data; Step 7-2: According to the HMDI interface protocol, set the relevant clock parameters, and connect the horizontal and vertical output signals of the output interface according to the video stream output in Step 7-1 to complete the processing and output of the video from inside the FPGA to the display screen.
[0047] This invention suppresses image noise while preserving as much detail and target outline as possible. Furthermore, in dynamic scenes or scenes with significant changes, it effectively suppresses the "trailing" phenomenon caused by moving objects in the image scene, such as... Figure 10 and Figure 11 As shown.
[0048] The parts of this invention not described in detail are common knowledge to those skilled in the art.
Claims
1. A hardware implementation method for a spatiotemporal filtering system incorporating image information, characterized in that... include: Step 1: Apply image downsampling and upsampling phase difference to each frame of the image to obtain multi-layer images at different resolutions, and obtain the preprocessed image of the original image. Step 2: Use edge filtering convolution kernels to process the data stream of each layer of the original image preprocessing map in a pipeline manner to obtain four-directional edge-filtered image video stream data; Step 3: Utilize the high-speed interaction channel between FPGA and DDR3 to obtain multi-frame image buffer data by reading and writing the filtered image video stream data of each layer; Step 4: Utilize the high-speed interaction channel between FPGA and DDR3 and the denoising formula between two frames, and combine the differential filtered image of the current frame with the motion-compensated result of the previous frame to perform motion compensation denoising on each layer of filtered image in the time domain. Step 5: The motion-compensated and denoised video stream is processed in a pipeline manner through a bilinear interpolation formula convolution kernel to obtain image data restored to the original resolution, and the video stream data of each layer is fused. Step 6: Use the edge filtering convolution kernel from Step 2 to process the fused image data in a pipeline manner to obtain spatially processed image and video stream data; Step 7: Using the clock parameters and image resolution of the spatially processed image video stream data, output it to the screen via the HDMI interface.
2. The hardware implementation method of a spatiotemporal filtering system combining image information according to claim 1, characterized in that: Step 1 involves downsampling and upsampling phase difference for each frame of image to obtain multi-layer images at different resolutions, thus obtaining the preprocessed original image. Specifically: Step 1-1: Perform downsampling on each frame of image 6 times at different magnifications, namely 2x, 4x, 8x, 16x, 32x and 64x. For each downsampling layer, the original image input clock is downsampled by frequency division. The 6 sets of clocks after frequency division are used as the control clock for downsampling. The sampling of the video stream is triggered by the rising edge of the control clock, thus completing the overall data downsampling process. Steps 1-2: The downsampled image data enters a 2x2 upsampling window constructed using the Shiftram Generator IP core in Vivado. The upsampling algorithm is triggered by the rising edge of the clock, and the downsampled image data stream undergoes 2x bilinear interpolation upsampling within the 2x2 upsampling window. After output, the data is buffered in the FIFO Generator IP core. When 64x sampled data arrives, all six layers of data are output simultaneously for difference analysis. Subtracting two images with different blur levels at the same scale space yields difference maps D1, D2, D3, D4, D5, and D6 at different resolutions, specifically represented as follows: In the formula, Indicates the first Layer differential pyramid downsampling image, Indicates the first Layered pyramid downsampling image, Indicates the first Layer-scale image restoration.
3. The hardware implementation method of a spatiotemporal filtering system combining image information according to claim 1, characterized in that: Step 2 utilizes edge-filtering convolution kernels to perform pipelined processing on the data stream of each layer of the original image preprocessing map, obtaining four-directional edge-filtered image video stream data, specifically: Step 2-1: Using the shiftram Generator IP core in Vivado, construct a 16×16 filtering algorithm window. The buffer size of each IP core is set to the number of row pixels of the current layer image. The original image preprocessing image video stream data is used as input data. Step 2-2: Calculate the time required for image and video stream processing in pipeline mode. Use the Shiftram Generator IP core in Vivado to build an image and video stream field signal buffer. Store the video stream field signal in the buffer. Steps 2-3: The video stream enters the 16*16 filtering algorithm window. The four-way edge filtering operator is used to perform calculations in the convolution window. The calculation process is carried out in a pipeline form inside the FPGA. After the calculation is completed, the filtered video data stream is output.
4. The hardware implementation method of a spatiotemporal filtering system combining image information according to claim 1, characterized in that: Step 3 utilizes the high-speed interaction channel between the FPGA and DDR3 to obtain multi-frame image buffer data through reading and writing of the filtered image video stream data of each layer, specifically as follows: Step 3-1: Configure the FDMA_control module and FDMA in the FPGA. The FDMA_control module includes a FIFO. The FDMA_control module controls the DDR3 interface timing, writing FIFO data from the FPGA to DDR3 or reading data from DDR3 to the FPGA. The FDMA manages the FIFO and detects its storage status. When the FIFO meets the conditions for writing to DDR3, it actively initiates a write request to the FDMA. Step 3-2: In the initial state, the video stream data line signal, field signal, and video stream data first enter the FIFO. When the FIFO is full of a burst, a write request state will be triggered. The FDMA_control module sends a write DDR request signal areq to the FDMA and waits for the FDMA to reply with a write enable signal wr_en to complete the handshake response; once the FDMA sends back the write enable signal wr_en, it enters the write state. During write operation, the write DDR request signal areq and the write enable signal wr_en are both active. Image data is read from the FIFO in the FDMAcontrol module and then written to DDR3 memory via the AXI4 bus. Step 3-2: When the initial state is idle, it means that no image data data_img has been written to the FIFO; when the valid signal rises, it means that the image data in the FIFO is buffered, and at the same time, it enters the read request state; the valid signal is a data validity identification signal, which is used to tell the FIFO that the data now transmitted is valid data that can be stored. In read request state, the write enable signal wr_en is set to valid && ready, indicating that in read request state, the image data in the FIFO buffer is written to DDR3 memory via the AXI bus; when the last image data in a group of FIFO buffers is transmitted, the system returns to the read initial state and the data stream stops transmitting.
5. The hardware implementation method of a spatiotemporal filtering system combining image information according to claim 4, characterized in that: Step 4 utilizes the high-speed interaction channel between the FPGA and DDR3, as well as the denoising formula between two frames, and combines the differential filtered image of the current frame with the motion-compensated result of the previous frame to perform motion-compensated denoising on each layer of filtered image in the time domain, specifically as follows: The differential filter image direction parameters of the current frame processed in steps 4-1 and 2 are buffered in the FIFO. The read enable terminal of the FIFO is controlled by the row enable of the previous frame motion-compensated video stream data obtained by the FPGA reading DDR3 in step 3-2. After the read enable signal becomes high bit, the differential filter image is read from the FIFO terminal and processed simultaneously with the motion-compensated video stream data of one frame. Step 4-2, Current Frame Parameter Processing: The denoising formula is described using Verilog. In Step 4-1, the two video streams are denoised using temporal motion compensation based on the denoising formula, which is as follows: in, This represents the result of the i-th layer of DOG motion compensation denoising for the n-th frame image. Indicates the first Image data after layer differential pyramid downsampling filtering. Represents the normalization parameters of the motion signal. Differential signal normalization parameters.
6. The hardware implementation method of a spatiotemporal filtering system combining image information according to claim 5, characterized in that: Step 5 involves pipelined convolution of the motion-compensated and denoised video stream using bilinear interpolation kernels to obtain image data restored to its original resolution, and then fusing the video stream data from each layer. Specifically: Step 5-1: Construct five 2*2 upsampling windows using Shiftram. The depth of each window is the row effective pixel value of the current layer image resolution. These windows correspond to the 6 layers of video stream image data and are used for subsequent upsampling processing. Step 5-2: Calculate the time required for processing each layer of image and video streams, and use the ShiftRamGenerator IP core in Vivado to build a buffer for the image and video stream field signals, and store the video stream field signals in the buffer. Step 5-3: Before each layer of video stream enters the buffer, a 2*2 upsampling window is constructed. Inside the window, the upsampling calculation is performed using a bilinear interpolation algorithm. The upsampling calculation process is carried out in a pipelined manner inside the FPGA. After the calculation is completed, the filtered video data stream is output. Step 5-4: After each layer of filtered video data stream is restored to its original resolution, it enters the FIFO for data buffering. Write enable is the line valid signal of the video data stream, and read enable is the line valid signal of the lowest layer data after it has been restored to its original resolution. When the line valid signal of the lowest layer video in the current frame is pulled high to restore the original video resolution, the data of the other 5 layers are output from the FIFO together, and the 6 layers of data are added at the same time to obtain the fused video data stream.
7. The hardware implementation method of a spatiotemporal filtering system combining image information according to claim 6, characterized in that: Step 7 utilizes the clock parameters and image resolution of the spatially processed image / video stream data to output it to the screen via the HDMI interface, specifically: Step 7-1: Based on the resolution of the image / video stream data processed in the spatial domain, construct the output interface for horizontal and vertical signals and output them. The output has six states, namely: S0 state: Idle state, waiting for a new output task; S1 state: Output FIFO data status; S2 state: Counting judgment state, used to determine whether the counter has reached the row count. If the row count has been reached, directly switch to S5 state; otherwise switch to S3 state. The counter is a logic counting module designed inside the FPGA to control the timing of HDMI row output, used to determine whether a row of image data has been output. S3 state: Horizontal blanking state. According to the timer, the horizontal blanking period is counted in one frame of the image. After the horizontal blanking period is counted, it is checked whether the amount of data read in the FIFO is sufficient for more than one row of data. If it is not sufficient, it transitions to the S4 state. S4 state: waiting state. When the amount of data read in the FIFO is sufficient to read more than one row of data, it transitions to S1 state and prepares to start outputting data. S5 state: End state. In this state, the next rising edge of the clock will cause the data output to re-enter the S0 state, ready to process the next output task. In state S0, the data stream buffering function is implemented. When the number of buffered data in the FIFO is greater than or equal to COL, it will enter state S1; COL represents the number of pixels in one row. In states S1 and S2, the new format data is output; after each line of data is output, the system will enter state S3. In state S3, a blanking period is generated between each row of image data; when the blanking period ends and the FIFO has not cached enough data for more than one row, it will enter state S4 and wait for the FIFO to cache the image data. After reading the ROW row data, it indicates that the format conversion of one frame of image data with a resolution of COL×ROW is complete, and the system enters state S5; at the beginning of the next clock cycle, the system will enter state S0 to prepare for the format conversion of a new frame of image data. Step 7-2: According to the HMDI interface protocol, set the relevant clock parameters, and connect the horizontal and vertical output signals of the output interface according to the video stream output in Step 7-1 to complete the processing and output of the video from inside the FPGA to the display screen.