Video image real-time noise reduction system based on FPGA

Abstract

The invention belongs to the field of digital image processing, and discloses an FPGA-based video image real-time noise reduction system, which comprises an OV5640 image sensor, an image acquisition module, an image processing module, an image caching module, an image transmission control module and display equipment. Wherein the OV5640 is responsible for collecting digital image data in real time; the image acquisition module is mainly used for completing initial configuration of an image sensor and receiving image data according to a data protocol format; the image processing module is used for realizing graying processing and filtering operation of the image; the image caching module is used for caching image data and writing the cached data into an SDRAM (Synchronous Dynamic Random Access Memory); and the image transmission control module transmits the image data to display equipment for real-time display by means of a data transmission protocol. According to the invention, a modular design and an assembly line technology are adopted, the real-time performance of the video noise reduction system is improved by utilizing the parallelism characteristic of the FPGA, and the occupancy rate of FPGA logic resources is reduced.

Description

Technical Field

[0001] The invention relates to the field of digital image processing, and in particular to a real-time video image noise reduction system based on FPGA. Background Art

[0002] As a carrier of external information, images are an important means for humans to collect, transmit and store information, so the collection, transmission and processing of image information are of great research significance. Digital images are usually interfered and affected by various external factors during the collection and transmission process, and are mixed with some invalid information. When performing advanced operations such as object recognition and detection, these invalid information will affect the relevant operation process and the operation results of the algorithm, causing the final image quality to deteriorate. Therefore, before executing advanced algorithms such as target recognition, it is necessary to filter the digital image to remove the interference information and invalid information.

[0003] Most of the interference to digital images during input and acquisition comes from various noises. Image noise mainly comes from image sensor noise and noise caused by channel transmission errors. In order to suppress or eliminate noise in the image, it is necessary to filter the image. At present, image noise reduction algorithms can be implemented by software or hardware. In contrast, the real-time performance of software is poor, and it is impossible to achieve real-time filtering and noise reduction of high-resolution images. However, in terms of hardware implementation, ASIC hardware is completely customized and cannot be modified. The functions that can be realized are relatively single and the versatility is poor. DSP has strong digital signal processing capabilities, but it uses a serial instruction system, which will be subject to certain limitations in communication. In contrast, FPGA uses a parallel instruction system, which can process more data information in the same time and has better real-time performance; FPGA is a programmable and reconfigurable device, which can realize more functions without changing the hardware structure, and has great advantages in flexibility. Summary of the invention

[0004] In order to solve the above problems in this field, the present invention proposes a real-time video image denoising system based on FPGA, which realizes real-time denoising of high frame rate and high resolution digital video stream, thereby meeting the real-time and reliability requirements of Gaussian filtering denoising algorithm.

[0005] The technical solution adopted by the present invention is as follows: a real-time video image noise reduction system based on FPGA, comprising:

[0006] OV5640 image sensor, image acquisition module, image processing module, image cache module, image transmission control module and display device;

[0007] The OV5640 image sensor is used to collect digital image data in real time and transmit the digital image data to the image acquisition module;

[0008] The image acquisition module is used to receive the digital image data collected by the OV5640 image sensor according to a specified data protocol format and transmit it to the image processing module;

[0009] The image processing module includes a grayscale processing module and a Gaussian filtering module, which are used to complete grayscale conversion and real-time noise reduction of digital image data;

[0010] The image cache module includes an SDRAM FIFO control module and an SDRAM controller module, and implements image data caching by writing the digital image data processed by the image processing module into the SDRAM;

[0011] The image transmission control module is used to transmit the digital image data of the image buffer module to the display device according to the data format supported by the display device;

[0012] The display device is used to display the processed image data in real time.

[0013] In the image processing module, a 3×3 pixel matrix cache module is added between the grayscale processing module and the Gaussian filter module; the pixel matrix cache module is a dual-end RAM IP core, including RAM1 and RAM0, which are respectively used to cache the pixel data of the first two rows, and the latest written data is used as the current row; after the data of the current two rows are read, the data of the second row is written into the first row, and then the data of the current row is written into the second row, so as to realize the dynamic update of the data in the RAM.

[0014] In the image cache module, the SDRAM FIFO control module encapsulates the SDRAM as a FIFO interface; the SDRAM controller module controls the working state of the SDRAM through a working state machine; the SDRAM FIFO control module calls the asynchronous FIFO IP core of the Quartus platform; converts the row and column addresses of the SDRAM into 24-bit linear addresses; and opens two storage spaces of the same size in the SDRAM to cache image data to realize ping-pong operation.

[0015] The SDRAM controller module includes an SDRAM state control module, an SDRAM data module and an SDRAM instruction module;

[0016] The SDRAM state control module uses two three-stage linear state machines to respectively control the SDRAM power-on initialization process and the SDRAM refresh, read and write process;

[0017] The SDRAM instruction module receives the initialization state and working state information sent by the SDRAM state control module, and assigns instruction control signals corresponding to different states to the signal lines and address lines of the SDRAM according to the instruction table in the SDRAM data manual, and outputs control instructions to the SDRAM;

[0018] The SDRAM data module controls the bidirectional data bus of the SDRAM according to the status information sent by the SDRAM status control module; when in the read operation state, it stores the data read from the SDRAM; when in the write operation state, it outputs the data to be written to the SDRAM data bus.

[0019] The Gaussian filter module selects a Gaussian filter template of σ=0.8, as shown in the following formula:

[0020]

[0021] The Gaussian filter module generates a 3×3 window of pixel data.

[0022] The pixel matrix is ​​convolved with the Gaussian filter template to obtain the following formula:

[0023]

[0024] Among them, f(x-1,y-1)~f(x+1,y+1) represent each pixel point in the 3×3 pixel matrix; driven by the clock signal and the clock enable signal, all operations in the braces in the formula are completed using two clock cycles; then, the operation result is shifted right by 4 bits using another clock cycle to obtain the final pixel value.

[0025] Beneficial effects of the invention: The invention provides a real-time video image noise reduction system based on FPGA, which can collect image information in real time through the OV5640 image sensor, and can realize the function of real-time video image noise reduction with low power consumption and logic resource occupancy rate, and finally can display the final processing result in real time through the display device. The invention effectively improves the real-time performance of the noise reduction system and reduces the hardware resource occupancy rate, and can be applied to high-resolution, high-frame-rate, and high-real-time application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is an overall structural diagram of the real-time video image noise reduction system based on FPGA provided by the present invention;

[0027] Figure 2 It is a workflow diagram of the image processing module in the present invention;

[0028] Figure 3This is the row-field timing diagram of the OV5640 image sensor;

[0029] Figure 4 It is a schematic diagram of the working principle of grayscale conversion;

[0030] Figure 5 It is a schematic diagram of the structural principle of the pixel matrix cache;

[0031] Figure 6 This is the functional structure diagram of the SDRAM controller module. DETAILED DESCRIPTION

[0032] The present invention will be further explained and illustrated in conjunction with the accompanying drawings.

[0033] like Figure 1 As shown, the FPGA-based real-time video image noise reduction system proposed in the present invention includes: an OV5640 image sensor, an image acquisition module, an image processing module, an image cache module, an image transmission control module and a display device. Among them, the image processing module includes a grayscale processing module and a Gaussian filter module.

[0034] The OV5640 image sensor is used to collect digital image data in real time and transmit the data to the image acquisition module;

[0035] The image acquisition module is used to receive image data acquired by the OV5640 image sensor according to a specified data protocol format;

[0036] The image processing module includes a grayscale processing module and a Gaussian filtering module, which are used to complete grayscale conversion and real-time noise reduction of the input image;

[0037] The image cache module includes an SDRAMFIFO control module and an SDRAM controller module, and implements image data caching by writing image data into SDRAM;

[0038] The image transmission control module is used to transmit the processed image data to the display device in a data format supported by the display device;

[0039] The display device is used to display the processed image data in real time.

[0040] The grayscale processing module is used to convert the data in RGB565 format into the Ycbcr format suitable for the digital video system, retaining 8 bits of grayscale data. The grayscale conversion formula is shifted and enlarged, and after the conversion is completed, it is shifted and reduced to obtain a grayscale image.

[0041] The Gaussian filter module is used to generate a pixel data window; perform Gaussian filtering on the input image data to suppress noise signals in the image;

[0042] The pixel data window is a 3×3 window; a pixel data window of corresponding size is generated by calling the RAM IP core;

[0043] The image cache module controls the processes of SDRAM power-on initialization, refresh, read and write operations through a three-stage linear state machine.

[0044] The specified data protocol format is the IIC protocol; the image acquisition module initializes and configures the OV5640 image sensor according to the IIC protocol, so that the OV5640 image sensor outputs image data in the RGB565 format; after the initialization is completed, the image acquisition module collects data according to the specified timing, and converts the 8-bit data into 16-bit RGB565 format data through the serial-to-parallel conversion circuit. In the present invention, the system first collects video image information through the OV5640 image sensor and transmits it to the image acquisition module. After the system is powered on, the image acquisition module initializes and configures the OV5640 according to the IIC protocol, so that the sensor outputs image data in the RGB565 format. After the initialization is completed, the image acquisition module Figure 2 The OV5640 shown collects data in timing and converts the 8-bit data into 16-bit RGB565 format data through a serial-to-parallel conversion circuit.

[0045] The specified timing is that the rising edge of the frame synchronization signal represents the start signal of each frame of the image, and the transmission of a frame of the image is completed between two rising edges. When the line synchronization signal becomes high, the sensor starts to output valid image data. In the output format of RGB565, each pixel is composed of 16 bits of data, and one pixel is output in two pixel clock cycles.

[0046] The image acquisition module comprises an IIC configuration module, an IIC driving module and a data acquisition module.

[0047] The IIC configuration module is used to store the address and data of the register to be configured of OV5640, configure the register through the case statement, and also complete the control of the initialization process.

[0048] The IIC driver module is used to drive the IIC bus and provide a clock signal for the aforementioned IIC configuration module.

[0049] The data acquisition module acquires image data according to a previously set data format.

[0050] The structure diagram of the image processing module is as follows: Figure 3 As shown in Figure 1, it includes grayscale processing, pixel matrix cache, Gaussian filtering and other links. First, the grayscale processing module follows Figure 4The method shown converts the RGB565 format data into the Ycbcr format for the digital video system, retaining 8 bits of grayscale data. Considering that Verilog does not support floating-point operations, in order to avoid occupying too many hardware resources, the grayscale conversion formula is shifted and enlarged, and then shifted and reduced after the conversion is completed to obtain a grayscale image; the final grayscale value formula is shown in formula (1):

[0051] Y=(77R+150G+29B)>>8 (1)

[0052] Among them, Y represents the grayscale component in the YCbCr format; R, G, and B represent the red, green, and blue components in the RGB565 format respectively; ">>" represents a right shift operation, and ">>8" means shifting the result right by 8 bits. Before performing Gaussian filtering, the pixel matrix cache is implemented by calling the dual-end RAM IP core. The specific structural principle is as follows Figure 5 As shown. RAM1 and RAM0 are used to cache the pixel data of the first two rows respectively, and the data just written is used as the current row. After the data of the current two rows are read, the data of the second row is written to the first row, and then the data of the current row is written to the second row, so as to realize the dynamic update of the data in RAM.

[0053] The present invention uses a Gaussian filter template with σ=0.8, as shown in the following formula:

[0054]

[0055] By convolving the pixel matrix with the Gaussian filter template, we can get the following formula:

[0056]

[0057] Where f(x-1, y-1) to f(x+1, y+1) represent the pixels in the 3×3 pixel matrix. Driven by the clock signal and the clock enable signal, all operations in the brackets in the formula are completed in two clock cycles. Then, the operation result is shifted right by 4 bits in another clock cycle to obtain the final pixel value.

[0058] The image cache module uses a state machine to complete the control of a series of operations such as SDRAM initialization, data reading and writing. The specific functional structure diagram is as follows Figure 6 shown.

[0059] The image transmission control module outputs corresponding control signals (such as backlight signals, horizontal and vertical synchronization signals, etc.) and 16-bit pixel data according to the display timing of the display device, and provides coordinate information and pixel data information corresponding to the pixel points.

[0060] In the present invention, FPGA is used as a development platform, and Intel Cyclone IV series development board is used to perform real-time grayscale processing and Gaussian filtering on a 50-frame digital video stream with a resolution of 640×480, and finally display it in real time through a display device.

Claims

1. A real-time video image noise reduction system based on FPGA, characterized in that: include: OV5640 image sensor, image acquisition module, image processing module, image cache module, image transmission control module and display device; The OV5640 image sensor is used to collect digital image data in real time and transmit the digital image data to the image acquisition module; The image acquisition module is used to receive the digital image data collected by the OV5640 image sensor according to a specified data protocol format and transmit it to the image processing module; The image processing module includes a grayscale processing module and a Gaussian filtering module, which are used to complete grayscale conversion and real-time noise reduction of digital image data; The image cache module includes an SDRAMFIFO control module and an SDRAM controller module, and realizes image data caching by writing the digital image data processed by the image processing module into the SDRAM; The image transmission control module is used to transmit the digital image data of the image buffer module to the display device according to the data format supported by the display device; The display device is used to display the processed image data in real time.

2. The FPGA-based real-time video image noise reduction system according to claim 1, characterized in that: In the image processing module, a 3×3 pixel matrix buffer module is added between the grayscale processing module and the Gaussian filter module; The pixel matrix cache module is a dual-end RAM IP core, including RAM1 and RAM0, which are used to cache the pixel data of the first two rows respectively, and the latest written data is used as the current row; After the data of the current two rows are read, the data of the second row is written into the first row, and then the data of the current row is written into the second row, so as to realize the dynamic update of the data in the RAM.

3. The real-time video image noise reduction system based on FPGA according to claim 1, characterized in that: In the image cache module, the SDRAM FIFO control module encapsulates the SDRAM as a FIFO interface; the SDRAM controller module controls the working state of the SDRAM through a working state machine; the SDRAM FIFO control module calls the asynchronous FIFO IP core of the Quartus platform; converts the row and column addresses of the SDRAM into 24-bit linear addresses; and opens two storage spaces of the same size in the SDRAM to cache image data to realize ping-pong operation.

4. The FPGA-based real-time video image noise reduction system according to claim 3, characterized in that: The SDRAM controller module includes an SDRAM state control module, an SDRAM data module and an SDRAM instruction module; The SDRAM state control module uses two three-stage linear state machines to respectively control the SDRAM power-on initialization process and the SDRAM refresh, read and write process; The SDRAM instruction module receives the initialization status and working status information sent by the SDRAM state control module, and assigns the instruction control signals corresponding to different states to the signal lines and address lines of the SDRAM according to the instruction table in the SDRAM data manual, and outputs the control instructions to the SDRAM; the SDRAM data module controls the bidirectional data bus of the SDRAM according to the status information sent by the SDRAM state control module; when in the read operation state, it stores the data read from the SDRAM; when in the write operation state, it outputs the data to be written to the SDRAM data bus.

5. The FPGA-based real-time video image noise reduction system according to any one of claims 1 to 4, characterized in that: The Gaussian filter module selects a Gaussian filter template of σ=0.8, as shown in the following formula:

6. The FPGA-based real-time video image noise reduction system according to claim 5, characterized in that: The Gaussian filter module generates a 3×3 window of pixel data.

7. The FPGA-based real-time video image noise reduction system according to claim 6, characterized in that: The pixel matrix is ​​convolved with the Gaussian filter template to obtain the following formula: Among them, f(x-1,y-1)~f(x+1,y+1) represent each pixel point in the 3×3 pixel matrix; driven by the clock signal and the clock enable signal, all operations in the braces in the formula are completed using two clock cycles; then, the operation result is shifted right by 4 bits using another clock cycle to obtain the final pixel value.

Images