A Night-time Video Image Enhancement System Based on FPGA + ARM

Abstract

The present invention discloses a night video image enhancement system based on FPGA + ARM, which includes four parts: an image acquisition unit, an arithmetic acceleration unit, a central control processing unit, and a display unit. The arithmetic acceleration unit utilizes the parallel processing and high-frequency characteristics of the FPGA to implement three different-scale Gaussian filters after simplification. The central control processing unit utilizes the strong logical operation ability and simple transplantation characteristics of the ARM to complete the more complex adaptive histogram equalization algorithm part and logical control. The video image acquisition unit of the present invention supports DVP and Cameralink interfaces. The arithmetic acceleration unit utilizes the parallel characteristics of the FPGA to improve the calculation rate. The central control unit performs complex logical and arithmetic operations through the ARM, and finally outputs the result through the display unit. Compared with the traditional system, the present invention can more accurately display the edge information of night images, and at the same time meets the real-time requirements of the system through the combination of FPGA + ARM.

Description

Technical Field

[0001] The present invention relates to the technical field of video image processing, in particular to nighttime intersection image monitoring, and specifically to a nighttime video image enhancement system based on FPGA+ARM and a construction method thereof. Background Art

[0002] Monitoring and night vision play an important role in the fields of road safety, railway supervision, and military detection. As people's living materials become more abundant, their nighttime activities become more frequent, and nighttime road safety has become an important issue that needs to be addressed urgently. Under traditional solutions, nighttime imaging systems usually use low-light cameras and thermal infrared cameras. Low-light cameras amplify some visible light into the infrared band, and infrared cameras convert the infrared spectrum of the target's thermal energy into a visible image. The imaging results of both methods will lose a lot of original details. In addition, although some improvements to the algorithm can improve the clarity of nighttime images, the resulting computational complexity also increases exponentially. To this end, the present invention proposes a method that can reproduce image details at night or under low-light conditions, and builds a video image enhancement system suitable for embedded devices. Summary of the Invention

[0003] The main technical problem addressed by this invention is to build a nighttime video image enhancement system suitable for embedded devices. This system improves upon the existing Retinex algorithm, enhancing image detail while maintaining image brightness. To address this technical problem, the present invention provides an FPGA+ARM-based nighttime video image enhancement system and its construction method.

[0004] A nighttime video image enhancement system based on an FPGA and ARM processor utilizes an FPGA and ARM architecture at its core. The FPGA is used to implement the Gaussian filtering process, taking on computational tasks that require simple but high computation times, and serving as the system's computational acceleration unit. The computational acceleration unit, with the FPGA as its core, implements the multi-scale Retinex algorithm. Before implementing the multi-scale Retinex algorithm in Verilog, the algorithm is simplified, specifying low, medium, and high format sizes. This simplification includes expanding the floating-point numbers required for the computation to integers and using pipelines to improve computational efficiency.

[0005] The image stretching algorithm is implemented using an ARM processor, serving as the system's central control processing unit. Adaptive histogram equalization is used to perform grayscale stretching on image data processed by the multi-scale Retinex algorithm. Histogram correction is performed by calculating a distribution function transformation, increasing the spacing between grayscale values in the original image and highlighting image details.

[0006] The night video image enhancement system further includes a video image acquisition unit and a display unit.

[0007] The video image acquisition unit includes video image acquisition devices, and converts the acquired image data into digital signals and transmits them to the operation acceleration unit with FPGA as the core. Among them, the video image acquisition unit supports two image device interfaces, DVP and Cameralink, and can receive various forms of acquisition device signals.

[0008] The display unit is used to convert the data processed by the ARM into an image and display it in the display unit, and then the enhanced image can be obtained. By displaying the images frame by frame, the enhanced video can be obtained. The central control processing unit with ARM as the core will transmit the processed image data to the display unit. The display unit supports connecting to a standard display through an HDMI interface or connecting to a PC through a gigabit network port.

[0009] The data acquired by the video image acquisition unit is received by the FPGA, and communication is carried out with the ARM through the Camera Sensor Interface (CSI). The image data received by the ARM is preprocessed and then directly output to the display device for display.

[0010] A construction method of a night video image enhancement system based on FPGA+ARM includes the following steps:

[0011] Step 1: Design a video image enhancement algorithm suitable for the embedded system;

[0012] Use the multi-scale Retinex algorithm as the core of image enhancement, and perform operations on the image at three scales: low, medium, and high.

[0013] Step 2: Design the video image acquisition unit;

[0014] The video image acquisition unit includes video image acquisition devices, and converts the acquired image data into digital signals and transmits them to the operation acceleration unit with FPGA as the core. Among them, the video image acquisition unit supports two image device interfaces, DVP and Cameralink, and can receive various forms of acquisition device signals.

[0015] Step 3: Analysis and simplification of the multi-scale Retinex algorithm;

[0016] The operation acceleration unit with FPGA as the core implements the multi-scale Retinex algorithm. Before implementing the multi-scale Retinex algorithm using Verilog, it is first simplified, and the format sizes of three sizes, low, medium, and high, are specified. The simplification includes expanding the floating-point numbers required for the operation into integers and using a pipeline to improve the operation efficiency.

[0017] Step 4: Use the image stretching algorithm to perform grayscale stretching on the image processed by the multi-scale Retinex algorithm;

[0018] Adaptive histogram equalization is used to perform grayscale stretching on the image data processed by the multi-scale Retinex algorithm. The histogram is corrected by calculating the transformation of the distribution function, so that the spacing between the gray values of the original image is increased, thereby highlighting the details of the image. An ARM-based central control unit is used to implement the image stretching algorithm.

[0019] Step 5, Transplantation of the multi-scale Retinex algorithm and the adaptive histogram equalization algorithm.

[0020] Step 6, Data interaction between the FPGA and the ARM. The FPGA transmits the processed data and the generated line and field signals to the processing unit in the ARM through the CSI bus.

[0021] Step 7: Convert the data processed by the ARM into an image through the display unit and display it in the display unit to obtain the enhanced image. By displaying the image frame by frame, the enhanced video can be obtained. The display unit supports connecting display devices through HDMI interface and gigabit network interface.

[0022] Furthermore, in Step 5, the multi-scale Retinex algorithm and the adaptive histogram equalization algorithm are transplanted. First, MATLAB is used to preliminarily implement and optimize the algorithm. Since the workload of directly implementing the algorithm in hardware language is too large, it is inconvenient to debug the input parameters in the algorithm, and the running and compilation time is long and the running results are not intuitive, so MATLAB is first used to preliminarily implement the algorithm. And MATALB is based on C language and has many basic functions, which is convenient for the transplantation of hardware language.

[0023] Since the FPGA is based on Verilog language and itself does not support the direct operation of MATLAB, it is necessary to complete the conversion of the algorithm between MATLAB and Verilog language so that the algorithm can run in the hardware system. Therefore, it is necessary to perform the conversion between MATLAB language and hardware language.

[0024] When implementing adaptive histogram equalization in C language, in order to reduce the excessive occupation of internal resources of the ARM, a four-byte float type is used instead of an eight-byte double type to reduce the memory occupation and improve the operation efficiency.

[0025] The beneficial effects of the present invention are as follows:

[0026] The video image acquisition unit of the present invention supports DVP and Cameralink interfaces. The operation acceleration unit utilizes the parallel characteristics of the FPGA to improve the calculation rate. The central control unit performs complex logical and arithmetic operations through the ARM, and finally outputs the results through the display unit. Compared with the traditional system, the present invention can more accurately display the edge information of night images, and at the same time meets the real-time requirements of the system through the combination of FPGA + ARM. Description of the Drawings

[0027] Figure 1 : Block diagram of the hardware processing system;

[0028] Figure 2 : Flowchart of the 5×5 filter processing;

[0029] Figure 3 : Interaction between FPGA and ARM;

[0030] Figure 4 : DVP interface design;

[0031] Figure 5 : Experimental effect diagram of image enhancement. Detailed Implementation Manner

[0032] The technical solution of the present invention will be described in more detail below in conjunction with the drawings and embodiments.

[0033] A construction method for a night video image enhancement system based on FPGA + ARM includes the following steps:

[0034] Step 1, design a video image enhancement algorithm suitable for the embedded system.

[0035] The collected image is divided into a reflection part and an illumination part. Among them, the reflection part is used to describe the high-frequency information of the original image, such as the edge contour and texture that need to be retained as much as possible during the processing; the illumination part describes the low-frequency information of the image and reflects the interference of the incident light on the image, which should be separated as much as possible during the processing.

[0036] Let the night vision image obtained by acquisition be I(x, y), the illumination image be L(x, y), and the reflected part be represented by R(x, y), then there is:

[0037] I(x, y) = R(x, y) * L(x, y) (1)

[0038] Since the human eye perception system perceives the light intensity in an exponential form, taking the logarithm of both sides of Equation (1) is sufficient. The single-scale Retinex algorithm has obvious advantages in image edge sharpening, dynamic range compression, and color constancy. Its mathematical form is:

[0039]

[0040] where \(i\) represents the color channel, \(r\) i (x,y) represents the output of the \(i\)-channel of the single-scale Retinex algorithm, \(*\) represents the convolution process, and \(F(x,y)\) represents the Gaussian function, which can be expressed as:

[0041]

[0042] where \(\sigma\) is the scale parameter of the single-scale Retinex algorithm, \(1 / 2\pi\sigma\) 2 is the normalization factor. The selected Gaussian function \(F(x,y)\) in the formula is a low-pass function. Therefore, the result obtained by convolving \(F(x,y)\) with \(I(x,y)\) corresponds to the low-frequency part of the original image. Subtracting the low-frequency part from the original image, the obtained result is the high-frequency information in the original image \(I(x,y)\). And this part of the high-frequency information belongs to the part that is easy for the human eye to capture, so the purpose of enhancing the edge information is achieved.

[0043] The single-scale Retinex algorithm cannot achieve a good balance between color fidelity and detail information restoration. After using a smaller scale parameter \(\sigma\), there may be a halo phenomenon in the image. While increasing \(\sigma\) reduces the occurrence of the halo phenomenon, it cannot achieve the expected detail restoration effect. Therefore, the improved multi-scale Retinex algorithm is used as the video image enhancement algorithm. The multi-scale Retinex algorithm is based on the single-scale and obtains the final calculation result by weighted filtering operations on the original image at low, medium, and high scales.

[0044] Step 2, Design of the video image acquisition unit.

[0045] The video image acquisition unit includes an image acquisition device and converts the acquired image data into digital signals and transmits them to the operation acceleration unit for processing. In the present invention, the video image acquisition unit is connected to the image acquisition device in the form of a board card, and can support a variety of image device interfaces, including the digital video interface DVP and the Cameralink interface. The DVP interface can directly output digital signals with the image acquisition device without data conversion and can be directly connected to the FPGA, as shown in the appendix Figure 4 . The Cameralink interface is a commonly used image acquisition interface for industrial cameras. It transmits signals using LVDS (low-voltage differential signals). When in use, it is necessary to convert the LVDS data into 28-bit CMOS / TTL data, and at the same time convert the LVDS clock signal into a single-ended CMOS / TTL clock signal for output, and connect to the FPGA to transmit the acquired image data.

[0046] Step 3, Optimization and implementation of the multi-scale Retinex algorithm based on FPGA.

[0047] An operation acceleration unit with FPGA as the core is used to implement the multi-scale Retinex algorithm. It should be noted that since the number of multipliers inside the FPGA is limited and too many multiplications will make the circuit very complex, the multi-scale Retinex algorithm cannot be directly used and needs to be optimized.

[0048] First, the FPGA processes the received data and converts it into a digital image matrix D, which participates in the subsequent Gaussian filtering. Then, a filter with an appropriate size is selected for the used Gaussian filtering function. Specifically, in this embodiment, three scales of 5×5, 15×15, and 30×30 are used as three different Gaussian filters. It should be noted that the present invention includes but is not limited to the division of these three specific scales, and modifications to the size should still be within the protection scope of the present invention. For the 5×5 Gaussian filter, first use MATLAB to obtain the required discrete Gaussian template, which is a 5×5 floating-point matrix, and expand it to an integer matrix B 5x5 . When performing Gaussian filtering, the Gaussian template B 5x5 is split into a horizontal one-dimensional Gaussian template B1 and a vertical one-dimensional Gaussian template B2, and performs a convolution operation with the digital image matrix D received by the FPGA, that is:

[0049] B 5×5 = B1×B2 (4)

[0050] When the FPGA caches the digital image matrix D, taking 5 rows as the processing unit, the data of the first 5 rows is cached in the first-level pipeline. When caching the 6-10th rows, the cached results of the first 5 rows enter the second-level pipeline. This not only improves the computing efficiency of the hardware but also saves the arithmetic operation resources inside the FPGA.

[0051] Specifically, for the pipeline design, in the first-level pipeline, the data of the first 5 rows is first cached and saved in 5 memories RAM. In the second-level pipeline, the first-column elements of the saved 5 rows of data are taken out and multiplied by the vector of B1, and the obtained result is stored in the register r11, as shown in the appendix Figure 2 . While saving the calculation results of the previous level in the third-level pipeline, calculate the vector multiplication result of the second column in the RAM and the Gaussian template B1, and save it in another register r22. And so on. After six levels of pipelines, the product r51~r55 of a 5-column data and B1 is obtained, and it is multiplied by B2 and output at the seventh clock.

[0052] For the 15×15 filter, its calculation process is roughly the same as that of the 5×5 one, except for the number of multipliers and memories required, and a 15×15 Gaussian template B is needed 15×15, calculated by Matlab, is also decomposed into two parts B3 and B4. During the calculation, the first-level pipeline caches the first 15 rows of data in D. The second-level pipeline multiplies the first column elements of the 15 rows with B3 in a vector multiplication, and the result is stored in register r11. While the third-level pipeline saves r11 to r21, it multiplies the cached data of the second column with the first Gaussian template in a vector multiplication, and the result is stored in r22. And so on. After 16 clock cycles, the calculation results of all 15 rows of data with B3 are obtained. Then, a one-dimensional vector composed of 15 results is multiplied with B4 in a vector multiplication to obtain the final result and output it.

[0053] The calculation idea of the 30×30 filter is roughly the same as the previous two, and the difference is that the required Gaussian template is B of 30×30 30×30 , and the processing steps of the pipeline are the same.

[0054] Step 4, implementation of the ARM image stretching algorithm. After the operation and processing by the FPGA, due to its strong logical operation ability, the ARM is used for the optimization after image enhancement and serves as the central control unit of the system. The enhanced image processed by the FPGA will have results such as lower contrast, the overall image being darker, and the enhancement effect being not obvious. After being processed by the ARM, the enhanced picture can be better displayed. Before running, the source code of the image stretching and enhancement algorithm is compiled using a cross-compiler in the Linux environment to generate an executable file that can be operated in the ARM, and this executable file is written into the ARM to run the file. Specifically as follows:

[0055] (1) First, store the data processed by the FPGA in the ARM so that the ARM can directly retrieve the picture data.

[0056] (2) Use the adaptive histogram equalization algorithm to evenly divide the image into equal-sized rectangles. First, for the image data of size M×N, divide the image into m×n equal rectangles (M and N are integer multiples of m and n respectively), and then calculate the gray probability of each small block, and obtain the gray histogram from the probability.

[0057] (3) Since there is no direct connection between the blocks after histogram partitioning, bilinear interpolation processing is performed on the rectangles after the histogram transformation in the previous step to optimize the image boundary effect and further achieve the enhancement effect.

[0058] (4) Send the processed image data to the display unit.

[0059] Step 5, transplantation of the multi-scale Retinex algorithm and the adaptive histogram equalization algorithm.

[0060] First, it is the FLASH for transplanting the multi-scale Retinex algorithm to the FPGA. After implementing Gaussian filtering using Verilog in step 3, it is also necessary to generate a bitstream file and solidify the generated bitstream file into the FLASH (flash memory) of the operation acceleration unit to prevent loss due to power failure.

[0061] Secondly, it is the transplantation of the adaptive histogram equalization algorithm based on ARM. The compiler used for the Linux system required by ARM is the GCC compiler with optimization functions. The GCC compiler is a programming language compiler developed by GNU and has a wide range of applications in the Linux system and is often used for C / C++ compilation. It can be used to create and use static libraries. We first preprocess the source file, insert the include file into the source file, expand the macro definitions, then compile the C++ code to generate a.i file, then use assembly to convert it into an OBJ file, and finally link the OBJ file with the files in the system library to generate an executable file that can run on a specific platform.

[0062] Step 6, data interaction between the FPGA and the ARM.

[0063] Specifically, the operation acceleration unit of the FPGA communicates with the ARM through the CSI. The FPGA transmits the processed data and the generated line and field signals to the memory in the ARM through the CSI bus. The memory in the ARM transmits the image data to the processing unit for image enhancement processing. The bus interface between the operation acceleration unit and the ARM is as shown in the appendix Figure 3 as follows.

[0064] The CSI bus supports a pixel clock of up to 200 MHz, and the timing generated by the FPGA fully meets the requirements of the CSI bus timing. At the same time, the operation acceleration unit reads the data written into the register at a clock of 200 MHz. The FPGA generates corresponding line and field signals and an enable signal data_en to help the CSI bus improve the data transmission rate.

[0065] Step 7, convert the data processed by the ARM into an image through the display unit and display it in the display unit to obtain the enhanced image. Displaying the image frame by frame can obtain the enhanced video. The display unit supports connecting to display devices through HDMI interfaces and gigabit network interfaces. Specifically, the output of this system supports two interfaces. Through the HDMI interface, the system is connected to a standard display to display the processed enhanced video image in real time. At the same time, it can also be connected to the PC through the gigabit network interface to display and save the real-time video image data collected.

[0066] In summary, different from the prior art, the present invention provides a video image enhancement method and system based on FPGA + ARM, asFigure 1 As shown, it includes four main parts: an image acquisition unit, an arithmetic addition unit, a central control processing unit, and a display unit. Specifically, the arithmetic acceleration unit utilizes the parallel processing and high-frequency characteristics of the FPGA to implement three different-scale Gaussian filters after simplification. The central control processing unit utilizes the strong logical operation ability and simple transplantation characteristics of the ARM to complete the more complex adaptive histogram equalization algorithm part and logical control. While accelerating the image processing operation, it ensures the accuracy requirements of edges and brightness. Figure 5 It is the experimental effect diagram of image enhancement.

[0067] The above description is only for the experimental illustration of the present invention and does not limit the invention itself. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A night video image enhancement system based on FPGA + ARM, characterized in that, The core of the night video image enhancement system adopts the FPGA+ARM architecture; The Gaussian filtering process is implemented using FPGA, undertaking the calculation links with simple operations but a large number of operation times, serving as the operation acceleration unit of the system; The operation acceleration unit with FPGA as the core implements the multi-scale Retinex algorithm; before implementing the multi-scale Retinex algorithm using Verilog, it is simplified first, stipulating the format sizes of three sizes: low, medium, and high; the said simplification includes expanding the floating-point numbers required for the operation into integers and using pipelines to improve the operation efficiency; The ARM is used to implement the image stretching algorithm and serves as the central control processing unit of the system; adaptive histogram equalization is used to perform gray-scale stretching on the image data processed by the multi-scale Retinex algorithm; Histogram correction is carried out by calculating the transformation of the distribution function, so that the spacing between the gray values of the original image is increased, thereby highlighting the details of the image; The night video image enhancement system also includes a video image acquisition unit and a display unit; The video image acquisition unit includes a video image acquisition device and converts the acquired image data into digital signals and transmits them to the operation acceleration unit with FPGA as the core; among them, the video image acquisition unit supports two image device interfaces: DVP and Cameralink and can receive various forms of acquisition device signals; The display unit is used to convert the data processed by the ARM into an image and display it in the display unit to obtain the enhanced image; by displaying the image frame by frame, the enhanced video can be obtained; the central control processing unit with ARM as the core will transmit the processed image data to the display unit, and the display unit supports connecting to a standard display through the HDMI interface or connecting to a PC through a gigabit network port; The data acquired by the video image acquisition unit is received by the FPGA and communicated with the ARM through the image sensor interface, and the image data received by the ARM is directly output to the display device for display after preprocessing.

2. A construction method of a night video image enhancement system based on FPGA + ARM, characterized in that, It includes the following steps: Step 1: Design a video image enhancement algorithm suitable for the embedded system; Use the multi-scale Retinex algorithm as the core of image enhancement and perform operations on the image at three scales: low, medium, and high; Step 2: Design the video image acquisition unit; The video image acquisition unit includes a video image acquisition device and converts the acquired image data into digital signals and transmits them to the operation acceleration unit with FPGA as the core; among them, the video image acquisition unit supports two image device interfaces: DVP and Cameralink and can receive various forms of acquisition device signals; Step 3: Analysis and simplification of the multi-scale Retinex algorithm; The operation acceleration unit with FPGA as the core implements the multi-scale Retinex algorithm; before implementing the multi-scale Retinex algorithm using Verilog, it is simplified first, stipulating the format sizes of three sizes: low, medium, and high; the said simplification includes expanding the floating-point numbers required for the operation into integers and using pipelines to improve the operation efficiency; Step 4: Use the image stretching algorithm to perform grayscale stretching on the image processed by the multi-scale Retinex algorithm; Adaptive histogram equalization is used to perform grayscale stretching on the image data processed by the multi-scale Retinex algorithm; histogram correction is performed by calculating the transformation of the distribution function, so that the spacing between the gray values of the original image is increased, thereby highlighting the details of the image; the image stretching algorithm is implemented using a central control unit with ARM as the core; Step 5, transplantation of the multi-scale Retinex algorithm and the adaptive histogram equalization algorithm; Step 6, data interaction between FPGA and ARM; The FPGA transmits the processed data and the generated line and field signals to the processing unit in the ARM through the CSI bus; Step 7: Convert the data processed by the ARM into an image through the display unit and display it in the display unit to obtain the enhanced image; display the image frame by frame to obtain the enhanced video; the display unit supports connecting display devices through the HDMI interface and the gigabit network interface.

3. The construction method of a night-time video image enhancement system based on FPGA + ARM according to claim 2, wherein, In Step 5, the multi-scale Retinex algorithm and the adaptive histogram equalization algorithm are transplanted; first, use MATLAB to preliminarily implement and optimize the algorithm; since FPGA is based on Verilog language and it cannot support the direct operation of MATLAB itself, it is necessary to complete the conversion of the algorithm between MATLAB and Verilog language so that the algorithm can run in the hardware system. Therefore, it is necessary to perform the conversion between MATLAB language and hardware language; When implementing adaptive histogram equalization in C language, in order to reduce the excessive occupation of internal resources in the ARM, a four-byte float type is used instead of an eight-byte double type to reduce the memory occupancy and improve the operation efficiency.

4. The construction method of a night-time video image enhancement system based on FPGA+ARM according to claim 2, characterized in that, The specific method of Step 1 is as follows: The collected image is divided into a reflection part and an illumination part, where the reflection part is used to describe the high-frequency information of the original image; the illumination part describes the low-frequency information of the image; Let the night vision image collected be I(x, y), the illumination image be L(x, y), and the reflected part be represented by R(x, y), then there is: I(x,y) = R(x,y) * L(x,y) (1) Since the human eye perception system perceives the light intensity in an exponential form, take the logarithm of both sides of Equation (1); the single-scale Retinex algorithm has obvious advantages in image edge sharpening, dynamic range compression, and color constancy, and its mathematical form is: where \(i\) represents the color channel, \(r\) i (\(x,y\)) represents the output of the \(i\)-th channel of the single-scale Retinex algorithm, \(*\) represents the convolution process, and \(F(x,y)\) represents the Gaussian function, which can be expressed as: where σ is the scale parameter of the single-scale Retinex algorithm, 1 / 2πσ 2 is the planning factor; the Gaussian function F(x, y) selected in the formula is a low-pass function. Therefore, the result obtained by convolving F(x, y) with I(x, y) corresponds to the low-frequency part of the original image. Subtracting the low-frequency part from the original image, the result obtained is the high-frequency information in the original image I(x, y); and this part of the high-frequency information belongs to the part that is easy for the human eye to capture, so the purpose of enhancing the edge information is achieved; 5. The construction method of a night-time video image enhancement system based on FPGA + ARM according to claim 4, characterized in that, Use the improved multi-scale Retinex algorithm as the video image enhancement algorithm; on the basis of the single scale, the multi-scale Retinex algorithm performs filtering operations on the original image at low, medium, and high scales, and obtains the final calculation result through a weighted method. The specific method of Step 2 is as follows: The video image acquisition unit includes an image acquisition device, and converts the acquired image data into digital signals and transmits them to the operation acceleration unit for processing; the video image acquisition unit is connected to the image acquisition device in the form of a board card, and can support a variety of image device interfaces, including the digital video interface DVP and the Cameralink interface; the DVP interface and the image acquisition device can directly output digital signals without data conversion and can be directly connected to the FPGA; the Cameralink interface is a commonly used image acquisition interface for industrial cameras, which uses low-voltage differential signals LVDS to transmit signals. When in use, the LVDS data needs to be converted into 28-bit CMOS / TTL data, and at the same time, the LVDS clock signal is converted into a single-ended CMOS / TTL clock signal for output, and then connected to the FPGA to transmit the acquired image data.

6. The construction method of a night-time video image enhancement system based on FPGA + ARM according to claim 5, characterized in that, The specific method of step three is as follows: Implement the multi-scale Retinex algorithm using an operation acceleration unit with FPGA as the core; First, the FPGA processes the received data, converts it into a digital image matrix D, and participates in the subsequent Gaussian filtering. Then, a filter of an appropriate size is selected for the Gaussian filtering function used. Specifically, three scales of 5×5, 15×15, and 30×30 are used as three different Gaussian filters. For the 5×5 Gaussian filter, first use MATLAB to find the required discrete Gaussian template, which is a 5×5 floating-point matrix, and expand it into an integer matrix B 5x5 ; When performing Gaussian filtering, the Gaussian template B 5x5 is participated in and split into a horizontal one-dimensional Gaussian template B1 and a vertical one-dimensional Gaussian template B2, and a convolution operation is performed with the digital image matrix D received by the FPGA, that is: B 5×5 = B1 × B2 (4) When the FPGA caches the digital image matrix D, taking 5 rows as the processing unit, the data of the first five rows is cached in the first-level pipeline. When caching the 6th - 10th rows, the cached results of the first 5 rows enter the second-level pipeline. This not only improves the hardware calculation efficiency but also saves the arithmetic operation resources inside the FPGA; Specifically, for the design of the pipeline, in the first-level pipeline, the data of the first 5 rows is first cached and saved in 5 memories RAM. In the second-level pipeline, the first-column elements of the saved 5 rows of data are taken out and multiplied by vector B1, and the obtained result is stored in the register r11; In the third-level pipeline, while saving the calculation result of the previous level, calculate the vector multiplication result of the second column in the RAM and the Gaussian template B1, and save it in another register r22; and so on. After six-level pipelines, the product r51 - r55 of a 5-column data and B1 is obtained, and it is multiplied by B2 at the seventh clock and the result is output; For a 15×15 filter, the calculation process is the same as that of a 5×5 filter. The differences lie in the number of multipliers and memories required, and the need for a 15×15 Gaussian template B 15×15 , which is calculated by Matlab and also decomposed into two parts B3 and B4. During the calculation, the first-level pipeline caches the first 15 rows of data in D. The second-level pipeline performs a vector multiplication of the first column elements of the 15 rows with B3, and the result is stored in register r11. While saving r11 to r21, the third-level pipeline performs a vector multiplication of the cached data of the second column with the first Gaussian template, and the result is stored in r22, and so on. After 16 clock cycles, the calculation results of all 15 rows of data with B3 are obtained. Then, a one-dimensional vector composed of 15 results is multiplied with B4 in a vector multiplication to obtain the final result and output it; The calculation idea of the 30×30 filter is the same as the previous two, and the difference is that the required Gaussian template is B of 30×30 30×30 , and the processing steps of the pipeline are the same.

7. The construction method of a night-time video image enhancement system based on FPGA + ARM according to claim 6, characterized in that, The specific method of step four is as follows: After the operation and processing by the FPGA, due to its strong logical operation ability, the ARM is used for the optimization after image enhancement and serves as the system central control unit; the enhanced image after FPGA processing will have results such as lower contrast, the overall image being darker, and the enhancement effect not being obvious. After being processed by the ARM, the enhanced picture can be better displayed; before running, use a cross-compiler to compile the source code of the image stretching and enhancement algorithm in the Linux environment and generate an executable file that can be operated in the ARM, and write this executable file into the ARM to run the file; specifically as follows: (1) First, store the data processed by the FPGA into the ARM so that the ARM can directly retrieve the picture data; (2) Use the adaptive histogram equalization algorithm to evenly divide the image into equal-sized rectangles; first, for the image data of size M×N, divide the image into m×n equal rectangles, where M and N are integer multiples of m and n respectively, and then calculate the gray probability of each small block, and obtain the gray histogram from the probability; (3) Because there is no direct connection between blocks after the histogram is divided, the rectangles after the histogram transformation in the previous step are subjected to bilinear interpolation to optimize the image boundary effect and further achieve the enhancement effect; (4) Send the processed image data to the display unit.

8. A construction method of a night-time video image enhancement system based on FPGA + ARM according to claim 7, characterized in that, The specific method of step five is as follows: First, the multi-scale Retinex algorithm is ported to the FPGA's FLASH. After implementing the Gaussian filter using Verilog in step 3, a bitstream file must be generated and then stored in the FLASH of the computational acceleration unit to prevent loss during power failure. The second is the transplantation of the adaptive histogram equalization algorithm based on ARM; the compiler used by the Linux system required by ARM is the GCC compiler with optimization function; first preprocess the source file, insert the include file into the source file, expand the macro definition, then compile the C++ code to generate the .i file, and then use assembly to convert it into an OBJ file, and finally link the OBJ file with the files in the system library to generate an executable file that can run on the selected platform.

9. The construction method of a night-time video image enhancement system based on FPGA + ARM according to claim 8, characterized in that, Step 6: The specific method is as follows: Specifically, the FPGA operation acceleration unit and the ARM communicate via the CSI bus. The FPGA transmits the processed data and the generated line and field signals to the memory in the ARM via the CSI bus. The memory in the ARM transmits the image data to the processing unit for image enhancement. The CSI bus supports a pixel clock of up to 200MHz, and the timing generated by the FPGA fully meets the CSI bus timing requirements; at the same time, the computing acceleration unit reads the data written to the register at a 200MHz clock; the FPGA will correspondingly generate the line field signal and enable signal data_en to help the CSI bus increase the data transmission rate.

Images