A multi-exposure high-speed image acquisition and processing system

Abstract

The present invention belongs to the field of image acquisition, transmission and processing, and specifically relates to a multi-exposure high-speed image acquisition and processing system. The system is composed of a multi-exposure image sensor module, an FPGA module, a DSP processor, a USB data transmission interface module and a host computer display module, and the FPGA module includes a DDR4 module and a clock module. Using FPGA and DSP as core modules, multi-core DSP processes data in parallel to achieve effective and rapid data processing. By configuring the above modules and adopting a dynamic multi-exposure image fusion algorithm, it is possible to achieve rapid processing and display of image data under multiple exposure modes, thereby improving the operating speed of the system and achieving the function of clearly displaying dynamic images.

Description

Technical Field

[0001] The invention belongs to the field of image acquisition, transmission and processing, and in particular relates to a multi-exposure high-speed image acquisition and processing system. Background Art

[0002] For traditional industrial cameras, different exposure values ​​have a great impact on the quality of photos. If the exposure is too high, the brighter the photo will be, the more image details will be lost. On the contrary, if the exposure is too low, the photo will be too dark. Both of these will cause the image to lose a lot of information, and the user will not be able to see the image details clearly. Existing high-dynamic cameras generally output two sets of HG and LG data streams at the same time, and then fuse the high and low gain images to achieve the high dynamic range of the human eye. However, fusing images with multiple exposure values ​​can achieve better visual effects and show more details, which requires the system to process multiple image data at the same time. However, the current data transmission system widely uses one data channel for transmission and processing. There is currently no reliable solution to achieve a system that can transmit multi-channel data with large amounts of data and finally display it at a high frame rate. In addition, surveillance videos are mostly used in outdoor dynamic scenes. In the captured multi-exposure image sequence, dynamic targets will appear in some exposure images or in different positions in each exposure image. If no special processing is performed and traditional fusion algorithms such as wavelet transform, Laplace fusion, Gaussian fusion, etc. are directly used, due to grayscale discontinuity, obvious multiple blurs or ghosting may occur. Summary of the invention

[0003] In order to overcome the existing technical difficulties, the present invention proposes a stable, reliable, high data transmission rate system, which can realize the rapid transmission and processing of image data, while ensuring real-time and accuracy. The present invention uses a dynamic fusion algorithm to obtain conventional high dynamic range image data. When the ambient light is unevenly distributed or the object is moving, the image information can still be clearly displayed and the image details can be restored. Image acquisition is performed on multiple exposure-worthy data channels, and rapid processing is performed to facilitate subsequent image fusion and display. It can be used for outdoor high-definition monitoring, and clear imaging can be achieved regardless of whether the ambient light is very bright or very dark, or there are moving objects in the image.

[0004] In order to achieve the above object, the technical solution of the present invention is as follows:

[0005] A multi-exposure high-speed image acquisition and processing system includes a multi-exposure image sensor module, an FPGA module, a DSP processor, a USB data transmission interface module and a host computer display module, wherein the FPGA module includes a DDR4 module and a clock module. Using FPGA and DSP as core modules, multi-core DSPs process data in parallel to achieve effective and rapid data processing.

[0006] The data output by the multi-exposure image sensor module is transmitted to the FPGA module via LVDS; after the FPGA module obtains the image data, it writes it into the cache unit DDR4 module. The DDR4 caches the image data sent by the image sensor to the FPGA module, and each image is packaged and sent once; the FPGA module generates a control signal to connect the multi-exposure image sensor module, DSP processor and USB data transmission interface module in the system; the multi-core DSP processor realizes real-time signal processing and image fusion, and processes the image data transmitted by the FPGA module in parallel; the USB data transmission interface module transmits the image data to the host computer, and transmits the instructions sent by the host computer to the FPGA module; the host computer completes the configuration of the multi-exposure image sensor module, and displays the fused image data transmitted by the USB data transmission interface module as an image mode; the clock module is used to ensure the correct timing of the entire system, and each module requires a clock with a corresponding operating frequency to work normally.

[0007] The FPGA module further configures the image sensor through SPI to control the multi-exposure image data to be cached in the asynchronous FIFO. The present invention needs to quickly transmit valid data, so an additional data arbitration module is required to determine whether the image data is valid data. The FPGA module first performs the first step of processing and judgment on the image, deletes the full-bright and full-dark image data streams, retains the data streams with more valid information, and then reads the data from the corresponding asynchronous FIFO and transmits it to the DDR4 module in the FPGA module.

[0008] Furthermore, each core of the DSP processor is connected, and the connections between the cores are the same and bidirectional. After receiving the data from the DDR4 module, the DSP processor assigns the task to the sub-core for image processing. For moving objects, a dynamic multi-exposure image fusion algorithm is used for image processing. The image is first divided into several groups. The parts at the same position in each group are assigned brightness information according to the constructed Gaussian weight formula, and finally the calculation and fusion are performed to obtain the final result. After each sub-core completes the corresponding task, it is sent to the main core, and the main core fusion result is transmitted to the FPGA module for transmission processing.

[0009] Furthermore, the USB data transmission interface module adopts the USB4 form, and the data transmission of USB4 uses the TBT3 mode. The USB4 tunnel technology encapsulates the PCIe protocol. The Lane0 Adapter channel and the Lane1 Adapter channel are used to transmit DP1.2 and PCIe 3.0 data at the same time. When transmitting PCIe data, the PCIe controller selects one of the two PCIe downstream port adapters to transmit the data, and then passes the PCIe data to one of the USB4 ports, and then passes it to the USB Type-C port. The USB4 hub for tunneling the PCIe data stream includes an internal PCIe switch for forwarding the PCIe data stream to its downstream port. The internal switch is connected to the router through the PCIe protocol adapter. The PCIe switch exposes one or more lower-level PCIe ports, which can be connected to a PCIe terminal, a PCIe switch, or a lower-level PCIe protocol adapter. The upstream port of the internal switch is connected to the upstream PCIe protocol adapter, which forwards the data packet to the upper-level opposite port. The USB4 device can transmit the PCIe data stream through the tunnel by merging the internal PCIe terminal or the internal PCIe switch. The maximum transmission bandwidth is 40Gbps, and 22Gbps of bandwidth can be allocated for data transmission, meeting the design requirements.

[0010] Furthermore, the upper computer selects and controls the timing of the lower computer, sets a single exposure value for the multi-exposure image sensor module, receives the fused image data transmitted by the USB data transmission interface module, displays the image, and can also realize the image storage function.

[0011] The multi-exposure high-speed image acquisition and processing in parallel of the present invention has the following advantages compared with the traditional image acquisition system:

[0012] (1) The image sensor is required to transmit multiple sets of different exposure value data each time, which reduces the requirement for a single exposure value. The dynamic range of the composite image is increased, and there will be no overexposure or underexposure problems.

[0013] (2) The FPGA and DSP architecture is used as the core processing module to ensure stable data transmission and accurate algorithm processing. DDR4 and USB4 ensure that the processed data can be quickly transmitted to the host computer and displayed in real time. An arbitration module is added to pre-determine whether the image is a valid image, further improving the speed.

[0014] (3) The dynamic multi-exposure image fusion algorithm constructs a new weight coefficient to allocate the proportion of image information in the superposition fusion according to the effectiveness of the image information. The final constructed image meets the high dynamic range and frame rate requirements of vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A system architecture diagram of the multi-exposure high-speed image acquisition and processing method of the present invention;

[0016] Figure 2 This is the working principle diagram of the image sensor;

[0017] Figure 3 It is the working block diagram of FPGA;

[0018] Figure 4 It is a block diagram of multi-core DSP data processing;

[0019] Figure 5 Schematic diagram of data processing for USB4;

[0020] Figure 6 Schematic diagram of the host computer display interface. DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions and advantages of the present invention more clearly understood, embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] When this embodiment is implemented, the FPGA module first configures the multi-exposure image sensor module through the SPI protocol. The SPI has a higher data transmission rate than the IIC and supports full-duplex operation. The multi-exposure image sensor module simultaneously outputs a plurality of image data streams with different exposure values, which are transmitted to the FPGA module through LVDS; after the FPGA module obtains the image data, it judges the image information. If the image under a certain exposure value is completely black or completely white, and the image under other exposure values ​​is not completely black or completely white, the data stream under the exposure value is no longer transmitted, and the information of the remaining exposure values ​​is written into the cache unit DDR4 module. The function of the DDR4 module is to cache the image data sent by the multi-exposure image sensor module to the FPGA module, and each image is packaged and sent once. The data rate of the DDR4 module reaches 6.4Gbps. It has a larger storage space than DDR3 and can carry 8 modules at the same time. Its storage capacity and access speed can well meet the system's requirements for data transmission speed.

[0023] In addition, the FPGA module also generates control signals to connect the various modules in the system. A multi-core DSP processor is used to implement real-time signal processing and image fusion algorithms, and parallel processing is performed on the image data transmitted from the FPGA module; the USB data transmission interface module is used to transmit image data to the host computer, and at the same time transmit the instructions sent by the host computer to the FPGA module. On the one hand, the host computer must complete the configuration of the multi-exposure image sensor module, and at the same time display the fused image data transmitted from the USB data transmission interface module as an image mode. The clock module is used to ensure the correct timing of the entire system. Each module requires a clock with a corresponding operating frequency to work normally.

[0024] Specific. Figure 1 As shown, the present invention provides a multi-exposure high-speed image acquisition and processing design system, including a multi-exposure image sensor module, an FPGA module (including a DDR4 module, a clock module), a DSP processor, a USB data transmission interface module and a host computer display module.

[0025] like Figure 2 As shown in the figure, multiple images with different exposures are acquired using the same single exposure. The intensity of the single exposure is controlled by the host computer. The generated image signal is digitized using the ADC on the chip. After digitization, multiple images are read out at the maximum speed in the channel. Figure 2 The image sequence sent from the multi-exposure image sensor module is shown. The frame image signals of multiple exposure value images are read out line by line from the multi-exposure image sensor module at the same time and transmitted to the I / O port of the FPGA module.

[0026] The specific implementation process of the FPGA module is as follows Figure 3 As shown, the multi-exposure image sensor module is configured through SPI, and all the multi-exposure image data obtained from the multi-exposure image sensor module are stored in the asynchronous FIFO, and the preset DDR4 module controller calibration signal is detected to be 0 or 1: 1 means it has been completed, and 0 means it has not been completed. If it has not been completed, it returns to the previous state; if it has been completed, the FPGA module controls the image data of all exposure values ​​to be cached in the FIFO, and the judgment module compares whether the image data of each exposure value is valid data, and takes the image data corresponding to the valid exposure value out of the FIFO and transmits it to the DSP processor.

[0027] The DDR4 module receives the data transmitted by the multi-exposure image sensor module, generates the read and write control commands of the DDR4 module, and passes them to the memory controller of the DDR4 module. The memory controller of the DDR4 module calculates different addresses for the read and write operations of the DDR4 module with different exposure values ​​according to the set read and write operation space. Perform status judgment. If it is a write command: end the idle state and write the image data of different exposure values ​​into the pre-allocated address space of the DDR4 module; if it is a read command: end the idle state, and the FPGA module reads the image data from the address space allocated by the corresponding DDR4 module for different exposure values ​​to complete the read process.

[0028] The DSP processor adopts a dual-core DSP architecture, and a DSP core is assigned to each image data corresponding to the exposure value to process the received data respectively. Multiple DSPs work in parallel, and the communication between DSPs is realized through a parallel bus. Each DSP core mainly includes two processes: a data receiving process and a data operation process. First, the data receiving process is performed. After the reception is completed, the signal is transmitted to the main core. After the data processing process is completed, the main core and the slave core operate. The information sent by the main core to the slave core mainly includes the first address of the picture processed by each slave core, and the information sent by the slave core to the main core is mainly the first address of the picture after processing by each core. Multiple main cores process the fusion algorithm at the same time, and transmit the fused image data back to the FPGA module. In order to realize the exchange of data between the DSP processor and the FPGA module, the EMIF external memory interface of the DSP processor and the BRAM in the FPGA module can be used. The BRAM adopts a dual-port BRAM, one port for the DSP EMIF interface, and the other port for the logic in the FPGA module. The DSP processor expands the DDR4 module as a buffer area for data operation.

[0029] For outdoor moving objects, if you want to obtain true and clear image information and achieve better fusion effects while allowing the image to present the most useful information, the dynamic multi-exposure fusion algorithm can meet the needs of users. The principle is to first divide the input image into several square image blocks of the same size, and assign different fusion weight values ​​to each image block. Finally, through the algorithm, multiple images with different exposure values ​​are superimposed together to integrate into a better image. The specific operations are as follows:

[0030] When a moving object is detected in the image, the difference method is first used to obtain the edge of the region of the moving object, and then the moving area is removed. The remaining data obtained is a static image, which can be processed using a traditional fusion algorithm, such as the PMF algorithm. However, for the same moving object corresponding to different exposure values, if a simple method is used to superimpose it in the static area, it will inevitably cause ghosting and blurring between blocks, and the final image effect is unacceptable to the human eye. In order to allow the edges of dynamic images to be naturally integrated into static images and provide users with good visual effects, the present invention proposes an algorithm for the extracted moving area: first define a fixed step size x, and take the data length with a specific step size of x as a group. The parts of each group at the same position are calculated according to the formula to obtain the final result. The specific step size value and the number of groups are jointly allocated according to the resolution of the multi-exposure image sensor module and the size of the moving area. The brightness of each group is determined by the formula, and a Gaussian function is constructed:

[0031]

[0032] Among them, μ x is the global average brightness, l x is the average brightness of the nth block, α g and α l is the weight of the brightness component. In this embodiment, α is taken g =0.2,α l =0.2 According to the weight, the fused brightness component can be obtained The display brightness of each group is multiplied by the coefficient of the Gaussian function and displayed as the final picture. The fused image not only has harmonious brightness, but also does not have ghosting due to the presence of moving objects, solving the grayscale discontinuity problem of traditional fusion.

[0033] The host computer plays a master role in the system. It can not only be used for image display, but also can realize the control of FPGA module and multi-exposure image sensor module. It can also display the multi-exposure image sensor module data uploaded by the lower computer in real time on the host computer. Finally, it can complete the archiving and recovery of data and real-time monitoring. You can use the QT platform to design a host computer interface such as Figure 5 As shown, it includes the selection of the timing of the multi-exposure image sensor module, the selection of the communication with USB4, the transmission speed detection, and the display of the fused image. QT can run across platforms and can be packaged into an APP application that can be used on Android or IOS systems, making it easier for users to manage on the mobile terminal.

[0034] The clock signal is the source of all operations and processing. The clock module can provide the corresponding operating frequency for each module of the present invention, which plays a very important role. Especially in the real-time system of the present invention, the requirements for clock synchronization are very high. The system clock uses the FPGA master clock. The DDR4 module requires a differential clock design CK and The role of is usually understood as the second trigger clock, but in fact it plays the role of trigger clock calibration. Due to various factors, the interval between the upper and lower edges of CK may change. It can play a role in correcting the deviation. Differential clock signals can be used to provide synchronous clock signals for DDR4. The differential clock signal has relatively strong anti-interference ability and meets the clock requirements of the DDR4 module. The FPGA module must have a second crystal oscillator to provide a clock signal for USB4, so when programming and driving the FPGA module, it is necessary to do a good job of delay to synchronize with the FPGA master clock. The DSP processor has a rich peripheral interface, and each peripheral interface has its own independent reference clock. Since the multi-core DSP has a faster data processing capability, the clock requirements for the peripheral interface are higher. The FPGA module crystal oscillator and the DSP processor's internal crystal oscillator circuit are used together to generate a clock to provide the clock for the DSP processor working core and the external DDR4 module.

Claims

1. A multi-exposure high-speed image acquisition and processing system, characterized in that: The multi-exposure high-speed image acquisition and processing system includes a multi-exposure image sensor module, an FPGA module, a DSP processor, a USB data transmission interface module and a host computer display module, and the FPGA module includes a DDR4 module and a clock module; The data output by the multi-exposure image sensor module is transmitted to the FPGA module via LVDS; after the FPGA module obtains the image data, it writes it into the cache unit DDR4 module. The DDR4 caches the image data sent by the image sensor to the FPGA module, and each image is packaged and sent once; the FPGA module generates a control signal to connect the multi-exposure image sensor module, DSP processor and USB data transmission interface module in the system; The multi-core DSP processor realizes real-time signal processing and image fusion, and processes the image data transmitted from the FPGA module in parallel; The USB data transmission interface module transmits the image data to the host computer, and transmits the instructions sent by the host computer to the FPGA module; the host computer completes the configuration of the multi-exposure image sensor module, and displays the fused image data transmitted by the USB data transmission interface module as an image mode; the clock module is used to ensure the correct timing of the entire system, and each module requires a clock of the corresponding operating frequency to work normally; The FPGA module further configures the image sensor through SPI, controls the multi-exposure image data to be cached in the asynchronous FIFO, and adds a data arbitration module to determine whether the image data is valid data; the FPGA module first processes and judges the image, deletes the full-bright and full-dark image data streams, retains the data streams with more valid information, and then reads the data from the corresponding asynchronous FIFO and transmits it to the DDR4 module in the FPGA module; Furthermore, each core of the DSP processor is connected, and the connections between the cores are the same and bidirectional. After receiving the data from the DDR4 module, the DSP processor assigns the task to the sub-core for image processing. For moving objects, a dynamic multi-exposure image fusion algorithm is used for image processing, that is, the image is first divided into several groups, and the parts at the same position in each group are assigned brightness information according to the Gaussian weight formula, and finally the calculation and fusion are performed to obtain the final result. After each sub-core completes the corresponding task, it is sent to the main core, and the fusion result of the main core is transmitted to the FPGA module for transmission processing. Furthermore, the USB data transmission interface module adopts the USB4 form, the USB4 data transmission selects the TBT3 mode, the USB4 tunnel technology encapsulates the PCIe protocol, and the Lane0 Adapter channel and the Lane1 Adapter channel are used to transmit DP 1.2 and PCIe 3.0 data at the same time. When transmitting PCIe data, one of the two PCIe downstream port adapters is selected through the PCIe controller to transmit the data, and then the PCIe data is passed to one of the USB4 ports, and then to the USB Type-C port. The USB4 hub that tunnels the PCIe data stream includes an internal PCIe switch for forwarding the PCIe data stream to its downstream port; The internal switch is connected to the router via a PCIe protocol adapter; The PCIe switch exposes one or more lower-level PCIe ports that are connected to a PCIe terminal, a PCIe switch, or a lower-level PCIe protocol adapter; The uplink port of the internal switch is connected to the uplink PCIe protocol adapter, which forwards the data packet to the upper-level opposite port; USB4 devices tunnel PCIe data streams by incorporating internal PCIe termination or internal PCIe switches; Furthermore, the upper computer realizes the selection and control of the timing of the lower computer, sets a single exposure value for the multi-exposure image sensor module, receives the fused image data transmitted by the USB data transmission interface module, displays the image, and can also realize the image storage function; The dynamic multi-exposure image fusion algorithm is as follows: for the extracted motion area, a fixed step length x is first defined, and the data length with a specific step length x is taken as a group; the parts at the same position of each group are calculated according to the formula to obtain the final result; the specific step length value and the number of groups are jointly allocated according to the resolution of the multi-exposure image sensor module and the size of the motion area; the brightness of each group is determined by the formula, and a Gaussian function is constructed: Among them, μ x is the global average brightness, l x is the average brightness of the nth block, α g and α l is the weight of the brightness component, and the fused brightness component _l is obtained; the display brightness of each group is multiplied by the coefficient of the Gaussian function, and displayed as the final picture.

Images