A dual-spectrum camera based on FPGA

Abstract

The present invention discloses a dual-spectrum camera based on FPGA, including a super starlight-level night vision full-color sensor, an infrared cutoff filter switching module, an SCCB interface, an infrared lighting device and an image processing device. The infrared lighting device realizes near-infrared light illumination for infrared exposure of the dual-spectrum camera under weak light conditions; the infrared cutoff filter switching module switches the light entering the super starlight-level night vision full-color sensor to visible light or infrared light; the super starlight-level night vision full-color sensor receives the driving control of the image processing device based on the SCCB interface, realizes the photoelectric conversion of visible light and the photoelectric conversion of infrared light to obtain image data; the image processing device is realized based on FPGA, and is used to drive and control the super starlight-level night vision full-color sensor, the infrared cutoff filter switching module and the external lighting device, and receive the image data output by the super starlight-level night vision full-color sensor.

Description

Technical Field

[0001] The present application belongs to the field of infrared optoelectronics and applied optical technology, and specifically relates to a dual-spectrum camera based on FPGA. Background Art

[0002] The high-definition and high-speed camera system proposed in the paper "High-definition and high-speed CMOS camera system design" is based on a high-performance CMOS image sensor AM41V4 developed by ALEXIMA in the United States. On this basis, Xilinx's Virtex-6 FPGA is used as the control core of the system, DDR3 dynamic memory is used as the imaging register, and the DVI display interface is used for image frame display. The Camera Link interface is used as the system camera interface for image data transmission. The solution proposed in this paper is mainly based on FPGA (Field-Programmable Gate Array) to achieve high-resolution image data acquisition, and does not achieve high-speed infrared acquisition and later infrared and visible light image processing.

[0003] The paper "Research on the Implementation of Infrared Thermal Imaging System and Image Enhancement Technology" designs an implementation scheme for infrared thermal imaging system and implements the image detail enhancement algorithm in order to solve the problems of image non-uniformity, blind pixels and image blur. However, the scheme does not implement a high-speed and high-frame-rate scheme.

[0004] The patent "A dual-spectrum high-speed camera based on FPGA" with publication number CN109951617A introduces a dual-spectrum high-speed camera based on FPGA, which includes: an infrared sensor, a visible light sensor, a serial communication master-slave interface realized by an I2C interface, an image processing device, and an image real-time display device. The I2C interface working timing is used to write the registers inside the CMOS sensor, and to control the exposure, automatic white balance on and off, frame rate and / or resolution of the CMOS sensor. Based on the high photosensitivity of infrared, the CMOS sensor increases the imaging speed to achieve high frame rate and high speed; the CMOS sensor adopts a global exposure mode to achieve the shooting of high-speed moving objects without distortion. The aforementioned dual-spectrum camera uses two chips, visible light CMOS and infrared CMOS. The infrared chip device cost is relatively high, and the FPGA needs to process two images at the same time, which is complicated to operate. Summary of the invention

[0005] The purpose of this application is to provide a dual-spectrum camera based on FPGA, which improves the image quality of infrared illumination and reduces the complexity of image processing.

[0006] To achieve the above purpose, the technical solution adopted by this application is:

[0007] A dual-spectrum camera based on FPGA, comprising a super starlight-level night vision full-color sensor, an infrared cutoff filter switching module, an SCCB interface, an infrared lighting device and an image processing device, wherein:

[0008] The infrared lighting device is used to realize near-infrared light illumination under the driving control of the image processing device, so as to provide infrared exposure for the dual-spectrum camera under weak light conditions;

[0009] The infrared cut-off filter switching module is used to switch the light entering the super starlight-level night vision full-color sensor to visible light or infrared light under the driving control of the image processing device;

[0010] The super starlight-level night vision full-color sensor receives the drive control of the image processing device based on the SCCB interface, realizes the photoelectric conversion of visible light and infrared light to obtain image data;

[0011] The image processing device is implemented based on FPGA, and is used to drive and control the super starlight level night vision full-color sensor, the infrared cutoff filter switching module and the external lighting device, and receive image data output by the super starlight level night vision full-color sensor.

[0012] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution, but are merely further supplements or preferences. Under the premise that there are no technical or logical contradictions, each optional method can be combined with the above-mentioned overall solution separately, and multiple optional methods can also be combined.

[0013] Preferably, the super starlight-level night vision full-color sensor is a CMOS image sensor, adopts a global exposure type CMOS shutter mode, and its corresponding infrared lighting device adopts a global FLASH type exposure mode.

[0014] Preferably, the infrared lighting device comprises a driver, a laser and a refractive optical element;

[0015] The driver receives the driving pulse output by the image processing device to control the operation of the laser;

[0016] The laser emits laser light under the drive of the driver, and the laser light is projected after passing through the refractive optical element. The projection area is a rectangular window, and the area of ​​the rectangular window is consistent with the lens photosensitive area of ​​the CMOS shutter.

[0017] Preferably, the super starlight-level night vision full-color sensor is a CMOS image sensor, adopts a rolling exposure type CMOS shutter mode, and its corresponding infrared lighting device is a one-dimensional line scanning exposure mode.

[0018] Preferably, the infrared lighting device comprises a driver, a laser, a scanning reflector and a refractive optical element;

[0019] The driver receives the driving pulse output by the image processing device to control the operation of the laser;

[0020] The laser emits laser light under the drive of the driver. The laser light first passes through a scanning reflector and then a refractive optical element and is output as a linear light spot. The galvanometer frequency of the scanning reflector corresponds to the exposure time of the rolling shutter.

[0021] Preferably, the galvanometer frequency of the scanning reflector corresponds to the rolling shutter exposure time, including:

[0022] The stepping time of the scanning mirror is consistent with the CMOS line-by-line exposure time of rolling shutter exposure.

[0023] Preferably, the image processing device controls the scanning mirror using an I2C interface.

[0024] Preferably, the infrared cut-off filter switching module comprises a filter switch, an infrared cut-off filter and a fully transparent glass sheet;

[0025] The thickness of the fully transparent glass sheet is l1, the thickness of the infrared cutoff filter is l2, the focal length of the visible light passing through the fully transparent glass sheet is f1, and the focal length of the infrared light after passing through the infrared cutoff filter is f2, then l1+f1=l2+f2.

[0026] The present application provides an FPGA-based dual-spectrum camera, which realizes dual-spectrum output using only a single-chip super-starlight-level night vision full-color sensor; uses an infrared lighting device to improve the image quality of infrared lighting; and uses a one-dimensional scanning infrared lighting device to reduce the power requirement of the infrared lighting device. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a structural schematic diagram of a FPGA-based dual-spectrum camera using a global exposure CMOS shutter mode in the present application;

[0028] Figure 2 This is a schematic structural diagram of a CMOS shutter mode using a rolling exposure type of FPGA-based dual-spectrum camera of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0031] Embodiment 1, as Figure 1 As shown, a dual-spectrum camera based on FPGA is provided, including a super starlight-level night vision full-color sensor, an infrared cutoff filter switching module, an SCCB interface, an infrared lighting device and an image processing device.

[0032] The infrared lighting device is used to realize near-infrared light illumination under the driving control of the image processing device, so as to provide infrared exposure for the dual-spectrum camera under weak light conditions.

[0033] The infrared cut-off filter switching module is used to switch the light entering the super starlight-level night vision full-color sensor to visible light or infrared light under the driving control of the image processing device.

[0034] The super starlight-level night vision full-color sensor receives the drive control of the image processing device based on the SCCB interface, realizes the photoelectric conversion of visible light and infrared light to obtain image data, and uploads the image data to the image processing device.

[0035] The image processing device is implemented based on FPGA and is used to drive and control the super starlight level night vision full-color sensor, the infrared cutoff filter switching module and the external lighting device, and receive the image data output by the super starlight level night vision full-color sensor.

[0036] The dual-spectrum camera in this embodiment uses only a single super-starlight-level night vision full-color sensor to achieve dual-spectrum output, which greatly reduces the complexity and manufacturing cost of the camera in terms of structure. In addition, the infrared lighting device is used to effectively improve the quality of infrared acquired images.

[0037] In this embodiment, the super starlight-level night vision full-color sensor is a CMOS image sensor, and the serial communication host interface implemented by the SCCB interface, that is, the super starlight-level night vision full-color sensor is configured by the FPGA through the SCCB interface, including using the SCCB interface working sequence to write the register inside the CMOS sensor, and control the exposure, automatic white balance on and off, frame rate, and resolution of the CMOS sensor.

[0038] And the FPGA is internally configured with an image data acquisition module, an image data real-time processing module, and an image data compression module. The image data acquisition module acquires the image data output by the CMOS image sensor through the MIPI interface (Mobile Industry Processor Interface), and processes it through the image data real-time processing module. The processing here can be image processing means such as filtering, scaling, and cropping, which can be configured according to actual needs and is not limited in this embodiment.

[0039] After the image data is processed, the image data is compressed by the image data compression module to facilitate rapid data transmission, and then the compressed image data is uploaded to the host computer via Ethernet for display and analysis. It should be noted that various types of drivers, image processing, compression, etc. implemented based on FPGA in this embodiment are implemented using existing technologies, which are not the focus of this application and are not subject to relevant restrictions.

[0040] In order to realize the dual-spectrum output of the single-chip super starlight-level night vision full-color sensor, an infrared cutoff filter switching module for switching between visible light and infrared light is provided in this embodiment. The infrared cutoff filter switching module of this embodiment includes a filter switcher, an infrared cutoff filter and a fully transparent glass sheet.

[0041] The filter switch is driven and controlled by the image processing device, and the filter switch completes the switching of the infrared cutoff filter, and the cutoff band is the near-infrared band, and the optional 808nm, 850nm, 905nm, and 940nm. Using the filter switch to switch the infrared cutoff filter, a single-chip super-starlight-level night vision full-color sensor can be realized to work in visible light and near-infrared light respectively. In this embodiment, the filter switch is mainly used to switch the filter, and its structure is not restricted on the premise of realizing the function.

[0042] In order to ensure the quality of the dual-spectrum image, in this embodiment, the thickness of the fully transparent glass sheet is taken as l1, the thickness of the infrared cut-off filter is taken as l2, the focal length of the visible light passing through the fully transparent glass sheet is f1, and the focal length of the infrared light after passing through the infrared cut-off filter is f2, then l1+f1=l2+f2.

[0043] The CMOS image sensor in this embodiment adopts a global exposure CMOS shutter mode, and the corresponding infrared illumination device adopts a global FLASH exposure mode. In this mode, the infrared illumination device includes a driver, a laser, and a refractive optical element.

[0044] The driver receives the driving pulse output by the image processing device to control the operation of the laser.

[0045] The laser emits laser when driven by a driver. The laser is projected after passing through a refractive optical element. The projection area is a rectangular window, and the area of ​​the rectangular window is consistent with the lens photosensitive area of ​​the CMOS shutter.

[0046] In this embodiment, the FPGA is configured with a laser drive control module, which uses the GPIO port to control the on and off of the infrared lighting device. The refractive optical element in this embodiment is an optical beam splitter.

[0047] Embodiment 2, as Figure 2 As shown, a dual-spectrum camera based on FPGA is provided. The difference between the dual-spectrum camera of this embodiment and the dual-spectrum camera in Example 1 is that the dual-spectrum camera in this embodiment adopts a rolling exposure type CMOS shutter mode, and its corresponding infrared lighting device is a one-dimensional line scanning exposure mode.

[0048] In a one-dimensional line scanning exposure mode, the infrared illumination device includes a driver, a laser, a scanning mirror and a refractive optical element.

[0049] The driver receives the driving pulse output by the image processing device to control the operation of the laser.

[0050] The laser emits laser light driven by the driver. The laser light first passes through a (one-dimensional) scanning mirror and then passes through a refractive optical element (optical beam splitter) to be output as a linear light spot. The galvanometer frequency of the scanning mirror corresponds to the exposure time of the rolling shutter.

[0051] In this embodiment, the stepping time of the scanning reflector is consistent with the row-by-row exposure time of the CMOS of the rolling shutter exposure. That is, the scanning step length of the scanning reflector is consistent with the row-by-row exposure time of the rolling shutter exposure camera, so that the infrared illumination exposure time of each row is consistent. In this embodiment, a one-dimensional scanning reflector is matched with the row-by-row scanning of the image sensor to achieve row-by-row infrared exposure.

[0052] And for the convenience of control, the image processing device uses the I2C interface to control the scanning mirror, including the rotation speed and stepping speed of the scanning mirror.

[0053] The other structures of the dual-spectrum camera in this embodiment are consistent with those of the dual-spectrum camera in Example 1, and will not be described in detail here.

[0054] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0055] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A dual-spectrum camera based on FPGA, characterized in that: The FPGA-based dual-spectrum camera includes a super starlight-level night vision full-color sensor, an infrared cutoff filter switching module, an SCCB interface, an infrared lighting device and an image processing device, wherein: The infrared lighting device is used to realize near-infrared light illumination under the driving control of the image processing device, so as to provide infrared exposure for the dual-spectrum camera under weak light conditions; The infrared cut-off filter switching module is used to switch the light entering the super starlight-level night vision full-color sensor to visible light or infrared light under the driving control of the image processing device; the infrared cut-off filter switching module includes a filter switch, an infrared cut-off filter and a fully transparent glass sheet; the thickness of the fully transparent glass sheet is l1, the thickness of the infrared cut-off filter is l2, the focal length of the visible light passing through the fully transparent glass sheet is f1, and the focal length of the infrared light after passing through the infrared cut-off filter is f2, then l1+f1=l2+f2; The super starlight-level night vision full-color sensor receives the driving control of the image processing device based on the SCCB interface, realizes the photoelectric conversion of visible light and the photoelectric conversion of infrared light to obtain image data; the super starlight-level night vision full-color sensor is a CMOS image sensor, adopts a global exposure type CMOS shutter mode, and its corresponding infrared lighting device is a global FLASH type exposure mode; or, the super starlight-level night vision full-color sensor is a CMOS image sensor, adopts a rolling exposure type CMOS shutter mode, and its corresponding infrared lighting device is a one-dimensional line scanning type exposure mode; The image processing device is implemented based on FPGA, and is used to drive and control the super starlight level night vision full-color sensor, the infrared cutoff filter switching module and the external lighting device, and receive image data output by the super starlight level night vision full-color sensor.

2. The FPGA-based dual-spectrum camera according to claim 1, characterized in that: The infrared lighting device comprises a driver, a laser and a refractive optical element; The driver receives the driving pulse output by the image processing device to control the operation of the laser; The laser emits laser light under the drive of the driver. The laser light is projected after passing through the refractive optical element. The projection area is a rectangular window, and the area of ​​the rectangular window is consistent with the lens photosensitive area of ​​the CMOS shutter.

3. The FPGA-based dual-spectrum camera according to claim 1, characterized in that: The infrared lighting device includes a driver, a laser, a scanning reflector and a refractive optical element; The driver receives the driving pulse output by the image processing device to control the operation of the laser; The laser emits laser light under the drive of the driver. The laser light first passes through a scanning reflector and then a refractive optical element and is output as a linear light spot. The galvanometer frequency of the scanning reflector corresponds to the exposure time of the rolling shutter.

4. The FPGA-based dual-spectrum camera according to claim 3, characterized in that: The galvanometer frequency of the scanning reflector corresponds to the rolling shutter exposure time, including: The stepping time of the scanning mirror is consistent with the CMOS line-by-line exposure time of rolling shutter exposure.

5. The FPGA-based dual-spectrum camera according to claim 3, characterized in that: The image processing device controls the scanning mirror using an I2C interface.

Images