A device and method for information fusion of multiple lenses and multiple sensors for visible light and mid-to-far infrared light

Abstract

The present invention discloses a visible light and mid-to-far infrared light multi-lens multi-sensor information fusion device, including an infrared sensor, a visible light sensor and a circuit board, the circuit board is integrated with a control and signal processing integrated circuit, the infrared sensor and the visible light sensor are respectively connected through a signal processing integrated control and signal processing integrated circuit; the image information collected by the infrared lens and the visible light lens is respectively transmitted to the control and signal processing integrated circuit through the infrared sensor and the visible light sensor, the control and signal processing integrated circuit respectively performs depth calculation on the image information collected by the infrared lens and the visible light lens, and the calculation results are fused. The present invention uses an infrared camera and an infrared sensor to collect images of dynamic animals, pedestrians and vehicles, and calculates the image depth through a control and signal processing integrated circuit, calculates the distance from the object to the camera device, thereby judging the road condition, and achieving safe driving.

Description

Technical Field

[0001] The present invention relates to the field of cameras, and in particular to a device and method for information fusion of multiple lenses and multiple sensors for visible light and mid-to-far infrared light. Background Art

[0002] With the development of science and technology and the improvement of people's living standards, cars have entered every household and become a common consumer product.

[0003] The existing conventional automobile driving technology has been developed to a very complete level and can basically meet everyone's usage needs. Unmanned driving technology has begun to develop slowly. In the process of conventional driving, because there are people driving, the road conditions can be judged very well. However, there is no human judgment in the process of unmanned driving. Therefore, during driving, it is necessary to use cameras to capture road conditions and analyze road conditions in a timely manner. It is necessary to judge whether obstacles are animals or static mountains, trees or other things. Therefore, this technology needs to be further improved. Summary of the invention

[0004] The object of the present invention is to overcome the above-mentioned deficiencies of the prior art and to provide a visible light and mid-to-far infrared light multi-lens multi-sensor information fusion device.

[0005] Another object of the present invention is to provide a method for an information fusion device using multiple lenses and multiple sensors of visible light and mid-to-far infrared light.

[0006] The technical solution of the present invention is implemented as follows: a visible light and mid-to-far infrared light multi-lens multi-sensor information fusion device comprises a circuit board, at least two mid-to-far infrared sensors and at least two visible light sensors, the infrared sensor is connected to an infrared lens, and the visible light sensor is connected to a visible light lens; a control and signal processing integrated circuit is integrated on the circuit board, and the infrared sensor and the visible light sensor are respectively connected to the control and signal processing integrated circuit through electrical signals or through an interface circuit;

[0007] in:

[0008] The infrared lens receives infrared electromagnetic waves and collects image information;

[0009] The visible light lens receives visible light electromagnetic waves and collects image information;

[0010] The image information collected by the infrared lens and the visible light lens is transmitted to the control and signal processing integrated circuit through the infrared sensor and the visible light sensor respectively. The control and signal processing integrated circuit performs depth calculation on the image information collected by the infrared lens and the visible light lens respectively, and the calculation results are fused and enhanced to obtain clear image information.

[0011] Preferably, the infrared sensor is a mid-to-far infrared sensor, and the infrared lens is a mid-to-far infrared lens.

[0012] Preferably, the visible light includes near infrared light.

[0013] Preferably, it is characterized in that the infrared sensor and the infrared lens are arranged in a one-to-one matching manner, and the visible light sensor and the visible light lens are arranged in a one-to-one matching manner, and the matched infrared sensor and infrared lens are arranged crosswise with the visible light sensor and the visible light lens.

[0014] Preferably, the wavelengths of infrared electromagnetic waves received by the mid- and far-infrared lens are: 3-8 microns, 8-15 microns and 15-1000 microns.

[0015] Preferably, the wavelengths of the visible light electromagnetic waves received by the visible light lens are: 0.4 microns-0.77 microns and 0.78 microns-1.1 microns (near infrared).

[0016] Preferably, the circuit board is also externally connected to a communication antenna and / or a communication interface.

[0017] A method based on the above-mentioned visible light and mid-to-far infrared light multi-lens multi-sensor information fusion device, the method comprising the following steps:

[0018] S1: Calibration and adjustment of the device: place a calibration object, turn on the device, the visible light lens and the infrared lens receive visible light electromagnetic waves and infrared light electromagnetic waves respectively and transmit them to the calibration object, use the principle of epipolar geometry to find the epipolar lines, and then find the corresponding points of the calibration object on the visible light sensor and the infrared sensor, and then find the distance of the calibration object relative to the visible light lens and the infrared lens, compare the calculated distance with the distance of the actual object, if the results are consistent, the calibration is completed, if the results are inconsistent, adjust the device and measure and compare again until the comparison results are consistent;

[0019] S2: Visible light depth measurement: the visible light lens emits visible light electromagnetic waves, which hit obstacles and are then received back by the visible light sensor. The visible light sensor transmits the received visible light image information to the control and signal processing integrated circuit. The control and signal processing integrated circuit calculates the image depth of the visible light image by calculating the maximum value of the correlation function.

[0020] S3: Infrared light depth measurement: The infrared lens emits visible light electromagnetic waves, which hit obstacles and are then received back by the infrared sensor. The infrared sensor transmits the received infrared light image information to the control and signal processing integrated circuit. The control and signal processing integrated circuit calculates the image depth of the infrared light image information using color classification based on the RGB model or the YCrCb model;

[0021] S4: Fusion of two calculation methods: When the two calculation methods are fused, if the visible light is stronger than the infrared light, the method of step S2 is used to calculate the image depth of the visible light and the infrared light; if the infrared light is stronger than the visible light, the method of step S3 is used to calculate the image depth of the visible light and the infrared light.

[0022] The beneficial effects of the present invention are as follows: the present invention utilizes a visible light camera and a visible light sensor to collect static images of mountains, trees or other objective materials, utilizes an infrared camera and an infrared sensor to collect dynamic images of animals, pedestrians and vehicles, and calculates the image depth through a control and signal processing integrated circuit, calculates the distance from the object to the camera device, thereby judging the road condition and achieving safe driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The present invention is a structural principle diagram of an information fusion device using multiple lenses and multiple sensors of visible light and mid-to-far infrared light.

[0024] Figure 2 It is a schematic diagram of the installation and setting of the infrared sensor and infrared lens and the visible light sensor and visible light lens in the present invention.

[0025] Figure 3 It is a schematic diagram of the installation and setting of the infrared sensor and infrared lens and the visible light sensor and visible light lens in the present invention, and a structural schematic diagram of the connection with the circuit.

[0026] Figure 4 It is a step diagram of a method for an information fusion device using multiple lenses and multiple sensors of visible light and mid-to-far infrared light according to the present invention.

[0027] The names corresponding to the numbers in the figure are: infrared sensor-1, visible light sensor-2, infrared lens-3, visible light lens-4. DETAILED DESCRIPTION

[0028] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings.

[0029] like Figure 1As shown, a visible light and mid-to-far infrared light multi-lens multi-sensor information fusion device comprises an infrared sensor 1, a visible light sensor 2 and a circuit board, wherein the infrared sensor is connected to an infrared lens 3, and the visible light sensor 2 is connected to a visible light lens 4; a control and signal processing integrated circuit is integrated on the circuit board, and the infrared sensor 1 and the visible light sensor 2 are respectively connected to the control and signal processing integrated circuit through electrical signals; wherein:

[0030] The infrared lens 3 receives infrared electromagnetic waves and collects image information;

[0031] The visible light lens 4 receives visible light electromagnetic waves and collects image information;

[0032] The image information collected by the infrared lens 3 and the visible light lens 4 is transmitted to the control and signal processing integrated circuit through the infrared sensor 1 and the visible light sensor 2 respectively. The control and signal processing integrated circuit performs depth calculation on the image information collected by the infrared lens 3 and the visible light lens 4 respectively, and the calculation results are fused and enhanced to obtain clear image information.

[0033] In this embodiment, the infrared sensor 1 is a mid-to-far infrared sensor, the infrared lens 3 is a mid-to-far infrared lens, and the infrared lens 3 receives infrared electromagnetic waves with wavelengths of 3-8 microns, 8-15 microns and 15-1000 microns. It is mainly used to detect animals, pedestrians and vehicles, and to judge road conditions. The visible light includes near-infrared light, and the wavelengths of visible electromagnetic waves received by the visible light lens 4 are 0.4 microns-0.77 microns and 0.78 microns-1.1 microns (near-infrared light waves). It is mainly used to detect objects such as hills, trees, houses, and cars. The distance from the lens to the object to be measured is measured by the infrared lens 3 and the visible light lens 4, so as to accurately make decisions and control the vehicle driven by integrated signal processing.

[0034] Depend on Figure 2-3 It can be seen that the infrared sensor 1 and the infrared lens 3 are matched in a one-to-one arrangement, that is, one infrared sensor 1 is connected to one infrared lens 3, and the infrared lens 3 is arranged on the outside. Similarly, the visible light sensor 2 and the visible light lens 4 are matched in a one-to-one arrangement, that is, one visible light sensor 1 is connected to one visible light lens 3, and the matched infrared sensor 1 and infrared lens 3 are arranged crosswise with the visible light sensor 2 and the visible light lens 4. In this embodiment, the number of the infrared sensor 1 and the infrared lens 3 is 2, and they are matched into two sets of infrared sensor 1 and infrared lens 3 combinations, and the number of the visible light sensor 2 and the visible light lens 4 are both 2, and they are matched into two pairs of visible light sensor 2 and visible light lens 4 combinations, and the four sets of sensor and lens combinations are arranged in a row.

[0035] The circuit board is also externally connected to a communication interface and (or) a communication antenna, and the communication interface and (or) the communication antenna are used to connect to a remote server, and the interface is an external interface used to connect to an external device.

[0036] like Figure 4 As shown, a method based on the above-mentioned visible light and mid-to-far infrared light multi-lens multi-sensor information fusion device comprises the following steps:

[0037] S1: Calibration and adjustment of the device: place a calibration object, turn on the device, the visible light lens and the infrared lens receive visible light electromagnetic waves and infrared light electromagnetic waves respectively and transmit them to the calibration object, use the principle of epipolar geometry to find the epipolar lines, and then find the corresponding points of the calibration object on the visible light sensor and the infrared sensor, and then find the distance of the calibration object relative to the visible light lens and the infrared lens, compare the calculated distance with the distance of the actual object, if the results are consistent, the calibration is completed, if the results are inconsistent, adjust the device and measure and compare again until the comparison results are consistent;

[0038] S2: Visible light depth measurement: the visible light lens emits visible light electromagnetic waves, which hit obstacles and are then received back by the visible light sensor. The visible light sensor transmits the received visible light image information to the control and signal processing integrated circuit. The control and signal processing integrated circuit calculates the image depth of the visible light image by calculating the maximum value of the correlation function.

[0039] S3: Infrared light depth measurement: The infrared lens emits visible light electromagnetic waves, which hit obstacles and are then received back and received by the infrared sensor. The infrared sensor transmits the received infrared light image information to the control and signal processing integrated circuit. The control and signal processing integrated circuit uses color classification based on the RGB model or the YCrCb model to calculate the image depth of the infrared light image information. The RGB model or the YCrCb model described here is an existing image component model, where Cr is the red component and Cb is the blue component.

[0040] S4: Fusion of two calculation methods: When the two calculation methods are fused, if the intensity of visible light is stronger than that of infrared light, the method of step S2 is used to calculate the image depth of the visible light and infrared light; if the intensity of infrared light is stronger than that of visible light, the method of step S3 is used to calculate the image depth of the visible light and infrared light.

[0041] According to the disclosure and teaching of the above description, those skilled in the art to which the present invention belongs may also make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the invention should also fall within the scope of protection of the claims of the present invention. In addition, although some specific terms are used in this specification, these terms are only for the convenience of description and do not constitute any limitation to the present invention.

Claims

1. A visible light and mid-to-far infrared light multi-lens multi-sensor information fusion device, characterized in that: It comprises a circuit board, at least two infrared sensors and at least two visible light sensors, wherein the infrared sensor is connected to an infrared lens, and the visible light sensor is connected to a visible light lens; a control and signal processing integrated circuit is integrated on the circuit board, and the infrared sensor and the visible light sensor are connected to the control and signal processing integrated circuit respectively through electrical signals or interface circuits, wherein: The infrared lens receives infrared electromagnetic waves and collects image information; The visible light lens receives visible light electromagnetic waves and collects image information; The image information collected by the infrared lens and the visible light lens is transmitted to the control and signal processing integrated circuit through the infrared sensor and the visible light sensor respectively. The control and signal processing integrated circuit performs depth calculation on the image information collected by the infrared lens and the visible light lens respectively, and fuses and enhances the calculation results to obtain clear image information; The infrared sensor is a mid-to-far infrared sensor, and the infrared lens is a mid-to-far infrared lens; The visible light includes near infrared light; The infrared sensor and the infrared lens are arranged in a one-to-one matching arrangement, and the visible light sensor and the visible light lens are arranged in a one-to-one matching arrangement; The control and signal processing integrated circuit calculates the image depth of the infrared light image information by color classification based on the RGB model or the YCrCb model; The infrared lens receives mid- and far-infrared electromagnetic waves with wavelengths of 3-8 microns, 8-15 microns and 15-1000 microns.

2. The information fusion device of visible light and mid-to-far infrared light with multiple lenses and multiple sensors as claimed in claim 1, characterized in that: The wavelengths of the visible light electromagnetic waves received by the visible light lens are: 0.4 microns-0.77 microns and 0.78 microns-1.1 microns.

3. The visible light and mid-to-far infrared light multi-lens multi-sensor information fusion device as claimed in claim 1, characterized in that: The circuit board is also externally connected to a communication antenna and / or a communication interface.

4. A method based on the information fusion device of visible light and mid-to-far infrared light multi-lens multi-sensor as claimed in any one of claims 1 to 3, characterized in that: The method comprises the following steps: S1: Calibration and adjustment of the device: place a calibration object, turn on the device, the visible light lens and the infrared lens receive visible light electromagnetic waves and infrared light electromagnetic waves respectively and transmit them to the calibration object, use the principle of epipolar geometry to find the epipolar lines, and then find the corresponding points of the calibration object on the visible light sensor and the infrared sensor, and then find the distance of the calibration object relative to the visible light lens and the infrared lens, compare the calculated distance with the distance of the actual object, if the results are consistent, the calibration is completed, if the results are inconsistent, adjust the device and measure and compare again until the comparison results are consistent; S2: Visible light depth measurement: the visible light lens emits visible light electromagnetic waves, which hit obstacles and are then received back by the visible light sensor. The visible light sensor transmits the received visible light image information to the control and signal processing integrated circuit. The control and signal processing integrated circuit calculates the image depth of the visible light image by calculating the maximum value of the correlation function. S3: Infrared light depth measurement: The infrared lens emits visible light electromagnetic waves, which hit obstacles and are then received back by the infrared sensor. The infrared sensor transmits the received infrared light image information to the control and signal processing integrated circuit. The control and signal processing integrated circuit calculates the image depth of the infrared light image information using color classification based on the RGB model or the YCrCb model; S4: Fusion of two calculation methods: When the two calculation methods are fused, if the visible light is stronger than the infrared light, the method of step S2 is used to calculate the image depth of the visible light and the infrared light; if the infrared light is stronger than the visible light, the method of step S3 is used to calculate the image depth of the visible light and the infrared light.

Images