Visible light and near infrared registration system and registration method based on laser radar

Abstract

The visible light and near infrared registration system and registration method based on laser radar, which relates to the image processing of vehicle-mounted visual system and the field of computer vision technology, solves the problems of slow speed, poor real-time performance and low efficiency of existing multi-sensor image registration. It includes a visible light camera, a near infrared camera, a laser radar, an image registration module and a display module for displaying the spliced ​​image and the original image; the image registration module is provided with an input end, an output end and an image registration internal processor, the image registration internal processor receives the image information from the input end and uses the image registration algorithm for calculation, the near infrared camera output interface and the visible light camera output interface are respectively connected to the input end of the image registration module, the output port of the laser radar is connected to the input end of the image registration module; the image display module is connected to the output end of the image registration module, and is used to display the images before and after registration. The method of the present invention has fast processing speed and high efficiency, and is suitable for visual systems that require real-time processing.

Description

Technical Field

[0001] The present invention relates to the field of vehicle-mounted vision system image processing and computer vision technology, and in particular to a laser radar-based visible light and near-infrared registration system and registration method. Background Art

[0002] Image registration is a method of matching images at different times, from different sensors, and at different locations. Image registration technology is the prerequisite for visual processing tasks such as image fusion and image stitching. A well-registered image has an important impact on subsequent processing tasks. It has a wide range of applications in robot visual navigation, medical image analysis, target detection and recognition, etc.

[0003] The image registration technology based on LiDAR point cloud solves the affine matrix between the two source images by converting the camera coordinate system and eliminating the wrong feature point pairs.

[0004] The traditional image registration method is to calculate the transformation matrix by calculating the feature descriptor as the feature point pair. However, this method is very time-consuming and not suitable for real-time visual processing systems. In addition, its sensor is likely to be affected by many factors such as light intensity and environmental changes. LiDAR has strong adaptability to the environment, good stability and high precision. Therefore, it can maintain its advantages in complex environments. Summary of the invention

[0005] In order to solve the problems of slow speed, poor real-time performance and low efficiency of multi-sensor image registration in the prior art, the present invention provides a visible light and near-infrared registration system and registration method based on laser radar, so as to improve the efficiency of the registration algorithm and reduce its registration time.

[0006] A visible light and near infrared registration system based on laser radar includes a visible light camera, a near infrared camera, a laser radar, an image registration module and a display module for displaying a spliced ​​image and an original image; the characteristics are:

[0007] The image registration module is provided with an input end, an output end and an internal image registration processor. The internal image registration processor receives image information from the input end and uses an image registration algorithm to perform calculations. The near-infrared camera output interface and the visible light camera output interface are respectively connected to the input end of the image registration module, and the output port of the laser radar is connected to the input end of the image registration module; the image display module is connected to the output end of the image registration module for displaying images before and after registration.

[0008] The visible light and near infrared registration method based on laser radar is implemented based on the visible light and near infrared registration system of laser radar. The registration method is specifically implemented by the following steps:

[0009] Step 1: Set the reference image and the image to be registered;

[0010] Input visible light image VIS, near infrared image NIR, 3D point cloud image information C P , visible light camera internal parameter IS VIS , visible light camera external parameter ES VIS , near infrared camera internal reference IS NIR , near infrared camera external parameters ES NIR ; Using the visible light image VIS as the image to be registered and the near infrared image NIR as the reference image;

[0011] Step 2: Convert the points contained in the 3D point cloud image information to the near-infrared and visible light image coordinate systems respectively:

[0012] The matrix relationship from the world coordinate system to the camera coordinate system is as follows:

[0013]

[0014] Where K is the camera’s internal parameter, [R t] is the camera’s external parameter, and X W ,Y W ,Z W Z is the coordinate of the 3D point cloud image acquisition point in the world coordinate system, and u and v are the coordinates in the image coordinate system. C The coordinates of the camera coordinate system are not related to the external parameters of the camera.

[0015] The visible light camera internal reference IS VIS External reference ES VIS and near infrared camera internal reference IS NIR External reference ES NIR Substitute the above formulas respectively, then construct the relationship between the 3D point cloud image and the image coordinates on the visible light and near-infrared images, and perform normalization, as shown in the following formula:

[0016]

[0017]

[0018] Where P VIS ,P NIR is the coordinate of the point cloud information point on the two camera images, P VIS is the feature point of the visible light camera, P NIR are the feature points of the visible light camera;

[0019] Step 3: Screen the obtained feature points;

[0020] Set the boundary threshold to remove feature points that are beyond the boundary and on the boundary. The specific expression is as follows:

[0021]

[0022] Where, C∈(RGB,NIR),(X P , Y P ) is P VIS , P NIR The feature point coordinates, M C ,N C is the size of the images collected by the two cameras, & represents the logical AND operation; retain all feature points covered in the image, remove feature points beyond the boundary and on the boundary, and obtain the filtered feature points LP VIS ,LP NIR ;

[0023] Step 4: Obtain the filtered feature points LP according to step 3 VIS ,LP NIR Obtain the homography matrix between the image to be registered VIS and the reference image NIR, and then perform affine transformation on the image to be registered according to the obtained homography matrix to obtain the final registered image.

[0024] Beneficial effects of the present invention: The near-infrared and visible light image registration system based on laser radar described in the present invention can solve the problems of slow image registration speed, poor real-time performance and insufficient reliability. The registration system adopted by the present invention can obtain registered images more quickly and with higher reliability, so that users can obtain better quality images and user experience.

[0025] The present invention is based on the visible light and near-infrared registration method of laser radar, which can realize fast and real-time image registration. It has high efficiency and strong stability. Compared with the traditional registration method of extracting feature points, the present invention is not affected by light intensity and can perform image registration under low light and high exposure conditions. Due to the stability of laser radar, it is not affected by harsh environments and has a stronger ability to cope with environmental changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic structural diagram of the visible light and near infrared registration system based on laser radar according to the present invention;

[0027] Figure 2 It is a schematic diagram of the three-dimensional structure of the visible light and near-infrared registration system based on laser radar according to the present invention;

[0028] Figure 3 This is a principle block diagram of the visible light and near-infrared registration system based on laser radar described in the present invention;

[0029] Figure 4This is a flow chart of the visible light and near-infrared registration method based on laser radar described in the present invention. DETAILED DESCRIPTION

[0030] Specific implementation method 1. Combination Figures 1 to 3 This embodiment describes that the visible light and near infrared registration system based on laser radar can solve the problems of slow image registration speed, poor real-time performance and insufficient reliability. The registration system adopted by the present invention can obtain the registered image more quickly and with higher reliability, so that users can obtain better quality images and user experience.

[0031] A near-infrared and visible light image registration system based on laser radar, comprising: a visible light camera 1, a near-infrared camera 2, a laser radar 3, an image registration module 4 and an image display module 5;

[0032] Among them, the visible light camera 1, the near infrared camera 2 and the laser radar 3 are respectively connected to the image registration module 4, and the image registration module 4 is connected to the display module 5. The image registration module 4 contains an input end, an output end and an internal processor for image registration. The output interface of the near infrared camera 2 and the output interface of the visible light camera 1 are respectively connected to the input end of the image registration module 4. The image display module 5 is connected to the image registration module 4 to display the images before and after registration.

[0033] The visible light camera 1 is used to obtain the visible light image of the measured target; the near infrared camera 2 is used to obtain the near infrared image of the measured target; the laser radar 3 is used to obtain the 3D point cloud information of the surrounding environment of the measured target; the image registration module 4 is used to assist the registration of the near infrared and visible light images by the laser radar; the display module 5 is used to display the spliced ​​image and the original image;

[0034] The visible light camera 1 , the near infrared camera 2 and the laser radar 3 are respectively connected to the image registration module 4 , and the image registration module is connected to the display module 5 .

[0035] The visible light camera 1 includes a GIGE output interface, a 12VDC (6V-30V) power interface, a visible light lens and a visible light sensor. The power interface provides power to the visible light sensor, the GIGE output interface is used to transmit the collected visible light image to the image registration module 4, and the visible light lens is used to capture color image information in the visible light frequency band.

[0036] The near-infrared camera 2 includes a GIGE output interface, a +6~+24VDC power interface, a near-infrared lens and a near-infrared sensor. The power interface provides power to the near-infrared sensor, the GIGE output interface is used to transmit the collected near-infrared image to the image registration module, and the near-infrared lens is used to receive and gather the near-infrared image information of the measured target.

[0037] In this embodiment, the laser radar 3 includes a GIGE output interface, a 24VDC power interface, and a laser scanning sensor. The laser scanning sensor is used to scan the surrounding environment and generate a 3D point cloud image. The GIGE output interface is used to transmit the point cloud image to the processing module.

[0038] The image registration module 4 includes an input terminal, an output terminal, a power supply terminal and an internal image registration processor. The input terminal is connected to the GIGE port of the near infrared and visible light cameras to transmit the image of the object to be measured to the image registration module, which is then processed by the internal processor and finally transmitted to the image display module through the output terminal.

[0039] In this embodiment, the visible light camera 1 and the near-infrared camera 2 need to be kept horizontal, and their lenses are on the same horizontal plane. The visible light camera 1 is located in the middle position, which can capture color images with a larger field of view and more conducive to visual perception. The near-infrared camera 2 is located on the left side of the visible light camera 1, in order to better cooperate with the visible light camera for image registration. The laser radar is located at the lower end of the visible light camera 1 and the near-infrared camera 2, and its front end should be on the same surface as the visible light camera lens and the near-infrared camera lens, and its structure should not change. The image registration module is located on the rear side of the device and below the laser radar, which is easy to connect the output ends of the two cameras and the laser radar with the input end of the image registration module, and can better maintain stability.

[0040] Specific implementation method 2: Combination Figure 4 This embodiment is described. This embodiment adopts the registration method implemented by the visible light and near infrared registration system based on laser radar described in the specific embodiment 1. The method is implemented by the following steps:

[0041] Step 1: Set the reference image and the image to be registered.

[0042] Input visible light image VIS, near infrared image NIR, 3D point cloud information C P , visible light camera internal parameter IS VIS , external reference ES VIS , visible light camera internal parameter IS NIR , external reference ES NIR In this process, the visible light image is set as the image to be registered and the near infrared image is set as the reference image.

[0043] Step 2: Convert the points contained in the 3D point cloud information into the image coordinate system.

[0044] The camera can capture three-dimensional objects in a two-dimensional picture. This process can be called transforming the target point in three-dimensional space into a two-dimensional projection.

[0045] It is known that the matrix relationship from the world coordinate system to the camera coordinate system is as follows:

[0046]

[0047] Where K is the camera’s internal parameter, [R t] is the camera’s external parameter, and X W ,Y W ,Z W Z is the coordinate of the 3D point cloud image acquisition point in the world coordinate system, and u and v are the coordinates in the image coordinate system. C The coordinates of the camera coordinate system are not related to the external parameters of the camera.

[0048] The visible light camera internal reference IS VIS External reference ES VIS and near infrared camera internal reference IS NIR External reference ES NIR Substitute the above formulas respectively, then construct the relationship between the 3D point image and the image coordinates on the visible light and near-infrared images, and perform normalization, as shown in the following formula:

[0049]

[0050]

[0051] Where P VIS ,P NIR is the coordinate of the point cloud information point on the two camera images, called P VIS is the feature point of the visible light camera, called P NIR are the feature points of the visible light camera.

[0052] In step 3, we have obtained the feature point pairs corresponding to the visible light and near-infrared images. However, many of these points have exceeded the boundaries of the image. This is because the image captured by the camera is much smaller than the range covered by the point cloud image, so we need to remove these feature point pairs that exceed the boundaries. To address this problem, we can set a boundary threshold to remove this type of feature point pairs. The specific expression is as follows:

[0053]

[0054] Where, C∈(RGB,NIR),(X P , Y P ) is P VIS , P NIR The feature point coordinates, M C ,N C is the size of the images collected by the two cameras, & represents the AND operation; retain all feature points covered in the image, remove feature points beyond the boundary and on the boundary, and obtain the filtered feature points LPVIS ,LP NIR ;

[0055] Step 4: Obtain the filtered feature points LP according to step 3 VIS ,LP NIR The homography matrix is ​​calculated for the image to be registered VIS and the reference image NIR, and then the image to be registered is affine transformed according to the obtained homography matrix to obtain the final registered image.

[0056] In this embodiment, the measured target is captured by visible light and near-infrared CCD cameras, and transmitted from the camera GIGE interface to the image registration module through the internal sensor of the near-infrared camera. At the same time, the laser radar captures 3D point cloud information and transmits the environmental information from the internal system of the radar to the image registration module. After being calculated by the internal processor of the image registration module, it is transmitted from the image registration module to the image display module for display. The image display module is connected to the image registration module to display the images before and after registration.

[0057] In this embodiment, the visible light camera is POE-C2010. The near infrared camera is MARS-1705. The laser radar is RS-LiDAR-M1. The image registration module is NVIDIA visual development component. The image display module is a 4K display screen.

Claims

1. A visible light and near infrared registration method based on laser radar, characterized by: The registration method is implemented based on a visible light and near infrared registration system of a laser radar, wherein the registration system includes a visible light camera, a near infrared camera, a laser radar, an image registration module, and a display module for displaying a spliced ​​image and an original image; The image registration module is provided with an input end, an output end and an image registration internal processor. The image registration internal processor receives image information from the input end and uses the image registration algorithm to perform calculations. The near-infrared camera output interface and the visible light camera output interface are respectively connected to the input end of the image registration module, and the output port of the laser radar is connected to the input end of the image registration module; the image display module is connected to the output end of the image registration module, and is used to display the images before and after registration; The registration method is specifically implemented by the following steps: Step 1: Set the reference image and the image to be registered; Input visible light image VIS, near infrared image NIR, 3D point cloud image information C P , visible light camera internal parameter IS VIS , visible light camera external parameter ES VIS , near infrared camera internal reference IS NIR , near infrared camera external parameters ES NIR ; Using the visible light image VIS as the image to be registered and the near infrared image NIR as the reference image; Step 2: Convert the points contained in the 3D point cloud image information to the near-infrared and visible light image coordinate systems respectively: The matrix relationship from the world coordinate system to the camera coordinate system is as follows: in K is the camera’s internal parameter, [ R t] is the external parameter of the camera, X W ,Y W ,Z W is the coordinate of the 3D point cloud image acquisition point in the world coordinate system, u, v are the coordinates in the image coordinate system, Z C The coordinates are in the camera coordinate system. The external parameters of the camera have nothing to do with its internal properties. The visible light camera internal reference IS VIS , external reference ES VIS and near infrared camera internal reference IS NIR , external reference ES NIR Substitute the above formulas respectively, then construct the relationship between the 3D point cloud image and the image coordinates on the visible light and near-infrared images, and perform normalization, as shown in the following formula: Where P VIS ,P NIR is the coordinate of the point cloud information point on the two camera images, P VIS is the feature point of the visible light camera, P NIR are the feature points of the visible light camera; Step 3: Screen the obtained feature points; Set the boundary threshold to remove feature points that are beyond the boundary and on the boundary. The specific expression is as follows: Where, C∈(RGB,NIR),(X P , Y P ) is P VIS , P NIR The feature point coordinates, M C ,N C is the size of the images collected by the two cameras, & represents the logical AND operation; retain all feature points covered in the image, remove feature points beyond the boundary and on the boundary, and obtain the filtered feature points LP VIS ,LP NIR ; Step 4: Obtain the filtered feature points LP according to step 3 VIS ,LP NIR Obtain the homography matrix between the image to be registered VIS and the reference image NIR, and then perform affine transformation on the image to be registered according to the obtained homography matrix to obtain the final registered image.

2. The visible light and near infrared registration method based on laser radar according to claim 1, characterized in that: The near-infrared camera includes a GIGE output interface, a +6~+24V DC power interface, a near-infrared lens and a near-infrared sensor; the power interface provides power to the near-infrared sensor, the GIGE output interface is used to transmit the collected near-infrared image to the image registration module, and the near-infrared lens is used to receive and gather near-infrared image information of the target being measured.

3. The visible light and near infrared registration method based on laser radar according to claim 1, characterized in that: The visible light camera includes a GIGE output interface, a 12V DC power interface, a visible light lens and a visible light sensor; the power interface provides power to the visible light sensor, the GIGE output interface is used to transmit the collected visible light image to the image registration device, and the visible light lens is used to capture color image information in the visible light frequency band.

4. The visible light and near infrared registration method based on laser radar according to claim 1, characterized in that: The laser radar includes a GIGE output interface, a 24V DC power supply interface and a laser scanning sensor; the laser scanning sensor is used to scan the surrounding environment information and generate a 3D point cloud image; the GIGE output interface is used to transmit the 3D point cloud image to the internal processor of the image processing.

Images