A multi-source information ranging method based on infrared laser radar and visible light camera

Abstract

The application discloses a kind of multi-source information ranging method based on infrared laser radar and visible light camera, and the ranging equipment and camera calibration device designed are integrated into one;First, the device is placed on flat ground, start the device, calibration device can automatically extend, ranging method integrates laser radar and visible light camera two kinds of sensor information;After starting measurement, rotating shaft drives camera and laser radar to rotate a circle;A number of image sequences are fused;Then, the point cloud image obtained by laser radar scanning is processed;Finally, the processed laser radar point cloud and fused visible light camera image are fused, and a 4-dimensional image containing depth information is obtained;The application realizes automation, module integration, and the device has very high portability, and has wide application value in building surveying and mapping, three-dimensional modeling and other fields.

Description

Technical Field

[0001] This invention relates to the field of multi-source information ranging, and more particularly to a multi-source information ranging method based on infrared lidar and a visible light camera. Background Technology

[0002] With national development and construction, the application of ranging systems is becoming increasingly widespread, finding crucial applications in fields such as building construction, scene modeling, and autonomous driving. Currently, there are many ranging methods, such as infrared ranging, laser ranging, and ultrasonic ranging. Each method has its advantages and disadvantages, and the ranging results obtained from a single ranging tool are often insufficient to meet industrial needs. Furthermore, the tools required for camera calibration before ranging must be carried separately, failing to meet portability requirements.

[0003] Therefore, a multi-source information ranging method is needed to integrate visible light and lidar information, fuse the two ranging results, and display the fused result. This provides better visualization of the ranging results, and the system output can be directly used for subsequent detection, modeling, and other fields. Furthermore, a calibration device needs to be integrated into the ranging system to address the issue of system portability. Summary of the Invention

[0004] To address the shortcomings of existing technologies, a multi-source information ranging method based on infrared lidar and visible light camera is provided to solve the problem that existing surveying and mapping systems require post-integration of multi-source information.

[0005] A multi-source ranging method based on infrared lidar and a visible light camera specifically includes the following steps:

[0006] Step 1: Set up the ranging equipment around a flat area to prepare for ranging;

[0007] The device is an integrated unit of an information sampling sensor and a calibration device. The information sampling sensor consists of a visible light camera and a lidar sensor, i.e., several lidar dot arrays, which are fixed on a rotating axis. The calibration device, i.e., the camera calibration device, opens to display a 10*10 chessboard image. After the device is started, the calibration device extends from the ranging device, the chessboard image unfolds, and after stabilization, the ranging device emits a prompt sound to indicate that the ranging device has reached stability. When the device is not in use, the calibration device is retracted into the ranging device.

[0008] Step 2: Establish the relationship between the pixel coordinates of objects in the image captured by the visible light camera and the actual world coordinates of the objects, and calibrate the visible light camera;

[0009] First, define the world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system;

[0010] World coordinate system: Xw, Yw, Zw; the coordinate system of the real three-dimensional world, introduced to describe the position of the target object in the real world. With the camera as the origin, three mutually perpendicular axes are arbitrarily chosen to establish a three-dimensional coordinate system. Xw, Yw, and Zw represent the coordinates of the object in the world coordinate system; the unit is meters.

[0011] Camera coordinate system: Xc, Yc, Zc; The camera coordinate system is a three-dimensional rectangular coordinate system established with the camera's focus center as the origin and the optical axis as the Z-axis; Xc, Yc, and Zc represent the coordinates of the object in the camera coordinate system; the unit is meters (m).

[0012] Image coordinate system: x, y; The origin of the image coordinate system is the image center, and x and y represent the position of the object in the image coordinate system, in meters (m);

[0013] Pixel coordinate system: u, v; the origin of the pixel coordinate system is the image center, and x and y represent the position of the object in the image in the pixel coordinate system, with the unit being pixels;

[0014] Step 2.1: Establish the coordinate transformation relationship between the world coordinate system and the camera coordinate system;

[0015]

[0016] Where R is the rotation matrix R = [r1, r2, r3], and t is the translation vector;

[0017] Step 2.2: Establish the coordinate transformation relationship between the camera coordinate system and the ideal image coordinate system;

[0018] Let point M be the location of the image point in the camera coordinate system, and let its coordinates in the image be:

[0019]

[0020] In the formula, f is the camera focal length, X M Y M Let X be the coordinates of a point in the camera coordinate system. p Y p These are the coordinates of the corresponding point in the ideal image coordinate system;

[0021] Step 2.3: Establish the coordinate transformation relationship from the ideal image coordinate system to the actual image coordinate system;

[0022] Considering radial distortion, the relationship between the ideal image coordinate system coordinates and the actual image coordinate system coordinates before and after correction is as follows:

[0023]

[0024] Considering tangential distortion, the relationship between the ideal image coordinate system coordinates and the actual image coordinate system coordinates before and after correction is as follows:

[0025]

[0026] Wherein, the distortion parameters are k1, k2, k3, p1, p2, which are real numbers; r 2 =x 2 +y 2 x and y are the coordinates before correction, and are real numbers. rcorr Y rcorr To determine the ideal image coordinate system coordinates after correcting radial distortion, X tcorr Y tcorr The ideal image coordinates after correcting tangential distortion are denoted as X. c Y c , represents the coordinates in the actual image coordinate system;

[0027] Step 2.4: Establish the coordinate transformation relationship between the actual image coordinate system and the pixel coordinate system;

[0028]

[0029] Where (u0, v0) are the coordinates of the origin of the pixel coordinate system in the image coordinate system, (μ, v) are the coordinates of the transformed pixel coordinate system, and d x d y This represents the dimensions of each pixel in the x-axis and y-axis directions in the image coordinate system.

[0030] Step 2.5: Define the intrinsic parameter matrix;

[0031]

[0032] Step 2.6: Perform feature point matching between the calibration image captured by the camera and the actual calibration image;

[0033] The unfolded calibration plate is fixed on the device. Each vertex of the square on the calibration plate is defined as a feature point. Points on the edge of the calibration plate are discarded. A total of 100 feature points are taken from the inside. The coordinates of the feature points in the world coordinate system can be known.

[0034] The calibration board is seen in the camera. The coordinates of each feature point on the calibration board in the pixel coordinate system are viewed in sequence to establish 100 pairs of correspondences.

[0035] Step 2.7: Based on Step 2.6, solve for the transformation parameters R and t in Step 2.1:

[0036] Using the 100 pairs of correspondences from step 2.6, the homography matrix H is solved using the LM algorithm. First, the homography matrix is ​​defined as:

[0037]

[0038] Where s is a scaling factor of arbitrary scale, solve the above equation to obtain coefficients r1, r2, t, and substitute them into steps 2.1-2.4 to obtain camera calibration;

[0039] Step 3: Activate the lidar sensor to scan the surrounding environment. The camera and lidar sensor are fixed on a rotating axis; the ranging device outputs the current rotation angle θ of the rotating axis in real time.

[0040] Step 4: When the rotation angle of the ranging device reaches the preset threshold T, i.e. (θmodT)=0, the visible light camera starts, takes an image, and stores it in the visible light image sequence. T is 90°. If θ≤360°, return to step 3; otherwise, proceed to step 5. The device rotates one revolution and obtains a total of 4 visible light images.

[0041] Step 5: The ranging device rotates one full circle, and based on the scanning information from the lidar sensor in Step 3, it obtains the point cloud information around the perimeter. The lidar is then turned off.

[0042] Step 6: Stitch together the four visible light images obtained in Steps 3 and 4;

[0043] Step 6.1: Arrange the four visible light images according to their acquired temporal order to obtain the sorted visible light sequence;

[0044] Step 6.2: Take img1 and img2 from the sorted visible light sequence, and use the Harris corner detection method to extract corners to obtain the corner sets of the two images;

[0045] Step 6.3: For each corner point, expand it outward by 5 pixels to obtain a circle with a radius of 5 pixels. Sum the pixels inside the circle to obtain the feature description value corresponding to that corner point:

[0046]

[0047] Step 6.4: Match the corner points in img1 and img2. If the feature description values ​​of the two corner points are less than the threshold T, they are considered to be matched. The threshold T is set to 20. Randomly select one corner point in img1 and one corner point in img2 as a combination. Traverse all combinations to complete the matching of all corner points in img1 and img2 and obtain the matching corner point combinations.

[0048] Step 6.5: Find matching corner point combinations and delete incorrect matches;

[0049] Take a pair of corner points A(x) that are farthest apart. a ,y a ),B(x b ,y b ), and take its matching corner point A'(xa ,y a ),B'(x b ,y b If the following conditions are met:

[0050]

[0051] If corner points A and B are considered to match, and if they do not match, select the next furthest corner point combination and repeat step 6.5 until the furthest, successfully matched corner point pair is found.

[0052] Step 6.6: For successfully matched images, perform a weighted fusion of the images between corner points A and B, with a weight of 0.5. For images outside the corner points, directly extract the values ​​to obtain the stitched image fusion.

[0053]

[0054] Step 6.7: If img2 is the last image in the sorted visible light sequence, then end, completing the stitching of 4 visible light images to obtain a single visible light image fusion; otherwise, take the next image in the image sequence as img2, take fusion as img1, and return to step 6.2.

[0055] Step 7: Fusion the point cloud obtained in Step 5 and the visible light image obtained in Step 6 to obtain a fused image containing depth information, and finally obtain the ranging result;

[0056] Step 7.1: Take each coordinate in the point cloud and execute steps 2.1-2.4 to map the point cloud world coordinates to the pixel coordinate system to obtain the corresponding pixel coordinates. At the same time, each world coordinate contains depth information, denoted as l;

[0057] The coordinates of the point cloud image after coordinate transformation are denoted as Lid, which is 2-dimensional and used only as a position label; the depth information is denoted as L, which is 1-dimensional.

[0058] Step 7.2: Concatenate Lid and L to obtain a matrix Lid-L containing depth information;

[0059] Step 7.3: Treat Lid-L as an image and use bicubic interpolation to upsample Lid-L. The size of the upsampled image is the same as the size of the fusion image in Step 6.

[0060] Step 7.4: Fuse the L-dimensional information from the Lid-L image into fusion to generate a 4D matrix fusion-l containing depth information, which is the final ranging result. Then turn off the device.

[0061] The beneficial effects of adopting the above technical solution are as follows:

[0062] 1. The ranging device of this invention has a built-in calibration device.

[0063] 2. The ranging device automatically performs calibration after startup, making it easy to operate.

[0064] 3. The ranging device automatically scans the surrounding information, and the ranging process does not require human intervention.

[0065] 3. The parameters obtained from the distance measurement are processed automatically by the device, without the need for manual follow-up processing.

[0066] 4. The ranging output is an image containing depth information, which has a good visualization effect. Attached Figure Description

[0067] Figure 1 This is a schematic diagram of the ranging device structure according to an embodiment of the present invention; wherein 1 is a visible light camera, 2 is a lidar sensor, 3 is a rotating shaft, 4 is a folding rod, and 5 is a camera calibration device;

[0068] Figure 2 This is a schematic diagram of a camera calibration device according to an embodiment of the present invention;

[0069] Figure 3 This is a flowchart of a multi-source information ranging method based on infrared lidar and visible light camera according to an embodiment of the present invention. Detailed Implementation

[0070] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0071] A multi-source ranging method based on infrared lidar and visible light camera, as shown in the appendix. Figure 3 As shown, the specific steps include:

[0072] Step 1: Place the ranging device near the center point of the surveying area, preferably on a flat surface, to prepare for the ranging.

[0073] The device is an integrated unit of an information sampling sensor and a calibration device, as shown in the attached document. Figure 1 As shown; the information sampling sensor consists of a visible light camera and a lidar sensor, i.e., several lidar dot arrays, with the camera and lidar sensor fixed on a rotating axis; the calibration device, i.e., the camera calibration device, opens to display a 10*10 chessboard image, as shown in the attached image. Figure 2 As shown; after the equipment is started, the calibration device extends from the ranging device, the chessboard image unfolds, and after stabilization, the ranging device emits a prompt sound to indicate that the ranging device has reached stability; when the equipment is not in use, the calibration device is retracted into the ranging device.

[0074] Step 2: Establish the relationship between the pixel coordinates of objects in the image captured by the visible light camera and the actual world coordinates of the objects, and calibrate the visible light camera;

[0075] Step 2.1: Start the visible light camera and wait for the calibration plate to appear clearly on the device display screen before starting the calibration; establish the coordinate transformation relationship between the world coordinate system and the camera coordinate system;

[0076] Step 2.2: Establish the coordinate transformation relationship between the camera coordinate system and the ideal image coordinate system;

[0077] Step 2.3: Establish the coordinate transformation relationship from the ideal image coordinate system to the actual image coordinate system;

[0078] Step 2.4: Establish the coordinate transformation relationship between the actual image coordinate system and the pixel coordinate system;

[0079] Step 2.5: Define the intrinsic parameter matrix;

[0080] Step 2.6: Perform feature point matching between the calibration image captured by the camera and the actual calibration image;

[0081] The unfolded calibration plate is fixed on the device. Each vertex of the square on the calibration plate is defined as a feature point. Points on the edge of the calibration plate are discarded. A total of 100 feature points are taken from the inside. Step 2.2 is repeated 100 times. The coordinates of the feature points in the world coordinate system can be known.

[0082] Select the world coordinates of a feature point, and manually select the corresponding pixel coordinates on the image. After selection, you need to confirm it three times, and the error each time should not exceed 10 pixels.

[0083] Step 3: Activate the lidar sensor to scan the surrounding environment. The camera and lidar sensor are fixed on a rotating axis; the ranging device outputs the current rotation angle θ of the rotating axis in real time.

[0084] Step 4: When the rotation angle of the ranging device reaches the preset threshold T, i.e. (θmodT)=0, the visible light camera starts, takes an image, and stores it in the visible light image sequence. T is 90°. If θ≤360°, return to step 3; otherwise, proceed to step 5. The device rotates one revolution and obtains a total of 4 visible light images.

[0085] Step 5: The ranging device rotates one full circle, and based on the scanning information from the lidar sensor in Step 3, it obtains the point cloud information around the perimeter. The lidar is then turned off.

[0086] Step 6: Stitch together the four visible light images obtained in Steps 3 and 4;

[0087] Step 6.1: Arrange the four visible light images according to their acquired temporal order to obtain the sorted visible light sequence;

[0088] Step 6.2: Take img1 and img2 from the sorted visible light sequence, and use the Harris corner detection method to extract corners to obtain the corner sets of the two images;

[0089] Step 6.3: For each corner point, expand it outward by 5 pixels to obtain a circle with a radius of 5 pixels. Sum the pixels inside the circle to obtain the feature description value corresponding to the corner point.

[0090] Step 6.4: Match the corner points in img1 and img2. If the feature description values ​​of the two corner points are less than the threshold T, they are considered to be matched. The threshold T is set to 20. Randomly select one corner point in img1 and one corner point in img2 as a combination. Traverse all combinations to complete the matching of all corner points in img1 and img2 and obtain the matching corner point combinations.

[0091] Step 6.5: Find matching corner point combinations and delete incorrect matches;

[0092] Step 6.6: For successfully matched images, perform a weighted fusion of the images between corner points A and B, with a weight of 0.5. For images outside the corner points, directly extract the values ​​to obtain the stitched image fusion.

[0093]

[0094] Step 6.7: If img2 is the last image in the sorted visible light sequence, then end, completing the stitching of 4 visible light images to obtain a single visible light image fusion; otherwise, take the next image in the image sequence as img2, take fusion as img1, and return to step 6.2.

[0095] Step 7: Fusion the point cloud obtained in Step 5 and the visible light image obtained in Step 6 to obtain a fused image containing depth information, and finally obtain the ranging result;

[0096] Step 7.1: Take each coordinate in the point cloud and execute steps 2.1-2.4 to map the point cloud world coordinates to the pixel coordinate system to obtain the corresponding pixel coordinates. At the same time, each world coordinate contains depth information, denoted as l;

[0097] The coordinates of the point cloud image after coordinate transformation are denoted as Lid, which is 2-dimensional and used only as a position label; the depth information is denoted as L, which is 1-dimensional.

[0098] Step 7.2: Concatenate Lid and L to obtain a matrix Lid-L containing depth information;

[0099] Step 7.3: Treat Lid-L as an image and use bicubic interpolation to upsample Lid-L. The size of the upsampled image is the same as the size of the fusion image in Step 6.

[0100] Step 7.4: Fuse the L-dimensional information from the Lid-L image into fusion to generate a 4D matrix fusion-l containing depth information, which is the final ranging result. Then turn off the device.

Claims

1. A multi-source information ranging method based on infrared lidar and a visible light camera, characterized in that, Specifically, the following steps are included: Step 1: Set up the ranging equipment around a flat area to prepare for ranging; Step 2: Establish the relationship between the pixel coordinates of objects in the image captured by the visible light camera and the actual world coordinates of the objects, and calibrate the visible light camera; Step 3: Activate the lidar sensor to scan the surrounding environment. The camera and lidar sensor are fixed on a rotating axis; the ranging device outputs the current rotation angle of the rotating axis in real time. ; Step 4: Once the rotation angle of the ranging device reaches the preset threshold T, that is... The visible light camera starts, takes a picture, and stores it in the visible light image sequence, with T set to 90; if Return to step 3; otherwise, proceed to step 5. The device rotates once to obtain a total of 4 visible light images. Step 5: The ranging device rotates one full circle, and based on the scanning information from the lidar sensor in Step 3, it obtains the point cloud information around the perimeter. The lidar is then turned off. Step 6: Stitch together the four visible light images obtained in Steps 3 and 4; Step 6.1: Arrange the four visible light images according to their acquired temporal order to obtain the sorted visible light sequence; Step 6.2: Take from the sorted visible light sequence The Harris corner detection method is used to extract corners, resulting in a set of corners for the two images. Step 6.3: For each corner point, expand it outward by 5 pixels to obtain a circle with a radius of 5 pixels. Sum the pixels inside the circle to obtain the feature description value corresponding to that corner point: ； Step 6.4: For The corner points are matched, and if the feature description values ​​of two corner points are less than a threshold, the matching is performed. If so, it is considered a match, threshold Take 20; randomly select one corner point from img1 and one corner point from img2 as a combination, traverse all combinations, complete the matching of all corner points of img1 and img2, and obtain the matching corner point combinations; Step 6.5: Find matching corner point combinations and delete incorrect matches; Take a pair of corner points that are furthest apart. Take its matching corner points If the following conditions are met: ； If corner points A and B are considered to match, and if they do not match, select the next furthest corner point combination and repeat step 6.5 until the furthest, successfully matched corner point pair is found. Step 6.6: For successfully matched images, perform weighted fusion on the images between corner points A and B, with a weight of 0.

5. For images outside the corner points, directly extract the values ​​to obtain the stitched image fusion. ； Step 6.7: If img2 is the last image in the sorted visible light sequence, then end the process and complete the stitching of 4 visible light images to obtain a single visible light image fusion; otherwise, take the next image in the image sequence as img2, take fusion as img1, and return to step 6.

2. Step 7: Fusion is performed on the point cloud obtained in Step 5 and the visible light image obtained in Step 6 to obtain a fused image containing depth information, and finally the ranging result is obtained.

2. The multi-source information ranging method based on infrared lidar and visible light camera according to claim 1, characterized in that, The device described in step 1 is an integrated device for information sampling sensor and calibration. The information sampling sensor consists of a visible light camera and a lidar sensor, i.e., several lidar dot arrays. The camera and lidar sensor are fixed on a rotating axis. The calibration device is a camera calibration device, which, when opened, becomes a... The device displays a chessboard image; after the device is started, the calibration device extends from the ranging device, the chessboard image unfolds, and after stabilization, the ranging device emits a prompt sound to indicate that the ranging device has reached stability; when the device is not in use, the calibration device is retracted into the ranging device.

3. The multi-source information ranging method based on infrared lidar and visible light camera according to claim 1, characterized in that, Step 2 should first define the world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system; World coordinate system: , , The coordinate system of the real three-dimensional world was introduced to describe the position of an object in the real world. It is established with the camera as the origin and three mutually perpendicular axes. , , These represent the coordinates of the object in the world coordinate system; the unit is meters (m). Camera coordinate system: , , The camera coordinate system is a three-dimensional rectangular coordinate system established with the camera's focal center as the origin and the optical axis as the Z-axis. , , This represents the coordinates of the object in the camera coordinate system; the unit is meters (m). Image coordinate system: x, y; The origin of the image coordinate system is the image center, and x and y represent the position of the object in the image coordinate system, in meters (m); Pixel coordinate system: u, v; the origin of the pixel coordinate system is the image center, and u and v represent the position of the object in the image in the pixel coordinate system, with the unit being pixels.

4. The multi-source information ranging method based on infrared lidar and visible light camera according to claim 3, characterized in that, Step 2 is as follows: Step 2.1: Establish the coordinate transformation relationship between the world coordinate system and the camera coordinate system; ； Let be a rotation matrix. , It is a translation vector; Step 2.2: Establish the coordinate transformation relationship between the camera coordinate system and the ideal image coordinate system; Let point M be the location of the image point in the camera coordinate system, and let its coordinates in the image be: ； In the formula, f is the camera focal length. The coordinates of the point in the camera coordinate system. These are the coordinates of the corresponding point in the ideal image coordinate system; Step 2.3: Establish the coordinate transformation relationship from the ideal image coordinate system to the actual image coordinate system; Considering radial distortion, the relationship between the ideal image coordinate system coordinates and the actual image coordinate system coordinates before and after correction is as follows: ； Considering tangential distortion, the relationship between the ideal image coordinate system coordinates and the actual image coordinate system coordinates before and after correction is as follows: ； Among them, the distortion parameter is , where is a real number; , The coordinates before correction are real numbers. The coordinates are the ideal image coordinates after correcting radial distortion. The ideal image coordinate system coordinates after correcting tangential distortion are denoted as... , representing the actual image coordinate system coordinates; Step 2.4: Establish the coordinate transformation relationship between the actual image coordinate system and the pixel coordinate system; ； in, These are the coordinates of the origin of the pixel coordinate system in the image coordinate system. These are the transformed pixel coordinates. This represents the dimensions of each pixel in the x-axis and y-axis directions in the image coordinate system. Step 2.5: Define the intrinsic parameter matrix; ； Step 2.6: Perform feature point matching between the calibration image captured by the camera and the actual calibration image; The unfolded calibration plate is fixed on the device. Each vertex of the square on the calibration plate is defined as a feature point. Points on the edge of the calibration plate are discarded. A total of 100 feature points are taken from the inside. The coordinates of the feature points in the world coordinate system can be known. The calibration board is seen in the camera. The coordinates of each feature point on the calibration board in the pixel coordinate system are viewed in sequence to establish 100 pairs of correspondences. Step 2.7: Based on Step 2.6, solve for the transformation parameters R and t in Step 2.1: Using the 100 pairs of correspondences from step 2.6, the homography matrix H is solved using the LM algorithm. First, the homography matrix is ​​defined as: ； Where s is a scaling factor of arbitrary scale, the coefficients are obtained by solving the above equation. Substitute steps 2.1-2.4 into the camera calibration.

5. The multi-source information ranging method based on infrared lidar and visible light camera according to claim 1, characterized in that, Step 7 specifically includes: Step 7.1: Take each coordinate in the point cloud and execute steps 2.1-2.4 to map the point cloud world coordinates to the pixel coordinate system to obtain the corresponding pixel coordinates. At the same time, each world coordinate contains depth information, denoted as l; The coordinates of the point cloud image after coordinate transformation are denoted as Lid, which is 2-dimensional and used only as a position label; the depth information is denoted as L, which is 1-dimensional. Step 7.2: Concatenate Lid and L to obtain a matrix Lid-L containing depth information; Step 7.3: Treat Lid-L as an image and use bicubic interpolation to upsample Lid-L. The size of the upsampled image is the same as the size of the fusion image in Step 6. Step 7.4: Fuse the L-dimensional information from the Lid-L image into fusion to generate a 4D matrix fusion-l containing depth information, which is the final ranging result. Then turn off the device.

Images