LiDAR and Image Calibration for Autonomous Vehicles
Through automated calculation and least squares analysis, the calibration problem of lidar system and camera system is solved, and efficient automatic calibration of lidar and camera system is realized. It is suitable for autonomous vehicles, simplifying the calibration process and improving accuracy.
Patent Information
- Application Number
- CN202180024007.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-07-08
- Filing Date
- 2021-07-07
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2041-07-07
AI Technical Summary
The existing lidar system has difficulties in calibrating camera coordinate systems in autonomous vehicles, especially the lack of automated and efficient methods, which leads to increased human participation and errors. Traditional hand-eye calibration is prone to deterioration when the vehicle posture changes, making it difficult to solve.
Using an automated method without targets, the pose change calculation and association between the camera and lidar system is calculated and correlated with the attitude change between the camera and lidar system, the rotation vector and the scaling rotation vector are used to perform the least squares analysis, and the coordinate system of the camera and lidar modules is automatically calibrated to reduce human participation. And improve calibration accuracy.
It realizes automatic calibration of lidar systems and camera systems, reduces human errors, improves calibration accuracy and efficiency, and is suitable for applications of autonomous and semi-autonomous vehicles, simplifying the calibration process.
Smart Images

Figure CN115427832B_ABST
Abstract
Description
[0001] Cross-references
[0002] This application claims priority to U.S. non-provisional patent application No. 16 / 924,000, filed on July 8, 2020, the entire contents of which are hereby incorporated by reference into this application. Background Art
[0003] Light steering generally involves projecting light in a predetermined direction to facilitate, for example, object detection and ranging, object illumination and scanning, etc. Light steering can be used in many different application areas including, for example, autonomous vehicles, medical diagnostic equipment, etc.
[0004] Modern vehicles are typically equipped with a suite of environmental detection sensors designed to detect objects and landscape features around the vehicle in real time, and these sensors can be used as the basis for many existing and emerging technologies, such as assisted lane change, collision avoidance, and autonomous driving features. Some commonly used sensing systems include optical sensors (e.g., infrared, cameras, etc.), radio detection and ranging (radar) for detecting the presence, direction, distance, and speed of other vehicles or objects, magnetometers (e.g., passive sensing of large ferrous objects such as trucks, cars, or railcars), and light detection and ranging (lidar).
[0005] LiDAR typically uses a pulsed light source and a detection system to estimate the distance to environmental features (e.g., vehicles, structures, etc.). In some systems, laser or light pulses (pulses) are emitted and focused through a lens assembly, and the pulses reflected from objects are collected by a receiver. The time of flight (TOF) of the pulse from the time it is emitted to the time the reflection is received can be measured, and / or the phase difference can be calculated, which can be represented as a single data point. This process can be repeated very quickly over any desired range (typically a 360° 2D plane for ground vehicles and a 3D area for aircraft) to form a collection of points that is dynamic and continuously updated in real time, forming a "point cloud." The point cloud data can be used to estimate the distance, size, and position of objects relative to the LiDAR system, typically with very high fidelity (e.g., to within 5 cm).
[0006] While LiDAR and other sensing systems hold promise for the continued development of fully automated transportation, several challenges remain that limit its widespread adoption. LiDAR systems are typically expensive, large, and cumbersome. In some cases, multiple transmitters may be required to accurately track a scene, especially for systems that require accuracy over a large range and field of view (FOV). While significant progress has been made in moving autonomous vehicle technology toward greater commercial adoption, further improvements are still needed. Summary of the invention
[0007] In some embodiments, the present application discloses a method for calibrating a camera using a lidar system without using a dedicated target.
[0008] In some embodiments, a system for calibrating a vision device with a multi-dimensional range finding device includes a scanner (eg, a lidar module), a camera including one or more image sensors; and one or more processors. One or more processors are configured to acquire a first image using the camera; calculate a first pose of the camera based on the first image; acquire a second image using the camera; calculate a second pose of the camera based on the second image; calculate a first pose change based on the first pose of the camera and the second pose of the camera; generate a first three-dimensional map of the scene using the lidar module; calculate the first pose of the lidar module based on the first three-dimensional map of the scene; generate a second three-dimensional map of the scene using the lidar module; calculate the second pose of the lidar module based on the second three-dimensional map of the scene; calculate a second pose change based on the first pose of the lidar module and the second pose of the lidar module; use a rotation vector to associate the spin of the first pose change with the spin of the second pose change; define a scaled rotation vector based on an initial value of the scaled rotation vector plus a scaled value; analyze the relationship between the first pose change, the second pose change, and the scaled rotation vector to solve the spin between the first pose of the camera and the first pose of the lidar module; and, based on the spin between the first pose of the camera and the first pose of the lidar module, calibrate the camera using the lidar module. In some embodiments, a lidar module is mounted on a vehicle; relative motion and rotation between the camera and the lidar module are prevented; the scaling value is multiplied by the sum of the rotation vector of the first posture change and the rotation vector of the second posture change before the scaling value is added to the initial value of the scaling rotation vector; and / or the analysis is a least squares analysis.
[0009] In some embodiments, a method for calibrating a visual device with a multi-dimensional distance measurement device includes acquiring a first image using a camera; calculating a first posture of the camera based on the first image; acquiring a second image using the camera; calculating a second posture of the camera based on the second image; calculating a first posture change based on the first posture of the camera and the second posture of the camera; generating a first three-dimensional image of a scene using a scanner; calculating a first posture of the scanner based on the first three-dimensional image of the scene; generating a second three-dimensional image of the scene using the scanner; calculating a second posture of the scanner based on the second three-dimensional image of the scene; calculating a second posture change based on the first posture of the scanner and the second posture of the scanner; using a rotation vector to associate the rotation of the first posture change with the rotation of the second posture change; defining a scaling rotation vector based on an initial value of the scaling rotation vector plus a scaling value; analyzing the relationship between the first posture change, the second posture change, and the scaling rotation vector to solve for the rotation between the first posture of the camera and the first posture of the scanner; and calibrating the camera using the scanner based on the rotation between the first posture of the camera and the first posture of the scanner. In some embodiments, the movement of the camera and the scanner is constrained to a plane so that the rotation around a first axis is much greater than the rotation around a second axis; and the second axis is orthogonal to the first axis; the rotation around the first axis is more than twice the rotation around the second axis; the movement of the camera and the scanner is from a vehicle traveling on flat ground; and also includes calibrating four or more cameras simultaneously with the scanner, solving the scaling value to solve for the rotation between the first posture of the camera and the first posture of the scanner, and / or synchronizing the time of the first posture change with the time of the second posture change by multiplying the scaling value by the sum of the rotation of the first posture change and the rotation of the second posture change before adding the scaling value to the initial value of the scaled rotation vector; the analysis is a least squares analysis; and / or the scanner is part of a lidar module.
[0010] Further areas of applicability of the present description will become apparent from the detailed description provided hereinafter.It should be understood that the detailed description and specific examples, while indicating various embodiments, are intended for purposes of illustration only and are not intended to limit the scope of the present description. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The detailed description is made with reference to the accompanying drawings.
[0012] Figure 1 An autonomous vehicle utilizing certain aspects of some embodiments of the disclosed technology is shown.
[0013] Figure 2 The center of the camera and the center of the lidar are shown.
[0014] Figure 3 The misalignment of the camera and the lidar and the alignment of the camera and the lidar are shown.
[0015] Figure 4 A flow chart of the alignment calculation is shown.
[0016] Figure 5 The corresponding matrix is shown.
[0017] Figure 6 The rotation vectors in different coordinate systems are shown.
[0018] Figure 7 A flow chart of an embodiment of a method for estimating a rotation vector for calibrating a camera system using a lidar system is shown.
[0019] Figure 8 A flow chart of an embodiment of a method for calibrating a camera system with a lidar system is shown.
[0020] Fig. 9 A block diagram of an embodiment of a computer system interfaced with a camera and lidar system is shown.
[0021] In the drawings, similar components and / or features may have the same reference label. In addition, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes the similar components. If only the first reference label is used in the specification, the description applies to any similar components having the same first reference label, regardless of the second reference label. DETAILED DESCRIPTION
[0022] Various aspects of the present specification generally relate to calibrating two optical systems. More specifically and without limitation, the present specification relates to calibrating a camera coordinate system using a lidar coordinate system without using a target, without using human interaction, or without using both a target and human interaction.
[0023] For autonomous and / or semi-autonomous vehicles, there are two ways to calibrate the camera's coordinate system to the lidar's coordinate system: 1. Target-based; 2. Manual. Target-based methods use objects of certain shapes and can usually only be performed at predefined locations. Manual methods usually compare the lidar and camera information in some way. For example, the user uses interaction information or common features to manually align the camera image with the lidar data. Manual methods traditionally do not use the camera's video stream. Calibration of the camera system to the lidar system can be performed at the factory, and then at regular intervals (for example, once a month) to verify the calibration of the sensor. Using targets to calibrate the coordinate system may require certain equipment and calibration knowledge. Using a human in the loop may take time and / or introduce human error. In addition, there are some calibration techniques that calibrate the camera separately. A better process and / or system is needed to calibrate the camera coordinate system with the lidar coordinate system.
[0024] The problem of determining the transformation of the robot and the camera is called the hand-eye calibration problem. The hand-eye calibration problem takes the matrix form AX=XB, where A and B are two systems and X is a transformation (e.g., Roger Y.Ysai and Reimar K.Lenz, "Real Time Versatile Robotics Hand / Eye Calibration using 3D Machine Vision," 1988; and
[0025] https: / / en.wikipedia.org / wiki / Hand_eye_calibration_problem, last accessed on May 25, 2020). In the camera / lidar system, matrix A can be used to represent the change in the camera's attitude, matrix B can be used to represent the change in the attitude of the lidar, and matrix X is the relationship between the attitude of the camera system and the attitude of the lidar system.
[0026] In a vehicle, the camera and lidar are rigidly coupled, so there is no relative motion or spin between the camera and lidar. The problem with using a traditional hand-eye system to calibrate a camera system with a lidar system is that the pose changes in a vehicle may include spins primarily or only about one axis (e.g., spin about the Y axis >> spins about the X and Z axes, where the Y axis is essentially parallel to the direction of gravity). The presence of degeneracy in the two spins makes the hand-eye problem difficult to solve.
[0027] To solve the degradation problem of the hand-eye problem, a direction vector with a scaler is used. The direction vector allows the rotation value to be solved by using the least squares fit. By using an automated calibration process, manpower can be reduced (e.g., no calibration plate; no environmental requirements such as specific targets; ordinary roads can be used for calibration, etc.).
[0028] In some configurations, the target-free approach uses a camera video stream, instead of extracting some explicit geometric information (e.g., plane-like and normal directions); runs a SLAM (simultaneous localization and mapping) algorithm to estimate the camera's pose changes; uses the pose changes for calibration (e.g., instead of using just one image); compares the camera's pose changes with the lidar's pose changes (e.g., via hand-eye calibration) to automatically handle temporal synchronization (e.g., because the pose can be relatively easily interpolated within the pose changes); and / or can be used with non-rotationally rich motions (e.g., classic hand-eye calibration uses rotationally rich motions to solve the hand-eye equation AX=XB).
[0029] In the following description, various examples of calibrating two local coordinate systems are described. For the purpose of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it is apparent to those skilled in the art that certain embodiments may be practiced or implemented without disclosing every detail. In addition, well-known features may be omitted or simplified to prevent any confusion about the novel features described herein.
[0030] Figure 1 An embodiment of a vehicle 100 (e.g., an autonomous vehicle or a vehicle including a driver) in which the disclosed technology may be implemented is described. The vehicle 100 includes a lidar module 102. The lidar module 102 allows the vehicle 100 to perform object detection and ranging in the surrounding environment. Based on the results of the object detection and ranging, the vehicle 100 may brake to avoid a collision with the object. The lidar module 102 may include an emitter 104 (e.g., a laser) and a receiver 106 (e.g., a detector) for light steering. The emitter 104 may project one or more light pulses 108 in a scanning pattern at different times in different directions, and the receiver 106 may monitor light pulses 110 generated by reflections of the light pulses 108 from an object. The lidar module 102 may detect an object based on the received light pulses 110, and may perform distance determination (e.g., the distance of the object) based on a time difference between the light pulses 108 and the light pulses 110 and / or based on a phase difference between the light pulses 108 and the light pulses 110. For example, as Figure 1As shown, the lidar module 102 can transmit a light pulse 108 in a direction directly in front of the vehicle 100 and receive a light pulse 110 reflected by an object 112 (e.g., another vehicle). Based on the received light pulse 110, the lidar module 102 can determine that the object 112 is directly in front of the vehicle 100. By transmitting and detecting many light pulses per second, the lidar module 102 can generate a point cloud, which is a three-dimensional representation of the object 112 in the environment. In addition, based on the time difference between the light pulses 108 and 110, the lidar module 102 can also determine the distance 114 between the vehicle 100 and the object 112. Based on the detection and ranging of the object 112 by the lidar module 102, the vehicle 100 can adjust its speed (e.g., slow down or stop) to avoid colliding with the object 112.
[0031] The vehicle 100 also includes a camera 116. The camera 116 is configured to acquire multiple images of a scene using an image sensor. The image sensor is sensitive to visible light. The camera 116 can be part of the lidar module 102 or separate from the lidar module 102. The camera 116 can be used to acquire images for image analysis and / or display to a vehicle user, a person viewing remotely, or a person viewing after the drive is over. In some embodiments, there are multiple cameras 116 (e.g., 2, 3, 4, 5, 10, or more cameras) integrated with the vehicle 100.
[0032] Figure 2 Embodiments of camera centers and lidar centers are described. Figure 2 Depicted are a camera center 204 and a lidar center 208. The camera center 204 is defined as a first local coordinate system (x, y, z). The lidar center 208 is defined as a second local coordinate system (X, Y, Z).
[0033] The camera center 204 can be defined as the z-axis passing through the center of the image sensor 212 of the camera, where the z-axis is perpendicular to the image sensor 212, the z-axis passes through the center of the image sensor 212, the distance f between the camera center 204 and the image sensor 212 is, and / or the distance f is the focal length of the camera lens. Thus, the image sensor 212 acquires an image in the x / y plane. The camera center 204 is Figure 1 The center of the camera 116.
[0034] The lidar center 208 can also be a local coordinate system of the car to which the lidar system belongs. The Y axis can be considered to be the vertical axis of the car (for example, when the car is on a horizontal surface, the Y axis is parallel to the direction of gravity). When the car turns on a horizontal surface, the lidar rotates around the Y axis. In some embodiments, the Z axis of the lidar center 208 is the direction in which the front of the car is pointing. The lidar center 208 is Figure 1 The center of the laser radar module 102.
[0035] In this specification, the position of the camera center 204 and the position of the lidar center 208 are measured in three translational degrees of freedom. For example, if the car moves forward 5 meters, the lidar center 208 will move 5 meters in the Z direction. The orientation of the camera center 204 and the orientation of the lidar center 208 are measured in three rotational degrees of freedom. For example, the measured directions are roll, pitch, and yaw, where roll is the rotation of the camera center 204 around the z-axis and the lidar center 208 around the z-axis; pitch is the rotation of the camera center 204 around the x-axis and the lidar center 208 around the x-axis; yaw is the rotation of the camera center 204 around the y-axis and the lidar center 208 around the y-axis. The pose of the camera and the pose of the lidar are measured in six degrees of freedom, where the pose is a combination of three positional degrees of freedom and three directional degrees of freedom. Although the examples of position, orientation, and pose are given in Cartesian coordinates, other coordinate systems can also be used (e.g., spherical coordinates or cylindrical coordinates; the camera pose can be defined in Cartesian coordinates, and the pose of the lidar can be defined in cylindrical coordinates).
[0036] Figure 3 Depicted is a diagram superimposed on image 316 (eg, by Figure 1 116) on a laser radar data 302 (e.g., from one of the plurality of images acquired by the camera 116). Figure 1 1 and 2. The lidar data 302 of an embodiment of the present invention is shown in FIG. 1 , and FIG. 2 shows a first frame 300-1 and a second frame 300-2 of an embodiment of a point cloud generated by the lidar module 102 in FIG. 2 . In the first frame 300-1, the lidar data 302 is not aligned with the image 316. For example, the lidar data 302 corresponds to the location of the pole 320 in the image 316, however, the lidar data 302 is not aligned with the pole 320 of the first frame 300-1 in the image 316.
[0037] In the second frame 300-2, the lidar data 302 is aligned with the image 316. For example, the lidar data 302 corresponding to the location of the pole 320 is aligned with the pole 320. Figure 2 The camera center 204 in is calibrated to the lidar center 208 so that the lidar data 302 is aligned with the image.
[0038] Figure 4A flowchart 400 of an embodiment of an alignment calculation for aligning a camera with a lidar is described. In some embodiments, the camera is aligned with the coordinate system of the car (e.g., the lidar system has been calibrated with the coordinate system of the vehicle), and / or the camera is aligned with the lidar by aligning both the camera and the lidar to a common reference system (e.g., the vehicle). Although aligning a camera with a lidar system is described, the description is applicable to alignment of other coordinate systems (e.g., camera with IMU, first camera to second camera, first lidar system to second lidar system, lidar system to IMU, and / or first IMU to second IMU).
[0039] In some embodiments, the calibration technique is accurate, automatic, and / or can be solved by Figure 4 to calibrate multiple cameras simultaneously. Figure 4 , place the camera (for example, Figure 1 The camera 116) in the image sensor is compared with the change in the coordinate system of the scanner (e.g., the lidar module 102). The camera includes an image sensor (e.g., Figure 2 The scanner includes a laser and a detector (e.g., Figure 1 The transmitter 104 and the receiver 106 in FIG. 104 ). The camera and the scanner are connected to the vehicle (e.g., Figure 1 The vehicle 100) is integrated together so that there is no relative motion between the camera and the scanner.
[0040] The camera acquires multiple images using an image sensor, including a first image and a second image. The first image is acquired at a first time t-1, and the second image is acquired at a second time t-2. The scanner generates multiple point clouds, including a first point cloud and a second point cloud. A point cloud is a three-dimensional map of a scene. The first point cloud is generated at a first time t-1, and the second point cloud is generated at a second time t-2. In some embodiments, the second point cloud includes one or more data points calculated during the first time t-1.
[0041] A first pose of camera A-1 is calculated based on the first image. A second pose of camera A-2 is calculated based on the second image. Transformation A, i.e., a change in the pose of the camera (sometimes referred to as the first pose change), is calculated based on the first pose of camera A-1 and the second pose of camera A-2. Transformation A is a transformation from the first pose of camera A-1 to the second pose of camera A-2.
[0042] A first pose of scanner B-1 is calculated based on the first point cloud. A second pose of scanner B-2 is calculated based on the second point cloud. Transformation B, the change in the pose of the scanner (sometimes called the second pose change), is calculated based on the first pose of scanner B-1 and the second pose of scanner B-2. Transformation B is the transformation from the first pose of scanner B-1 to the second pose of scanner B-2. Transformation X is the difference in pose between the camera and the scanner. Transformations A, B, and X are matrices containing information about rotations and translations.
[0043] A first change in posture, transform A, is synchronized in time with a second change in posture, transform B (e.g., using a first time t-1 and a second time t-2). Transform A, transform B, and transform X are related by the following equation: XA=BX. In some embodiments, transform A is calculated, transform B is calculated, and then transform X is solved using the equation XA=BX.
[0044] The transformation A can be calculated using a visual SLAM algorithm. However, monocular visual impact may be difficult to solve. SLAM can be divided into two parts: initialization and tracking. In some embodiments, in order to calculate the transformation A, one or more of the following is used for initialization: extract features with descriptors (e.g., function cv::goodFeaturesToTrack (e.g., see https: / / docs.opencv.org / 2.4 / modules / imgproc / doc / feature_detection.html#goodfeaturestotrack) + BRIEF descriptor); match feature descriptors (low ratio + bidirectional + maximum distance); estimate pose (function cv:findEssentialMat + minimize symmetric pole distance); triangulate (function cv::triangulatePoint + minimum disparity); and / or discard if the quality is low (points with large differences > failure + points with small differences). In some embodiments, to compute the transformation A, tracking is performed using one or more of the following: matching with a reference frame (low ratio + bidirectional + maximum distance); estimating pose (minimizing reprojection loss); skipping if not a keyframe (small movement and similar to the reference frame); matching with prior points (filtering total outliers based on initial values); estimating pose (minimizing projection loss); and / or triangulating more points (e.g., selecting 3 frames with maximum commonality).
[0045] Figure 5 Depicted Figure 4The corresponding matrices of transformation A, transformation B, and transformation X are arranged in the equation XA=BX. One challenge in solving the equation XA=BX comes from degenerate motion. For example, if the car is driving on relatively flat ground, there are rotations around the Y axis, but fewer rotations around the X or Z axis (e.g., degenerate in planar motion). Therefore, the spins around the X and Z axes may be degenerate, which can make solving the equation XA=BX challenging.
[0046] like Figure 5 As shown, the transformation matrices A, B, and X can be divided into components of the spin R (3×3 matrix) and the translation t (3x1 matrix), where R_X is the spin of transformation X and t_X is the translation of transformation X; R_A is the spin of transformation A and t_A is the translation of transformation A; R_B is the spin of transformation B and t_B is the translation of transformation B. It is worth noting that the scale ambiguity v is multiplied by the translation t_A of transformation A. The scale ambiguity v is used because the monocular camera SLAM does not know the actual length between two points; on the contrary, the monocular camera SLAM knows the relative distance between points. The following formulas involve various spins R and translations t:
[0047] Formula 1: R_X*R_A=R_B*R_X
[0048] Formula 2: (I-R_B)*t_x=t_B–R_X*t_A*v
[0049] The camera can be calibrated to the scanner by calculating R_X and t_X, where there is a fixed relative rotation and translation between the camera and the scanner.
[0050] Figure 6 Describes the rotation vector e in different coordinate systems. Figure 6 Camera center 204 and lidar center 208 are shown. In some embodiments, t_X is measured (eg, the offset of camera center 204 to lidar center 208 is measured). Figure 6 Also described is the rotation vector e_A of transformation A relative to camera center 204 and the rotation vector e_B of transformation B relative to lidar center 208. Since the relative motion (rotation and translation) between camera center 204 and lidar center 208 is fixed, both e_A and e_B have the same theta rotation, and equation 1 can be rewritten as:
[0051] Formula 3: e_B = R_X*e_A
[0052] Formula 3 relates the rotation of the first pose change to the rotation of the second pose change using a rotation vector R_X, which is the difference in rotation between the camera coordinate system and the scanner coordinate system. In addition:
[0053] Formula 4: (e_A+e_B)×e_X=e_A-e_B,
[0054] where (e_A+e_B) is level 2; and
[0055] Formula 5: e_X=e_0+u*(e_A+e_B).
[0056] In Formula 5, e_X is a scaled rotation vector based on the initial value of the rotation vector e_0 plus a scaling value u multiplied by the sum of the spin e_A of the first pose and the spin e_B of the second pose. The scaling value u is 1 rotational degree of freedom.
[0057] Combining Equation 5 with Equation 2 yields:
[0058] Formula 6: (I-R_B)*t=t_B–v*R_X(u)*t_A, where R_X is a function of the scaling value u.
[0059] Formula 6 is a nonlinear equation that can be solved for u (scaling value) and R_X (rotation vector between the first pose of the camera and the first pose of the lidar system) by applying a nonlinear least squares fitting analysis. For example, a nonlinear least squares fitting is used to minimize the residual to obtain u and R_X.
[0060] In some embodiments, the scale ambiguity v can be estimated (e.g., for setting an initial condition for the scale ambiguity v). In a first example for estimating the scale ambiguity v, the height of the camera above the ground is known (e.g., 1.5 meters). The point cloud is generated by camera SLAM. A ground detection algorithm is run (e.g., points with small y in the plane fitting coordinate system), which can provide the ground height without scale (e.g., the ground height is normalized to 1.0). Using the known height, the scale of the ground height can be estimated (e.g., 1.0 = 1.5 meters).
[0061] In the second example for estimating the scale ambiguity v, the vehicle moves in a straight line. The lidar module can report the movement distance (e.g., 3 meters). The camera SLAM also reports the movement distance (e.g., normalized to 1.0). The movement distance of the lidar module is used to provide a scale estimate for the scale ambiguity of the camera SLAM (e.g., 1.0 = 3 meters). The applicant found that the second example is more stable in some cases.
[0062] Figure 7 A flow chart of an embodiment of an estimated spin process 700 for calibrating a camera system with a lidar system is described. The process 700 begins with step 704 of acquiring a plurality of images. The plurality of images are obtained by Figure 1 The images are acquired by one or more cameras 116. In step 708, a SLAM algorithm is run on the plurality of images. Figure 4The steps of some example SLAM algorithms are described. In step 712, the camera pose is calculated. For example, based on running the SLAM algorithm on multiple images, the camera pose is calculated. Figure 4 The first camera pose A-1 and the second camera pose A-2 in .
[0063] In step 716, multiple point clouds are obtained. Figure 1 The laser radar module 102 acquires multiple point clouds. Step 720, run the laser radar SLAM algorithm on the multiple point clouds, and calculate the laser radar posture in step 724. For example, the first scanner posture B-1 and the second scanner posture B-2 are calculated based on running the laser radar SLAM algorithm on the multiple point clouds. In some embodiments, steps 716, 720 and 724 can be performed simultaneously with steps 704, 708 and 712. In some embodiments, steps 704, 708 and 712 (together with steps 716, 720, 724) are repeated based on the video image.
[0064] In step 728, pose queue interpolation is performed. A pose can be interpolated from a first pose at a first time and a second pose at a second time, where the interpolation provides a pose between the first time and the second time. (For example, using spline interpolation, https: / / math.stackexchange.com / questions / 2092456 / spline-interpolation-in-se3). The poses (e.g., poses A-1 and B-1; and A-2 and B-2) are time synchronized, step 732. In step 736, the degenerate motion of the equation XA=BX is solved (for example, see Figure 5 and Figure 6 From solving the equation XA=BX, a rotation vector R_X is estimated, step 740, indicating how the camera center and the lidar center are related in three rotational degrees of freedom. The distance between the camera center and the lidar center (e.g., t_X) can be measured in three translational degrees of freedom (e.g., with a ruler or electronic measuring device).
[0065] Figure 8 A flow chart of an embodiment of a process 800 for calibrating a camera with a scanner is described. The process 800 begins at step 804 of acquiring a first image using a camera. For example, the first image is obtained by Figure 1 The camera 116 acquires the first pose of the camera based on the first image (eg, Figure 4 The first posture A-1 in the image is calculated, step 808. For example, the first posture of the camera is calculated using a SLAM algorithm. The second image is acquired using the camera, step 812, and the second posture of the camera is calculated based on the second image (for example, Figure 4Calculate a first pose change (e.g., Figure 4 Transformation A in ), step 820.
[0066] At step 824, a scanner (e.g., Figure 1 The laser radar module 102 in the embodiment is used to generate a first three-dimensional map of the scene. A first pose of the scanner is calculated from the first three-dimensional map (e.g., Figure 4 The first posture B-1 in the image is calculated, step 828. For example, a first posture of the scanner is calculated based on the first three-dimensional image using a SLAM algorithm. A second three-dimensional image is generated by the scanner, step 832, and a second posture of the scanner is calculated using the second three-dimensional image (e.g., Figure 4 B-2 in step 836. Calculate a second posture change based on the first posture of the scanner and the second posture of the scanner (e.g., Figure 4 Transformation B), step 840. Steps 824-840 may be performed simultaneously with steps 804-820.
[0067] The first posture change is synchronized with the second posture change, step 844. For example, Figure 4 In step 848, the first posture change of the camera and the second posture change of the scanner are calculated from time t-1 to time t-2. In step 848, the rotation of the first posture change is associated with the rotation of the second posture change. For example, e_B=R_X*e_A (Formula 3). The scaling rotation vector is defined in step 852. For example, e_X (scaling rotation vector) is a function of the initial value e_0, the scaling value u, e_A and e_B (Formula 5). In step 856, the rotation (e.g., R_X) and the scaling value (e.g., scaling value u) between the camera and the scanner are solved. For example, Formula 6 solves the scaling value u and R_X(u) by applying a nonlinear least squares fit analysis to the relationship between the first posture change, the second posture change and the scaled rotation vector. By solving the rotation R_X (because t_X is previously known and / or can be measured), the camera can be calibrated to the scanner.
[0068] In some embodiments, the motion of the camera and scanner is constrained to a plane so that the rotation about the first axis is much greater than the rotation about the second axis. For example, the motion is that the vehicle is traveling in the X / Z plane so that the rotation about the Y axis is greater (e.g., equal to or greater than 2, 5, 10, 15, 20, or 24 times) than the rotation about the X and / or Z axes. For example, a car can rotate 360° about the Y axis and no more than 5°, 10°, or 15° about the X axis. In some embodiments, the flat ground has a slope equal to or less than 1°, 2°, or 5°. The Y axis is orthogonal to the X and Z axes. In some embodiments, multiple cameras can be calibrated to the scanner at once. For example, two, three, four, or more cameras integrated with a vehicle can be calibrated to the scanner (calibrated to the vehicle) at the same time.
[0069] Fig. 9 900 is a simplified block diagram of a computing device 900. The computing device 900 may implement some or all of the functions, behaviors, and / or capabilities described above using electronic storage or processing, as well as other functions, behaviors, or capabilities not explicitly described. The computing device 900 includes a processing subsystem 902, a storage subsystem 904, a user interface 906, and / or a communication interface 908. The computing device 900 may also include other components (not explicitly shown), such as batteries, power controllers, and other components operable to provide various enhanced functions. In various embodiments, the computing device 900 may be in a vehicle, a laptop computer, a mobile device (e.g., a tablet, a smartphone, a mobile phone), a wearable device, a media device, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the above functions or a combination of functions.
[0070] The storage subsystem 904 can be implemented using local storage and / or removable storage media, for example, using a disk, flash memory (e.g., a secure digital card, a universal serial bus flash drive), or any other non-temporary storage media, or a combination thereof, and can include volatile and / or non-volatile storage media. Local storage can include random access memory (RAM), including dynamic RAM (DRAM), static RAM (SRAM), or battery-backed RAM. In some embodiments, the storage subsystem 904 can store one or more application programs and / or operating system programs to be executed by the processing subsystem 902, including programs for implementing some or all of the operations that will be performed using a computer as described above. For example, the storage subsystem 904 can store one or more code modules 910 for implementing one or more of the method steps described above.
[0071] Firmware and / or software implementations may be implemented with modules (e.g., procedures, functions, etc.). Machine-readable media tangibly embodying instructions may be used to implement the methods described herein. Code modules 910 (e.g., instructions stored in memory) may be implemented within a processor or external to a processor. As used herein, the term "memory" refers to a type of long-term, short-term, volatile, non-volatile, or other storage medium, and is not limited to any particular type of memory or amount of memory or type of media on which memory is stored.
[0072] In addition, the term "storage medium" or "storage device" may refer to one or more memories for storing data, including read-only memory (ROM), RAM, magnetic RAM, core memory, magnetic disk storage media, optical storage media, flash memory devices for storing information, and / or other machine-readable media. The term "machine-readable medium" includes, but is not limited to, portable or fixed storage devices, optical storage devices, wireless channels, and / or various other storage media capable of storing instructions and / or data.
[0073] In addition, embodiments can be implemented by hardware, software, scripting language, firmware, middleware, microcode, hardware description language and / or any combination thereof. When implemented with software, firmware, middleware, scripting language and / or microcode, the program code or code segment for performing tasks can be stored in a machine-readable medium such as a storage medium. Code segment (e.g., code module 910) or machine executable instructions can represent a combination of process, function, subroutine, program, routine, subroutine, module, software package, script, class or instruction, data structure and / or program statement. By transmitting and / or receiving information, data, independent variable, parameter and / or memory content, code segment can be coupled to another code segment or hardware circuit. Information, command line parameters, parameters, data, etc. can be transmitted, forwarded or transmitted in a suitable manner, including memory sharing, message passing, token passing, network transmission, etc.
[0074] The implementation of the above techniques, blocks, steps and devices can be accomplished in various ways. For example, these techniques, blocks, steps and devices can be implemented in hardware, software or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more ASICs, DSPs, DSPDs, PLDs, FPGAs, processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the above functions and / or combinations thereof.
[0075] Each code module 910 may include a set of instructions (code) embodied on a computer-readable medium that directs the processor of the computing device 900 to perform a corresponding action. The instructions may be configured to run sequentially, in parallel (e.g., under different processing threads), or in a combination thereof. After loading the code module 910 on a general-purpose computer system, the general-purpose computer is converted into a special-purpose computer system.
[0076] Computer programs incorporating various features described herein (e.g., in one or more code modules 910) may be encoded and stored on various computer-readable storage media. Computer-readable media encoded with program code may be packaged with compatible electronic devices, or the program code may be provided separately from the electronic device (e.g., downloaded via the Internet or as a separately packaged computer-readable storage medium). The storage subsystem 904 may also store information useful for establishing a network connection using the communication interface 908.
[0077] The user interface 906 may include input devices (e.g., touch pads, touch screens, scroll wheels, click wheels, dials, buttons, switches, keypads, microphones, etc.), and output devices (e.g., video screens, indicator lights, speakers, headphone jacks, virtual or augmented reality displays, etc.), and supporting electronic devices (e.g., digital-to-analog or analog-to-digital converters, signal processors, etc.). A user may operate the input devices of the user interface 906 to invoke functions of the computing device 900 and may view and / or hear output from the computing device 900 through the output devices of the user interface 906. For some embodiments, the user interface 906 may not be present (e.g., for processes using an ASIC).
[0078] The processing subsystem 902 may be implemented as one or more processors (e.g., integrated circuits, one or more single-core or multi-core microprocessors, microcontrollers, central processing units, graphics processing units, etc.). In operation, the processing subsystem 902 may control the operation of the computing device 900. In some embodiments, the processing subsystem 902 may execute various programs in response to program code and may maintain multiple programs or processes that are executed simultaneously. At a given time, some or all of the program code to be executed may reside in the processing subsystem 902 and / or a storage medium, such as the storage subsystem 904. Through programming, the processing subsystem 902 may provide various functions for the computing device 900. The processing subsystem 902 may also execute other programs to control other functions of the computing device 900, including programs that may be stored in the storage subsystem 904.
[0079] The processing subsystem 902 is in electronic communication (wired or wireless) with the camera system 912 and the scanner 916. For example, the camera system 912 is Figure 1 The camera 116 in the scanner 916 is Figure 1The laser radar module 102 in.
[0080] The communication interface 908 can provide voice and / or data communication capabilities for the computing device 900. In some embodiments, the communication interface 908 may include components for accessing wireless data networks (e.g., Wi-Fi networks; 3G, 4G / LTE; etc.), mobile communication technologies, components for short-range wireless communications (e.g., using Bluetooth communication standards, NFC, etc.), other components or technology combinations. In some embodiments, in addition to or instead of a wireless interface, the communication interface 908 can provide a wired connection (e.g., a universal serial bus, Ethernet, a universal asynchronous receiver / transmitter, etc.). The communication interface 908 can be implemented using a combination of hardware (e.g., driver circuits, antennas, modulators / demodulators, encoders / decoders, and other analog and / or digital signal processing circuits) and software components. In some embodiments, the communication interface 908 can support multiple communication channels simultaneously. In some embodiments, the communication interface 908 is not used.
[0081] It should be understood that computing device 900 is illustrative and that variations and modifications are possible. The computing device may have various functions not specifically described (eg, voice communications via a cellular telephone network) and may include components suitable for such functions.
[0082] In addition, although the computing device 900 is described with reference to specific blocks, it should be understood that these blocks are defined for ease of description and do not imply a specific physical arrangement of component parts. For example, the processing subsystem 902, storage subsystem 904, user interface 906, and / or communication interface 908 can be in one device or distributed in multiple devices.
[0083] Furthermore, blocks need not correspond to physically distinct components. Blocks may be configured to perform various operations, for example, by programming a processor or providing appropriate control circuitry, and various blocks may or may not be reconfigurable depending on how the initial configuration is obtained. Embodiments of the present invention may be implemented in various devices including electronic devices implemented using a combination of circuits and software. The electronic devices described herein may be implemented using a computing device 900.
[0084] The various features described herein, such as methods, apparatus, computer-readable media, etc., can be implemented using a combination of dedicated components, programmable processors, and / or other programmable devices. The processes described herein can be implemented on the same processor or on different processors. Where components are described as being configured to perform certain operations, such configurations can be implemented, for example, by designing electronic circuits to perform the operations, by programming programmable electronic circuits (e.g., microprocessors) to perform the operations, or a combination thereof. In addition, although the above-described embodiments may refer to specific hardware and software components, it will be appreciated by those skilled in the art that different combinations of hardware and / or software components may also be used, and that specific operations described as being implemented in hardware may be implemented in software, and vice versa.
[0085] Specific details are given in the above description to provide an understanding of the embodiments. However, it should be understood that the embodiments can be implemented without these specific details. In some cases, well-known circuits, processes, algorithms, structures and techniques can be described without unnecessary details to avoid obscuring the embodiments.
[0086] The following description only provides preferred exemplary embodiments and is not intended to limit the scope, applicability or configuration of this specification. On the contrary, the subsequent description of the preferred exemplary embodiments will provide those skilled in the art with an effective description for implementing the preferred exemplary embodiments. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope set forth in the appended claims.
[0087] The specific details of a particular embodiment may be combined in any suitable manner without departing from the spirit and scope of the embodiments of the invention. However, other embodiments of the invention may be directed to specific embodiments relating to each individual aspect or specific combinations of these individual aspects.
[0088] In addition, it should be noted that the embodiments may be described as processes of a job diagram, a flow chart, a data flow diagram, a structure diagram, or a block diagram. Although a flow chart may describe operations as a sequential process, many operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but there may be other steps not included in the figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.
[0089] References to "a," "an," or "the" are intended to mean "one or more" unless expressly stated to the contrary.
[0090] All patents, patent applications, publications, and descriptions mentioned herein are incorporated by reference in their entirety for all purposes. None are prior art.
Claims
1. A system for calibrating a visual device with a multi-dimensional distance measuring device, characterized in that: The system comprises: LiDAR module; a camera including one or more image sensors; One or more processors configured to: Acquire a first image using the camera; Calculating a first pose of the camera based on the first image; acquiring a second image using the camera; Calculating a second pose of the camera based on the second image; calculating a first posture change based on the first posture of the camera and the second posture of the camera; generating a first three-dimensional map of a scene using the lidar module; Calculating a first posture of the laser radar module based on the first three-dimensional image of the scene; generating a second three-dimensional map of the scene using the lidar module; Calculating a second posture of the laser radar module based on the second three-dimensional map of the scene; Calculating a second posture change based on the first posture of the laser radar module and the second posture of the laser radar module; Using a rotation vector, relating the spin of the first posture change to the spin of the second posture change; Define the scaled rotation vector based on the initial value of the scaled rotation vector plus the scale value; Analyzing the relationship between the first posture change, the second posture change, and the scaling rotation vector to solve the rotation between the first posture of the camera and the first posture of the lidar module; and Based on a rotation between the first pose of the camera and the first pose of the lidar module, the camera is calibrated using the lidar module.
2. The system according to claim 1, characterized in that The laser radar module is installed on the vehicle.
3. The system according to claim 1, characterized in that Relative motion and rotation between the camera and the lidar module are prevented.
4. The system according to claim 1, characterized in that The scaling value is multiplied by a sum of the rotation of the first change in pose and the rotation of the second change in pose before the scaling value is added to the initial value of the scaled rotation vector.
5. The system according to claim 1, characterized in that The analysis was a least squares analysis.
6. A method for calibrating a visual device with a multi-dimensional distance measuring device, characterized in that: The method comprises: Acquire a first image using a camera; Calculating a first pose of the camera based on the first image; acquiring a second image using the camera; Calculating a second pose of the camera based on the second image; calculating a first posture change based on the first posture of the camera and the second posture of the camera; generating a first three-dimensional image of the scene using a scanner; calculating a first pose of the scanner based on the first three-dimensional map of the scene; generating a second three-dimensional map of the scene using the scanner; calculating a second pose of the scanner based on the second three-dimensional map of the scene; calculating a second posture change based on the first posture of the scanner and the second posture of the scanner; Using a rotation vector, relating the spin of the first posture change to the spin of the second posture change; Define the scaled rotation vector based on the initial value of the scaled rotation vector plus the scale value; analyzing a relationship between the first pose change, the second pose change, and the scaling rotation vector to solve for a rotation between the first pose of the camera and the first pose of the scanner; and The camera is calibrated using the scanner based on a rotation between the first pose of the camera and the first pose of the scanner.
7. The method according to claim 6, wherein: The motion of the camera and the scanner is constrained to a plane such that the rotation about the first axis is much greater than the rotation about the second axis; as well as The second axis is orthogonal to the first axis.
8. The method according to claim 7, characterized in that The rotation about the first axis is more than twice the rotation about the second axis.
9. The method according to claim 7, characterized in that: The motion of the camera and the scanner is from a vehicle driving on flat ground.
10. The method according to claim 6, characterized in that Also included is calibrating four or more cameras simultaneously using the scanner.
11. The method according to claim 6, characterized in that Also included is solving the scaling value to solve for rotation between the first pose of the camera and the first pose of the scanner.
12. The method according to claim 6, characterized in that The method also includes synchronizing a time of the first posture change with a time of the second posture change.
13. The method according to claim 6, characterized in that The scaling value is multiplied by a sum of the rotation of the first change in posture and the rotation of the second change in posture before the scaling value is added to the initial value of the scaled rotation vector.
14. The method according to claim 6, characterized in that The analysis was a least squares analysis.
15. The method according to claim 6, characterized in that The scanner is part of the lidar module.
16. A storage device having instructions, characterized in that: When executing the instructions, one or more processors are caused to perform the following steps: Acquire a first image using a camera; Calculating a first pose of the camera based on the first image; acquiring a second image using the camera; Calculating a second pose of the camera based on the second image; calculating a first posture change based on the first posture of the camera and the second posture of the camera; generating a first three-dimensional image of the scene using a scanner; calculating a first pose of the scanner based on the first three-dimensional map of the scene; generating a second three-dimensional map of the scene using the scanner; calculating a second pose of the scanner based on the second three-dimensional map of the scene; calculating a second posture change based on the first posture of the scanner and the second posture of the scanner; Using a rotation vector, relating the spin of the first posture change to the spin of the second posture change; Define the scaled rotation vector based on the initial value of the scaled rotation vector plus the scale value; analyzing a relationship between the first pose change, the second pose change, and the scaling rotation vector to solve for a rotation between the first pose of the camera and the first pose of the scanner; and The camera is calibrated using the scanner based on a rotation between the first pose of the camera and the first pose of the scanner.
17. The storage device according to claim 16, characterized in that: When the instructions are executed, the one or more processors are caused to synchronize the timing of the first posture change with the timing of the second posture change.
18. The storage device according to claim 16, characterized in that: The scaling value is multiplied by a sum of the rotation of the first change in posture and the rotation of the second change in posture before the scaling value is added to the initial value of the scaled rotation vector.
19. The storage device according to claim 16, characterized in that: The analysis was a least squares analysis.
20. The storage device according to claim 16, characterized in that The scanner is part of the lidar module.
