Sensor calibration method, device, vehicle and storage medium

Abstract

The present application relates to a sensor calibration method, device, vehicle and storage medium. The sensor calibration method comprises: determining a first external parameter between a first coordinate system and a second coordinate system based on the difference between a first pose and a second pose, wherein the first pose is the pose of a calibration sensor disposed on a mobile platform in the first coordinate system, and the second pose is the pose of a sensor to be calibrated disposed on the mobile platform in the second coordinate system; determining a second external parameter between the first coordinate system and the second coordinate system based on ground detection; if the first external parameter and the second external parameter match, calibrating the sensor to be calibrated based on the first external parameter and / or the second external parameter. The solution provided by the present application can effectively improve the accuracy of the calibration results of the online calibration sensor.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a sensor calibration method, device, vehicle and storage medium. Background Art

[0002] Radar can detect targets and determine their spatial positions. LiDAR has the advantages of reasonable cost, high resolution, strong anti-interference ability, good low-altitude detection performance, light weight and flexibility, and is widely used in technical fields such as assisted driving and autonomous driving.

[0003] After the laser radar collects the position information of the obstacle, the relative position relationship between the obstacle and the vehicle can be obtained based on the position of the laser radar in the vehicle coordinate system. The accuracy of this relative position relationship depends in part on the accuracy of the laser radar position in the vehicle coordinate system. Therefore, after installing the laser radar, the external parameters of the laser radar need to be calibrated.

[0004] In the related art, the radar can be calibrated by an offline calibration method, which is not suitable for an online calibration process. For example, the accuracy of the online calibration result cannot be guaranteed. Summary of the invention

[0005] In order to solve or partially solve the problems existing in the related art, the present application provides a sensor calibration method, device, vehicle and storage medium, which can effectively improve the accuracy of online calibration results.

[0006] On the one hand, the present application provides a sensor calibration method, including: determining a first external parameter between a first coordinate system and a second coordinate system based on a difference between a first pose and a second pose, wherein the first pose is a pose of a calibration sensor disposed on a mobile platform in the first coordinate system, and the second pose is a pose of a sensor to be calibrated disposed on the mobile platform in the second coordinate system; determining a second external parameter between the first coordinate system and the second coordinate system based on ground detection; and calibrating the sensor to be calibrated based on the first external parameter and / or the second external parameter if the first external parameter matches the second external parameter.

[0007] In one embodiment, the sensor to be calibrated includes a laser radar, and the laser radar collects multiple frames of point clouds. Accordingly, determining the second external parameter between the first coordinate system and the second coordinate system based on ground detection includes: fitting a ground plane based on the multiple frames of point clouds; determining a normal vector of the fitted ground plane; and calculating the second external parameter between the first coordinate system and the second coordinate system based on the normal vector, wherein the second external parameter includes at least one of a roll angle, a pitch angle, and a height.

[0008] In one embodiment, fitting a ground plane based on a multi-frame point cloud includes: fitting a ground plane in a full point cloud using a first method to obtain a ground plane point cloud with a first roughness; based on the ground plane point cloud with the first roughness, fitting the ground plane using a second method to obtain a ground plane point cloud with a second roughness, wherein the second roughness is finer than the first roughness.

[0009] In one embodiment, the sensor to be calibrated includes a laser radar, which collects multiple frames of point clouds; the second posture is determined in the following manner: for at least part of the frames in the multiple frames of point clouds collected by the laser radar, the posture differences of adjacent frames located between the start frame and the end frame are accumulated to obtain the second posture, wherein the posture differences of adjacent frames are determined in the following manner: motion compensation is performed on a point cloud of a frame using the relative posture of a frame to obtain a point cloud of a frame after distortion is eliminated; the line and surface features of the point cloud of the previous frame and the point cloud of the current frame after distortion is eliminated are respectively extracted; and the posture difference of the sensor to be calibrated between the previous frame and the current frame is obtained by line and surface feature alignment.

[0010] In one embodiment, the above method also includes: for a frame of point cloud, determining the point cloud in the frame of point cloud corresponding to the moving object; removing the point cloud in the frame of point cloud corresponding to the moving object; respectively extracting the line and surface features of the point cloud of the previous frame after eliminating the distortion and the point cloud of the current frame, including: respectively extracting the line and surface features from the point cloud of the previous frame after eliminating the distortion and removing the moving object and the point cloud of the current frame.

[0011] In one embodiment, removing the point cloud corresponding to the moving object in a frame of point cloud includes: deleting the point cloud of a first specific object from a frame of point cloud, the first specific object includes a movable object; or obtaining the point cloud of a second specific object from a frame of point cloud, the second specific object includes a fixed object.

[0012] In one embodiment, a first external parameter between a first coordinate system and a second coordinate system is determined based on the difference between a first pose and a second pose, including: aligning a first pose queue including a first pose and a second pose queue including a second pose based on a timestamp to obtain an aligned first pose and a second pose; adjusting the aligned first pose and / or the aligned second pose in a coarse-to-fine manner to reduce the difference between the aligned first pose and the second pose to obtain a first external parameter between the first coordinate system and the second coordinate system, the first external parameter including at least one of a pitch angle or a yaw angle.

[0013] In one embodiment, a first pose queue including a first pose and a second pose queue including a second pose are aligned based on timestamps to obtain the aligned first pose and second pose, including: using the timestamp of the second pose queue to interpolate the corresponding second pose in the first pose queue.

[0014] In one embodiment, the calibration sensor includes an inertial measurement unit (IMU) or an inertial measurement unit and at least one of the following: a speedometer, a steering wheel sensor or a positioning system; when the calibration sensor includes an inertial measurement unit and at least one of the following: a speedometer, a steering wheel sensor or a positioning system, the first pose is determined by performing Kalman filtering on data collected by the inertial measurement unit and data collected by at least one of the speedometer, the steering wheel sensor or the positioning system to obtain the first pose.

[0015] On the other hand, the present application provides a sensor calibration device, including: an external parameter determination module, a ground detection module and a cross calibration module. The external parameter determination module is used to determine the first external parameter between the first coordinate system and the second coordinate system based on the difference between the first pose and the second pose, the first pose is the pose of the calibration sensor set on the mobile platform in the first coordinate system, and the second pose is the pose of the sensor to be calibrated set on the mobile platform in the second coordinate system; the ground detection module is used to determine the second external parameter between the first coordinate system and the second coordinate system based on the ground detection; the cross calibration module is used to calibrate the sensor to be calibrated based on the first external parameter and / or the second external parameter if the first external parameter and the second external parameter match.

[0016] On the other hand, the present application provides a vehicle, including: a body; a laser radar, arranged on the body, for collecting point clouds; a processor, coupled to the laser radar, for receiving point clouds; a memory, on which executable code is stored, and when the executable code is executed by the processor, the processor executes the method as described above.

[0017] On the other hand, the present application provides a computer-readable storage medium having executable codes stored thereon. When the executable codes are executed by a processor of an electronic device, the processor executes the above method.

[0018] On the other hand, the present application provides a computer program product, including executable codes, which implement the above method when the executable codes are executed.

[0019] The technical solution provided by this application may have the following beneficial effects:

[0020] The technical solution of the embodiment of the present application determines the first external parameter between the first coordinate system and the second coordinate system based on the difference between the first pose of the same object in the first coordinate system and the second pose of the same object in the second coordinate system. In addition, the second external parameter between the first coordinate system and the second coordinate system is determined based on a ground detection algorithm. If the first external parameter and the second external parameter calculated according to different methods match, it indicates that the reliability of the first external parameter is high, and the sensor to be calibrated can be calibrated based on the first external parameter and / or the second external parameter, which effectively improves the reliability of the calibration result of the sensor to be calibrated.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above and other objects, features and advantages of the present application will become more apparent by describing in more detail exemplary embodiments of the present application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of the present application.

[0023] Figure 1 It is an exemplary system architecture to which a sensor calibration method, device, vehicle, and storage medium can be applied as shown in an embodiment of the present application;

[0024] Figure 2 is a schematic diagram of the structure of a vehicle, a laser radar and an IMU shown in an embodiment of the present application;

[0025] Figure 3 is a flow chart of a sensor calibration method shown in an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of aligning postures determined based on different sensors according to an embodiment of the present application;

[0027] Figure 5 is a data flow diagram of a sensor calibration process according to an embodiment of the present application;

[0028] Figure 6 is a data flow diagram of a sensor calibration process according to another embodiment of the present application;

[0029] Figure 7 is a schematic diagram of a cross-validation process according to an embodiment of the present application;

[0030] Figure 8 is a flow chart of a sensor calibration method shown in another embodiment of the present application;

[0031] Fig. 9 is a block diagram of a sensor calibration device according to an embodiment of the present application;

[0032] Fig.10 It is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to make the present application more thorough and complete, and to fully convey the scope of the present application to those skilled in the art.

[0034] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "an" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0035] It should be understood that although the terms "first", "second", "third", etc. may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0036] With the rapid development of Advanced Driving Assistance System (ADAS) and unmanned driving technology, a variety of sensors are loaded on cars to perceive the surrounding environment of the car, such as millimeter wave radar, lidar, monocular / binocular cameras and satellite navigation.

[0037] Laser Detecting and Ranging (Lidar) determines the distance by emitting and receiving laser beams, measuring the time difference and phase difference of laser signals, measuring the angle by horizontal rotation scanning, and establishing a two-dimensional polar coordinate system based on these two parameters, and then obtaining three-dimensional height information by obtaining signals at different pitch angles. High-frequency lasers can obtain a large amount of (such as 1.5 million) position point information (called point clouds) within one second, and perform three-dimensional modeling based on this information. Lidar has the advantages of high resolution, strong anti-active interference ability, good low-altitude detection performance, and light weight. As a large-scale environmental perception module, Lidar has been widely used in map creation and lane perception for autonomous driving.

[0038] The laser radar in the autonomous driving vehicle has fixed its position in the vehicle coordinate system when it is installed on the vehicle, that is, the coordinate data of the laser radar in the vehicle coordinate system has been fixed. Therefore, the point cloud generated by the laser radar on the vehicle is relative to its own local coordinate system. It is necessary to obtain the expression of the laser radar's position in the global coordinate system (i.e., external parameters) to improve the accuracy of the point cloud. Specifically, the laser radar obtains point cloud information of the environment and can obtain the coordinate data of the surrounding objects in the laser radar coordinate system. The vehicle processor can convert the coordinate data of the surrounding objects in the vehicle coordinate system based on the coordinate data obtained by the laser radar and the coordinate data of the laser radar itself in the vehicle coordinate system, and then realize the vehicle's obstacle avoidance and other functions based on the coordinate data. Therefore, in order for the vehicle to obtain the correct coordinate data of the surrounding objects, it is necessary to calibrate the laser radar so that the coordinate data of the surrounding objects in the laser radar coordinate system obtained by it is correct.

[0039] Related technologies can calibrate sensors through offline calibration. For example, lidar can be calibrated offline based on a calibration plate. This calibration method requires professional technicians and professional equipment such as calibration plates to complete the calibration process. However, the calibrated sensor may be deformed or displaced due to interference, age, etc., resulting in inaccurate original calibration results.

[0040] Therefore, online calibration of LiDAR is a key issue that needs to be solved in autonomous driving. The online calibration method means that the calibration is completed automatically when the autonomous driving system is running, which does not require professional equipment such as calibration boards and can also ensure sufficient accuracy.

[0041] In summary, there are many problems with the method of calibrating LiDAR in the related art as shown below.

[0042] For example, traditional calibration methods are mostly offline and require a series of specific calibration operations using professional equipment such as calibration boards, which is time-consuming and labor-intensive.

[0043] For example, the online calibration method based on specific scenarios depends on specific scenarios, such as lane lines, walls, and poles. It cannot be calibrated when the conditions are not met.

[0044] For example, an online calibration method based on joint optimization of Lidar-Inertial Odometry (LIO) is computationally complex and has high requirements on the computing unit.

[0045] In the embodiment of the present application, a first posture of the same target object in a first coordinate system (the coordinate system where the calibration sensor is located) and a second posture in a second coordinate system (the coordinate system where the sensor to be calibrated is located) are calibrated by a calibration algorithm (such as hand-eye calibration) to obtain a first external parameter of the sensor to be calibrated. In addition, a second external parameter between the first coordinate system and the second coordinate system is determined based on ground detection, so that the credibility of the first external parameter and / or the second external parameter can be determined by cross-validation. If the cross-validation shows that the credibility is high, the sensor to be calibrated can be calibrated based on the first external parameter and / or the second external parameter.

[0046] In addition, in some embodiments of the present application, the motion of the vehicle in the first coordinate system (IMU coordinate system, or vehicle coordinate system) is calculated based on at least the inertial data collected by the IMU, and the motion of the vehicle in the second coordinate system (laser radar coordinate system) is calculated using the lidar odometer (Lidar-Odometry), and the first external parameter between the first coordinate system and the second coordinate system is obtained by calibration using algorithms such as hand-eye calibration. This method has a moderate amount of computation, is less dependent on the scene, and can be used as an online calibration for other sensors, such as cameras, RGBD cameras, etc.

[0047] The following will be passed Figures 1 to 10 A method, device and electronic device for storing video in an embodiment of the present application are described in detail.

[0048] Figure 1 This is an exemplary system architecture that can be applied to the sensor calibration method, device, vehicle and storage medium shown in an embodiment of the present application. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present application can be applied, in order to help those skilled in the art understand the technical content of the present application, but it does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0049] See also Figure 1According to the embodiment, the system architecture 100 may include mobile platforms 101, 102, 103, a network 104 and a cloud 105. The network 104 is used to provide a medium for communication links between the mobile platforms 101, 102, 103 and the cloud 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The mobile platforms 101 and 102 may be equipped with sensors, such as laser radars, millimeter wave radars, cameras, etc., to sense the surrounding environment information of the mobile platforms 101 and 102.

[0050] Users can use mobile platforms 101, 102, and 103 to interact with other mobile platforms and cloud 105 through network 104 to receive or send information, such as sending model training requests, model parameter download requests, and receiving trained model parameters. The trained model can classify objects in image data or point cloud data, such as fixed objects or moving objects, such as obstacles or non-obstacles. Mobile platforms 101, 102, and 103 can be installed with various communication client applications, such as assisted driving applications, autonomous driving applications, car applications, web browser applications, database applications, search applications, instant messaging tools, email clients, social platform software, and other applications.

[0051] Mobile platforms 101 , 102 , 103 include, but are not limited to, electronic devices such as cars, robots, tablet computers, and laptop computers.

[0052] The cloud 105 can receive model training requests, model parameter download requests, etc., adjust model parameters for model training, send model topology structures, send trained model parameters, etc., and can also send weather information, real-time traffic information, etc. to the mobile platforms 101, 102, 103. For example, the cloud 105 can be a background management server, a server cluster, a car network, etc.

[0053] It should be noted that the number of mobile platforms, networks and cloud servers is only illustrative, and any number of mobile platforms, networks and cloud servers may be provided as required.

[0054] Figure 2 It is a schematic diagram of the structure of a vehicle, a laser radar and an IMU shown in one embodiment of the present application.

[0055] See also Figure 2 The vehicle 100 may include one or more of the following components: a vehicle body 110 , a laser radar 120 , and an IMU 130 . The laser radar 120 and the IMU 130 are disposed in the vehicle body 110 .

[0056] For example, the laser radar 120 can be installed on both sides of the front bumper. In addition, the laser radar 120 can also be installed on the roof or above the front window of the vehicle, etc., which is not limited here. The laser radar 120 can include a pulse laser radar and a continuous wave laser radar, and can use ultraviolet light, visible light or near infrared light as a light source.

[0057] For example, the detection distance of the laser radar 120 set up in the front and side dual stations can reach 150 meters (such as for a target with a reflectivity of 10%), with a horizontal 120° field of view (FOV), and the field of view is increased to 150°. If calculated according to the point cloud density of 144 lines, the angular resolution can reach 0.16°, and the central area refresh rate is 20Hz, which can realize the detection of distant obstacles in scenes such as highways and urban roads, and improve the driving safety of assisted driving.

[0058] In addition, one or more ultrasonic sensors, millimeter wave radars, etc. may be provided on the vehicle 100 to improve the comprehensiveness and accuracy of the perceived vehicle environment information. For example, ultrasonic sensors may be provided inside the front and rear bumpers. For example, millimeter wave radars may be provided inside the front and rear bumpers.

[0059] Figure 3 It is a flow chart of a sensor calibration method shown in one embodiment of the present application.

[0060] See also Figure 3 The sensor calibration method may include operations S310 to S330.

[0061] In operation S310, a first external parameter between the first coordinate system and the second coordinate system is determined based on a difference between a first pose and a second pose, wherein the first pose is a pose of a calibration sensor disposed on the mobile platform in the first coordinate system, and the second pose is a pose of a sensor to be calibrated disposed on the mobile platform in the second coordinate system.

[0062] In this embodiment, the mobile platform may be a car, a robot, an aircraft, or a ship. The calibration sensor may be an IMU, a camera, a millimeter-wave radar, an ultrasonic radar, a laser radar, etc. The sensor to be calibrated may be a camera, a millimeter-wave radar, an ultrasonic radar, a laser radar, etc. Taking a car as an example, the coordinates of the calibration sensor may be set to the car coordinates. The calibration process of the sensor to be calibrated is to reduce the deviation or error of the collected data caused by the difference in spatial position between the sensor to be calibrated and the calibration sensor.

[0063] For ease of understanding, the following is an illustrative explanation using an example in which the calibrated sensor includes an IMU and the sensor to be calibrated includes a lidar.

[0064] The first posture, the second posture, and the first external parameter may respectively include: at least one of the roll angle, the pitch angle, the yaw angle, the X-axis displacement, the Y-axis displacement, or the Z-axis displacement. For example, the yaw angle of the laser radar may refer to the angle between the vehicle's driving direction in the horizontal plane and the straight line where the preset direction (X-axis) in its laser radar coordinate system is located, that is, the angle between the X-axis in the laser radar coordinate system and the XOZ plane in the vehicle coordinate system (the Z-axis is the vertical upward direction, and O is the origin of the coordinate system). The pitch angle may be the angle between the straight line where the X-axis in the laser radar coordinate system is located and the XOY plane in the vehicle coordinate system (the X-axis, Y-axis, and Z-axis are orthogonal). The roll angle may be the angle between the Y-axis in the laser radar coordinate system and the XOY plane in the vehicle coordinate system. If there is no angle deviation in the laser radar, that is, the laser radar coordinate system OX ′ Y ′ Z ′ If the laser radar is aligned with the standard coordinate system OXYZ, the yaw angle, pitch angle and roll angle corresponding to the laser radar should all be zero. It should be noted that the above description of the X-axis, Y-axis, Z-axis, yaw angle, pitch angle and roll angle is only for illustrative purposes and cannot be construed as a limitation on the present application.

[0065] In operation S320 , a second extrinsic parameter between the first coordinate system and the second coordinate system is determined based on the ground detection.

[0066] In this embodiment, after the laser radar to be calibrated collects the point cloud, the ground can be fitted by ground detection technology to obtain the normal vector of the fitted ground. In this way, the second external parameter between the first coordinate system and the second coordinate system can be calculated based on the difference between the normal vector of the fitted ground and the normal vector of the XOY plane of the first coordinate system. The parameters included in the second external parameter can refer to the parameters of the first external parameter.

[0067] In operation S330 , if the first extrinsic parameter and the second extrinsic parameter match, the sensor to be calibrated is calibrated based on the first extrinsic parameter and / or the second extrinsic parameter.

[0068] In this embodiment, if the first external parameter and the second external parameter match, such as the similarity between the first external parameter and the second external parameter exceeds a preset threshold, it can be determined that the first external parameter and the second external parameter are accurate, and the sensor to be calibrated can be calibrated based on the first external parameter and / or the second external parameter.

[0069] It should be noted that different sensors have different sensitivities (data accuracy) for different parameters. For example, the accuracy of attitude angle detected by an automotive-grade IMU is better than the accuracy of displacement. As long as the attitude angle in the first external parameter is the same as the attitude angle in the second external parameter, it can be determined that the first external parameter and the second external parameter match. For another example, as long as it can be determined that the pitch angle in the first external parameter and the pitch angle in the second external parameter are the same (the difference is less than the preset angle threshold), it can be determined that the first external parameter and the second external parameter match.

[0070] In this embodiment, after obtaining the first external parameter between the first coordinate system and the second coordinate system, the sensor to be calibrated is not calibrated directly using the first external parameter, but the second external parameter between the first coordinate system and the second coordinate system is obtained in another way. When the first external parameter and the second external parameter match, the accuracy of the first external parameter and / or the second external parameter can be guaranteed to be high, and then the sensor to be calibrated is calibrated based on the first external parameter and / or the second external parameter. The accuracy of the calibration result of the sensor to be calibrated can be effectively improved.

[0071] In some embodiments, the sensor to be calibrated includes a laser radar, and the laser radar collects multiple frames of point clouds.

[0072] Accordingly, determining the second external parameter between the first coordinate system and the second coordinate system based on ground detection may include the following operations.

[0073] First, the ground plane is fitted based on multi-frame point clouds.

[0074] Then, determine the normal vector that fits the ground plane.

[0075] Next, a second extrinsic parameter between the first coordinate system and the second coordinate system is calculated based on the normal vector, where the second extrinsic parameter includes at least one of a roll angle, a pitch angle, and a height.

[0076] In some embodiments, the normal vector of the fitted ground plane can be obtained based on the ground detection algorithm. Specifically, the pitch angle and roll angle of the laser radar can be obtained by converting the normal vector of the fitted ground plane. Among them, the laser radar can obtain the normal vector of the fitted ground plane of the road surface on which the current vehicle passes based on the ground detection algorithm (ground detection of ground plane fitting) to obtain the pitch angle and roll angle of the laser radar. It can be understood that if there is an angle deviation in the laser radar, that is, there is a deviation between the laser radar coordinate system and the standard coordinate system (the IMU coordinate system is based on the IMU), then there will also be an angle deviation between the normal vector of the fitted ground plane obtained by the laser radar and the actual standard normal vector. Therefore, the angle deviation can be converted into the pitch angle and roll angle of the laser radar, and then the laser radar is calibrated.

[0077] After obtaining the normal vector of the fitted ground plane, the pitch angle and roll angle of the laser radar can be calculated based on the normal vector of the fitted ground plane. For example, the normal vector of the fitted ground plane is converted into the pitch angle and roll angle corresponding to the laser radar, and the laser radar is calibrated based on the obtained pitch angle and roll angle. It can be understood that if the laser radar itself has a bias, and there is a deviation between its pitch angle or roll angle and the standard value, then there must be an angle between the normal vector of its fitted ground plane and the normal vector that should actually be obtained. At this time, the vehicle can calibrate the pitch angle and roll angle of the laser radar based on the normal vector of the fitted ground plane. Among them, the normal vector of the fitted ground plane can only calibrate the pitch angle and roll angle of the laser radar, but cannot calibrate the yaw angle of the laser radar. That is, if the yaw angle of the laser radar is biased, the bias cannot be reflected in the normal vector of the fitted ground plane obtained by the laser radar.

[0078] For example, the standard normal vector (Nx, Ny, Nz) can be considered as the normal vector in the horizontal plane, that is, the vector (0, 0, 1), and the normal vector of the fitted ground plane (Nx ′ , Ny ′ , Nz ′ ) and the standard normal vector are shown in equations (1) to (3).

[0079] Nx ′ = sin(pitch(rad)) Formula (1)

[0080] Ny ′ = -sin(roll(rad))× cos(pitch(rad)) Formula (2)

[0081] N ′ = cos(roll(rad))× cos(pitch(rad)) Formula (3)

[0082] Therefore, after determining the standard normal vector and the normal vector of the kernel's ground plane, the pitch and roll angles of the lidar can be calculated.

[0083] For example, the pitch angle and roll angle of the laser radar can be determined based on the normal vector of the fitted ground plane in the following way: the angle between the normal vector of the fitted ground plane and the standard normal vector is obtained as the first angle, the standard normal vector is the normal vector of the fitted ground plane in a preset state, and the preset state is that the normal vector of the fitted ground plane is parallel to the normal vector of the horizontal plane.

[0084] After the vehicle obtains the normal vector of the fitted ground plane measured by the laser radar, the angle between the normal vector of the fitted ground plane and the standard normal vector can be used as the first angle. Among them, the standard normal vector is the normal vector of the fitted ground plane in a preset state, that is, the normal vector of the ground when the ground is parallel to the horizontal plane. The standard normal vector can also be understood as a vector parallel to the normal vector of the horizontal plane. From the perspective of the coordinate system, the standard normal vector can be the normal vector corresponding to the XOY plane in the standard coordinate system, that is, the vector (0, 0, 1). The standard coordinate system is the coordinate system corresponding to the laser radar when there is no bias. Based on the standard normal vector, the vehicle can use the angle between the normal vector of the fitted ground plane and the standard normal vector as the first angle. It can be understood that if the laser radar itself is biased, that is, the coordinate system of the laser radar does not coincide with the standard coordinate system, the first angle obtained by the vehicle is not zero. At this time, the first angle can be converted into the pitch angle and roll angle corresponding to the laser radar.

[0085] The following is an exemplary description of the method of obtaining the first pose.

[0086] In some embodiments, for an inertial measurement unit with sufficiently high accuracy, the first pose can be directly solved based on the inertial data output by the IMU.

[0087] In some embodiments, in order to further improve the accuracy of the first pose, multi-sensor data fusion may be used. Alternatively, in order to reduce hardware costs, reduce the space occupied by sensors, etc., an inertial measurement unit with lower accuracy may be used. In order to ensure the accuracy of the first pose, multi-sensor data fusion may be used.

[0088] For example, the calibration sensor includes an inertial measurement unit or an inertial measurement unit and at least one of the following: a wheel speed meter, a steering wheel sensor, or a positioning system. For example, the vehicle can obtain the current vehicle speed and travel distance by obtaining the wheel speed of the vehicle and calculating it through a built-in formula. The sensor for obtaining the wheel speed can be a Hall sensor, etc. For example, the vehicle can obtain the current driving direction by obtaining the steering angle of the steering wheel sensor of the vehicle and calculating it through a built-in formula.

[0089] Accordingly, when the calibration sensor includes an inertial measurement unit and at least one of the following: a speedometer, a steering wheel sensor or a positioning system, the first pose is determined by performing Kalman filtering on data collected by the inertial measurement unit and data collected by at least one of the speedometer, the steering wheel sensor or the positioning system to obtain the first pose.

[0090] In a specific embodiment, a wheel speed meter and an IMU are used to perform Kalman filtering to calculate the first position of the current frame. In this way, the first position can be determined by using an IMU and a wheel speed meter with general accuracy, thereby reducing hardware costs.

[0091] The following is an exemplary description of the method of obtaining the second posture.

[0092] In a certain embodiment, the sensor to be calibrated includes a laser radar, and the laser radar collects multiple frames of point clouds.

[0093] Accordingly, the second posture is determined in the following manner: for at least part of the frames in the multi-frame point cloud collected by the laser radar, the posture differences of adjacent frames between the start frame and the end frame are accumulated to obtain the second posture.

[0094] The pose differences between adjacent frames are determined as follows.

[0095] First, use the relative position of one frame to perform motion compensation on the point cloud of one frame to obtain a point cloud of one frame after distortion is eliminated. The frame rate of LiDAR is generally 10Hz, that is, 100ms per frame. When the vehicle is driving at high speed or turning, the points in one frame of point cloud are not the measurement results obtained in the same coordinate system. For the same target, distortion will appear in the three-dimensional point cloud. At this time, in order to obtain accurate measurement, it is necessary to perform motion compensation on the point cloud, or even correction, to unify all points in the same frame of point cloud into a coordinate system at a certain point in time.

[0096] Then, the line and surface features of the point cloud of the previous frame and the point cloud of the current frame after the distortion is eliminated are extracted respectively.

[0097] Then, the pose difference between the previous frame and the current frame of the sensor to be calibrated is obtained through line-surface feature registration.

[0098] For example, the pose transformation solution of adjacent frames can be realized based on the inter-frame transformation solution realized by least squares. Specifically, the point-to-line distance and the point-to-plane distance can be used as the residual of nonlinear optimization. And because feature encoding is used, all neighbor searches are realized through binary radar ring characteristics. Because there is no need to use the multidimensional tree search method in traditional point cloud matching, this matching method can further improve the matching speed.

[0099] The pose transformation parameter between two adjacent frames (such as the i+1th frame and the ith frame) is a six-dimensional vector, including the rotation parameters between the two frames, which are Euler angles (including roll angle, pitch angle, and yaw angle), and the translation vector between the two frames. The optimization method for solving pose transformation has better stability when the pose transformation is small.

[0100] Before completing the one-to-one correspondence, two transformations are performed. The first transformation is to map the features horizontally or vertically, and the second transformation is to cyclically shift the true value of the annular features. When the correspondence between the two laser point cloud lines is determined, the above two mapping parameters are essentially included. Among them, the second transformation characterizes the yaw relationship between two adjacent frames. Based on the yaw of the second transformation and the angle difference between the horizontal or vertical mapping of the two features of the first transformation, the change of the transformation parameters between the two frames can be determined.

[0101] The horizontal and vertical projection formulas already include roll and pitch information in their projection angle calculations. When the coordinates of the rotation axis relative to the IMU are known, they are aligned with the X-axis or Y-axis. At this time, the rotation angle of the rotation axis relative to the IMU is the pitch or roll. At this time, the other angle remains unchanged, and the pitch or roll plus the rotation angle of the rotation axis is the current pitch or roll angle. It should be noted that in one frame, each sampling point has the same roll angle, the same pitch angle, and the same yaw angle.

[0102] The following is an exemplary description of the method of obtaining the second external parameter.

[0103] In some embodiments, fitting a ground plane based on multi-frame point clouds may include the following operations.

[0104] First, a ground plane is fitted in the full point cloud using a first method to obtain a ground plane point cloud with a first roughness.

[0105] Then, based on the ground plane point cloud with the first roughness, the ground plane is fitted using a second method to obtain a ground plane point cloud with a second roughness, where the second roughness is finer than the first roughness.

[0106] In a specific embodiment, ground detection calculates roll and pitch, and performs cross-validation. Specifically, ground detection is also performed in a coarse-to-fine manner. First, a relatively loose parameter method is used to fit the ground plane in the full point cloud to obtain a relatively rough ground plane point cloud, and then a relatively strict parameter method is used to fit the ground plane to obtain a fitted ground plane with higher precision. Next, the normal of the fitted ground plane with higher precision is calculated, and roll, pitch, and height are calculated based on the normal vector. Then, check whether the first extrinsic parameter and the second extrinsic parameter match. If they match, calibrate the laser radar based on the first extrinsic parameter and the second extrinsic parameter. If they do not match, repeat the above operations until the first extrinsic parameter and the second extrinsic parameter match, or the preset number of repetitions is reached.

[0107] In some embodiments, since different types of sensors have different sampling frequencies, when obtaining the posture by solving the collected data of different sensors, the posture needs to be aligned to improve the accuracy of the result.

[0108] For example, determining a first external parameter between the first coordinate system and the second coordinate system based on the difference between the first pose and the second pose may include the following operations.

[0109] First, a first pose queue including a first pose and a second pose queue including a second pose are aligned based on timestamps to obtain an aligned first pose and second pose.

[0110] Then, the aligned first posture and / or the aligned second posture is adjusted in a coarse-to-fine manner to reduce the difference between the aligned first posture and the second posture, thereby obtaining a first external parameter between the first coordinate system and the second coordinate system, wherein the first external parameter includes at least one of a pitch angle or a yaw angle.

[0111] Specifically, aligning a first pose queue including a first pose and a second pose queue including a second pose based on timestamps to obtain the aligned first pose and second pose may include the following operations: interpolating the corresponding second pose in the first pose queue using the timestamp of the second pose queue.

[0112] Figure 4 It is a schematic diagram of aligning postures determined based on different sensors according to an embodiment of the present application.

[0113] See also Figure 4 , the frequency of the first pose queue (such as the pose queue determined based on inertial data collected by IMU) is relatively high, at more than 100Hz. The frequency of the second pose queue (such as the pose queue determined based on point cloud data collected by radar) is relatively low, at around 10Hz. In order to accurately determine the difference between the first pose and the second pose, the two pose queues need to be aligned. As shown in the figure, the timestamp of the second pose queue is used to interpolate the corresponding second pose in the first pose queue, and then the difference between the first pose and the second pose is calculated to determine the external parameters between the first coordinate system and the second coordinate system.

[0114] Figure 5 It is a data flow diagram of the sensor calibration process of one embodiment of the present application.

[0115] See also Figure 5, after motion compensation of the point cloud, line and surface features are extracted from the compensated point cloud, and then the normal vector of the fitted ground plane is calculated based on the line and surface features to determine the second pose. Kalman filtering is performed on the data from the IMU and / or wheel speed meter to obtain fused data, and then the pose calculation is performed on the fused data to obtain the first pose. Hand-eye calibration is performed on the first pose and the second pose to obtain the first external parameter, such as yaw / pitch. In addition, ground detection is performed on the compensated point cloud, and then the fitted ground plane is refined to obtain a normal vector with higher accuracy. In this way, the second external parameter, such as roll / pitch / z, can be calculated based on the normal vector. If the cross-validation passes, calibration can be performed based on the first external parameter and the second external parameter.

[0116] In some embodiments, considering that when a moving object moves, its point cloud data will interfere with the calibration process, in order to further improve the accuracy of the first external parameter, the point cloud of the moving object in the point cloud data can be removed.

[0117] Specifically, for a frame of point cloud, the above method may also include the following operations.

[0118] First, the point cloud corresponding to the moving object in a frame of point cloud is determined.

[0119] Then, the point cloud corresponding to the moving object in a frame of point cloud is removed.

[0120] Accordingly, respectively extracting line and surface features of the point cloud of the previous frame and the point cloud of the current frame after eliminating distortion may include: respectively extracting line and surface features from the point cloud of the previous frame and the point cloud of the current frame after eliminating distortion and removing moving objects.

[0121] Figure 6 It is a data flow diagram of the sensor calibration process of another embodiment of the present application.

[0122] See also Figure 6 ,and Figure 5 The difference is, Figure 6 The point cloud after motion compensation needs to be detected as a moving object, and the point cloud data corresponding to the moving object in the point cloud after motion compensation is deleted. This can effectively reduce the problem of inaccurate second position calculated by line and surface features due to the movement of the moving object.

[0123] In some embodiments, removing the point cloud corresponding to the moving object in a frame of point cloud includes: deleting the point cloud of the first specific object from the frame of point cloud, and the first specific object includes a movable object. Among them, movable objects include but are not limited to: cars, pedestrians, bicycles, motorcycles, etc. In this embodiment, the point cloud data corresponding to cars, pedestrians, etc. are directly deleted without determining whether the car or pedestrian is in a moving state, which helps to reduce the amount of calculation and improve the response speed. In addition, since the point cloud data corresponding to cars, pedestrians, etc. in the point cloud is only a part (even a small part), deleting the point cloud data corresponding to cars, pedestrians, etc. that have not moved will not cause inaccurate calculated posture due to insufficient point cloud data.

[0124] In some embodiments, removing the point cloud corresponding to the moving object in a frame of point cloud includes: obtaining the point cloud of the second specific object from the frame of point cloud, and the second specific object includes a fixed object. Among them, the fixed objects include but are not limited to: houses, telephone poles, lane lines, traffic lights, street lights, green plants, etc. In this embodiment, only objects such as houses and telephone poles are retained, and there is no need to determine whether the target object is in a moving state, which helps to reduce the amount of calculation and improve the response speed. In addition, since there is a lot of point cloud data corresponding to objects such as houses and telephone poles, it is sufficient to accurately calculate the second pose, etc.

[0125] For example, the classification model used to determine the type of the target object may be a pre-trained model that can determine whether the target object in the input point cloud data is the first specific object or the second specific object. Classification models include, but are not limited to: Linear Regression, Logistic Regression, Decision Trees, Naive Bayes, K-Nearest Neighbors, Support Vector Machines, Random Decision Forests or Bagging, Neural Networks, etc.

[0126] In some embodiments, the classification model may use a deep neural network (Deep Neural Networks) or the like. For example, the classification model may perform feature extraction on the input point cloud data to obtain a feature map. Then, based on the feature map or the feature map and other features, the point cloud data is classified. For another example, the classification model may perform feature extraction on each frame in the input point cloud data to obtain a feature map, and then determine a candidate box corresponding to the target object from the feature map, and then obtain feature data of the target object from the feature map based on the candidate box, and then determine the category of the candidate box based on the feature data of the target object. It should be noted that the first specific object and / or the second specific object may also be identified using traditional geometric methods.

[0127] Figure 7 It is a schematic diagram of a cross-validation process according to an embodiment of the present application.

[0128] See also Figure 7 During the cross-validation process, all parameters of the first external parameter and the second external parameter can be compared, and the verification is successful after the comparison is passed. Figure 7 As shown by the dotted line in the middle, the first external parameter and the second external parameter may each include Yaw, pitch and roll. When all parameters are successfully compared, the verification is successful.

[0129] In addition, see Figure 7 During the cross-validation process, some parameters of the first external parameter and the second external parameter can be compared, and the verification is successful after the comparison. Different sensors have different detection accuracy for different parameters. If all parameters of the first external parameter and the second external parameter are compared, it may lead to inaccurate verification results. For example, the GPS height detection is not accurate enough, and the horizontal displacement accuracy is high. If the GPS height is used for verification, it may lead to wrong verification results. Reference Figure 7 As shown by the solid line in the middle, the parameters of the first external parameter may include: Yaw / pitch, the parameters of the second external parameter may include: roll / pitch / z, and the credibility of the parameters pitch of the first external parameter and the parameters pitch of the second external parameter is high, then only the parameters pitch of the first external parameter and the parameters pitch of the second external parameter can be compared to see if they are consistent. When the pitch comparison is successful, the verification is successful. If roll / z is missing from the parameters of the first external parameter, it can be obtained from the parameters of the second external parameter.

[0130] Specifically, the consistency of the pitch is checked, and if it is consistent, the laser radar is calibrated based on the first external parameter and the second external parameter. If it is inconsistent, the above operation is repeated until the pitch is consistent or the preset number of repetitions is reached.

[0131] In a specific embodiment, reference Figure 6As shown, regarding the relative posture calculation of the first coordinate system, it can include performing Kalman filtering on the wheel speed meter and IMU to calculate the first posture.

[0132] Regarding the second posture calculation, operations a1 to a4 may be included.

[0133] In operation a1, motion compensation is performed on the point cloud using the relative pose of the previous frame.

[0134] In operation a2, obstacle detection, deep learning is used to detect obstacles, including vehicles, pedestrians, tricycles and other obstacles that may move. It should be noted that obstacle detection can be performed using both deep learning methods and traditional geometric methods.

[0135] In operation a3, obstacle removal, the point cloud of obstacles that may move is removed. These obstacles that may move may affect the registration accuracy of the subsequent lidar odometry.

[0136] In operation a4, lidar odometry, lidar odometry is performed on the point cloud after obstacle point cloud removal, and the method of extracting line and surface features for registration is similar to the registration method of lidar odometry and mapping (LOAM). This registration method is used to calculate the relative pose between frames. It should be noted that lidar odometry can also use iterative closest point (ICP), GICP, NDT and other methods for point cloud registration.

[0137] In addition, the first pose and the second pose may be processed based on hand-eye calibration. Specifically, the following operations may be included.

[0138] First, the poses are aligned based on timestamp interpolation, and then the aligned first pose and second pose are calculated. Next, hand-eye calibration is performed. For example, the input is the first pose queue and the second pose queue. Usually there are at least 40 pairs in the queue, and the selection of relative pose pairs must ensure that the angles have a certain change, and the change must be greater than 0.4°. Specifically, a coarse-to-fine approach can be used for hand-eye calibration. For example, the SVD method is first used to calculate the rough initial value, and then the nonlinear optimization method is used to further align the first pose and the second pose to solve the first external parameters pitch, yaw and height.

[0139] In addition, ground detection is also performed in a coarse-to-fine manner. Specifically, first, a relatively loose parameter method is used to fit the ground plane in the full point cloud to obtain a relatively rough fitted ground plane, and then a relatively strict parameter method is used to fit the plane to obtain a higher-precision fitted ground plane. Next, the normal vector of the fitted ground plane is calculated, and the second external parameters: roll, pitch, and height are calculated based on the normal vector.

[0140] Cross-validate the consistency of the pitch in the first external parameter and the second external parameter, and if they are consistent, merge the first external parameter and the second external parameter. In addition, if the cross-validation result is inconsistent, the above operation can be repeated until the verification passes or the number of cycles is reached.

[0141] The above method can calibrate the external parameters of the laser radar relative to the first coordinate system (vehicle body coordinate system).

[0142] Figure 8 It is a flow chart of a sensor calibration method shown in another embodiment of the present application.

[0143] See also Figure 8 ,When calibrating the laser radar, first obtain a frame of point cloud.

[0144] Then, the lidar odometer result is used to perform motion compensation on the frame point cloud.

[0145] Next, obstacle detection is performed on the frame point cloud. The obstacle detection method can use a deep learning method or a traditional geometric method.

[0146] Then, the point cloud of the detected obstacle is removed from the motion-compensated point cloud of the one frame.

[0147] Next, the lidar odometer is calculated by extracting line and surface features for registration and calculating the second posture.

[0148] While determining the second posture based on the lidar, the mileage data obtained by the wheel speed meter and the inertial data measured by the IMU can also be obtained.

[0149] Next, Kalman filtering is performed on the mileage data and inertial data to calculate the first posture of the current frame.

[0150] Then, a first extrinsic parameter is calculated based on the difference between the first posture and the second posture.

[0151] After performing motion compensation on a frame of point cloud data, ground detection can also be performed on the point cloud to obtain the normal vector of the fitted ground plane.

[0152] Next, calculate roll, pitch, and height based on the ground normal vector.

[0153] Then, the consistency of the pitch in the first extrinsic parameter and the second extrinsic parameter is cross-checked. If they are consistent, the lidar is calibrated based on the first extrinsic parameter and the second extrinsic parameter.

[0154] Another aspect of the present application provides a radar.

[0155] The radar may be a laser radar, a millimeter wave radar, an ultrasonic radar, etc. The radar may be a scanning radar or a non-scanning radar.

[0156] The following is an exemplary description using a scanning laser radar.

[0157] For example, MEMS-based LiDAR can dynamically adjust its scanning pattern to focus on specific objects, collect detailed information about smaller objects farther away, and identify them. MEMS-based LiDARs do not have a large moment of inertia and can move quickly, so fast that they can track a 2D scanning pattern in less than a second.

[0158] For example, Flash LiDAR can quickly record the entire scene, avoiding the trouble caused by the movement of the target or LiDAR during the scanning process. The radar system uses an array of micro sensors to collect laser beams reflected from different directions.

[0159] For example, a phased-array lidar uses a row of transmitters to change the direction of a laser beam by adjusting the relative phase of the signals.

[0160] For example, mechanical rotating laser radar is an early developed laser radar and the technology is relatively mature. However, the structure of the mechanical rotating laser radar system is very complex and the core components are quite expensive. These mainly include lasers, scanners, optical components, photodetectors, receiving ICs, and positioning and navigation devices.

[0161] Taking MEMS solid-state laser radar as an example, since MEMS solid-state laser radar is scanned by simple harmonic vibration of the galvanometer, its scanning path can be realized in spatial order, for example, a scanning field of view with a slow axis from top to bottom and a fast axis from left to right. Therefore, the detection range of MEMS solid-state laser radar is divided by the field of view angle corresponding to the slow axis. For example, the vertical field of view angle corresponding to the slow axis of MEMS solid-state laser radar is 13° to 13°.

[0162] Taking the mechanical laser radar in the scanning sensor as an example, since the mechanical laser radar uses a mechanical drive device to drive the optical system to rotate 360 ​​degrees to achieve scanning, a cylindrical detection area with the laser radar as the center is formed. Therefore, the detection range corresponding to the rotation of the mechanical laser radar by 360 degrees is the detection range corresponding to the detection of one frame of data, so the division of the detection range of a mechanical laser radar cycle is generally divided by the degree of rotation.

[0163] For non-scanning LiDAR, the image is processed by the internal photosensitive component circuit and control component and converted into a digital signal that can be recognized by the computer. The digital signal is then input into the computer through a parallel port or USB connection, and the image is restored by the software.

[0164] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0165] Another aspect of the present application provides a sensor calibration device.

[0166] Fig. 9 It is a block diagram of a sensor calibration device according to an embodiment of the present application.

[0167] See also Fig. 9 The sensor calibration device 900 includes: an external parameter determination module 910, a ground detection module 920 and a cross calibration module 930.

[0168] The external parameter determination module 910 is used to determine the first external parameter between the first coordinate system and the second coordinate system based on the difference between the first pose and the second pose, wherein the first pose is the pose of the calibration sensor arranged on the mobile platform in the first coordinate system, and the second pose is the pose of the sensor to be calibrated arranged on the mobile platform in the second coordinate system.

[0169] The ground detection module 920 is used to determine a second extrinsic parameter between the first coordinate system and the second coordinate system based on ground detection.

[0170] The cross calibration module 930 is used to calibrate the sensor to be calibrated based on the first external parameter and / or the second external parameter if the first external parameter matches the second external parameter.

[0171] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated again here.

[0172] Another aspect of the present application also provides a vehicle.

[0173] Fig.10 A block diagram of a vehicle according to an embodiment of the present application is schematically shown.

[0174] See also Fig.10 , vehicle 1000 includes a vehicle body, a memory 1010 , a processor 1020 and a laser radar 1030 .

[0175] The memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. Among them, ROM can store static data or instructions required by the processor 1020 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose the stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as a permanent storage device. In some other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as a dynamic random access memory. The system memory may store some or all instructions and data required by the processor at run time. In addition, the memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor storage chips (such as DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a readable and / or writable removable storage device, such as a laser disc (CD), a read-only digital versatile disc (such as a DVD-ROM, a double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a mini SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer-readable storage medium does not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0176] The processor 1020 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0177] The memory 1010 stores executable codes, and when the executable codes are processed by the processor 1020 , the processor 1020 can execute part or all of the methods described above.

[0178] The laser radar 1030 is arranged on the vehicle body to collect point clouds. Please refer to the above related contents and will not be described in detail here.

[0179] It is understandable that, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, or each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0180] If the integrated unit / module is implemented in the form of hardware, the hardware can be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not otherwise specified, the artificial intelligence processor can be any appropriate hardware processor, such as CPU, GPU, FPGA, DSP, and ASIC, etc. If not otherwise specified, the storage module can be any appropriate magnetic storage medium or magneto-optical storage medium, such as resistive random access memory RRAM (Resistive Random Access Memory), dynamic random access memory DRAM (Dynamic Random Access Memory), static random access memory SRAM (Static Random-Access Memory), enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), high-bandwidth memory HBM (High-Bandwidth Memory), hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0181] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0182] In a possible implementation, a board is also disclosed, which includes a storage device, an interface device, a control device and the above-mentioned processor; wherein the processor is connected to the storage device, the control device and the interface device respectively; the storage device is used to store data; the interface device is used to realize data transmission between the processor and an external device; and the control device is used to monitor the status of the processor.

[0183] In a possible implementation, an electronic device is disclosed, which includes the above-mentioned processor. The electronic device includes a data processing device, a robot, a computer, a printer, a scanner, a tablet computer, a smart terminal, a mobile phone, a driving recorder, a navigator, a sensor, a camera, a server, a cloud server, a camera, a video camera, a projector, a watch, a headset, a mobile storage, a wearable device, a means of transportation, a household appliance, and / or a medical device. The means of transportation include an airplane, a ship, and / or a vehicle; the household appliance includes a television, an air conditioner, a microwave oven, a refrigerator, an electric rice cooker, a humidifier, a washing machine, an electric lamp, a gas stove, and a range hood; the medical device includes an MRI, an ultrasound machine, and / or an electrocardiograph.

[0184] In addition, the method according to the present application may also be implemented as a computer program or a computer program product, which includes computer program code instructions for executing some or all of the steps in the above method of the present application.

[0185] Alternatively, the present application can also be implemented as a computer-readable storage medium (or non-transitory machine-readable storage medium or machine-readable storage medium) on which executable code (or computer program or computer instruction code) is stored. When the executable code (or computer program or computer instruction code) is executed by a processor of an electronic device (or server, etc.), the processor executes part or all of the steps of the above-mentioned method according to the present application.

[0186] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A sensor calibration method, characterized in that: include: Determine a first external parameter between a first coordinate system and a second coordinate system based on a difference between a first pose and a second pose, wherein the first pose is a pose of a calibration sensor disposed on a mobile platform in the first coordinate system, and the second pose is a pose of a sensor to be calibrated disposed on the mobile platform in the second coordinate system; Determine a second extrinsic parameter between the first coordinate system and the second coordinate system based on ground detection; If the first external parameter matches the second external parameter, calibrating the sensor to be calibrated based on the first external parameter and / or the second external parameter; Wherein, determining a first external parameter between the first coordinate system and the second coordinate system based on the difference between the first posture and the second posture includes: Aligning a first pose queue including the first pose and a second pose queue including the second pose based on the timestamp to obtain an aligned first pose and a second pose; The aligned first pose and / or the aligned second pose are adjusted in a coarse-to-fine manner to reduce a difference between the aligned first pose and the second pose, thereby obtaining a first extrinsic parameter between the first coordinate system and the second coordinate system.

2. The method according to claim 1, characterized in that The sensor to be calibrated includes a laser radar, and the laser radar collects multiple frames of point clouds; The determining a second extrinsic parameter between the first coordinate system and the second coordinate system based on ground detection includes: Fitting a ground plane based on the multi-frame point cloud; Determine the normal vector of the fitted ground plane; A second extrinsic parameter between the first coordinate system and the second coordinate system is calculated based on the normal vector, where the second extrinsic parameter includes at least one of a roll angle, a pitch angle, and a height.

3. The method according to claim 2, characterized in that The step of fitting a ground plane based on the multi-frame point cloud comprises: Fitting a ground plane in the full point cloud using the first method to obtain a ground plane point cloud with a first roughness; Based on the ground plane point cloud with the first roughness, the ground plane is fitted by using a second method to obtain a ground plane point cloud with a second roughness, where the second roughness is finer than the first roughness.

4. The method according to claim 1, characterized in that The sensor to be calibrated includes a laser radar, and the laser radar collects multiple frames of point clouds; The second posture is determined by: for at least part of the frames in the multi-frame point cloud collected by the laser radar, accumulating the posture differences of adjacent frames between the start frame and the end frame to obtain the second posture; The pose difference between adjacent frames is determined by: Using the relative position of a frame, motion compensation is performed on the point cloud of the frame to obtain a point cloud of the frame after distortion is eliminated; Extract line and surface features of the point cloud of the previous frame and the point cloud of the current frame after eliminating distortion respectively; The position difference of the sensor to be calibrated between the previous frame and the current frame is obtained by line-surface feature registration.

5. The method according to claim 4, characterized in that Also includes: For a frame of point cloud, Determine a point cloud corresponding to the moving object in the one frame of point cloud; Removing the point cloud corresponding to the moving object in the one frame of point cloud; The extracting of line and surface features of the point cloud of the previous frame and the point cloud of the current frame after eliminating distortion respectively includes: Line and surface features are respectively extracted from the point cloud of the previous frame and the point cloud of the current frame after eliminating distortion and removing the moving object.

6. The method according to claim 5, characterized in that The removing of the point cloud corresponding to the moving object in the one frame of point cloud comprises: Deleting the point cloud of a first specific object from the one frame of point cloud, where the first specific object includes a movable object; or A point cloud of a second specific object is obtained from the one frame of point cloud, where the second specific object includes a fixed object.

7. The method according to claim 1, characterized in that The first external parameter includes at least one of a pitch angle or a yaw angle.

8. The method according to claim 7, characterized in that The method of aligning a first pose queue including the first pose and a second pose queue including the second pose based on the timestamp to obtain the aligned first pose and second pose comprises: The timestamp of the second pose queue is used to interpolate a corresponding second pose in the first pose queue.

9. The method according to any one of claims 1 to 8, characterized in that The calibration sensor includes an inertial measurement unit or an inertial measurement unit and at least one of the following: a wheel speed meter, a steering wheel sensor, or a positioning system; When the calibration sensor includes an inertial measurement unit and at least one of the following: a wheel speed meter, a steering wheel sensor, or a positioning system, the first posture is determined by: Kalman filtering is performed on data collected by the inertial measurement unit and data collected by at least one of the wheel speed meter, the steering wheel sensor, or the positioning system to obtain the first posture.

10. A sensor calibration device, characterized in that: include: An external parameter determination module, used to determine a first external parameter between a first coordinate system and a second coordinate system based on a difference between a first pose and a second pose, wherein the first pose is a pose of a calibration sensor arranged on a mobile platform in the first coordinate system, and the second pose is a pose of a sensor to be calibrated arranged on the mobile platform in the second coordinate system; A ground detection module, configured to determine a second extrinsic parameter between the first coordinate system and the second coordinate system based on ground detection; a cross calibration module, configured to calibrate the sensor to be calibrated based on the first external parameter and / or the second external parameter if the first external parameter matches the second external parameter; Among them, the external parameter determination module is used to: align the first posture queue including the first posture and the second posture queue including the second posture based on the timestamp to obtain the aligned first posture and the second posture; adjust the aligned first posture and / or the aligned second posture in a coarse-to-fine manner to reduce the difference between the aligned first posture and the second posture, so as to obtain the first external parameter between the first coordinate system and the second coordinate system.

11. A vehicle, characterized in that: include: The main body of the vehicle; A laser radar, arranged on the vehicle body, for collecting point clouds; a processor, coupled to the laser radar, configured to receive the point cloud; A memory having executable codes stored thereon, which, when executed by the processor, causes the processor to execute the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the method according to any one of claims 1 to 9.

Images