lidar-to-lidar alignment and lidar-to-vehicle alignment online verification

Abstract

A LIDAR-to-LIDAR alignment system includes a memory and an autonomous driving module. The memory stores first and second points based on outputs of first and second LIDAR sensors. The autonomous driving module performs a verification process to determine whether an alignment of the LIDAR sensors satisfies an alignment condition. The verification process includes: aggregating the first and second points in a vehicle coordinate system to provide aggregated LIDAR points; based on the aggregated LIDAR points, performing (i) a first method that includes determining a pitch and roll difference between the first and second LIDAR sensors, (ii) a second method that includes determining a yaw difference between the first and second LIDAR sensors, or (iii) a point cloud registration to determine a rotation and translation difference between the first and second LIDAR sensors; and based on a result of the first method, the second method, or the point cloud registration, determining whether the alignment condition is satisfied.

Description

[0001] introduction

[0002] The information provided in this section is intended to provide a general overview of the background of this disclosure. To the extent described in this section, the work of the currently named inventors, and aspects of the description that may not conform to the prior art at the time of filing, are neither explicitly nor implicitly considered to be prior art of this disclosure. Technical Field

[0003] This disclosure relates to vehicle object detection systems, and more particularly, to vehicle optical detection and ranging (LIDAR) systems. Background Technology

[0004] The vehicle may include various sensors for detecting the surrounding environment and objects within that environment. Sensors may include cameras, radio detection and ranging (RADAR) sensors, LiDAR sensors, etc. The vehicle controller may perform various operations in response to the detected environment. These operations may include performing partially and / or fully autonomous vehicle operations, collision avoidance operations, and information reporting operations. The accuracy of the performed operations may be based on the accuracy of the data collected from the sensors. Summary of the Invention

[0005] A LIDAR-to-LIDAR alignment system is provided, comprising a memory and an autonomous driving module. The memory is configured to store (i) a first data point provided based on the output of a first LIDAR sensor, and (ii) a second data point provided based on the output of a second LIDAR sensor. The autonomous driving module is configured to perform a verification process to determine whether the alignment of the first and second LIDAR sensors satisfies one or more alignment conditions. The verification process includes: clustering a first data point and a second data point in a vehicle coordinate system to provide clustered LIDAR points; based on the clustered LIDAR points, performing at least one of the following: (i) performing a first method including determining pitch and roll differences between the first and second LIDAR sensors, (ii) performing a second method including determining yaw differences between the first and second LIDAR sensors, or (iii) performing point cloud registration to determine rotation and translation differences between the first and second LIDAR sensors; determining whether one or more alignment conditions are met based on the result of at least one of the first method, the second method, or point cloud registration; and recalibrating at least one of the first or second LIDAR sensors in response to not meeting the one or more alignment conditions.

[0006] Among other features, the autonomous driving module is further configured to: determine whether a plurality of enabling conditions are met, including two or more of the following: determining whether the vehicle is moving, determining whether the vehicle is in a known position, determining whether the vehicle's speed is greater than a predetermined speed, determining whether the vehicle's acceleration is greater than a predetermined acceleration, or determining whether the yaw rate is greater than a predetermined yaw rate; and in response to the fulfillment of the enabling conditions, aggregate a first data point and a second data point.

[0007] Among other features, the aggregation of the first data point and the second data point includes: performing point registration to map the first data point at the first time point in the coordinate system of the first LIDAR sensor at the second time point; aggregating the mapped first data points from the first LIDAR sensor to provide a first aggregation point; performing point registration to map the second data point at the first time point in the coordinate system of the second LIDAR sensor at the second time point; and aggregating the mapped second data points from the second LIDAR sensor to provide a second aggregation point.

[0008] Among other features, the aggregation of the first data point and the second data point also includes: mapping the first aggregation point to the vehicle coordinate system based on the rotation calibration value and translation calibration value of the first LIDAR sensor; and mapping the second aggregation point to the vehicle coordinate system based on the rotation calibration value and translation calibration value of the second LIDAR sensor.

[0009] Among other features, aggregating the first data point and the second data point includes: mapping the first data point to the vehicle coordinate system based on the rotation calibration value and translation calibration value of the first LIDAR sensor; and mapping the second data point to the vehicle coordinate system based on the rotation calibration value and translation calibration value of the second LIDAR sensor.

[0010] Among other features, the autonomous driving module is configured to: perform ground fitting and selection based on aggregated LIDAR points to provide a first selection point of a first LIDAR sensor in the ground region and a second selection point of a second LIDAR sensor in the ground region; and perform a first method based on the first and second selection points.

[0011] Among other features, the autonomous driving module is configured to execute a first method and, based on the result of the first method, determine whether one or more alignment conditions are met.

[0012] Among other features, the autonomous driving module is configured to execute a second method, based on the results of the second method, to determine whether one or more alignment conditions are met.

[0013] Among other features, the autonomous driving module is configured to perform point cloud registration and, based on the results of the point cloud registration, determine whether one or more alignment conditions are met.

[0014] Among other features, the autonomous driving module is configured to perform a first method, a second method, and point cloud registration, and based on the results of the first method, the second method, and point cloud registration, determine whether one or more alignment conditions are met.

[0015] Among other features, the autonomous driving module is configured to: integrate results from one or more of the first method, the second method, and the point cloud registration, including determining a weighted sum of difference vectors for six degrees of freedom; and in response to one or more of the differences exceeding a predetermined threshold, at least one of the following: (i) identifying one or more of the first LIDAR sensor and the second LIDAR sensor as suspicious, or (ii) recalibrating the one or more of the first LIDAR sensor and the second LIDAR sensor.

[0016] Among other features, the autonomous driving module is configured to determine whether the transformation from the second LiDAR sensor coordinate system to the vehicle coordinate system is accurate based on knowledge that the transformation from the first LiDAR sensor coordinate system to the vehicle coordinate system is accurate and a weighted sum of the difference vectors.

[0017] Among other features, the autonomous driving module is configured to calculate the six-degree-of-freedom difference between the first and second LiDAR sensors based on the alignment results of the first and second LiDAR sensors with the vehicle.

[0018] Among other features, a LIDAR-to-LIDAR alignment verification method is provided, the method comprising: clustering first data points and second data points in a vehicle coordinate system to provide clustered LIDAR points, wherein the first data points are provided based on the output of a first LIDAR sensor and the second data points are provided based on the output of a second LIDAR sensor; based on the clustered LIDAR points, performing at least one of the following: (i) performing a first method including determining pitch and roll differences between the first and second LIDAR sensors, (ii) performing a second method including determining yaw differences between the first and second LIDAR sensors, or (iii) performing point cloud registration to determine rotation and translation differences between the first and second LIDAR sensors; determining whether one or more alignment conditions are met based on the result of at least one of the first method, the second method, or the point cloud registration; and recalibrating at least one of the first or second LIDAR sensors in response to the non-metance of the one or more alignment conditions.

[0019] Among other features, the method further includes: determining whether a plurality of enabling conditions are met, including two or more of the following: determining whether the vehicle is moving, determining whether the vehicle is in a known position, determining whether the vehicle's speed is greater than a predetermined speed, determining whether the vehicle's acceleration is greater than a predetermined acceleration, or determining whether the yaw rate is greater than a predetermined yaw rate; and in response to the satisfaction of the enabling conditions, aggregating a first data point and a second data point.

[0020] Among other features, the aggregation of the first data point and the second data point includes: performing point registration to map the first data point at the first time in the coordinate system of the first LIDAR sensor at the second time; aggregating the mapped first data point from the first LIDAR sensor to provide a first aggregation point; performing point registration to map the second data point at the first time in the coordinate system of the second LIDAR sensor at the second time; aggregating the mapped second data point from the second LIDAR sensor to provide a second aggregation point; mapping the first aggregation point to the vehicle coordinate system based on rotation calibration values ​​and translation calibration values ​​of the first LIDAR sensor; and mapping the second aggregation point to the vehicle coordinate system based on rotation calibration values ​​and translation calibration values ​​of the second LIDAR sensor.

[0021] Among other features, the method further includes: performing ground fitting and selection based on aggregated LIDAR points to provide a first selection point for a first LIDAR sensor in the ground region and a second selection point for a second LIDAR sensor in the ground region; and performing a first method based on the first and second selection points.

[0022] Among other features, the method also includes performing a first method and a second method, and determining, based on the results of the first method and the second method, whether one or more alignment conditions are met.

[0023] Among other features, the method also includes performing point cloud registration and, based on the results of the point cloud registration, determining whether one or more alignment conditions are met.

[0024] Among other features, the method also includes performing a first method, a second method, and point cloud registration, and determining whether one or more alignment conditions are met based on the results of the first method, the second method, and point cloud registration.

[0025] This invention provides the following technical solutions:

[0026] 1. A LiDAR-to-LiDAR alignment system, comprising:

[0027] A memory configured to store (i) a first data point provided based on the output of a first LIDAR sensor, and (ii) a second data point provided based on the output of a second LIDAR sensor; and

[0028] An autonomous driving module is configured to perform a verification process to determine whether the alignment of the first LIDAR sensor and the second LIDAR sensor meets one or more alignment conditions, the verification process including...

[0029] The first data point and the second data point are aggregated in the vehicle coordinate system to provide aggregated LIDAR points.

[0030] Based on clustered LiDAR points, at least one of the following:

[0031] Perform a first method including determining the pitch and roll difference between the first LIDAR sensor and the second LIDAR sensor.

[0032] Perform a second method including determining the yaw difference between the first LIDAR sensor and the second LIDAR sensor, or

[0033] Point cloud registration is performed to determine the rotation and translation difference between the first and second LiDAR sensors.

[0034] Based on the result of at least one of the first method, the second method, or the point cloud registration, determine whether one or more alignment conditions are met, and

[0035] In response to failure to meet one or more alignment conditions, at least one of the first LIDAR sensor or the second LIDAR sensor is recalibrated.

[0036] 2. The LIDAR-to-LIDAR alignment system according to technical solution 1, wherein the autonomous driving module is further configured to:

[0037] Determine whether multiple enabling conditions are met, wherein the enabling conditions include two or more of the following:

[0038] Determine whether the vehicle is moving.

[0039] Determine whether the vehicle is in a known location.

[0040] Determine whether the vehicle's speed is greater than a predetermined speed.

[0041] Determine whether the vehicle's acceleration is greater than a predetermined acceleration, or

[0042] Determine whether the yaw rate is greater than the predetermined yaw rate; and

[0043] In response to the fulfillment of the plurality of enabling conditions, the first data point and the second data point are aggregated.

[0044] 3. The LIDAR-to-LIDAR alignment system according to technical solution 1, wherein the aggregation of the first data point and the second data point includes:

[0045] Perform point registration to map the first data point at the first time point into the coordinate system of the first LIDAR sensor at the second time point;

[0046] First data points mapped from the first LIDAR sensor are aggregated to provide a first aggregation point;

[0047] Perform point registration to map the second data point at the first time point into the coordinate system of the second LIDAR sensor at the second time point; and

[0048] The mapped second data points from the second LIDAR sensor are aggregated to provide a second aggregation point.

[0049] 4. The LIDAR-to-LIDAR alignment system according to technical solution 3, wherein the aggregation of the first data point and the second data point further includes:

[0050] The first aggregation point is mapped to the vehicle coordinate system based on the rotation and translation calibration values ​​of the first LIDAR sensor; and

[0051] The second aggregation point is mapped to the vehicle coordinate system based on the rotation calibration value and translation calibration value of the second LIDAR sensor.

[0052] 5. The LIDAR-to-LIDAR alignment system according to technical solution 1, wherein the aggregation of the first data point and the second data point includes:

[0053] The first data points are mapped to the vehicle coordinate system based on the rotation and translation calibration values ​​from the first LIDAR sensor; and

[0054] The second data point is mapped to the vehicle coordinate system based on the rotation calibration value and translation calibration value of the second LIDAR sensor.

[0055] 6. The LIDAR-to-LIDAR alignment system according to technical solution 1, wherein the autonomous driving module is configured to:

[0056] Based on the aggregated LIDAR points, ground fitting and selection are performed to provide a first selected point for a first LIDAR sensor and a second selected point for a second LIDAR sensor in the ground region; and

[0057] The first method is executed based on the first selection point and the second selection point.

[0058] 7. The LIDAR-to-LIDAR alignment system according to technical solution 1, wherein the autonomous driving module is configured to execute the first method and determine, based on the result of the first method, whether one or more alignment conditions are met.

[0059] 8. The LIDAR-to-LIDAR alignment system according to technical solution 1, wherein the autonomous driving module is configured to execute the second method and, based on the result of the second method, determine whether one or more alignment conditions are met.

[0060] 9. The LIDAR-to-LIDAR alignment system according to technical solution 1, wherein the autonomous driving module is configured to perform the point cloud registration and, based on the result of the point cloud registration, determine whether one or more alignment conditions are met.

[0061] 10. The LIDAR-to-LIDAR alignment system according to technical solution 1, wherein the autonomous driving module is configured to execute the first method, the second method and the point cloud registration, and determine whether one or more alignment conditions are met based on the results of the first method, the second method and the point cloud registration.

[0062] 11. The LIDAR-to-LIDAR alignment system according to technical solution 1, wherein the autonomous driving module is configured to:

[0063] Integrating results from one or more of the first method, the second method, and point cloud registration, including determining a weighted sum of the difference vectors for the six degrees of freedom; and

[0064] In response to one or more of the differences exceeding a predetermined threshold, at least one of the following is performed: (i) one or more of the first LIDAR sensor and the second LIDAR sensor is identified as suspicious, or (ii) one or more of the first LIDAR sensor and the second LIDAR sensor are recalibrated.

[0065] 12. The LIDAR-to-LIDAR alignment system according to technical solution 11, wherein the autonomous driving module is configured to determine whether the transformation from the second LIDAR sensor coordinate system to the vehicle coordinate system is accurate based on knowing that the transformation from the first LIDAR sensor coordinate system to the vehicle coordinate system is accurate and the weighted sum of the difference vectors.

[0066] 13. The LIDAR-to-LIDAR alignment system according to technical solution 1, wherein the autonomous driving module is configured to calculate the six-degree-of-freedom difference between the first LIDAR sensor and the second LIDAR sensor based on the alignment result of the first LIDAR sensor to the vehicle and the alignment result of the second LIDAR sensor to the vehicle.

[0067] 14. A LiDAR-to-LiDAR alignment verification method, comprising:

[0068] The first data point and the second data point are aggregated in the vehicle coordinate system to provide aggregated LIDAR points, wherein the first data point is provided based on the output of the first LIDAR sensor and the second data point is provided based on the output of the second LIDAR sensor;

[0069] Based on clustered LiDAR points, at least one of the following:

[0070] Perform a first method including determining the pitch and roll difference between the first LIDAR sensor and the second LIDAR sensor.

[0071] Perform a second method including determining the yaw difference between the first LIDAR sensor and the second LIDAR sensor, or

[0072] Perform point cloud registration to determine the rotation and translation difference between the first and second LiDAR sensors;

[0073] Based on the result of the first method, the second method, or at least one of the point cloud registration, determine whether one or more alignment conditions are met; and

[0074] In response to failure to meet one or more alignment conditions, at least one of the first LIDAR sensor or the second LIDAR sensor is recalibrated.

[0075] 15. The method according to technical solution 14 further includes:

[0076] Determine whether multiple enabling conditions are met, including two or more of the following:

[0077] Determine whether the vehicle is moving.

[0078] Determine whether the vehicle is in a known location.

[0079] Determine whether the vehicle's speed is greater than a predetermined speed.

[0080] Determine whether the vehicle's acceleration is greater than a predetermined acceleration, or

[0081] Determine whether the yaw rate is greater than a predetermined yaw rate; and

[0082] In response to the fulfillment of the plurality of enabling conditions, the first data point and the second data point are aggregated.

[0083] 16. The method according to technical solution 14, wherein the aggregation of the first data point and the second data point includes:

[0084] Perform point registration to map the first data point at the first time point into the coordinate system of the first LIDAR sensor at the second time point;

[0085] First data points mapped from the first LIDAR sensor are aggregated to provide a first aggregation point;

[0086] Perform point registration to map the second data point at the first time point into the coordinate system of the second LIDAR sensor at the second time point;

[0087] Aggregate the second data points mapped from the second LIDAR sensor to provide a second aggregation point;

[0088] The first aggregation point is mapped to the vehicle coordinate system based on the rotation and translation calibration values ​​from the first LIDAR sensor; and

[0089] The second aggregation point is mapped to the vehicle coordinate system based on the rotation calibration value and translation calibration value of the second LIDAR sensor.

[0090] 17. The method according to technical solution 14 further includes:

[0091] Based on the aggregated LIDAR points, ground fitting and selection are performed to provide a first selected point for a first LIDAR sensor and a second selected point for a second LIDAR sensor in the ground region; and

[0092] The first method is executed based on the first selection point and the second selection point.

[0093] 18. The method according to technical solution 14 further includes performing the first method and the second method, and determining whether the one or more alignment conditions are satisfied based on the results of the first method and the second method.

[0094] 19. The method according to technical solution 14 further includes performing the point cloud registration and determining whether the one or more alignment conditions are met based on the result of the point cloud registration.

[0095] 20. The method according to technical solution 14 further includes performing the first method, the second method and the point cloud registration, and determining whether the one or more alignment conditions are met based on the results of the first method, the second method and the point cloud registration.

[0096] Further applications of this disclosure will become apparent from the detailed description, claims, and accompanying drawings. The detailed description and specific examples are for illustrative purposes only and are not intended to limit the scope of the invention. Attached Figure Description

[0097] This disclosure will be more fully understood from the detailed description and accompanying drawings, wherein:

[0098] Figure 1 This is a functional block diagram of an example vehicle system including a verification alignment module according to this disclosure;

[0099] Figure 2 This is a functional block diagram of an example autonomous driving module operating based on LIDAR sensor alignment performed by the verification alignment module, according to this disclosure.

[0100] Figure 3 This is a functional block diagram of an example alignment system including a LIDAR alignment module and a LIDAR alignment verification module according to the present disclosure.

[0101] Figure 4 An example method for verifying LIDAR sensor alignment according to this disclosure is illustrated;

[0102] Figure 5 The illustration shows an example enable condition check method according to this disclosure;

[0103] Figure 6 The illustration shows an example ground fitting and selection method according to this disclosure;

[0104] Figure 7 This is an example diagram illustrating yaw difference detection according to this disclosure; and

[0105] Figure 8 This is the example object detection method.

[0106] In the accompanying drawings, reference numerals may be used repeatedly to identify similar and / or identical elements. Detailed Implementation

[0107] The autonomous driving module can perform sensor alignment and fusion operations, perception and localization operations, and path planning and vehicle control operations. These operations can be performed based on data collected from various sensors, such as LiDAR sensors, RADAR sensors, cameras, and inertial measurement sensors (or inertial measurement units). Sensor alignment and fusion may include aligning the coordinate system of each sensor with a reference coordinate system (such as the vehicle coordinate system). Fusion may refer to the collection and combination of data from various sensors.

[0108] Perception refers to the monitoring of the vehicle's surroundings and the detection and identification of various features and / or objects within those features. This can include determining aspects of features and objects. As used herein, a "feature" refers to one or more detection points that can be reliably used to determine the location of an object. This differs from other detected data points that do not provide reliable information about the object's location, such as points on leaves or branches. Determined aspects can include object distance, location, size, shape, orientation, trajectory, etc. This can include determining the type of object detected, such as whether it is a traffic sign, vehicle, utility pole, pedestrian, ground, etc. Lane marking information can also be detected. Features can refer to the surface, edge, or corner of a building. Localization refers to the determined information about the primary vehicle, such as its position, speed, heading, etc. Path planning and vehicle control (e.g., braking, steering, and acceleration) are performed based on the collected perception and localization information.

[0109] Vehicles may include multiple LiDAR sensors. LiDAR sensor alignment, including LiDAR-to-vehicle alignment and LiDAR-to-LIDAR alignment, affects the accuracy of the determined perception and localization information, including feature and object information, such as those described above. LiDAR alignment may require periodic recalibration to address any misalignment caused by different driving conditions. Online verification (meaning in-vehicle verification during runtime) is challenging because the target object is not predetermined.

[0110] Examples described herein include LiDAR-to-LiDAR alignment and LiDAR-to-vehicle alignment verification methods. Verification is used to detect LiDAR misalignment to initiate a recalibration process. Verification methods include an integration of ground fitting, object detection, and point cloud registration. In one embodiment, roll, pitch, and yaw differences between LiDAR sensors are determined based on objects (e.g., ground, traffic signs, light poles, etc.). In the same or alternative embodiments, rotation and translation differences of the LiDAR sensors are determined based on differences in point cloud registration of the LiDAR sensors. Verification methods include (i) a first method for determining a first six-parameter vector of differences between LiDAR sensors in pitch, roll, yaw, x, y, z values, and / or (ii) a second method for determining a second six-parameter vector of pitch, roll, yaw, x, y, z values. The first method determines roll, pitch, and yaw based on the selection of certain objects. The second method determines rotation and translation differences based on the point cloud of the LiDAR sensors. The results of these methods can be weighted and aggregated to provide a synthetic six-parameter vector upon which alignment is determined.

[0111] Figure 1 An example vehicle system 100 is shown, including an alignment verification module 104 for LiDAR-to-LiDAR and LiDAR-to-vehicle alignment. The operations performed by the alignment verification module 104 are described below with at least reference. Figure 2-8 Further description.

[0112] Vehicle system 100 may include an autonomous driving module 105, a body control module (BCM) 107, a telematics module 106, a propulsion control module 108, a power steering system 109, a braking system 111, a navigation system 112, an infotainment system 114, an air conditioning system 116, and other vehicle systems and modules 118. The autonomous driving module 105 may include an alignment verification module 104, and may also include a sensor alignment and fusion module 113, a perception and localization module 115, and a path planning module 117. The operation of these modules is further described below.

[0113] Modules and systems 104-108, 112-115, and 118 can communicate with each other via a Controller Area Network (CAN) bus, Ethernet, Local Area Network (LIN) bus, another bus or communication network, and / or wirelessly. Object 119 may refer to and / or include a CAN bus, Ethernet, LIN bus, and / or other bus and / or communication network. This communication may include other systems, such as systems 109, 111, and 116. A power supply 120 may be included to power the autonomous driving module 105 and other systems, modules, devices, and / or components. Power supply 120 may include an accessory power module, one or more batteries, a generator, and / or other power sources.

[0114] The telematics module 106 may include a transceiver 130 and a telematics control module 132. The propulsion control module 108 controls the operation of the propulsion system 136, which may include an engine system 138 and / or one or more electric motors 140. The engine system 138 may include an internal combustion engine 141, a starter motor 142 (or starter), a fuel system 144, an ignition system 146, and a throttle system 148.

[0115] The autonomous driving module 105 can control modules and systems 106, 108, 109, 111, 112, 114, 116, 118, and other devices and systems based on data from sensors 160. Other devices and systems may include door and window actuators 162, interior lights 164, exterior lights 166, a trunk motor and lock 168, a seat position motor 170, a seat temperature control system 172, and a vehicle reflector motor 174. Sensors 160 may include temperature sensors, pressure sensors, flow sensors, position sensors, etc. Sensors 160 may include a LiDAR sensor 180, a RADAR sensor 182, a camera 184, an inertial measurement sensor 186, a map sensor 188, a GPS system 190, and / or other environmental and feature detection sensors and / or systems. The GPS system 190 may be implemented as part of a navigation system 112.

[0116] Figure 2 An example autonomous driving module 200 based on LIDAR sensor alignment operations performed by alignment verification module 104 is shown. Autonomous driving module 200 can replace... Figure 1 One or more autonomous driving modules 105, and may include a sensor alignment and fusion module 113, an alignment verification module 104, a perception and localization module 115, and a path planning module 117.

[0117] Sensor alignment and fusion module 113 can perform sensor alignment and fusion operations based on the output of sensor 160 (e.g., sensors 180, 182, 184, 186, 188, 190), as described above. Alignment verification module 104 determines whether the LIDAR sensors are aligned, meaning that the differences in information provided by the LIDAR sensors for the same one or more features and / or objects are within predetermined ranges from each other. Alignment verification module 104 determines the differences in the six degrees of freedom of the LIDAR sensors, including roll, pitch, yaw, x, y, and z differences, and determines whether the LIDAR sensors are aligned based on this information. If they are not aligned, one or more of the LIDAR sensors can be recalibrated. In one embodiment, when one LIDAR sensor is determined to be misaligned, the misaligned LIDAR sensor is recalibrated. In another embodiment, when one LIDAR sensor is determined to be misaligned, two or more LIDAR sensors, including the misaligned LIDAR sensor, are recalibrated. In another embodiment, the misaligned LIDAR sensor is isolated and no longer used, and an indication signal is generated indicating that the LIDAR sensor requires maintenance. Data from the misaligned sensor may be discarded. Additional data may be collected after the misaligned LIDAR sensor has been recalibrated and / or maintained.

[0118] After verification, the perception and localization module 115 can perform perception and localization operations based on the collected and aggregated sensor data to determine the corresponding master vehicle (e.g., Figure 1 The path planning module 117 can determine the vehicle's path based on the output of the perception and localization module 115. The path planning module 117 can control the vehicle's operation based on the determined path, including controlling the operation of the power steering system, propulsion control module, and braking system via actuator 204. Actuator 204 can include motors, drives, valves, switches, etc.

[0119] The autonomous driving module 105 may include a memory 192, which may store sensor data, historical data, and other data and information mentioned herein.

[0120] Figure 3 An alignment system 300 is shown, comprising a LiDAR alignment module 302 and a LiDAR alignment verification module 304. The alignment system 300 can represent the operations performed in relation to the LiDAR sensor 180. Figure 1-2This is part of the sensor alignment and fusion module 113 and the alignment verification module 104. The LIDAR alignment module 302 can collect sensor outputs and / or stored sensor data, such as data collected from and / or generated based on the outputs of the LIDAR sensor 180, the inertial measurement sensor 182, and the GPS system 190. Sensor data 306 is shown.

[0121] The LIDAR alignment module 302 can independently measure the output of the LIDAR sensors using corresponding coordinate systems, one coordinate system for one LIDAR sensor. Each LIDAR sensor generates point cloud data, where each point includes x, y, and z coordinates. The x-coordinate can refer to the horizontal direction. The y-coordinate can refer to the front-back or longitudinal direction, and the z-coordinate can refer to the vertical direction. In other coordinate systems, x, y, and z can be defined differently. Each LIDAR sensor can be mounted in a corresponding position on the vehicle and has a corresponding orientation. Because the LIDAR sensors are in different positions and can be in different orientations, the LIDAR sensors can report different values ​​for the same detected object. The LIDAR alignment module 302 performs a LIDAR-to-vehicle transformation for each LIDAR sensor and provides six alignment parameters 308 for one or more LIDAR sensors, representing roll, pitch, yaw, x, y, and z transformation values. The parameters can be represented as matrices (e.g., 4×4 matrices).

[0122] LIDAR alignment verification module 304 receives the output of LIDAR alignment module 302, which may include six parameter values ​​and / or a representative value matrix for each LIDAR sensor. Based on the received values, LIDAR alignment verification module 304 performs online verification of LIDAR-to-LIDAR alignment and provides diagnostic result data 310 indicating whether the alignment meets specific conditions. Online verification refers to verification performed within the vehicle during runtime, rather than verification performed in a back office (or central office) or later in the vehicle (referred to as offline). This can be performed for partially or fully autonomous vehicles. The distinction between online and offline is primarily related to runtime. If the verification is performed during the vehicle's regular driving tasks, it is considered online. Therefore, online verification can be performed in a back office as long as it is during runtime. Vehicles can perform verification processes outside of their regular driving tasks, such as at a dealership where the vehicle is being serviced. In this case, even if the verification is performed within the vehicle, it is considered offline.

[0123] LiDAR alignment verification includes: monitoring the enabling conditions for initiating an online verification process based on vehicle dynamics and localization, as well as perception results; clustering LiDAR point clouds into a vehicle coordinate system; determining pitch, roll, and yaw errors; determining rotation and translation errors; and / or making decisions based on the integration of results from multiple different methods. LiDAR point cloud clustering may include mapping each point to a vehicle coordinate system using a LiDAR-to-vehicle transformation when the corresponding primary vehicle is in a static state (i.e., not moving). For example, the vehicle coordinate system may have the centroid or other points on the vehicle as reference points. LiDAR point cloud clustering may also include, when the primary vehicle is in a dynamic state (i.e., moving), first clustering multiple LiDAR coordinate systems using point registration, followed by applying the LiDAR-to-vehicle transformation.

[0124] The first method can be used to determine pitch and roll difference based on data received from two LiDAR sensors. As described below, pitch and roll difference can be determined based on ground fitting and selection. The first method may include using spatial filters, slope-based algorithms, morphological transformations, etc. Spatial filters use a three-dimensional (3D) region in space to select points within that region. For example, a spatial filter can be defined as having a range... ,in This is a predetermined value (or threshold). A point (x, y, z) is selected by the spatial filter if it satisfies a predetermined condition within that region. Slope-based algorithms detect points on the ground (e.g., the ground surface or road surface) by checking whether the slope between a point and its neighbors satisfies a predetermined condition (e.g., the slope is less than a predetermined threshold). Morphological transformation refers to the theories and techniques used to analyze and manipulate geometric structures. These theories and techniques can be based on set theory, lattice theory, topology, and random functions.

[0125] The determination of the yaw difference between two LiDARs can be based on target detection and can include the use of spatial filters, intensity filters, clustering, and / or point cloud registration. Intensity filters can include intensity ranges, defined as points whose intensity values ​​fall within that range. For example, an intensity filter can be defined as... , where i is the intensity and It is a predetermined threshold. Clustering is the task of grouping a set of objects such that objects in the same group (called a cluster) are more similar to each other (in some sense) than objects in other groups (clusters). Point cloud registration is the process of finding spatial transformations (e.g., scaling, rotation, and translation) that align two point clouds.

[0126] Point cloud registration determines the rotational and translational difference between two LiDAR sensors using methods such as Iterative Closest Point (ICP), Generalized ICP, and Normal Distribution Transform (NDT). ICP is an algorithm for minimizing the difference between two point clouds. ICP can include calculating the correspondence between two scans and calculating a transformation that minimizes the distance between corresponding points. Generalized ICP is similar to ICP and can include a minimization operation that appends a probability model to ICP. NDT is a representation of a distance scan that can be used when matching 2D distance scans. Similar to an occupancy grid, the 2D plane is subdivided into cells. A normal distribution is assigned to each cell, which locally models the probability of the measurement point. The result of the transformation is a piecewise continuous and differentiable probability density, which can be used to match another scan using Newton's algorithm.

[0127] Modules 302 and 304 can perform LIDAR-to-LIDAR alignment and verification, which includes transforming point data from one LIDAR sensor to the coordinate system of another LIDAR sensor and verifying the transformation. In addition to LIDAR-to-vehicle alignment, this can also be performed for redundancy and verification purposes, and to account for errors in performing these transformations. LIDAR-to-vehicle transformations can be performed on the dataset from each LIDAR sensor. Generally, if the difference between the LIDAR-to-LIDAR and LIDAR-to-vehicle transformations is small (e.g., below a predetermined threshold), the LIDAR sensor is aligned. If not, one or more of the LIDAR sensors may require recalibration and / or repair. The results of performing LIDAR-to-LIDAR transformations can be used to infer the LIDAR-to-vehicle transformation. A LIDAR-to-LIDAR transformation can be calculated using two LIDAR-to-vehicle transformations. Verifying the accuracy of the LIDAR-to-LIDAR transformations infers the accuracy of each LIDAR-to-vehicle transformation. When validating LiDAR-to-LiDAR transformations and / or LiDAR-to-vehicle transformations, at least two LiDAR sensors are used.

[0128] Alignment verification

[0129] Figure 4 An example method for verifying LiDAR sensor alignment is shown, which can be performed iteratively. This method can be... Figure 1-3 The alignment verification modules 104 and 304 execute this process. The method can begin at 402, which includes checking enable condition parameters to determine whether to begin the verification process. The enable conditions are related to vehicle dynamics, vehicle position, and the detection and localization of surrounding objects.

[0130] Check enable conditions

[0131] At 404, the alignment verification module determines whether the enable condition is met. If so, operation 406 is executed. Operations 402 and 404 may include information about... Figure 5 The method describes the operation.

[0132] Figure 5 The method may begin at 502, which includes measuring, reading, and / or determining the vehicle's speed and acceleration. At 504, the alignment verification module determines whether the vehicle is not moving (or is static). If so, operation 506 is performed; otherwise, operation 516 is performed.

[0133] At 506, the alignment verification module collects vehicle GPS data. At 508, the alignment verification module determines whether the vehicle is at the predetermined location based on the vehicle GPS data. If yes, operation 514 is performed; otherwise, operation 510 is performed. At 510, the alignment verification module detects the calibration target. This may include using sensing operations and / or performing... Figure 6 The target detection method will be further described below with reference to operations 408 and 412.

[0134] At 512, the alignment verification module determines whether the calibration target exists. If it does, operation 514 is executed; otherwise, operation 518 is executed. At 514, the alignment verification module determines that the enable condition is met and continues with the target to operation 408.

[0135] At 516, the alignment verification module determines whether the dynamic parameters are within a predetermined range. Dynamic parameters may include vehicle speed, acceleration, and yaw rate conditions. For example, acceleration, speed, and yaw rate may exceed certain predetermined thresholds. If they are not within the predetermined range, operation 518 is performed; otherwise, operation 522 is performed.

[0136] At step 518, the alignment verification module determines whether the number of feature points is greater than or equal to a predetermined number. If so, operation 520 is executed; otherwise, operation 522 is executed. At step 520, the alignment verification module determines that the enable condition is met and continues point registration to operation 408. At step 522, the alignment verification module determines that the enable condition is not met and does not continue. Figure 4 The method can end after operation 522.

[0137] LIDAR point clustering

[0138] Refer again Figure 4At 406, the alignment verification module aggregates LIDAR points in the vehicle coordinate system. Operation 406 may include static aggregation performed on points collected when the vehicle is not moving, and may additionally include dynamic aggregation performed on points collected when the vehicle is moving. Static aggregation includes performing a LIDAR-to-vehicle transformation for each LIDAR sensor. As an example, two LIDAR sensors A and B can be used. Points from LIDAR A (denoted as...) ) and rotation for LiDAR A R A Peaceful relocation t A The calibration values ​​are used to aggregate and / or map points from LiDAR A to the vehicle coordinate system. The resulting points are represented as... This can be done using Equation 1, where i is the number of points.

[0139] (1).

[0140] Points from LiDAR B (represented as) ) and for rotation R B Peaceful relocation t B The LiDAR B calibration values ​​are used to aggregate and / or map points from LiDAR B to the vehicle coordinate system. The resulting points are represented as... This can be accomplished using Equation 2. In this example, the LIDAR-to-LIDAR comparison is not a direct comparison, but rather an indirect comparison via a LIDAR-to-vehicle transformation.

[0141] (2).

[0142] Dynamic aggregation includes time We obtain a point from each of LiDAR A and B, denoted as LiDAR j, where j is either A or B, and perform point registration. We perform point registration to map the point at time k' in the coordinate system of LiDAR j at time k. This can be done using Equation 3, such that... and overlapping.

[0143] (3).

[0144] The registration method performed can include ICP, generalized ICP, edge and surface feature point registration, NDT, etc. Then, Equation 4 is used to aggregate the mapped points for LIDAR j (A or B).

[0145] (4).

[0146] Then, generated by performing LIDAR registration and The values ​​are used as input to perform the same aggregation as static aggregation, and are input into Equations 1 and 2 respectively to provide the resulting aggregation and mapping points. , Aggregation and mapping points , It can be provided as input for operating 408, 412 and 416.

[0147] Because the two LiDAR sensors detect a position differently, pitch and roll differences, yaw differences, rotation differences, and / or translational differences can be determined even when mapped to the same coordinate system. These differences are determined in operations 410, 414, and 416 below. To determine pitch and roll differences, two corresponding ground positions can be selected, one on the side of the vehicle and one in front of the vehicle. The vertical height difference when detecting the ground position on the side of the vehicle can be used to determine the roll difference. The vertical height difference when detecting the ground position in front of the vehicle can be used to determine the pitch difference. For yaw difference, a vertically extending object (e.g., a light pole) can be selected. The horizontal difference in the detected object's position can be used to determine the yaw difference.

[0148] Ground Fitting and Selection

[0149] In operation 408, the alignment verification module performs ground fitting and selection. Operations 408 and 410 can be referred to as the pitch and roll difference method. Operation 408 may include... Figure 6 The operation can begin at 602. At 602, the alignment verification module reads the aggregated LIDAR points, as determined above, from the memory. At 604, the alignment verification module determines whether the vehicle is at the predetermined known location. If not, operation 606 is performed; otherwise, operation 608 is performed.

[0150] In step 606, ground fitting is performed. In step 606A, the alignment verification module projects points onto a grid, where each row of the grid represents a beam and each column represents an azimuth. In step 606B, the alignment verification module calculates the slope of each point relative to its corresponding point in the previous row. In step 606C, the alignment verification module uses z < θ to determine the ground point in the lowest row of the grid (or coordinate system), where θ is a predetermined threshold. A point is considered a ground point in the lowest row if its z value is less than the predetermined threshold.

[0151] In step 606D, the alignment verification module determines whether adjacent points and ground points have been marked. If so, operation 608 can be performed; otherwise, operation 606E can be performed. In step 606E, the alignment verification module selects ground point P. i Unmarked point P jIn the 606F, the alignment verification module determines P. j Is the slope close to P? i The slope. If yes, then perform operation 606G; otherwise, perform operation 606H. In 606G, the alignment verification module will P j Marked as a ground point. At 606H, the alignment verification module will... j Mark as a non-ground point. After operations 606G and 606H, perform operation 606D.

[0152] At 608, the alignment verification module selects candidate points based on spatial filters, where . Different pitch and roll values ​​can be used. In step 610, the alignment verification module uses principal component analysis to fit the plane to the selected ground points and removes points that are more than a predetermined distance from the plane.

[0153] In section 612, the alignment verification module selects points from LiDAR A and B that are associated with the same ground area using a method that selects points within a predetermined range of each other. This method may include the use of a kd-tree, a binary tree where each leaf node is a k-dimensional point. The kd-tree can be used to select points that are close in distance.

[0154] Δz and pitch and roll differences

[0155] Refer again Figure 4 The method, at 410, involves the alignment verification module calculating the z-difference (or Δz) between the output data points (referred to as "points") of the LiDAR sensors and determining the pitch and roll differences between the LiDAR sensors. To determine the pitch difference, the ground points selected above are obtained. , Use Equation 5 to determine the difference in z.

[0156] (5).

[0157] Use Equation 6 to determine the pitch difference.

[0158] (6)

[0159] Wherein, Pitch_diff: pitch difference.

[0160] To determine the roll difference, obtain the ground point selected above. , Use Equation 7 to determine the difference in z.

[0161] (7).

[0162] Use Equation 8 to determine the roll difference.

[0163] (8)

[0164] Roll_diff: Roll difference.

[0165] Target detection

[0166] At position 412, the alignment verification module detects the target (e.g., features and / or objects). Unlike large buildings, a target can refer to a small object that can be fully detected. Target points are feature points, but not all feature points are target points. Target detection can be implemented using spatial filters, where... ,distance Intensity filtering and clustering; and point cloud registration. If the vehicle's location is known (e.g., in a parking lot with known objects, such as a lamppost in a known location), spatial filtering can be used. If the vehicle's location is unknown, intensity filtering and clustering can be performed together with point cloud registration to select targets, and then the angular difference with the targets is calculated. The yaw difference is then determined based on the angular difference.

[0167] Yaw difference

[0168] In step 414, the alignment verification module determines the yaw difference between points on the LiDAR sensor. Operations 412 and 414 may refer to the yaw difference method. Figure 7 As shown, the yaw error can be determined using Equation 9-11, where, It is the angle between the reference line and the target as seen by LiDAR A. It is the angle between the reference line and the target as seen by LiDAR B, and the yaw difference is... Point 700 refers to the vehicle's reference point (e.g., center of gravity). Point 702 refers to the target as seen by LIDAR A, and point 704 refers to the target as seen by LIDAR B.

[0169] (9)

[0170] (10)

[0171] (11)

[0172] Where, Yaw_diff: yaw difference.

[0173] Point cloud registration

[0174] In step 416, the alignment verification module performs point cloud registration to determine translation and rotation information. Operation 416 can be referred to as the point cloud difference method and includes determining the rotation and translation differences. This is done using aggregated points from LiDAR A and B, or... The alignment and verification module performs point cloud registration, including determining the transformation. T B-A This satisfies equation 12, where and alignment.

[0175] (12).

[0176] The alignment verification module will transform T B-A The values ​​are converted into rotation angles and translation values, which are roll difference, pitch difference, yaw difference, x difference, y difference, and z difference values. The aforementioned technique involves moving a first point cloud to overlap with a second point cloud and determining how much the first point cloud has moved, represented by rotation and translation values. In one embodiment, the first point cloud is moved to overlap and match the second point cloud as closely as possible. The rotation and translation values ​​are then converted into six degrees of freedom. Pitch, roll, yaw, x, y, and z errors can be calculated and indicate the degree of misalignment between the LiDAR sensors.

[0177] From operations 410, 414, and 416, two sets of values ​​for the six degrees of freedom of the LIDAR sensor are determined. The first set comes from operations 410 and 414, and the second set comes from operation 416.

[0178] Integration of Results and Final Decision

[0179] In step 418, the alignment verification module integrates the results of the pitch and roll difference method, the yaw difference method, and the point cloud difference method in operations 410, 414, and 416. Operation 418 may include using results from the first method (or operations 408, 410, 412, 414) and / or results from the second method (operation 416). A first vector of the six differences can be obtained from the combined results of operations 410 and 414. A second vector of the six differences can be obtained from performing operation 416. The method number is indicated by variable i. The vector of the six parameter differences. As shown in Equation 13, similar vectors can be determined for each method performed.

[0180] (13)

[0181] The integration result of these methods can be determined using Equation 14, where, w i These are the calibrated weights applied to each vector. The same weight can be applied to every value of a vector, or different weights can be applied to each vector. Alternatively, different weights can be applied to each difference between each vector.

[0182] (14).

[0183] At 420, the alignment verification module performs the final decision, such as determining whether the alignment conditions are met. If the conditions are met, the LIDAR sensor is aligned. If the obtained vector... If the value is greater than a predetermined threshold of the predetermined vector, the alignment verification is considered to have failed. The predetermined vector may include six thresholds corresponding to six different parameters. When one or more of the thresholds are exceeded, the alignment verification can be considered to have failed. Then, one or more of the LIDAR sensors can be recalibrated and / or repaired and / or replaced to provide new calibration values, and the above process can be repeated with the new calibration values. For example, when one or more of the roll, pitch, and yaw differences are greater than or equal to 0.3°, the condition is not met, and one or more LIDAR sensors can be recalibrated. As another example, when the x, y, z difference is greater than or equal to 5.0 cm, the condition is not met, and one or more LIDAR sensors can be recalibrated. If the vector If the result value is less than or equal to the value of the predetermined threshold vector, then the alignment verification is considered to have passed.

[0184] When there are more than two LiDARs, a vector can be determined for each pair of LiDAR sensors. .if If a vector exceeds a predetermined threshold, one or more of the evaluated LIDAR sensor pairs can be identified as suspicious. Suspicious LIDARs can be recalibrated, or the intersection of suspicious LIDARs can be calibrated. For example, if the difference between LIDAR sensors A and B is large, the difference between LIDAR sensors A and C is large, and the difference between LIDAR sensors B and C is small, then LIDAR sensor A can be recalibrated, while LIDAR sensors B and C do not need to be recalibrated.

[0185] In one embodiment, alignment results for a first LiDAR sensor to a vehicle transformation and alignment results for a second LiDAR sensor to a vehicle transformation are provided. Based on these results, six degrees of freedom differences are determined for the two LiDAR sensors.

[0186] In another embodiment, when the alignment result of the first LIDAR sensor is known to be accurate and the result from the first LIDAR sensor to vehicle transformation (or alignment) is known, it can be determined whether the second LIDAR to vehicle transformation (or alignment) is accurate. The first LIDAR to vehicle transformation (or alignment) can provide a first six-value difference, and the first LIDAR to second LIDAR transformation (or alignment) can provide a second six-value difference. To obtain accurate values ​​(or ground reality) of the six values ​​of the second LIDAR to vehicle transformation (or alignment), the first six-value difference can be added to the second six-value difference. The difference between this ground reality of the six values ​​of the second LIDAR to vehicle transformation (or alignment) and the existing calibration result of the second LIDAR to vehicle transformation (or alignment) is generated based on the six-value difference. The synthesized six-value difference can then be compared with six thresholds to determine whether the second LIDAR to vehicle transformation (or alignment) is accurate.

[0187] If alignment verification is deemed to have failed, operation 422 can be performed; otherwise, operation 402 can be performed. In 422, the alignment verification module can trigger a LIDAR alignment process to recalibrate identified LIDAR sensors that have been identified as failing the alignment verification process. Suspected LIDAR sensors may not be used and / or relied upon until they are recalibrated and / or repaired.

[0188] The above operations are intended as illustrative examples. Depending on the application, operations may be performed sequentially, synchronously, simultaneously, continuously, during overlapping time periods, or in a different order. Furthermore, depending on the implementation method and / or the order of events, any operation may be omitted or skipped.

[0189] Table 1 below indicates the advantages of performing operations 408, 410, 412, 414 (method 1), 416 (method 2), and 418.

[0190] Compare Based on the goal Based on point registration Integration of the two methods Able to verify orientation difference yes yes yes Capable of verifying translation difference no yes yes Can be applied to any location no yes yes Can be applied during dynamic states no yes yes Computing resources Low high high change high Low Low

[0191] Table 1 - Execution Figure 4 Some of the advantages of this operation.

[0192] In one embodiment, the above examples include dynamic LiDAR-to-vehicle verification. Dynamic LiDAR-to-vehicle verification includes point registration in dynamic states, motion distortion removal of LiDAR data, and inertial navigation system (INS) error compensation. These examples may include comparisons with references, including providing LiDAR alignment results and determining references such as ground reality, historical results, and fleet statistics. "Ground reality" refers to known correct points and / or information that can then be used as a reference for generating information and / or making decisions.

[0193] Figure 8 An object detection method is shown, which can be integrated into the above method to detect objects. This method can begin at 802, where... Figure 1 The autonomous driving module 105 cycles through each scan of data collected using LIDAR sensors for each type of object being considered.

[0194] In 802A, the autonomous driving module 105 reads the data collected via the LiDAR sensor and aggregates the data in the world coordinate system. In 802B, the autonomous driving module 105 performs LiDAR data segmentation. In 802C, the autonomous driving module 105 filters out low-intensity points or points with intensity levels less than a predetermined threshold (e.g., T1).

[0195] In 802D, the autonomous driving module 105 filters out low and high range (distance) data or data outside the predetermined range T2-T3. In 802E, the autonomous driving module 105 filters data location (mean drift clustering) and spatial dimension (range in x, y, z).

[0196] In 802F, the autonomous driving module 105 eliminates potential objects with fewer than N1 data points, where N1 is a predetermined threshold number of points. In 802G, the autonomous driving module 105 detects one or more objects for each scan.

[0197] At 804, the autonomous driving module 105 determines whether the considered object exists in at least N2 consecutive scans, where N2 is a predetermined threshold number of scans. If yes, operation 806 is executed; otherwise, operation 808 is executed. At 806, the autonomous driving module 105 indicates that an object has been detected. At 808, the autonomous driving module 105 indicates that no object has been detected.

[0198] The values ​​T1, T2, T3, N1, N2, and the ranges of x, y, z are calibrable and can be specific to each object considered. Objects with truly known locations can also be detected, for example, using high-definition maps, vehicle-to-vehicle communication, and / or vehicle-to-infrastructure communication.

[0199] The foregoing description is merely illustrative in nature and is in no way intended to limit this disclosure, its application, or its use. The broad teachings of this disclosure can be implemented in many forms. Therefore, although this disclosure includes specific examples, its true scope should not be so limited, as other modifications will become apparent upon examination of the drawings, specification, and appended claims. It should be understood that one or more steps in the method may be performed in a different order (or simultaneously) without altering the principles of this disclosure. Furthermore, although each embodiment is described above as having certain features, any one or more of those features described with respect to any embodiment of this disclosure may be implemented in any other embodiment and / or combined with features of any other embodiment, even if such combination is not explicitly described. In other words, the described embodiments are not mutually exclusive, and substitutions of one or more embodiments for each other remain within the scope of this disclosure.

[0200] Various terms are used to describe spatial and functional relationships between elements (e.g., modules, circuit elements, semiconductor layers, etc.), including “connected,” “joined,” “coupled,” “adjacent,” “right next to,” “on top of,” “above,” “below,” and “set.” Unless explicitly described as “direct,” when describing a relationship between first and second elements in the foregoing disclosure, the relationship can be a direct relationship in which no other intervening element exists between the first and second elements, or an indirect relationship (spatial or functional) in which one or more intervening elements exist between the first and second elements. As used herein, the phrase “at least one of A, B, and C” should be interpreted as meaning the use of the non-exclusive logic “OR” to represent the logic “A or B or C,” and should not be interpreted as meaning “at least one of A, at least one of B, and at least one of C.”

[0201] In a diagram, the direction of the arrows typically indicates the flow of information (such as data or instructions) of interest to the diagram. For example, when components A and B exchange various types of information, but the information transmitted from component A to component B is relevant to the diagram, the arrow can point from component A to component B. This unidirectional arrow does not mean that no other information is transmitted from component B to component A. Furthermore, for information sent from component A to component B, component B can send a request for or confirmation of receipt of the information to component A.

[0202] In this application, including the following definitions, the term "module" or "controller" may be replaced by the term "circuit". The term "module" or "controller" may refer to, be part of, or include: application-specific integrated circuit (ASIC); digital, analog, or mixed-signal analog / digital discrete circuit; digital, analog, or mixed-signal analog / digital integrated circuit; combinational logic circuit; field-programmable gate array (FPGA); processor circuitry (shared, dedicated, or grouped) that executes code; memory circuitry (shared, dedicated, or grouped) that stores code executed by the processor circuitry; other suitable hardware components that provide the functions described above; or some or all of the above combinations, such as in a system-on-a-chip.

[0203] A module or controller may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces that connect to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module disclosed herein may be distributed across multiple modules connected via the interface circuits. For example, multiple modules may allow for load balancing. In another example, a server (also referred to as a remote or cloud) module may perform some functions on behalf of a client module.

[0204] As described above, the term "code" can include software, firmware, and / or microcode, and can refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuitry" covers a single processor circuitry that executes some or all of the code from multiple modules. The term "group processor circuitry" covers a processor circuitry that, in conjunction with additional processor circuitry, executes some or all of the code from one or more modules. The reference to "multiple processor circuitry" covers multiple processor circuitry on a discrete die, multiple processor circuitry on a single die, multiple cores of a single processor circuitry, multiple threads of a single processor circuitry, or a combination of the above. The term "shared memory circuitry" covers a single memory circuitry that stores some or all of the code from multiple modules. The term "group memory circuitry" covers a memory circuitry that, in conjunction with additional memory, stores some or all of the code from one or more modules.

[0205] The term "memory circuit" is a subset of the term "computer-readable medium." As used herein, the term "computer-readable medium" does not cover transient electrical or electromagnetic signals propagating through a medium (such as on a carrier wave); therefore, the term "computer-readable medium" can be considered tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or masked read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0206] The apparatus and methods described in this application can be implemented, in part or in whole, by a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions contained in a computer program. The function blocks, flowchart components, and other elements described above serve as software specifications that can be translated into a computer program through the routine work of a skilled technician or programmer.

[0207] A computer program includes processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. A computer program may also include or depend on stored data. A computer program may encompass a basic input / output system (BIOS) that interacts with the hardware of a special-purpose computer, device drivers that interact with specific devices of a special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0208] Computer programs may include: (i) descriptive text to be parsed, such as HTML (Hypertext Markup Language), XML (Extensible Markup Language), or JSON (JavaScript Object Notation); (ii) assembly code; (iii) object code generated from source code by a compiler; (iv) source code executed by an interpreter; (v) source code compiled and executed by a just-in-time compiler; and so on. As an example only, source code may be written using syntax from the following languages: C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java. ® Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5 (Hypertext Markup Language, Fifth Revision), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash ® Visual Basic® Lua, MATLAB, SIMULINK, and Python ® .

Claims

1. A LIDAR-to-LIDAR alignment system, comprising: The memory is configured to store (i) a first data point provided based on the output of a first LIDAR sensor, and (ii) a second data point provided based on the output of a second LIDAR sensor; and An autonomous driving module is configured to perform a verification process to determine whether the alignment of the first LIDAR sensor and the second LIDAR sensor meets one or more alignment conditions, the verification process including... The first data points and the second data points are aggregated in the vehicle's vehicle coordinate system to provide aggregated LIDAR points, wherein the aggregation of the first data points and the second data points includes: Perform point registration to map the first data point at the first time point into the coordinate system of the first LIDAR sensor at the second time point; First data points mapped from the first LIDAR sensor are aggregated to provide a first aggregation point; Perform point registration to map the second data point at the first time point into the coordinate system of the second LIDAR sensor at the second time point; and The mapped second data points from the second LIDAR sensor are aggregated to provide a second aggregation point. Based on clustered LiDAR points, at least one of the following: Perform a first method including determining the pitch and roll difference between the first LIDAR sensor and the second LIDAR sensor. Perform a second method including determining the yaw difference between the first LIDAR sensor and the second LIDAR sensor, or Point cloud registration is performed to determine the rotation and translation difference between the first and second LiDAR sensors. Based on the result of at least one of the first method, the second method, or the point cloud registration, determine whether one or more alignment conditions are met, and In response to failure to meet one or more alignment conditions, at least one of the first LIDAR sensor or the second LIDAR sensor is recalibrated.

2. The LIDAR-to-LIDAR alignment system according to claim 1, wherein, The autonomous driving module is also configured to: Determine whether multiple enabling conditions are met, wherein the enabling conditions include two or more of the following: Determine whether the vehicle is moving. Determine whether the vehicle is in a known location. Determine whether the vehicle's speed is greater than a predetermined speed. Determine whether the vehicle's acceleration is greater than a predetermined acceleration, or Determine if the yaw rate is greater than the predetermined yaw rate; as well as In response to the fulfillment of the plurality of enabling conditions, the first data point and the second data point are aggregated.

3. The LIDAR-to-LIDAR alignment system according to claim 1, wherein, The aggregation of the first data point and the second data point also includes: The first aggregation point is mapped to the vehicle coordinate system based on the rotation and translation calibration values ​​of the first LIDAR sensor; and The second aggregation point is mapped to the vehicle coordinate system based on the rotation calibration value and translation calibration value of the second LIDAR sensor.

4. The LIDAR-to-LIDAR alignment system according to claim 1, wherein, The aggregation of the first data point and the second data point includes: The first data points are mapped to the vehicle coordinate system based on the rotation and translation calibration values ​​from the first LIDAR sensor; and The second data point is mapped to the vehicle coordinate system based on the rotation calibration value and translation calibration value of the second LIDAR sensor.

5. The LIDAR-to-LIDAR alignment system according to claim 1, wherein, The autonomous driving module is configured to: Based on the clustered LIDAR points, perform ground fitting and selection to provide a first selected point for a first LIDAR sensor in the ground region and a second selected point for a second LIDAR sensor in the ground region; as well as The first method is executed based on the first selection point and the second selection point.

6. The LIDAR-to-LIDAR alignment system according to claim 1, wherein, The autonomous driving module is configured to execute the first method and, based on the result of the first method, determine whether one or more alignment conditions are met.

7. The LIDAR-to-LIDAR alignment system according to claim 1, wherein, The autonomous driving module is configured to execute the second method and, based on the result of the second method, determine whether one or more alignment conditions are met.

8. The LIDAR-to-LIDAR alignment system according to claim 1, wherein, The autonomous driving module is configured to perform the point cloud registration and, based on the result of the point cloud registration, determine whether one or more alignment conditions are met.

9. The LIDAR-to-LIDAR alignment system according to claim 1, wherein, The autonomous driving module is configured to execute the first method, the second method, and the point cloud registration, and based on the results of the first method, the second method, and the point cloud registration, determine whether one or more alignment conditions are met.

10. The LIDAR-to-LIDAR alignment system according to claim 1, wherein, The autonomous driving module is configured to: Integrating results from one or more of the first method, the second method, and point cloud registration, including determining a weighted sum of the difference vectors for the six degrees of freedom; and In response to one or more of the differences exceeding a predetermined threshold, at least one of the following is performed: (i) one or more of the first LIDAR sensor and the second LIDAR sensor is identified as suspicious, or (ii) one or more of the first LIDAR sensor and the second LIDAR sensor are recalibrated.

11. The LIDAR-to-LIDAR alignment system according to claim 10, wherein, The autonomous driving module is configured to determine whether the transformation from the second LiDAR sensor coordinate system to the vehicle coordinate system is accurate based on the knowledge that the transformation from the first LiDAR sensor coordinate system to the vehicle coordinate system is accurate and the weighted sum of the difference vectors.

12. The LIDAR-to-LIDAR alignment system according to claim 1, wherein, The autonomous driving module is configured to calculate the six-degree-of-freedom difference between the first LIDAR sensor and the second LIDAR sensor based on the alignment results of the first LIDAR sensor and the second LIDAR sensor.

13. A LiDAR-to-LiDAR alignment verification method, comprising: Aggregating first and second data points in the vehicle's vehicle coordinate system to provide aggregated LIDAR points, wherein the first data points are provided based on the output of a first LIDAR sensor, and the second data points are provided based on the output of a second LIDAR sensor, wherein the aggregation of the first and second data points includes: Perform point registration to map the first data point at the first time point into the coordinate system of the first LIDAR sensor at the second time point; First data points mapped from the first LIDAR sensor are aggregated to provide a first aggregation point; Perform point registration to map the second data point at the first time point into the coordinate system of the second LIDAR sensor at the second time point; Aggregate the second data points mapped from the second LIDAR sensor to provide a second aggregation point; The first aggregation point is mapped to the vehicle coordinate system based on the rotation and translation calibration values ​​from the first LIDAR sensor; and The second aggregation point is mapped to the vehicle coordinate system based on the rotation calibration value and translation calibration value of the second LiDAR sensor; Based on clustered LiDAR points, at least one of the following: Perform a first method including determining the pitch and roll difference between the first LIDAR sensor and the second LIDAR sensor. Perform a second method including determining the yaw difference between the first LIDAR sensor and the second LIDAR sensor, or Perform point cloud registration to determine the rotation and translation difference between the first and second LiDAR sensors; Based on the result of the first method, the second method, or at least one of the point cloud registration methods, determine whether one or more alignment conditions are met; and In response to failure to meet one or more alignment conditions, at least one of the first LIDAR sensor or the second LIDAR sensor is recalibrated.

14. The method of claim 13, further comprising: Determine whether multiple enabling conditions are met, including two or more of the following: Determine whether the vehicle is moving. Determine whether the vehicle is in a known location. Determine whether the vehicle's speed is greater than a predetermined speed. Determine whether the vehicle's acceleration is greater than a predetermined acceleration, or Determine if the yaw rate is greater than the predetermined yaw rate; as well as In response to the fulfillment of the plurality of enabling conditions, the first data point and the second data point are aggregated.

15. The method of claim 13, further comprising: Based on the clustered LIDAR points, perform ground fitting and selection to provide a first selected point for a first LIDAR sensor in the ground region and a second selected point for a second LIDAR sensor in the ground region; as well as The first method is executed based on the first selection point and the second selection point.

16. The method of claim 13, further comprising performing the first method and the second method, and determining, based on the results of the first method and the second method, whether one or more alignment conditions are satisfied.

17. The method of claim 13, further comprising performing the point cloud registration and determining, based on the result of the point cloud registration, whether one or more alignment conditions are satisfied.

18. The method of claim 13, further comprising performing the first method, the second method, and the point cloud registration, and determining, based on the result of the first method, the second method, and the point cloud registration, whether one or more alignment conditions are satisfied.

Images