Cooperative Estimation and Correction of LIDAR Line-of-Sight Alignment Error and Host Vehicle Positioning Error

Abstract

A LIDAR-to-vehicle alignment system includes a memory and an autonomous driving module. The memory stores data points provided based on the output of a LIDAR sensor and a GPS position. The autonomous driving module performs an alignment process that includes performing feature extraction on the data points to detect one or more features of one or more predetermined types of objects having one or more predetermined characteristics. It is determined that the features correspond to one or more targets because the features have the predetermined characteristics. One or more GPS positions are of the targets. The alignment process further includes: determining a ground truth position of the feature; correcting the GPS position based on the ground truth position; calculating a LIDAR-to-vehicle transformation based on the corrected GPS position; and determining, based on the result of the alignment process, whether one or more alignment conditions are satisfied.

Description

BACKGROUND OF THE INVENTION

[0001] The information provided in this section is for the purpose of generally presenting the background of the present disclosure. To the extent that the work of the currently named inventors described in this section and aspects of this specification that may not have been prior art at the time of filing are not expressly or impliedly considered prior art to the present disclosure.

[0002] The present disclosure relates to vehicle object detection systems and, more particularly, to vehicle light detection and ranging (LIDAR) systems.

[0003] Vehicles can include various sensors for detecting the surrounding environment and objects in that environment. The sensors can include cameras, radio detection and ranging (RADAR) sensors, LIDAR sensors, and the like. A vehicle controller can perform various operations in response to the detected surrounding environment. The operations can include performing partial and / or fully autonomous vehicle operations, collision avoidance operations, and information reporting operations. The accuracy of the operations performed can be based on the accuracy of the data collected from the sensors. SUMMARY OF THE INVENTION

[0004] A LIDAR-to-vehicle alignment system is provided that includes a memory and an autonomous driving module. The memory is configured to store data points provided based on the output of a LIDAR sensor and a global positioning system location. The autonomous driving module is configured to perform an alignment process that includes: obtaining the data points; performing feature extraction on the data points to detect one or more features of one or more predetermined types of objects having one or more predetermined characteristics, wherein the one or more features are determined to correspond to one or more targets because the one or more features have the one or more predetermined characteristics and wherein one or more of the global positioning system locations are of the one or more targets; determining a ground truth location of the one or more features; correcting the one or more global positioning system locations based on the ground truth location; calculating a LIDAR-to-vehicle transformation based on the corrected one or more global positioning system locations; determining whether one or more alignment conditions are met based on the results of the alignment process; and in response to the LIDAR-to-vehicle transformation not meeting the one or more alignment conditions, recalibrating at least one of the LIDAR-to-vehicle transformations or recalibrating the LIDAR sensor.

[0005] Among other features, the autonomous driving module is configured to detect at least one of the following while performing feature extraction: (i) a first object of a first predetermined type, (ii) a second object of a second predetermined type, or (ii) a third object of a third predetermined type. The first predetermined type is a traffic sign. The second predetermined type is a light pole. The third predetermined type is a building.

[0006] Among other features, the autonomous driving module is configured to detect the edges or planar surfaces of a third object while performing feature extraction.

[0007] Among other features, the autonomous driving module is configured to operate in an offline mode while performing an alignment process.

[0008] Among other features, the autonomous driving module is configured to operate in an online mode while performing an alignment process.

[0009] Among other features, when performing feature extraction, the autonomous driving module is configured to: transform data from the LIDAR sensor to the vehicle coordinate system and then to the world coordinate system; and aggregate the resulting world coordinate system data to provide data points.

[0010] Among other features, when determining the ground truth position, the autonomous driving module is configured to: assign weights to the data points based on the vehicle speed, the type of acceleration actuator, and the global positioning system signal strength to indicate the confidence level in the data points; remove data points having weight values less than a predetermined weight from the data points; and determine a model of the feature corresponding to the remaining data points in the data points to generate ground truth data.

[0011] Among other features, the model is a plane or a line.

[0012] Among other features, the ground truth data includes a model, eigenvectors, and mean vectors.

[0013] Among other features, the ground truth data is determined using principal component analysis.

[0014] Among other features, the LIDAR-to-vehicle alignment system is implemented at the vehicle. The memory stores inertial measurement data. The autonomous driving module is configured to, during the alignment process: determine the orientation of the vehicle based on the inertial measurement data; and correct the orientation based on the ground truth data.

[0015] Among other features, the autonomous driving module is configured to perform interpolation based on a previously determined corrected global positioning system position to correct one or more of the global positioning system positions.

[0016] Among other features, the autonomous driving module is configured to: use a ground truth model for traffic signs or light poles to correct the one or more global positioning system positions; project LIDAR points of the traffic signs or the light poles onto a plane or a line; calculate an average global positioning system offset for a plurality of timestamps; apply the average global positioning system offset to provide one or more corrected global positioning system positions in the global positioning system positions; and update the vehicle-to-world transformation based on the one or more corrected global positioning system positions in the global positioning system positions.

[0017] Among other features, the autonomous driving module is configured to: use ground truth point matching to correct one or more global positioning system positions and inertial measurement data, including running an iterative closest point algorithm to find a transformation between current data and ground truth data, calculating an average global positioning system offset and a vehicle orientation offset for a plurality of timestamps, and applying the average global positioning system offset and the vehicle orientation offset to generate the one or more corrected global positioning system positions in the global positioning system positions and a corrected vehicle orientation; and update the vehicle-to-world transformation based on the one or more corrected global positioning system positions in the global positioning system positions and the corrected inertial measurement data.

[0018] Among other features, a LIDAR-to-vehicle alignment process is provided, and the process includes: obtaining data points provided based on an output of a LIDAR sensor; performing feature extraction on the data points to detect one or more features of one or more predetermined types of objects having one or more predetermined characteristics, wherein the one or more features are determined to correspond to one or more targets because the one or more features have the one or more predetermined characteristics; determining a ground truth position of the one or more features; correcting one or more global positioning system positions of the one or more targets based on the ground truth position; calculating a LIDAR-to-vehicle transformation based on the corrected one or more global positioning system positions; determining whether one or more alignment conditions are satisfied based on a result of the alignment process; and in response to the LIDAR-to-vehicle transformation not satisfying the one or more alignment conditions, recalibrating at least one of the LIDAR-to-vehicle transformation or the LIDAR sensor.

[0019] Among other features, the LIDAR-to-vehicle alignment process further includes, when determining the ground truth position: assigning weights to data points based on vehicle speed, type of acceleration actuator, and global positioning system signal strength to indicate the confidence level in the data points; removing data points having weight values less than a predetermined weight; and using principal component analysis to determine a model of the features corresponding to the remaining data points in the data points to generate ground truth data, where the model is a plane or a line, and where the ground truth data includes the model, eigenvectors, and mean vectors.

[0020] Among other features, the LIDAR-to-vehicle alignment process further includes: determining the orientation of the vehicle based on inertial measurement data; and correcting the orientation based on the ground truth data.

[0021] Among other features, the one or more global positioning system positions are corrected by performing interpolation based on a previously determined corrected global positioning system position.

[0022] Among other features, the LIDAR-to-vehicle alignment process further includes: using a ground truth model for traffic signs or light poles to correct the one or more global positioning system positions; projecting LIDAR points for the traffic signs or the light poles onto a plane or a line; calculating an average global positioning system offset for a plurality of timestamps; applying the average global positioning system offset to provide the corrected one or more global positioning system positions; and updating the vehicle-to-world transformation based on the corrected one or more global positioning system positions.

[0023] Among other features, the LIDAR-to-vehicle alignment process further includes: using ground truth point matching to correct one or more global positioning system positions and inertial measurement data, including running an iterative closest point algorithm to find the transformation between current data and ground truth data, calculating an average global positioning system offset and a vehicle orientation offset for a plurality of timestamps, and applying the average global positioning system offset and the vehicle orientation offset to generate the corrected one or more global positioning system positions and corrected vehicle orientation; and updating the vehicle-to-world transformation based on the corrected one or more global positioning system positions and corrected inertial measurement data.

[0024] The present invention further includes the following solutions:

[0025] Solution 1. A LIDAR-to-vehicle alignment system, comprising:

[0026] a memory configured to store data points provided based on the output of a LIDAR sensor and global positioning system positions; and

[0027] an autonomous driving module configured to perform an alignment process, the alignment process including:

[0028] Obtain the data points,

[0029] Perform feature extraction on the data points to detect one or more features of one or more predetermined types of objects having one or more predetermined characteristics, wherein, since the one or more features have the one or more predetermined characteristics, the one or more features are determined to correspond to one or more targets, and

[0030] wherein one or more of the global positioning system positions are of one or more targets,

[0031] Determine the ground truth position of the one or more features,

[0032] Correct the one or more global positioning system positions in the global positioning system positions based on the ground truth position,

[0033] Calculate the LIDAR-to-vehicle transformation based on the corrected one or more global positioning system positions in the global positioning system positions,

[0034] Based on the result of the alignment process, determine whether one or more alignment conditions are satisfied, and

[0035] In response to the LIDAR-to-vehicle transformation not satisfying the one or more alignment conditions, recalibrate at least one of the LIDAR-to-vehicle transformations or recalibrate the LIDAR sensor.

[0036] Solution 2. The LIDAR-to-vehicle alignment system according to Solution 1, wherein:

[0037] The autonomous driving module is configured to detect at least one of (i) a first object of a first predetermined type, (ii) a second object of a second predetermined type, or (iii) a third object of a third predetermined type when performing feature extraction; and

[0038] The first predetermined type is a traffic sign;

[0039] The second predetermined type is a lamp post; and

[0040] The third predetermined type is a building.

[0041] Solution 3. The LIDAR-to-vehicle alignment system according to Solution 2, wherein the autonomous driving module is configured to detect an edge or a planar surface of the third object when performing feature extraction.

[0042] Solution 4. The LIDAR-to-vehicle alignment system according to Solution 1, wherein the autonomous driving module is configured to operate in an offline mode when performing the alignment process.

[0043] Solution 5. The LIDAR-to-vehicle alignment system according to Solution 1, wherein the autonomous driving module is configured to operate in an online mode when performing the alignment process.

[0044] Solution 6. The LIDAR-to-vehicle alignment system according to Solution 1, wherein the autonomous driving module is configured to, when performing feature extraction:

[0045] Convert the data from the LIDAR sensor to the vehicle coordinate system and then to the world coordinate system; and

[0046] Aggregate the resulting world coordinate system data to provide the data points.

[0047] Solution 7. The LIDAR-to-vehicle alignment system according to Solution 1, wherein the autonomous driving module is configured to, when determining the ground truth position:

[0048] Based on the vehicle speed, the type of acceleration actuator, and the global positioning system signal strength, assign weights to the data points to indicate the confidence level of the data points;

[0049] Remove the data points with weight values less than a predetermined weight from the data points; and

[0050] Determine a model corresponding to the features of the remaining data points among the data points to generate the ground truth data.

[0051] Solution 8. The LIDAR-to-vehicle alignment system according to Solution 7, wherein the model is a plane or a line.

[0052] Solution 9. The LIDAR-to-vehicle alignment system according to Solution 7, wherein the ground truth data includes the model, eigenvectors, and mean vectors.

[0053] Solution 10. The LIDAR-to-vehicle alignment system according to Solution 7, wherein principal component analysis is used to determine the ground truth data.

[0054] Solution 11. The LIDAR-to-vehicle alignment system according to Solution 1, wherein:

[0055] The LIDAR-to-vehicle alignment system is implemented at a vehicle;

[0056] The memory stores inertial measurement data; and

[0057] The autonomous driving module is configured to, during the alignment process,

[0058] determine the orientation of the vehicle based on the inertial measurement data, and

[0059] correct the orientation based on the ground truth data.

[0060] Scheme 12. The LIDAR-to-vehicle alignment system according to Scheme 1, wherein the autonomous driving module is configured to perform interpolation to correct one or more global positioning system positions in the global positioning system positions based on previously determined corrected global positioning system positions.

[0061] Scheme 13. The LIDAR-to-vehicle alignment system according to Scheme 1, wherein the autonomous driving module is configured to:

[0062] Use a ground truth model for traffic signs or light poles to correct the one or more global positioning system positions;

[0063] Project the LIDAR points of the traffic sign or the light pole onto a plane or a line;

[0064] Calculate the average global positioning system offset for multiple timestamps;

[0065] Apply the average global positioning system offset to provide the corrected one or more global positioning system positions in the global positioning system positions; and

[0066] Update the vehicle-to-world transformation based on the corrected one or more global positioning system positions in the global positioning system positions.

[0067] Scheme 14. The LIDAR-to-vehicle alignment system according to Scheme 1, wherein the autonomous driving module is configured to:

[0068] Use ground truth point matching to correct the one or more global positioning system positions and inertial measurement data, including

[0069] Running an iterative closest point algorithm to find the transformation between the current data and the ground truth data,

[0070] Calculating the average global positioning system offset and vehicle orientation offset for multiple timestamps, and

[0071] Applying the average global positioning system offset and the vehicle orientation offset to generate the corrected one or more global positioning system positions in the global positioning system positions and the corrected vehicle orientation; and

[0072] Update the vehicle-to-world transformation based on one or more corrected global positioning system (GPS) positions among the GPS positions and the corrected inertial measurement data.

[0073] Solution 15. A LIDAR-to-vehicle alignment process, comprising:

[0074] Obtain data points provided based on the output of a LIDAR sensor;

[0075] Perform feature extraction on the data points to detect one or more features of one or more predetermined types of objects having one or more predetermined characteristics, wherein, since the one or more features have the one or more predetermined characteristics, the one or more features are determined to correspond to one or more targets;

[0076] Determine the ground truth position of the one or more features;

[0077] Correct one or more GPS positions of the one or more targets based on the ground truth position;

[0078] Calculate the LIDAR-to-vehicle transformation based on the corrected one or more GPS positions;

[0079] Based on the result of the alignment process, determine whether one or more alignment conditions are met; and

[0080] In response to the LIDAR-to-vehicle transformation not meeting the one or more alignment conditions, recalibrate at least one of the LIDAR-to-vehicle transformation or recalibrate the LIDAR sensor.

[0081] Solution 16. The LIDAR-to-vehicle alignment process according to Solution 15, further comprising, when determining the ground truth position:

[0082] Based on the vehicle speed, the type of the acceleration actuator, and the GPS signal strength, assign weights to the data points to indicate the confidence level in the data points;

[0083] Remove data points having weight values less than a predetermined weight from the data points; and

[0084] Use principal component analysis to determine a model corresponding to the features of the remaining data points in the data points to generate the ground truth data, wherein the model is a plane or a line, and the ground truth data includes the model, eigenvectors, and mean vectors.

[0085] Solution 17. The LIDAR-to-vehicle alignment process according to Solution 15, further comprising:

[0086] Determine the orientation of the vehicle based on inertial measurement data; and

[0087] Correct the orientation based on the ground truth data.

[0088] Scheme 18. The LIDAR-to-vehicle alignment process according to Scheme 15, wherein the one or more global positioning system positions are corrected by performing interpolation based on a previously determined corrected global positioning system position.

[0089] Scheme 19. The LIDAR-to-vehicle alignment process according to Scheme 15, further comprising:

[0090] Use a ground truth model for traffic signs or light poles to correct the one or more global positioning system positions;

[0091] Project the LIDAR points of the traffic sign or the light pole onto a plane or a line;

[0092] Calculate the average global positioning system offset for multiple timestamps;

[0093] Apply the average global positioning system offset to provide the corrected one or more global positioning system positions; and

[0094] Update the vehicle-to-world transformation based on the corrected one or more global positioning system positions.

[0095] Scheme 20. The LIDAR-to-vehicle alignment process according to Scheme 15, further comprising:

[0096] Use ground truth point matching to correct the one or more global positioning system positions and inertial measurement data, including

[0097] Running an iterative closest point algorithm to find the transformation between the current data and the ground truth data,

[0098] Calculate the average global positioning system offset and vehicle orientation offset for multiple timestamps, and

[0099] Apply the average global positioning system offset and the vehicle orientation offset to generate the corrected one or more global positioning system positions and the corrected vehicle orientation; and

[0100] Update the vehicle-to-world transformation based on the corrected one or more global positioning system positions and the corrected inertial measurement data.

[0101] Other application areas of the present disclosure will become apparent according to the detailed description, claims, and drawings. The detailed description and specific examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Description of the Drawings

[0102] The present disclosure will become more fully understood from the following detailed description and the accompanying drawings, in which:

[0103] Figure 1 is a functional block diagram of an example vehicle system in accordance with the present disclosure that includes a sensor alignment and fusion module and a mapping and positioning module;

[0104] Figure 2 is a functional block diagram of an example alignment system in accordance with the present disclosure that includes an autonomous driving module that performs global positioning system (GPS), LIDAR, and vehicle positioning correction;

[0105] Figure 3 illustrates an example alignment method in accordance with the present disclosure that includes GPS, LIDAR, and vehicle positioning correction;

[0106] Figure 4 illustrates an example portion of an alignment method implemented when operating in an offline mode in accordance with the present disclosure Figure 3 thereof;

[0107] Figure 5 illustrates an example portion of an alignment method implemented when operating in an online mode with or without cloud-based network support in accordance with the present disclosure Figure 3 thereof;

[0108] Figure 6 illustrates an example feature data extraction method in accordance with the present disclosure;

[0109] Figure 7 illustrates an example ground truth data generation method in accordance with the present disclosure;

[0110] Figure 8 illustrates an example GPS and inertial measurement correction and LIDAR-to-vehicle alignment method in accordance with the present disclosure;

[0111] Figure 9 illustrates an example GPS correction method using a ground truth model in accordance with the present disclosure; and

[0112] Figure 10 illustrates an example GPS and inertial measurement correction method using ground truth point matching in accordance with the present disclosure.

[0113] In the drawings, reference numerals may be reused to identify similar and / or identical elements. Detailed Description

[0114] The autonomous driving module can perform sensor alignment and fusion operations, perception and positioning operations, as well as path planning and vehicle control operations. These operations can be performed based on data collected from various sensors such as LIDAR sensors, RADAR sensors, cameras, and inertial measurement sensors (or inertial measurement units), as well as data collected from the Global Positioning System (GPS). Sensor alignment and fusion can include aligning the coordinate system of each sensor with a reference coordinate system, such as the vehicle coordinate system. Fusion can refer to collecting and combining data from various sensors.

[0115] Perception refers to monitoring the vehicle's surrounding environment and detecting and identifying various features and / or objects in the surrounding environment. This can include determining various aspects of the features and objects. As used herein, the term "feature" refers to one or more detected points that can be reliably used to determine the position of an object. This is different from other detected data points that do not provide reliable information about the position of the object (e.g., points on the leaves or branches of a tree). The determined aspects can include object distance, position, size, shape, orientation, trajectory, etc. This can include determining the type of the detected object, e.g., whether the object is a traffic sign, vehicle, pole, pedestrian, ground, etc. Lane marking information can also be detected. A feature can refer to the surface, edge, or corner of a building. Positioning refers to information determined about the host vehicle, such as position, speed, heading, etc. Path planning and vehicle control (e.g., braking, steering, and acceleration) are performed based on the collected perception and positioning information.

[0116] The vehicle can include multiple LIDAR sensors. LIDAR sensor alignment, including LIDAR-to-vehicle alignment and LIDAR-to-LIDAR alignment, affects the accuracy of the determined perception and positioning information, which includes feature and object information such as those described above. GPS measurements are used for vehicle positioning, mapping, and LIDAR alignment. GPS signals can be degraded and result in a blurred image of the environment, especially when the corresponding vehicle is near large (or tall) buildings, under bridges, or inside tunnels where GPS signals can be blocked. This is known as the multipath effect on GPS signals. High-precision GPS (e.g., real-time kinematic GPS) can also experience this same problem. Real-time kinematic GPS uses carrier-based positioning. This degradation can cause, for example, a stationary object to appear as if it is moving. For example, a traffic sign may appear to be moving while in fact it is stationary. Inaccurate GPS data negatively affects the accuracy and quality of the aggregated LIDAR data for estimating the vehicle's position and orientation based on GPS and inertial measurements.

[0117] The examples described herein include using LIDAR, inertial, and GPS measurements to estimate LIDAR boresight alignment and correct the position of the host vehicle. This includes correcting GPS data and vehicle orientation. The examples include a collaborative framework that iteratively implements processes to generate an accurate vehicle position and provide accurate LIDAR boresight alignment. The iterative process corrects GPS data while performing LIDAR calibration. GPS and inertial measurement signal data are corrected based on LIDAR data associated with multiple features. Data of explicit and / or selected road elements (e.g., traffic signs and light poles) are used to determine ground truth. "Ground truth" refers to points and / or information that are known to be correct and can then be used as a reference based on which information is generated and / or decisions are made. Principal component analysis (PCA) is used to characterize features. The corrected position information is used to calibrate the alignment of the vehicle with the LIDAR. Feature data from the past driving history of the host vehicle and / or other vehicles are used to improve algorithm performance.

[0118] Figure 1 An example vehicle system 100 of vehicle 102 is shown, which includes a sensor alignment and fusion module 104 and a mapping and positioning module 113. The operations performed by modules 104 and 113 will be described below with reference to Figures 1 to 10 further described.

[0119] Vehicle system 100 may include an autonomous driving module 105, a body control module (BCM) 107, a telematics module 106, a propulsion control module 108, a power steering system 109, a braking system 111, a navigation system 112, an infotainment system 114, an air conditioning system 116, and other vehicle systems and modules 118. Autonomous driving module 105 includes sensor alignment and fusion module 104 and mapping and positioning module 113, and may also include a validation module 115, a perception module 117, and a path planning module 121. Sensor alignment and fusion module 104 and mapping and positioning module 113 may communicate with each other and / or be implemented as a single module. Mapping and positioning module 113 may include a GPS correction module, as Figure 2 shown. The operations of these modules are further described below.

[0120] Modules and systems 104-108, 112-115, 121, and 118 can communicate with each other via a Controller Area Network (CAN) bus, Ethernet, a Local Interconnect Network (LIN) bus, another bus or communication network, and / or wirelessly. Item 119 can refer to and / or include a CAN bus, an Ethernet network, a LIN bus, and / or other buses and / or communication networks. This communication can include other systems, such as systems 109, 111, 116. A power supply 122 can be included and power the autonomous driving module 105 and other systems, modules, devices, and / or components. The power supply 122 can include an accessory power module, one or more batteries, a generator, and / or other power sources.

[0121] The telematics module 106 can include a transceiver 130 and a telematics control module 132. The propulsion control module 108 can control the operation of a propulsion system 136, which can include an engine system 138 and / or one or more electric motors 140. The engine system 138 can include an internal combustion engine 141, a starter motor 142 (or starter), a fuel system 144, an ignition system 146, and a throttle system 148.

[0122] The autonomous driving module 105 can control the modules and systems 106, 108, 109, 111, 112, 114, 116, 118, and other devices and systems based on data from sensors 160. Other devices and systems can include window and door actuators 162, interior lights 164, exterior lights 166, trunk motors and locks 168, seat position motors 170, seat temperature control systems 172, and vehicle rearview mirror motors 174. The sensors 160 can include temperature sensors, pressure sensors, flow rate sensors, position sensors, etc. The sensors 160 can include a LIDAR sensor 180, a RADAR sensor 182, a camera 184, an inertial measurement sensor 186, a GPS system 190, and / or other environmental and feature detection sensors. The GPS system 190 can be implemented as part of a navigation system 112. The LIDAR sensor 180, the inertial measurement sensor 186, and the GPS system 190 can provide the LIDAR data points, inertial measurement data, and GPS data mentioned below.

[0123] The autonomous driving module 105 can include a memory 192, which can store sensor data, historical data, alignment information, etc. The memory 192 can include dedicated buffers, which will be mentioned below.

[0124] Figure 2 An example alignment system 200 is shown, which includes an autonomous driving module that performs Global Positioning System (GPS), LIDAR, and vehicle positioning correction. The system 200 can include a first (or primary) vehicle (e.g., Figure 1Vehicle 102) and / or other vehicles, distributed communication system 202, and back office 204. The host vehicle includes an autonomous driving module 206, which can replace Figure 1 Autonomous driving module 105, vehicle sensors 160, telematics module 106, and actuators 210 of Figure 1 Autonomous driving module 105, vehicle sensors 160, telematics module 106, and actuators 210 of

[0125] The back office 204 can be a central office that provides services including data collection and processing services for the vehicle. The back office 204 can include a transceiver 212 and a server 214 having a control module 216 and a memory 218. Additionally or alternatively, the vehicle can communicate with other cloud-based network devices other than the server.

[0126] The autonomous driving module 206 can replace Figure 1 Autonomous driving module 105 of

[0127] The sensor alignment and fusion module 104 can perform sensor alignment and fusion operations based on the outputs of sensors 160 (e.g., sensors 180, 182, 184, 186, 190), as further described below. The mapping and positioning module 113 performs the operations further described below. The GPS correction module 220 can be included in one of the modules 104, 113. The alignment confirmation module 115 determines whether the LIDAR sensor and / or other sensors are aligned, which means that the differences in the information provided by the LIDAR sensor and / or other sensors for the same one or more features and / or objects are within a predetermined range of each other. The alignment confirmation module 115 can determine the differences in the six degrees of freedom of the LIDAR sensor, the degrees of freedom including roll, pitch, yaw, differences in x, y, and z, and determine whether the LIDAR sensor is aligned based on this information. The x coordinate can refer to the lateral horizontal direction. The y coordinate can refer to the front-back or longitudinal direction, and the z direction can refer to the vertical direction. The x, y, z coordinates can be switched and / or defined differently. If not aligned, one or more LIDAR sensors can be recalibrated. In one embodiment, when one of the LIDAR sensors is determined to be misaligned, the misaligned LIDAR sensor is recalibrated. In another embodiment, when it is determined that one of the LIDAR sensors is misaligned, two or more LIDAR sensors including the misaligned LIDAR sensor are recalibrated. In another embodiment, the misaligned LIDAR sensor is isolated and no longer used, and an indication signal is generated that indicates that the LIDAR sensor needs to be repaired. Data from the misaligned sensor can be discarded. After the recalibration and / or repair of the misaligned LIDAR sensor, additional data can be collected.

[0128] The mapping and positioning module 113 and the sensor alignment and fusion module 104 provide accurate results of the GPS position and LIDAR alignment, such that the data provided to the perception module 117 is accurate for perception operations. After verification, the perception module 117 can perform perception operations based on the collected, corrected, and aggregated sensor data to determine aspects of the environment around the corresponding host vehicle (e.g., Figure 1 vehicle 102). This can include generating the perception information as described above. This can include the detection and identification of features and objects (if not already performed), and determining the position, distance, and trajectory of the features and objects relative to the host vehicle. The path planning module 121 can determine the path of the vehicle based on the outputs of the perception and positioning module 113. The path planning module 121 can control the operation of the vehicle based on the determined path, including controlling the operation of the power steering system, the propulsion control module, and the braking system via the actuator 210.

[0129] The autonomous driving module 206 can operate in an offline mode or an online mode. The offline mode means when the backend 204 collects data and performs data processing for the autonomous driving module 206. This can include, for example, collecting GPS data from the vehicle and performing GPS positioning correction and LIDAR alignment for data annotation, and providing the corrected GPS data and data annotation back to the autonomous driving module 206. The neural network of the autonomous driving module 206 can be trained based on the data annotation. GPS position correction can be performed before data annotation. Although Figure 2 not shown in Figure 2 , the control module 216 of the server 214 can include one or more of modules 104, 113, and / or 115, and / or perform operations similar to one or more of modules 104, 113, and / or 115.

[0130] During the offline mode, the server 214 processes data collected over a previous extended period of time. During this online mode, the autonomous driving module 206 performs the GPS positioning correction and / or the LIDAR alignment. This can be achieved with or without the help of a cloud-based network device such as the server 214. During this online mode, the autonomous driving module 206 performs real-time GPS positioning and LIDAR alignment using the collected and / or historical data. This can include data collected from other vehicles and / or infrastructure devices. The cloud-based network device can provide historical data, historical results, and / or perform other operations to assist in real-time GPS positioning and LIDAR alignment. Real-time GPS positioning refers to GPS information providing the current position of the host vehicle. Generate LIDAR alignment information for the current state of one or more LIDAR sensors.

[0131] Figure 3 An alignment method including GPS, LIDAR, and vehicle positioning correction is shown. Figure 3 The operation, as if Figures 4 to 5 the operation of Figures 4 to 5 , can be performed by Figures 1 to 2 one or more of modules 104, 113, 220 of the module.

[0132] The alignment method is executed to dynamically calibrate the LIDAR to the vehicle's line of sight alignment and correct the GPS and inertial measurement positioning results. This method is applicable to LIDAR dynamic calibration and GPS and inertial measurement correction. PCA and plane fitting are used to determine the ground truth, such as the position of traffic signs. Multi-feature fusion is performed to determine the ground truth position data of objects such as traffic signs and lamp posts. Ground truth data for point registration can also be executed. When the LIDAR sensor does not scan traffic signs within a specific range of the host vehicle, interpolation is used to correct GPS measurements. When searching for the best transformation, the ground truth points can be weighted to improve algorithm performance. The ground truth data is stored in the vehicle memory and / or cloud-based network memory and is applied during the upcoming trips of the host vehicle and / or other vehicles.

[0133] The method can start at 300, which includes collecting data from sensors (such as Figure 1 sensor 160). The GPS data collected includes longitude, latitude, and attitude data. Inertial measurements are determined via the inertial measurement sensor 186 for roll, pitch, yaw rate, acceleration, and orientation, and the angles are estimated. At 302 and 304, feature extraction is performed. At 302, feature detection and characterization are performed for the first feature type (e.g., traffic signs, lamp posts, etc.). At 304, other feature detection and characterization are performed for the second feature type (e.g., building edges, corners, flat surfaces, etc.).

[0134] At 306, the ground truth position is calculated for one or more features and / or objects. Different road elements are monitored, including signs, buildings, and lamp posts, which provide sufficient coverage and a robust system. Road elements are detected, such as traffic signs, for determining the ground truth position. This can include using the entire point cloud from the LIDAR sensor. Utilizing the prior knowledge of the traffic sign, which has the characteristics of a flat plane, high-intensity reflectivity, and is stationary (i.e., not moving), the algorithm implemented at 306 can easily characterize the sign data using the PCA method to determine the ground truth position of the sign.

[0135] At 308, one or more ground truth positions are used to correct the GPS position. This can include performing interpolation to address missing LIDAR gaps in the data, where the LIDAR data is unavailable and / or unusable for certain time periods and / or timestamps. This improves the positioning accuracy.

[0136] At 310, the corrected GPS position is used to calculate the alignment. During this operation, the alignment of the LIDAR with the vehicle can be recalibrated to address alignment drift.

[0137] At 312, operations may be performed to determine whether the alignment is acceptable. This may include performing an alignment verification process. The alignment verification process may include performing a variety of methods. The methods include the integration of ground fitting, target detection, and point cloud registration. In one embodiment, the roll, pitch, and yaw differences between LIDAR sensors are determined based on targets (e.g., the ground, traffic signs, light poles, etc.). In the same or alternative embodiments, the rotational and translational differences of the LIDAR sensors are determined based on the differences in the point cloud registration of the LIDAR sensors.

[0138] The verification method includes (i) a first method for determining a first six-parameter vector of the differences in pitch, roll, yaw, x, y, and z values between LIDAR sensors is determined, and / or (ii) a second method for determining a second six-parameter vector of pitch, roll, yaw, x, y, and z values is determined. The first method is based on selecting certain objects to determine roll, pitch, and yaw. The second method is based on determining the rotational and translational differences from the point cloud of the LIDAR sensors. As described above, the results of these methods can be weighted and summed to provide a resulting six-parameter vector, based on which the determination of alignment is made. If the result is accepted, the method may end, and the GPS position information as well as the output of the LIDAR sensors can be used to make autonomous driving decisions including the control of systems 109, 111, 136, etc. Another variation is that if the alignment result is accepted, the alignment algorithm is not executed, but the GPS correction algorithm is continuously run to correct the GPS coordinates. The alignment result generally does not change over time, unless there is long-term degradation and / or an accident, but the GPS correction result can be updated and is useful at any time since the vehicle has moved (i.e., the position of the vehicle has changed). If the result is not accepted, then (i) the LIDAR-to-vehicle transformation can be recalibrated, and / or (ii) at least one of the one or more LIDAR sensors can be recalibrated and / or repaired.

[0139] In an embodiment, the alignment verification is performed after operation 310 and verifies the alignment performed at 310. In another embodiment, the alignment verification is performed at least before operations 308 and 310. In this embodiment, operations are performed Figure 3 to perform GPS correction and correct the alignment as a result of the verification process indicating an invalid alignment.

[0140] The following references Figures 4 to 5 the method to further describe operations 302, 306, 308, and 310. Operations 302 are also further described with respect to Figure 6 the method. Operations 306 are further described with respect to Figure 7 the method. Operations 308 and 310 are further described with reference to Figures 8 to 10 the method.

[0141] Figure 4 Part of the alignment method implemented when operating in offline mode. The method can start at 400, which includes loading the latest LIDAR-to-vehicle and vehicle-to-world transforms and / or the output of the next algorithm into the vehicle's memory (e.g., Figure 3 the memory 192 of Figure 1 ).

[0142] At 402, the LIDAR points are transformed from the LIDAR frame to the world frame using the above two transforms. The LIDAR coordinates are converted to vehicle coordinates and then to world coordinates (e.g., east, north, up (ENU) coordinates). Matrix transformations can be performed to convert to world coordinates. When performing the vehicle-to-world transform and the resulting image generated from the aggregated LIDAR data is blurred, the GPS data is inaccurate. The GPS position is corrected at 414 so that after correction, the resulting image is clear.

[0143] At 404, the LIDAR points of the continuously aggregated data (referred to as LIDAR points) are used for the next batch. As an example, this can be done for a batch of 500 frames. At 406, Figure 6 the feature extraction method of

[0144] is used to extract feature data from the aggregated LIDAR point cloud. At 408, the feature data is saved to a dedicated buffer in the memory. Figure 7 At 410, it is determined whether there is data collected for the feature. If so, operation 404 can be performed, otherwise operation 412 is performed. At 412,

[0145] the ground truth data generation method of Figure 8 is used to calculate the ground truth of the data.

[0146] In one embodiment, during the offline mode, after the data and / or other data are aggregated and the ground truth is generated, the data is reprocessed to be corrected.

[0147] Figure 5 Part of the alignment method implemented when operating in online mode with or without cloud-based network support. Figure 3Part of the alignment method. The method can start at 500, which includes loading the latest LIDAR-to-vehicle and vehicle-to-world transformations and / or the output of the next algorithm into the vehicle's memory (e.g., Figure 1 the memory 192).

[0148] At 502, LIDAR data is read and the LIDAR points are transformed from the LIDAR frame to the world frame using the above two transformations. The LIDAR coordinates are converted to vehicle coordinates and then to world coordinates (or East, North, Up (ENU) coordinates). Matrix transformations can be performed to convert to world coordinates. When performing the vehicle-to-world transformation and the resulting image generated from the aggregated LIDAR data is blurred, the GPS data is inaccurate. At 510, the GPS position is corrected so that after correction, the resulting image is clear.

[0149] At 504, ground truth data from the memory and / or a cloud-based network device is loaded (or obtained) by, for example, Figure 2 the autonomous driving module 206. At 506, the Figure 6 feature data extraction method is used to determine whether there are potential features. If so, operation 508 can be performed, otherwise operation 502 is performed.

[0150] At 508, it is determined whether the potential feature data is part of the ground truth data. If so, operation 510 is performed, otherwise operation 512 is performed.

[0151] At 510, the ground truth data is used to correct the position and update the LIDAR-to-vehicle transformation. This is done based on GPS and inertial measurement corrections and by implementing the Figure 8 LIDAR-to-vehicle alignment method. At 512, the vehicle transformation data is continuously aggregated, and the ground truth is generated using the Figure 7 method.

[0152] At 514, the ground truth data stored in the memory and / or a cloud-based network device is updated using the ground truth data generated at 512. Operation 502 can be performed after operation 514. The updated ground truth data (i) is saved for the next set of data and the next or subsequent timestamp and / or time period, and (ii) is not used for the current data set and the current timestamp and / or time period. For the online correction mode, there may not be enough time to accumulate data for ground truth generation before position correction, and thus the ground truth is generated after the correction of the next set of data.

[0153] Figure 6An example feature data extraction method is shown. The method can start at 600, which includes applying a spatial filter to find static points. As an example, the static points can be located at positions where z is greater than 5 meters and the distance of the points from the vehicle is greater than 20 meters.

[0154] The spatial filter uses a three-dimensional (3D) region in space to pick up points within that region. For example, the spatial filter can be defined to have a range x, y, z: , where is a predetermined value (or threshold). If the (x, y, z) of a point satisfies a predetermined condition of being within the region, the point is selected by the spatial filter.

[0155] At 602, an intensity filter and a shape filter are used to detect a first object (e.g., a traffic sign). The traffic sign has predetermined characteristics, such as being planar and having a specific geometry. The intensity filter can include an intensity range defined to select points with intensity values within an intensity range (e.g., a range of 0 - 255). As an example, the intensity filter can select points with intensity values greater than 200. The intensity value is proportional to the reflectivity of the material of the feature and / or object where the point is located. For example, the intensity filter can be defined as , where i is the intensity, and is a predetermined threshold. The shape filter can include detecting an object with a predetermined shape. At 604, an intensity filter and a shape filter are used to detect a second object (e.g., a lamp post). The lamp post has predetermined characteristics, such as being long and cylindrical.

[0156] At 606, an edge detection algorithm is used to detect a first feature (e.g., the edge of a building). The edge detection algorithm can be a method stored in Figure 1 the memory 192 or in a point cloud library stored in a cloud-based network device. At 608, a plane detection algorithm is used to detect a second feature (e.g., the plane of a building). The plane detection algorithm can be a method stored in Figure 1 the memory 192 or in a point cloud library stored in a cloud-based network device.

[0157] At 610, the feature data and the feature type (e.g., traffic sign, lamp post, edge of a building, or plane of a building) are saved together with the previously determined vehicle position and orientation. The method can end after operation 610.

[0158] Figure 7 An example ground truth data generation method is shown. The method can start at 700, which includes loading the feature data.

[0159] At 702, weights are assigned to features based on parameters. For example, based on vehicle speed, the type of acceleration actuator being executed, GPS signal strength, weights are assigned to traffic sign LIDAR points to indicate the confidence level of these points. If the position of the traffic sign does not move frame by frame, but the position of the host vehicle has changed, there is a problem and the weighted value of these points is set low. However, if the traffic sign does move frame by frame in the expected manner, a higher weight value is given to the points with a higher confidence level among the indicated points.

[0160] At 704, it is determined whether the feature data is of a first type (e.g., traffic sign). If so, operation 706 is performed, otherwise operation 710 is performed.

[0161] At 706, low-weight points (e.g., points assigned a weight value less than a predetermined weight value) are filtered out. At 708, principal component analysis (PCA) or plane fitting is performed after removing the low-weight points to determine the model of the feature. The model can be represented in the form of Equation 1, where and . a and b are the vertical eigenvectors in the plane, and c is the third eigenvector perpendicular to the plane of the feature and calculated by PCA. m is the mean (or average) vector of the remaining points.

[0162]

[0163] At 710, it is determined whether the feature data is of a second type (e.g., lamp post). If so, operation 712 is performed, otherwise operation 716 is performed.

[0164] At 712, low-weight points are filtered out. At 714, the remaining points are fitted to a 3D line model represented by Equation 2, where t is a parameter, e 1 is the eigenvector corresponding to the largest eigenvalue from PCA, and m is the mean vector. The eigenvectors e 2 and e 3 are very small. The first eigenvector e 1 extends in the longitudinal direction of the lamp post (e.g., in the z direction for a vertically extending post).

[0165]

[0166] At 716, the generated model, the data of the remaining points, and the corresponding weights are saved in the memory 192 such as Figure 1 .

[0167] Figure 8 Illustrates an example GPS and inertial measurement correction and LIDAR to vehicle alignment method. The method can start at 800, which includes reading LIDAR data points and obtaining GPS and inertial measurement data.

[0168] At 802, project the LIDAR data points into world coordinates. At 804, aggregate the projected LIDAR data points.

[0169] At 806, determine whether there are projected LIDAR data points within the ground truth data range. If so, perform operation 808; otherwise, perform operation 804.

[0170] At 808, use one or more of the methods in Figures 9 to 10 to correct the GPS and inertial measurement data and update the vehicle-to-world transformation. At 810, use the corrected GPS and LIDAR data to calculate the LIDAR-to-vehicle transformation. At 812, save the LIDAR-to-vehicle transformation to the Figure 1 memory 192 of

[0171] At 814, determine whether there are more LIDAR data points. If so, perform operation 804; otherwise, the method can end after operation 814.

[0172] Figure 9 Illustrates an example GPS correction method using a ground truth model. The method can start at 900, which includes loading LIDAR data points and GPS and inertial measurement data.

[0173] At 902, load the ground truth data for the first object (e.g., traffic sign) or the second object (e.g., lamp post).

[0174] At 904, determine whether there are LIDAR points belonging to one or more targets. The one or more targets can refer to, for example, one or more currently detected static objects such as traffic signs and / or lamp posts. If so, perform operation 906; otherwise, perform operation 900.

[0175] At 906, the LIDAR points belonging to the target are projected onto a plane and / or a line. This can be done using Equation 3-5 below, where X_ Lidar is a LIDAR point with x, y, and z components, X'_ Lidar is the projected point, center is the average vector, norm is the third eigenvector e 3 and normT is norm the transpose of. This includes removing center 、projecting onto a plane and adding back center .

[0176]

[0177] At 908, the average GPS offset is calculated for each timestamp and saved to the dedicated buffer of the memory 192 of Figure 1 . The difference between the original point and the projected point is determined to provide the GPS offset.

[0178] At 910, it is determined whether the corrected offset is reasonable by comparing the corrected offset with adjacent offsets on the corrected offset curve (generated based on the corrected offset). If the corrected offset is an outlier (e.g., greater than a predetermined distance from the corrected offset curve), the corrected offset is not used. If the corrected offset is on or within a predetermined distance of the corrected offset curve, it is used. Then, the corrected offset curve can be updated based on the used corrected offset. If the corrected offset is used, operation 912 is performed, otherwise operation 914 is performed.

[0179] At 912, the corrected offset is applied and the correct GPS position is calculated. At 914, the timestamps of two adjacent corrected GPS positions are determined. Adjacent corrected GPS positions refer to the corrected GPS positions that are closest in time to the GPS position to be corrected.

[0180] At 916, it is determined whether the time difference between the timestamp of the current corrected GPS position and the timestamps of the two adjacent corrected GPS positions is greater than a predetermined threshold. If not, operation 918 is performed, otherwise operation 920 is performed.

[0181] At 918, interpolation (e.g., linear interpolation) is performed to calculate the corrected GPS position of the target. If operation 912 is performed, the average of the corrected GPS positions determined at 912 can be averaged with the corrected GPS position determined at 918. At 920, the corrected GPS position and other corresponding information (such as timestamps and adjacent corrected GPS positions) can be stored in the memory 192.

[0182] Figure 9 The method of

[0183] Figure 10 illustrates an example GPS and inertial measurement correction method using ground truth point matching. The method can be performed Figure 10 instead of performing Figure 9 the method, or can be performed in addition to Figure 9 the method and in parallel with Figure 9 the method. The method can start at 1000, which includes loading LIDAR data points and GPS and inertial measurement data. Figure 10 At 1002, ground truth data is loaded. This can include the ground truth data determined when performing

[0184] the method. Figure 7 At 1004, it is determined whether there are LIDAR points belonging to one or more targets. If so, operation 1006 is performed; otherwise, operation 1000 is performed.

[0185] At 1006, other LIDAR points that do not correspond to one or more targets are filtered out.

[0186] At 1008, LIDAR point cloud registration (e.g., Iterative Closest Point (ICP) algorithm and / or Generalized Iterative Closest Point (GICP) algorithm) is performed to find the transformation between the current data and the ground truth data. Weights can be used in the ICP and / or GICP optimization functions.

[0187] ICP is an algorithm for minimizing the difference between two point clouds. ICP can include calculating the correspondence between two scans and calculating the transformation that minimizes the distance between corresponding points. Generalized ICP is similar to ICP and can include attaching a probability model to the minimization operation of ICP. The ICP algorithm can be performed iteratively by alternating between (i) given a transformation, finding the closest point in S for each point in M, and (ii) given the correspondence, finding the best rigid transformation by solving a least squares problem. Point set registration is the process of aligning two point sets.

[0188] At 1010, the average GPS offset and vehicle orientation offset are calculated for each timestamp and saved to a dedicated buffer in

[0189] the memory 192 of Figure 1 .

[0190] At 1012, it is determined whether the average GPS offset is reasonable by comparing the average GPS offset with one or more adjacent offset and orientation values. If so, operation 1014 is performed; otherwise, operation 1016 is performed.

[0191] At 1014, an average GPS offset is applied and a corrected GPS position and a corrected vehicle orientation are calculated.

[0192] At 1016, timestamps of two adjacent corrected GPS positions are determined, as similarly performed at 914 in Figure 9 .

[0193] At 1018, it is determined whether a time difference between the timestamp of the GPS position to be corrected and the timestamps of the two adjacent corrected GPS positions is greater than a predetermined threshold. If not, operation 1020 is performed; otherwise, operation 1022 is performed.

[0194] At 1020, interpolation (e.g., linear interpolation) is performed to calculate a corrected GPS position and a corrected vehicle orientation. If operation 1014 is performed, then (i) the average of the corrected GPS positions determined at 1014 can be averaged with the corrected GPS position determined at 1020, and (ii) the average of the corrected vehicle orientations determined at 1014 can be averaged with the corrected vehicle orientation determined at 1020.

[0195] At 1022, the corrected GPS position, the corrected vehicle orientation, and other corresponding information, such as timestamps and adjacent corrected GPS position information, can be stored in the memory 192.

[0196] The above operations are intended as illustrative examples. Depending on the application, the operations can be performed sequentially, synchronously, simultaneously, continuously, during overlapping time periods, or in a different order. Also, depending on the implementation and / or the order of events, any operation may not be performed or may be skipped.

[0197] The above examples include using updated position and orientation to update the LIDAR-to-vehicle alignment to improve the LIDAR-to-vehicle alignment accuracy. Correcting GPS data using LIDAR data and specific detection features (e.g., traffic signs, light poles, etc.). Contrary to general point registration methods, this does provide a system that is robust to initial guesses, hyperparameters, and dynamic objects. Examples include using intensity and spatial filters, clustering, and features of PCA and / or object detection and characterization. A flexible feature fusion architecture is provided to calculate the ground truth positions of features and / or objects (e.g., lane markings, light poles, road surfaces, building surfaces, and corners) to improve the accuracy and robustness of the entire system. Using ground truth information to correct GPS positions using interpolation to provide a system that is robust to noisy data and lost frames. As a result, precise LIDAR boresight alignment and precise GPS positioning and mapping are provided, which improves autonomous feature coverage and performance.

[0198] The feature data mentioned in this document may include feature data received at the host vehicle from other vehicles via vehicle-to-vehicle communication and / or vehicle-to-infrastructure communication. In the above example, the historical feature data may be used for the same current driving route of the host vehicle. The historical feature data may be stored on-board and / or received from a remote server (e.g., a server in a back office, central office, and / or cloud-based network).

[0199] The foregoing description is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses. The broad teachings of the disclosure may be implemented in a variety of forms. Thus, while the disclosure includes specific examples, the true scope of the disclosure should not be so limited since other modifications will become apparent after study of the drawings, the specification, and the appended claims. It should be understood that one or more steps in a method may be performed in a different order (or concurrently) without changing the principles of the disclosure. Further, although each embodiment above is described as having certain features, any one or more of those features described with respect to any embodiment of the disclosure may be implemented in and / or combined with the features of any other embodiment, even if not explicitly described in that combination. In other words, the described embodiments are not mutually exclusive, and permutations of one or more of the embodiments with each other remain within the scope of the disclosure.

[0200] Spatial and functional relationships between elements (e.g., between modules, circuit elements, semiconductor layers, etc.) are described using various terms, including "connected," "engaged," "coupled," "adjacent," "next to," "on top," "above," "below," and "disposed." Unless explicitly described as "direct," when the relationship between a first and a second element is described in the above disclosure, the relationship may be a direct relationship, where no other intermediate elements exist between the first and second elements, but may also be an indirect relationship, where one or more intermediate elements (spatially or functionally) exist between the first and second elements. As used herein, the phrase at least one of A, B, and C should be construed to represent the logic (A OR B OR C) using a non-exclusive logical OR, and should not be construed to mean at least one of A, at least one of B, and at least one of C.

[0201] In the drawings, the direction of an arrow, as indicated by the arrowhead, generally represents the flow of information (such as data or instructions) relevant to the illustration. For example, when element A and element B exchange various information, but the information sent from element A to element B is relevant to the illustration, the arrow may point from element A to element B. This one-way arrow does not mean that no other information is sent from element B to element A. Further, for the information sent from element A to element B, element B may send a request for that information or a receipt confirmation of that information to element A.

[0202] In this application, including the following definitions, the term "module" or the term "controller" may be replaced by the term "circuit". The term "module" may refer to, be part of, or include the following: application specific integrated circuit (ASIC); digital, analog, or mixed analog / digital discrete circuit; digital, analog, or mixed analog / digital integrated circuit; combinational logic circuit; field programmable gate array (FPGA); processor circuit (shared, dedicated, or group) that executes code; memory circuit (shared, dedicated, or group) that stores code executed by the processor circuit; other suitable hardware components that provide the functionality; or some or all of the combinations of the above, such as in a system on a chip.

[0203] The module may include one or more interface circuits. In some examples, the interface circuit may include a wired or wireless interface connected to a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any given module of the present disclosure may be distributed among multiple modules connected via the interface circuit. For example, multiple modules may allow load balancing. In another example, a server (also referred to as remote or cloud) module may perform some functions on behalf of a client module.

[0204] As described above, the term "code" may include software, firmware, and / or microcode, and may refer to programs, routines, functions, classes, data structures, and / or objects. The term "shared processor circuit" includes a single processor circuit that executes some or all of the code from multiple modules. The term "group processor circuit" includes a processor circuit that, in combination with additional processor circuits, executes some or all of the code from one or more modules. References to multiple processor circuits include multiple processor circuits on separate die, multiple processor circuits on a single die, multiple cores of a single processor circuit, multiple threads of a single processor circuit, or combinations of the above. The term "shared memory circuit" includes a single memory circuit that stores some or all of the code from multiple modules. The term "group memory circuit" includes a memory circuit that, in combination with additional memory, stores some or all of the code from one or more modules.

[0205] The term "memory circuit" is a subset of the term "computer-readable medium". As used herein, the term "computer-readable medium" does not include transitory electrical or electromagnetic signals propagated through a medium, such as on a carrier wave; thus, the term "computer-readable medium" can be considered tangible and non-transitory. Non-limiting examples of non-transitory tangible computer-readable media are non-volatile memory circuits (such as flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (such as static random access memory circuits or dynamic random access memory circuits), magnetic storage media (such as analog or digital magnetic tape or hard disk drives), and optical storage media (such as CDs, DVDs, or Blu-ray discs).

[0206] The devices and methods described in this application can be implemented in part or in whole by a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions implemented in a computer program. The above functional blocks, flowchart components, and other elements serve as software specifications, which can be converted into a computer program by routines worked by a skilled technician or programmer.

[0207] The computer program includes processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. The computer program may also include or rely on stored data. The computer program may include a basic input / output system (BIOS) that interacts with the hardware of the special-purpose computer, device drivers that interact with specific devices of the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0208] The computer program may include: (i) descriptive text to be parsed, such as HTML (HyperText Markup Language), XML (eXtensible Markup Language), or JSON (JavaScript Object Notation) (ii) assembly code, (iii) object code generated from source code by a compiler, (iv) source code executed by an interpreter, (v) source code for compilation and execution by a just-in-time compiler, etc. By way of example only, the source code may be written using the syntax of languages including C, C++, C#, Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl, Pascal, Curl, OCaml, Javascript®, HTML5 (HyperText Markup Language, 5th Edition), Ada, ASP (Active Server Pages), PHP (PHP: Hypertext Preprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, VisualBasic®, Lua, MATLAB, SIMULINK, and Python®.

Claims

1. A LIDAR-to-vehicle alignment system, which comprises: a memory configured to store data points provided based on the output of a LIDAR sensor and a global positioning system location; and an autonomous driving module configured to perform an alignment process, the alignment process including: obtaining the data points, performing feature extraction on the data points to detect one or more features of one or more predetermined types of objects having one or more predetermined characteristics, wherein, since the one or more features have the one or more predetermined characteristics, the one or more features are determined to correspond to one or more targets, and wherein one or more of the global positioning system locations in the global positioning system location are of the one or more targets, determining the ground truth location of the one or more features, correcting the one or more global positioning system locations in the global positioning system location based on the ground truth location, calculating a LIDAR-to-vehicle transformation based on the corrected one or more global positioning system locations in the global positioning system location, determining whether one or more alignment conditions are met based on the result of the alignment process, and responsive to the LIDAR-to-vehicle transformation not meeting the one or more alignment conditions, recalibrating at least one of the LIDAR-to-vehicle transformations or recalibrating the LIDAR sensor.

2. The LIDAR-to-vehicle alignment system according to claim 1, wherein: the autonomous driving module is configured to detect at least one of (i) a first object of a first predetermined type, (ii) a second object of a second predetermined type, or (iii) a third object of a third predetermined type when performing feature extraction; and the first predetermined type is a traffic sign; the second predetermined type is a lamp post; and the third predetermined type is a building.

3. The LIDAR-to-vehicle alignment system according to claim 2, wherein the autonomous driving module is configured to detect an edge or a planar surface of the third object when performing feature extraction.

4. The LIDAR-to-vehicle alignment system according to claim 1, wherein the autonomous driving module is configured to operate in an offline mode when performing the alignment process.

5. The LIDAR-to-vehicle alignment system according to claim 1, wherein the autonomous driving module is configured to operate in an online mode when performing the alignment process.

6. The LIDAR-to-vehicle alignment system according to claim 1, wherein the autonomous driving module is configured to, when performing feature extraction: convert data from the LIDAR sensor to a vehicle coordinate system and then to a world coordinate system; and aggregate the resulting world coordinate system data to provide the data points.

7. The LIDAR-to-vehicle alignment system according to claim 1, wherein the autonomous driving module is configured to, when determining the ground truth location: assign weights to the data points based on vehicle speed, the type of an acceleration actuator, and global positioning system signal strength to indicate the confidence level of the data points; Remove data points in the data points that have weight values less than a predetermined weight; and Determine a model corresponding to the characteristics of the remaining data points in the data points to generate ground truth data.

8. The LIDAR-to-vehicle alignment system according to claim 7, wherein the model is a plane or a line.

9. The LIDAR-to-vehicle alignment system according to claim 7, wherein the ground truth data includes the model, eigenvectors, and mean vectors.

10. The LIDAR-to-vehicle alignment system according to claim 7, wherein principal component analysis is used to determine the ground truth data.

11. The LIDAR-to-vehicle alignment system according to claim 1, wherein: The LIDAR-to-vehicle alignment system is implemented at a vehicle; The memory stores inertial measurement data; and The autonomous driving module is configured to, during the alignment process, Based on the inertial measurement data, determine the orientation of the vehicle, and Correct the orientation based on the ground truth data.

12. The LIDAR-to-vehicle alignment system according to claim 1, wherein the autonomous driving module is configured to perform interpolation to correct one or more global positioning system positions in the global positioning system positions based on previously determined corrected global positioning system positions.

13. The LIDAR-to-vehicle alignment system according to claim 1, wherein the autonomous driving module is configured to: Use a ground truth model for traffic signs or light poles to correct the one or more global positioning system positions; Project the LIDAR points of the traffic sign or the light pole onto a plane or a line; Calculate the average global positioning system offset for multiple timestamps; Apply the average global positioning system offset to provide the corrected one or more global positioning system positions in the global positioning system positions; and Update the vehicle-to-world transformation based on the corrected one or more global positioning system positions in the global positioning system positions.

14. The LIDAR-to-vehicle alignment system according to claim 1, wherein the autonomous driving module is configured to: Use ground truth point matching to correct the one or more global positioning system positions and inertial measurement data, including running an iterative closest point algorithm to find the transformation between the current data and the ground truth data, Calculate the average global positioning system offset and vehicle orientation offset for multiple timestamps, and Apply the average global positioning system offset and the vehicle orientation offset to generate the corrected one or more global positioning system positions in the global positioning system positions and the corrected vehicle orientation; and Update the vehicle-to-world transformation based on the corrected one or more global positioning system positions in the global positioning system positions and the corrected inertial measurement data.

15. A LIDAR-to-vehicle alignment process, which includes: Obtain data points provided based on the output of a LIDAR sensor; Perform feature extraction on the data points to detect one or more features of one or more predetermined types of objects having one or more predetermined characteristics, wherein, since the one or more features have the one or more predetermined characteristics, the one or more features are determined to correspond to one or more targets; Determine the ground truth position of the one or more features; Correct one or more global positioning system positions of the one or more targets based on the ground truth position; Calculate the LIDAR-to-vehicle transformation based on the corrected one or more global positioning system positions; Based on the result of the alignment process, determine whether one or more alignment conditions are satisfied; and In response to the LIDAR-to-vehicle transformation not satisfying the one or more alignment conditions, recalibrate at least one of the LIDAR-to-vehicle transformations or recalibrate the LIDAR sensor.

16. The LIDAR-to-vehicle alignment process according to claim 15, further comprising, when determining the ground truth position: Assign weights to the data points based on vehicle speed, type of acceleration actuator, and global positioning system signal strength to indicate the confidence level in the data points; Remove data points having weight values less than a predetermined weight from the data points; And Use principal component analysis to determine a model of the features corresponding to the remaining data points in the data points to generate ground truth data, wherein the model is a plane or a line, and wherein the ground truth data includes the model, eigenvectors, and mean vectors.

17. The LIDAR-to-vehicle alignment process according to claim 15, further Comprising: Determine the orientation of the vehicle based on inertial measurement data; And Correct the orientation based on the ground truth data.

18. The LIDAR-to-vehicle alignment process according to claim 15, wherein the one or more global positioning system positions are corrected by implementing interpolation based on previously determined corrected global positioning system positions.

19. The LIDAR-to-vehicle alignment process according to claim 15, further Comprising: Use a ground truth model for traffic signs or light poles to correct the one or more global positioning system positions; Project the LIDAR points of the traffic sign or the light pole onto a plane or a line; Calculate the average global positioning system offset for multiple timestamps; Apply the average global positioning system offset to provide the corrected one or more global positioning system positions; and Update the vehicle-to-world transformation based on the corrected one or more global positioning system positions.

20. The LIDAR-to-vehicle alignment process according to claim 15, further Comprising: Use ground truth point matching to correct the one or more global positioning system positions and inertial measurement data, including running an iterative closest point algorithm to find the transformation between the current data and the ground truth data, Calculate the average global positioning system offset and vehicle orientation offset for multiple timestamps, and Apply the average GPS offset and the vehicle orientation offset to generate the corrected one or more GPS positions and the corrected vehicle orientation; And Update the vehicle-to-world transformation based on the corrected one or more GPS positions and the corrected inertial measurement data.

Images