Automatic detection of the alignment state of a laser radar to a vehicle using positioning data

Abstract

A system in a vehicle includes a lidar system to obtain lidar data in a lidar coordinate system, and a processing circuit to obtain the lidar data and positioning data. The positioning data is indicative of a position and orientation of the vehicle. The processing circuit automatically determines an alignment state of a lidar-to-vehicle transformation matrix that projects the lidar data from the lidar coordinate system to a vehicle coordinate system to provide lidar-to-vehicle data. The alignment state is determined using the positioning data.

Description

Technical Field

[0001] This topic discloses information about using location data to automatically detect lidar alignment of vehicles. Background Technology

[0002] Vehicles (e.g., cars, trucks, construction equipment, farm equipment) increasingly include sensors that acquire information about the vehicle and its environment. This information contributes to the semi-autonomous or autonomous operation of the vehicle. For example, sensors (e.g., cameras, radar systems, lidar systems, inertial measurement units (IMUs), steering angle sensors) can facilitate semi-autonomous maneuvering, such as automatic braking, collision avoidance, or adaptive cruise control. Typically, sensors such as cameras, radar systems, and lidar systems have a coordinate system different from the vehicle's coordinate system. The sensor coordinate system must be correctly aligned with the vehicle's coordinate system to obtain information from the sensors that is readily applicable to vehicle operation. Therefore, it is desirable to use positioning data to provide automatic detection of the lidar's alignment with the vehicle. Summary of the Invention

[0003] In one exemplary embodiment, the system in the vehicle includes a lidar system for acquiring lidar data in a lidar coordinate system and processing circuitry for acquiring lidar data and positioning data. The positioning data indicates the vehicle's position and orientation. The processing circuitry automatically determines an alignment state that results in the lidar reaching a vehicle transformation matrix, which projects the lidar data from the lidar coordinate system to the vehicle coordinate system to provide lidar-to-vehicle data. The alignment state is determined using the positioning data.

[0004] In addition to one or more features described herein, the processing circuitry also obtains a vehicle-to-world transformation matrix to project data from the vehicle coordinate system to the world coordinate system, which is a fixed coordinate system, based on the location.

[0005] In addition to one or more features described herein, the processing circuit obtains multiple lidar data frames at corresponding timestamps, determines transformation matrices, each transformation matrix being between the first of the multiple lidar data frames and the subsequent one of the multiple lidar data frames, and obtains residual sums based on the transformation matrices, the lidar-to-vehicle transformation matrix of each frame, and the vehicle-to-world transformation matrix of each frame.

[0006] In addition to one or more features described herein, the processing circuitry also determines the alignment state as aligned based on the average of the sum of the residuals for each frame being less than or equal to a threshold.

[0007] In addition to one or more features described herein, the processing circuit aggregates multiple frames of LiDAR data at corresponding timestamps to obtain aggregated LiDAR data, uses the LiDAR to the vehicle transformation matrix and the vehicle-to-world transformation matrix to obtain aggregated LiDAR data in the world coordinate system, and identifies at least a predefined minimum number of objects.

[0008] In addition to one or more features described herein, the processing circuitry performs principal component analysis or determines the density of points in aggregated lidar data based on the object type of at least a predefined minimum number of objects associated with the point, and determines the alignment state based on the minimum eigenvalue generated by the principal component analysis or density.

[0009] In addition to one or more features described herein, the processing circuit aggregates multiple frames of lidar data at corresponding timestamps to obtain aggregated lidar data, uses lidar to vehicle transformation matrix and vehicle-to-world transformation matrix to obtain aggregated lidar data in world coordinate system, and identifies objects in aggregated lidar data in world coordinate system.

[0010] In addition to one or more features described herein, the processing circuitry acquires a high-definition map, a pre-saved point cloud indicating stationary objects, or vehicle-to-everything (V2X) communication, identifies one or more objects as ground reality objects, and determines the alignment state based on the distance between the objects identified in the aggregated lidar data and the corresponding ground reality objects in the ground reality objects.

[0011] In addition to one or more features described herein, the processing circuitry trains a deep learning neural network based on collected lidar data, positioning data, and alignment lidar to vehicle transformation matrix, injects different levels of alignment faults into the collected lidar data to generate modified lidar data and modified lidar data according to alignment fault labels, and implements supervised learning to train the deep learning neural network to classify alignment as good or faulty.

[0012] In addition to one or more features described in this paper, the processing circuit aggregates multiple frames of LiDAR data at corresponding timestamps to obtain aggregated LiDAR data, uses the LiDAR to vehicle transformation matrix and vehicle to world transformation matrix to obtain aggregated LiDAR data in the world coordinate system, implements a deep learning neural network, and determines the alignment status based on the indication of good or faulty alignment.

[0013] In another exemplary embodiment, the method in the vehicle includes obtaining lidar data in a lidar coordinate system from a lidar system by processing circuitry, and obtaining positioning data by processing circuitry, wherein the positioning data indicates the position and orientation of the vehicle. The method also includes automatically determining, by processing circuitry, an alignment state resulting from the lidar to a vehicle transformation matrix, which projects the lidar data from the lidar coordinate system to the vehicle coordinate system to provide lidar-to-vehicle data. The alignment state is determined using the positioning data.

[0014] In addition to one or more features described herein, the method further includes: the processing circuitry obtaining a vehicle-to-world transformation matrix to project data from the vehicle coordinate system to the world coordinate system, which is a fixed coordinate system, based on the positioning.

[0015] In addition to one or more features described herein, the method also includes processing circuitry that acquires multiple lidar data frames at corresponding timestamps, determines transformation matrices, each transformation matrix being between the first of the multiple lidar data frames and the subsequent one of the multiple lidar data frames, and obtains residual sums based on the transformation matrices, the lidar-to-vehicle transformation matrix for each frame, and the vehicle-to-world transformation matrix for each frame.

[0016] In addition to one or more features described herein, determining the alignment state involves determining that the alignment state is aligned based on the average of the residual sums of each frame being less than or equal to a threshold.

[0017] In addition to one or more features described herein, the method also includes processing circuitry that aggregates multiple frames of LiDAR data at corresponding timestamps to obtain aggregated LiDAR data, uses LiDAR to a vehicle transformation matrix and a vehicle-to-world transformation matrix to obtain aggregated LiDAR data in the world coordinate system, and identifies at least a predefined minimum number of objects.

[0018] In addition to one or more features described herein, the method also includes processing circuitry that performs principal component analysis or determines the density of a point based on the object type of at least a predefined minimum number of objects associated with the point, and determines the alignment state based on the minimum eigenvalue obtained from the principal component analysis or density.

[0019] In addition to one or more features described herein, the method also includes processing circuitry that aggregates multiple frames of lidar data at corresponding timestamps to obtain aggregated lidar data, uses lidar to a vehicle transformation matrix and a vehicle-to-world transformation matrix to obtain aggregated lidar data in the world coordinate system, and identifies objects in the aggregated lidar data in the world coordinate system.

[0020] In addition to one or more features described herein, the method also includes processing circuitry to obtain a high-definition map, a pre-saved point cloud indicating stationary objects, or vehicle-to-everything (V2X) communication, to identify one or more objects as ground reality objects, and to determine an alignment state based on the distance between the objects identified in the aggregated lidar data and the corresponding ground reality objects in the ground reality objects.

[0021] In addition to one or more features described herein, the method also includes processing circuitry to train a deep learning neural network based on collected lidar data, positioning data, and an alignment lidar to vehicle transformation matrix; injecting different levels of alignment faults into the collected lidar data to generate modified lidar data and modified lidar data based on alignment fault labels; and implementing supervised learning to train the deep learning neural network to classify alignment as good or faulty.

[0022] In addition to one or more features described herein, the method also includes processing circuitry that aggregates multiple frames of lidar data at corresponding timestamps to obtain aggregated lidar data, using lidar to a vehicle transformation matrix and a vehicle-to-world transformation matrix to obtain aggregated lidar data in the world coordinate system, implementing a deep learning neural network, and determining the alignment status based on an indication of good or faulty alignment.

[0023] The above-described features and advantages, as well as other features and advantages of this disclosure, will become apparent when taken in conjunction with the accompanying drawings and the following detailed description. Attached Figure Description

[0024] Other features, advantages, and details appear by way of example only in the following detailed specification, which refers to the accompanying drawings, wherein:

[0025] Figure 1 It is a block diagram of a vehicle including automatic detection of vehicle alignment status by laser radar according to one or more embodiments;

[0026] Figure 2 This is a processing flow of a method for automatically detecting the alignment state between a lidar coordinate system and a vehicle coordinate system according to an exemplary embodiment;

[0027] Figure 3 This is a processing flow of a method for automatically detecting the alignment state between a lidar coordinate system and a vehicle coordinate system according to another exemplary embodiment;

[0028] Figure 4 This is a processing flow of a method for automatically detecting the alignment state between a lidar coordinate system and a vehicle coordinate system according to another exemplary embodiment;

[0029] Figure 5This is a processing flow of a method for automatically detecting the alignment state between a lidar coordinate system and a vehicle coordinate system according to another exemplary embodiment; and

[0030] Figure 6 This is a processing flow of a method for automatically detecting the alignment state between a lidar coordinate system and a vehicle coordinate system according to another exemplary embodiment. Detailed Implementation

[0031] The following description is exemplary in nature only and is not intended to limit this disclosure, its application, or use. It should be understood that throughout the drawings, corresponding reference numerals denote the same or corresponding parts and features.

[0032] As mentioned earlier, the sensors in a lidar system have a coordinate system different from the vehicle's coordinate system. Therefore, information from the lidar system (e.g., the position of objects around the vehicle) must be projected onto the vehicle coordinate system via a transformation matrix in order to use this information to directly control vehicle operation. The transformation matrix is ​​essentially a representation of the alignment between the two coordinate systems. That is, the alignment process is the process of finding the transformation matrix. Therefore, when the two coordinate systems are correctly aligned, the transformation matrix correctly projects the lidar information onto the vehicle coordinate system, and when the two coordinate systems are misaligned, the transformation matrix will not correctly project the lidar information onto the vehicle coordinate system. Knowing the alignment state (i.e., aligned or misaligned) is important for correcting the transformation matrix as needed. Furthermore, monitoring the alignment state over time (e.g., dynamically detecting the alignment state) is important because aging, vibration, accidents, or other factors can change the alignment state.

[0033] Existing methods for ensuring alignment between a lidar system and a vehicle involve manually observing the lidar point cloud in the lidar coordinate system and those identical lidar point clouds projected onto the vehicle coordinate system to determine if misalignment exists in the transformation matrix visible in the projected lidar point cloud. This method has several drawbacks, including the time required and the fact that the evaluation is not suitable for real-time execution during vehicle operation. Embodiments of the systems and methods detailed herein involve automatically detecting the lidar alignment status (i.e., aligned or misaligned) to the vehicle using positioning data. The positioning data indicates the vehicle's position and orientation.

[0034] According to an exemplary embodiment, Figure 1 This is a block diagram of a vehicle 100 that includes automatic detection of the alignment status of the LiDAR and the vehicle. Detecting the alignment status means determining whether the existing transformation matrix correctly projects data from the LiDAR coordinate system 115 to the vehicle coordinate system 105 (i.e., the alignment status is aligned) or incorrectly projects data to the vehicle coordinate system 105 (i.e., the alignment status is not aligned). Figure 1The exemplary vehicle 100 shown is a car. Vehicle 100 is shown as including a lidar system 110 with a lidar coordinate system 115. In addition to the vehicle coordinate system 105 and the lidar coordinate system 115, a world coordinate system 102 is also shown. World coordinate system 102 remains constant, while the other coordinate systems 105, 115 may shift as vehicle 100 moves. Three exemplary objects 140a, 140b, and 140c (generally referred to as 140) are shown. Object 140a is a lamppost, and object 140b is a traffic sign. For example, object 140c could be another car or a pedestrian.

[0035] While a lidar system 110 is shown, the exemplary illustration is not intended to limit the number or location of lidar systems 110. Vehicle 100 may include any number of lidar systems 110 or other sensors 120 (e.g., cameras, radar systems, IMU 126, Global Navigation Satellite System (GNSS) such as GPS 125) at any location around vehicle 100. For example, other sensors 120 may provide positioning information (e.g., the position and orientation of vehicle 100). Movements associated with yaw (Y), pitch (P), and roll (R) are indicated. The yaw angle is the angle between the vehicle 100's direction of travel and its x-axis (xv), and rotates about the z-axis (vz) of vehicle 100, while the pitch angle is relative to the y-axis (vy) of vehicle 100, and the roll angle is relative to the z-axis (vz) of vehicle 100 and rotates about the x-axis (xv). This information can be obtained using other sensors 120 such as GPS 125 and IMU 126.

[0036] in addition, Figure 1 The coordinate systems 105 and 115 shown are exemplary. Furthermore, although... Figure 1 Three exemplary objects 140 are shown, but one or more sensors can detect any number of objects 140. An exemplary lidar field of view (FOV) 111 is outlined. As previously mentioned, a transformation matrix helps project data from one coordinate system to another. It is also noted that the process of aligning two coordinate systems is the process of determining the transformation matrix to project data from one coordinate system to another. Misalignment refers to errors in this transformation matrix.

[0037] Vehicle 100 includes a controller 130 that controls one or more operations of vehicle 100. Controller 130 can perform an alignment process between lidar system 110 and vehicle 100 (i.e., determining the transformation matrix between lidar coordinate system 115 and vehicle coordinate system 105). Controller 130 can also perform communication with other vehicles in the form of vehicle-to-vehicle (V2V) messages, communication with infrastructure in the form of vehicle-to-infrastructure (V2I) messages, and typically communication with a cloud server or any other system via vehicle-to-everything (V2X) messages. This will refer to... Figure 5 Further discussion. Controller 130 can additionally perform the training and implementation of machine learning (e.g., deep learning neural networks). This will refer to... Figure 6 Further discussion.

[0038] Controller 130 can implement a reference separately. Figure 2-6 The discussion process aims to determine the alignment status between the LIDAR system 110 and the vehicle 100. Depending on different exemplary embodiments, the controller 130 may implement a reference... Figure 2-6 One of the methods discussed, or two or more methods may be implemented in parallel. Controller 130 may include processing circuitry, which may include application-specific integrated circuits (ASICs), electronic circuitry, a processor (shared, dedicated, or grouped) and memory executing one or more software or firmware programs, combinational logic circuitry, and / or other suitable components that provide the described functionality.

[0039] Figure 2 This is a processing flow of a method 200 for automatically detecting the alignment state between a lidar coordinate system 115 and a vehicle coordinate system 105, according to an exemplary embodiment. (See reference...) Figure 2-6 The embodiments discussed rely on positioning data (e.g., the position and orientation of vehicle 100) obtained in world coordinate system 102. This positioning data may be obtained, for example, based on one of the other sensors 120 that are GPS 125.

[0040] At box 210, the process includes, for example, acquiring lidar data from lidar system 110 and positioning data from GPS 125 and IMU 126. At box 220, a check is performed to determine if activation conditions are met. Activation conditions refer to maneuvers such as U-turns, right turns, or left turns. Activation conditions can be quantified as yaw angles greater than a predefined threshold (e.g., 90 degrees) and vehicle translation (e.g., changes in xv and yv) greater than a predefined threshold (e.g., 20 meters) during which activation conditions are not met. If the check at box 220 indicates that activation conditions are not met, the process of acquiring lidar data and positioning data (at box 210) continues. Based on the check at box 220, if activation conditions are met (i.e., vehicle 100 is performing a maneuver that helps determine the alignment state according to method 200), the process at box 230 is executed.

[0041] At box 230, for a series of frames of lidar data (obtained at box 210 via a set of timestamps, indicated by index i), the transformation matrix Ai between the first frame and each subsequent frame is determined. This is accomplished using known techniques known as lidar registration, such as Laser Ranging and Mapping (LOAM), Iterative Closest Point (ICP), Generalized ICP (GICP), and Normal Distribution Transform (NDT). Outliers are then filtered out. That is, smooth movement of vehicle 100 is assumed during enabling conditions (e.g., U-turns). Therefore, any transformation matrix Ai outside a predefined range defined by extrapolation between frames is considered an outlier. Thus, after removing outlier frames and their corresponding transformation matrices in the transformation matrix Ai, the remaining transformation matrix Ai is retained.

[0042] Continuing the discussion of the process at box 230, for frames that retain the transformation matrix Ai, the vehicle-to-world transformation matrix T is obtained based on the positioning data. VWi That is, for each frame indicated by index i, the yaw, pitch, and roll angles of the components Tx, Ty, and Tz that constitute the rotation matrix R and translation matrix T are obtained from other sensors 120 such as GPS 125 and IMU 126. Then, the vehicle-to-world transformation matrix T... vwi It was obtained as:

[0043]

[0044] At box 240, the laser radar of interest, representing the alignment of its state, is used to reach the vehicle transformation matrix T. LV To calculate each pair of transformation matrices Ai and T VWi The residuals and S i :

[0045] S i =(A i TLV -T LV T VWi [Equation 2]

[0046] The average of the residual sums is also determined. At box 250, a check is performed to see if the average residual sum exceeds a predefined threshold. If it does, misalignment is reported as aligned at box 260. Conversely, if the check at box 250 indicates that the average residual sum does not exceed the threshold, alignment is determined as aligned at box 270.

[0047] Figure 3 This is a processing flow of a method 300 for automatically detecting the alignment state between a lidar coordinate system 115 and a vehicle coordinate system 105, according to another exemplary embodiment. At block 310, the process includes obtaining lidar data and positioning data from the lidar system 110, and at block 320, the process includes checking whether an enable condition is met. These processes are similar to those described in the reference... Figure 2 The procedures at boxes 210 and 220 are discussed. If the check indication at box 320 does not meet the activation conditions, the procedure for obtaining lidar data and positioning data (at box 310) continues. If the activation conditions are met according to the check at box 320 (i.e., vehicle 100 is performing the operation that facilitates the determination of the alignment state according to method 300), the procedure at box 330 is executed.

[0048] At box 330, the process includes aggregating the LiDAR data obtained at box 310, transforming the LiDAR data to world coordinate system 102, and identifying at least a minimum number of N objects 140 (e.g., N≥3) visible to manipulation based on the activation conditions identified at box 320. Transformation to world coordinate system 102L W The lidar data L (i.e., the point cloud obtained by lidar system 110) is obtained as follows:

[0049] L W =T VW T LV L [Equation 3]

[0050] For reference Figure 2 As stated, T VW It is the vehicle-to-world transformation matrix, and T LV This is the LiDAR-vehicle transformation matrix, representing the alignment of interest in the state. The aggregation at box 330 refers to the alignment in world coordinate system 102L. W Connect LiDAR data in series. Identify the world coordinate system 102L. WThe minimum number of N objects 140 in the cascaded LiDAR data may require the use of known identification techniques with some additional conditions. For example, the conditions may include a minimum number of points in the point cloud. At box 330, identifying objects 140 includes determining whether each of the N objects 140 is a lamppost (object 140A) or a traffic sign (object 140B).

[0051] At box 340, principal component analysis (PCA) is performed only on the points of the aggregated LiDAR data (i.e., point cloud) associated with each object 140 identified at box 330 as a lamppost (object 140a) or a traffic sign (object 140b). Performing PCA on the points of the LiDAR data results in three eigenvalues ​​corresponding to the eigenvectors indicating ambiguity in object 140. At box 350, it is checked whether the minimum eigenvalue is less than a threshold. If so, the alignment is determined to be in an aligned state at box 380.

[0052] On the other hand, if the check at box 350 indicates that the minimum feature value is not less than a threshold, then misalignment is reported as aligned at box 390. According to an exemplary embodiment, the result based on the check at box 350 may be sufficient. According to an alternative embodiment, the process at box 360 and the check at box 370 may be performed additionally or alternatively. When the check at box 370 is performed additionally, if both the check and boxes 350 and 370 indicate misalignment (i.e., both checks reach box 390), misalignment may be reported only at box 390.

[0053] At box 360, point cloud density is calculated for points in aggregated LiDAR data (i.e., point cloud) associated with each object 140 that is determined at box 330 to be neither a lamppost (object 140a) nor a traffic sign (object 140b). The calculation of point cloud density can be performed according to different exemplary embodiments. According to an exemplary embodiment, a set (e.g., 20) of the nearest points to any point associated with an object 140 that is neither a lamppost (object 140a) nor a traffic sign (object 140b) can be selected. The density can be determined as the reciprocal of the average distance from that point to each of the nearest points in that set. According to another exemplary embodiment, the density can be calculated based on the number of other points within a distance (e.g., 1 meter) of the point associated with an object 140 that is neither a lamppost (object 140a) nor a traffic sign (object 140b). According to yet another exemplary embodiment, the density can be calculated based on the sum of three characteristic values ​​obtained by performing singular value decomposition on a set (e.g., 20) of points associated with object 140 that is neither a lamppost (object 140a) nor a traffic sign (object 140b). At box 370, a check is performed to determine whether the density is greater than a threshold density. If so, then at box 380, alignment is determined to be in an aligned state. On the other hand, if the check at box 370 indicates that the density does not exceed the threshold density, then misalignment is reported as an aligned state at box 390.

[0054] Figure 4 This is a processing flow of a method 400 for automatically detecting the alignment state between a lidar coordinate system 115 and a vehicle coordinate system 105, according to another exemplary embodiment. At block 410, the process includes, for example, obtaining lidar data from the lidar system 110 and positioning data from the GPS 125 and IMU 126. At block 420, the process includes aggregating the lidar data obtained at block 410 and transforming it to the world coordinate system according to Equation 3. It also includes loading a high-resolution (HD) map or pre-saved point cloud representing the ground situation and indicating the current proximity of stationary objects 140 to the vehicle 100.

[0055] At box 430, the process includes identifying more than a minimum number N objects 140 (e.g., N>3) in aggregated LiDAR data indicated on the HD map or in a pre-saved point cloud of the objects, and calculating the distance between the points in the LiDAR data corresponding to the objects 140 and their associated points in the HD map or pre-saved point cloud. For example, the distance can be calculated based on Euclidean distance, plane angle, or line orientation. At box 440, it is checked whether the distance between any associated points exceeds a threshold distance. If not, alignment is determined as aligned at box 460. Conversely, if the check at box 440 indicates that the distance between any associated points exceeds the threshold distance, misalignment is reported as aligned at box 450.

[0056] Figure 5 This is a processing flow of a method 500 for automatically detecting the alignment state between a lidar coordinate system 115 and a vehicle coordinate system 105, according to another exemplary embodiment. At block 510, the process includes, for example, obtaining lidar data from the lidar system 110 and positioning data from the GPS 125. At block 520, the process includes aggregating the lidar data obtained at block 510 and transforming it to the world coordinate system according to Equation 3.

[0057] At box 530, the process includes identifying and locating another vehicle (object 140) in the aggregated LiDAR data in the world coordinate system, and also obtaining the position of the other vehicle via a V2X message. The process at box 530 then includes calculating the distance between a point corresponding to the other vehicle's LiDAR data and the position obtained via the V2X message. For example, the distance can be calculated based on Euclidean distance, plane angle, or line orientation. At box 540, a check is performed to determine if the distance exceeds a threshold distance. If not, alignment is determined as aligned at box 560. Conversely, if the check at box 540 indicates that the distance exceeds the threshold distance, misalignment is reported as aligned at box 550.

[0058] Figure 6 This is a processing flow of a method 600 for automatically detecting the alignment state between a lidar coordinate system 115 and a vehicle coordinate system 105, according to another exemplary embodiment. Method 600 requires processes for training a deep learning neural network (at boxes 603, 605, 607). At box 603, the process includes collecting lidar data and positioning data (e.g., according to reference) when an enable condition is met. Figure 2 or Figure 3 The process involves (as discussed in the previous section) and additionally obtaining a ground-based transformation matrix (i.e., an alignment transformation matrix). At block 605, the process includes injecting different levels of alignment faults (i.e., altering the alignment transformation matrix to different degrees), aggregating the transformed data, and automatically labeling the data based on the injected faults (e.g., aligned, unaligned). For example, an exemplary alignment fault might involve adding 1 degree to the yaw. At block 607, the process includes performing supervised learning to train a deep learning neural network to classify alignment as good or faulty. Once training is performed, the deep learning neural network can be implemented to automatically evaluate the alignment status. That is, training can be performed offline before implementation in vehicle 100.

[0059] At block 610, the process includes, for example, obtaining lidar data from lidar system 110 and positioning data from GPS 125 and IMU 126. At block 615, the process includes checking whether an enable condition is met. These processes are similar to those in the reference... Figure 2 The processes at boxes 210 and 220 are discussed. At box 620, this process includes aggregating the LiDAR data obtained at box 610 and transforming the aggregated LiDAR data to the world coordinate system using Equation 3. At box 625, applying a deep neural network trained according to the processes at boxes 603, 605, and 607 helps to obtain an indication at box 630 whether the alignment is good or faulty. If the alignment is indicated as faulty at box 630, then misalignment is reported as aligned at box 635. On the other hand, if the alignment is not indicated as faulty at box 630, then alignment is determined as aligned at box 640.

[0060] While the foregoing disclosure has been described with reference to exemplary embodiments, those skilled in the art will understand that various changes can be made and elements can be substituted with equivalents without departing from its scope. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of this disclosure without departing from the basic scope of this disclosure. Therefore, this disclosure is not intended to be limited to the specific embodiments disclosed, but will include all embodiments falling within its scope.

Claims

1. A system in a vehicle, comprising: The lidar system is configured to acquire lidar data in a lidar coordinate system; as well as A processing circuit configured to acquire the lidar data and positioning data, wherein the positioning data indicates the position and orientation of the vehicle, and automatically determine an alignment state that causes the lidar to a vehicle transformation matrix, the lidar to vehicle transformation matrix projecting the lidar data from the lidar coordinate system to the vehicle coordinate system to provide lidar to vehicle data, wherein the positioning data is used to determine the alignment state. The processing circuit obtains a vehicle-to-world transformation matrix to project data from the vehicle coordinate system to a world coordinate system, which is a fixed coordinate system, based on the positioning; and The processing circuit is configured to aggregate multiple frames of lidar data at corresponding timestamps to obtain aggregated lidar data, use lidar to vehicle transformation matrix and vehicle-to-world transformation matrix to obtain aggregated lidar data in world coordinate system, identify at least a predefined minimum number of objects, and perform principal component analysis or density determination of points in aggregated lidar data based on object types among at least a predefined minimum number of objects associated with points, and determine alignment state based on minimum eigenvalues ​​generated by principal component analysis or density.

2. The system according to claim 1, wherein, The processing circuit is configured to acquire multiple lidar data frames at corresponding timestamps, determine transformation matrices, each transformation matrix being between the first and subsequent lidar data frames, and obtain residual sums based on the transformation matrices, the lidar-to-vehicle transformation matrix for each frame, and the vehicle-to-world transformation matrix for each frame, and determine the alignment state as aligned based on the average of the residual sums for each frame being less than or equal to a threshold.

3. The system according to claim 1, wherein, The processing circuit is configured to aggregate multiple frames of LiDAR data at corresponding timestamps to obtain aggregated LiDAR data. It uses LiDAR to obtain a vehicle-to-world transformation matrix and a vehicle-to-world transformation matrix to obtain aggregated LiDAR data in the world coordinate system. It also identifies objects in the aggregated LiDAR data in the world coordinate system and obtains a high-definition map, a pre-saved point cloud indicating stationary objects, or vehicle-to-everything communication. It identifies one or more objects as ground reality objects and determines the alignment state based on the distance between the objects identified in the aggregated LiDAR data and their corresponding ground reality objects. The processing circuit is configured to collect LiDAR data... The system uses LiDAR data, positioning data, and alignment data to train a deep learning neural network from the vehicle transformation matrix. Different levels of alignment faults are injected into the collected LiDAR data to generate modified LiDAR data. The modified LiDAR data is then labeled based on alignment faults. Supervised learning is implemented to train the deep learning neural network to classify alignment as good or faulty. Multiple frames of LiDAR data are aggregated at corresponding timestamps to obtain aggregated LiDAR data. The LiDAR data is then used to obtain aggregated LiDAR data in the world coordinate system using the vehicle transformation matrix and the vehicle-to-world transformation matrix. Finally, the deep learning neural network is implemented to determine the alignment status based on the indication of good or faulty alignment.

4. A method in a vehicle, the method comprising: The processing circuit obtains lidar data in the lidar coordinate system from the lidar system. The processing circuit obtains positioning data, wherein the positioning data indicates the position and orientation of the vehicle; and The processing circuit automatically determines the alignment state of the lidar to the vehicle transformation matrix, which projects the lidar data from the lidar coordinate system to the vehicle coordinate system to provide lidar to vehicle data, wherein the alignment state is determined using the positioning data. The processing circuit obtains a vehicle-to-world transformation matrix to project data from the vehicle coordinate system to the world coordinate system, which is a fixed coordinate system, based on the positioning. as well as The processing circuit obtains multiple lidar data frames at corresponding timestamps, determines transformation matrices, with each transformation matrix falling between the first and subsequent lidar data frames, and obtains residual sums based on the transformation matrices, the lidar-to-vehicle transformation matrix for each frame, and the vehicle-to-world transformation matrix for each frame.

5. The method according to claim 4, wherein, Determining the alignment state includes determining that the alignment state is aligned based on the average of the sum of the residuals for each frame being less than or equal to a threshold.

6. The method according to claim 4, further comprising: The processing circuit aggregates multiple frames of lidar data at corresponding timestamps to obtain aggregated lidar data, uses the lidar to vehicle transformation matrix and the vehicle to world transformation matrix to obtain the aggregated lidar data in the world coordinate system, identifies at least a predefined minimum number of objects, performs principal component analysis or determines the density of points in the aggregated lidar data based on the object type among the at least predefined minimum number of objects associated with points, and determines the alignment state based on the minimum eigenvalue generated by principal component analysis or density.

7. The method according to claim 4, further comprising: The processing circuit aggregates multiple frames of LiDAR data at corresponding timestamps to obtain aggregated LiDAR data. It uses the LiDAR to vehicle transformation matrix and the vehicle to world transformation matrix to obtain the aggregated LiDAR data in the world coordinate system. It also identifies objects in the aggregated LiDAR data in the world coordinate system, obtains a high-definition map, a pre-saved point cloud indicating stationary objects, or vehicle-to-everything communication, identifies one or more objects as ground reality objects, and determines the alignment state based on the distance between the objects identified in the aggregated LiDAR data and the corresponding ground reality objects. The processing circuit trains a deep learning neural network based on the collected LiDAR data, positioning data, and the aligned LiDAR to vehicle transformation matrix. It injects different levels of alignment faults into the collected LiDAR data to generate modified LiDAR data and modifies the LiDAR data according to the alignment fault markers. The system implements supervised learning to train a deep learning neural network to classify alignment as good or faulty, and aggregates multiple frames of LiDAR data at corresponding timestamps to obtain aggregated LiDAR data. It uses the LiDAR to the vehicle transformation matrix and the vehicle-to-world transformation matrix to obtain aggregated LiDAR data in the world coordinate system, implements a deep learning neural network, and determines the alignment status based on the indication that the alignment is good or faulty.

Images