Time registration method and device of vehicle sensor and automatic driving vehicle

Abstract

The invention provides a time registration method and device of a vehicle sensor and an automatic driving vehicle, and relates to the technical field of computers, in particular to the technical fields of automatic driving, multi-sensor fusion, sensor time registration and the like. According to the specific implementation scheme, points belonging to a target object are obtained from to-be-processed data frames collected by a to-be-calibrated sensor in a vehicle and serve as to-be-optimized points; in the world coordinate system, the space distance between the point to be optimized and the reference point of the target object is minimized, and the time compensation amount of the sensor to be calibrated is obtained; the spatial distance is determined based on a time compensation amount as a variable; and correcting the time of the sensor to be calibrated based on the time compensation amount.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of autonomous driving, multi-sensor fusion, and sensor time registration. Background Technology

[0002] In the field of driving, sensor technology provides crucial information for driving decisions. With the development of multi-sensor fusion technology, vehicles are equipped with various types of sensors, such as LiDAR, cameras, and millimeter-wave radar. These sensors can perceive the environmental information around the vehicle in real time, providing key data support for the vehicle's autonomous driving, driver assistance, and safety systems.

[0003] However, the time differences between different sensors may cause data to be inaccurately aligned in the time dimension, which in turn affects the accuracy of fusion, the accuracy of environmental perception, and the reliability of driving decisions. Summary of the Invention

[0004] This disclosure provides a method, apparatus, and autonomous vehicle for time registration of vehicle sensors.

[0005] According to one aspect of this disclosure, a time registration method for a vehicle sensor is provided, comprising: From the data frames to be processed collected by the sensors to be calibrated in the vehicle, the points belonging to the target object are obtained as the points to be optimized; In the world coordinate system, minimizing the spatial distance between the point to be optimized and the reference point of the target object yields the time compensation amount of the sensor to be calibrated; the spatial distance is determined based on the time compensation amount as a variable. The time of the sensor to be calibrated is corrected based on the time compensation amount.

[0006] According to another aspect of this disclosure, a time registration device for a vehicle sensor is provided, comprising: The acquisition module is used to acquire points belonging to the target object from the data frames to be processed collected by the sensors to be calibrated in the vehicle, as points to be optimized; The determination module is used to minimize the spatial distance between the point to be optimized and the reference point of the target object in the world coordinate system, so as to obtain the time compensation amount of the sensor to be calibrated; the spatial distance is determined based on the time compensation amount as a variable. The correction module is used to correct the time of the sensor to be calibrated based on the time compensation amount.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0009] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0010] According to another aspect of this disclosure, a vehicle is provided, including the electronic equipment provided in this disclosure.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic flowchart of a time registration method for vehicle sensors according to an embodiment of the present disclosure; Figure 2 This is a schematic flowchart illustrating the process of determining the time compensation amount of a sensor to be calibrated according to an embodiment of the present disclosure; Figure 3 This is a schematic flowchart illustrating the process of determining a time correction value based on the compensation amount to be optimized and the timestamp of the point to be optimized, according to an embodiment of the present disclosure. Figure 4 This is another schematic diagram of a process for determining a time correction value based on the compensation amount to be optimized and the timestamp of the point to be optimized, according to an embodiment of the present disclosure; Figure 5 This is a flowchart illustrating the process of determining the vehicle pose corresponding to the time compensation amount to be optimized according to an embodiment of the present disclosure. Figure 6 This is a schematic diagram of the structure of a time registration device for a vehicle sensor according to an embodiment of the present disclosure; Figure 7 This is a block diagram of an electronic device used to implement the time registration method for vehicle sensors according to embodiments of the present disclosure. Detailed Implementation

[0013] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0014] The terms “first,” “second,” etc., used in this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0015] It should be noted that, unless it is explicitly stated that there is a sequential order of execution between different operations, or that there is a sequential order of execution between different operations in terms of technical implementation, the execution order between multiple operations may not be significant, and multiple operations may be executed simultaneously.

[0016] Sensor technology plays a crucial role in the fields of driver assistance and autonomous driving. Different types of sensors have their own unique working principles and characteristics, so they need to cooperate to provide effective information for driving decisions.

[0017] Time synchronization issues during multi-sensor data fusion can affect the accuracy of the fusion results. This time asynchrony can lead to inaccurate alignment of data collected by different sensors in the time dimension, thus affecting subsequent data fusion and analysis. For example, during multi-sensor data fusion, if the clock errors of LiDAR and cameras are greater than a preset error, the fusion of their data may result in incorrect matching of obstacle positions and states, leading to deviations in the vehicle's perception of the surrounding environment.

[0018] In view of this, this disclosure provides a time registration method for vehicle sensors. This method is used to dynamically fine-tune the timestamps of vehicle sensors to solve the perception deviation of the same target object caused by small clock deviations between different sensors, thereby improving the vehicle's perception accuracy and reliability of the surrounding environment.

[0019] like Figure 1 The diagram shown is a flowchart illustrating the time registration method for vehicle sensors provided in this disclosure, including the following: S101: Obtain points belonging to the target object from the data frames to be processed collected by the sensors to be calibrated in the vehicle, and use them as points to be optimized.

[0020] The sensor to be calibrated refers to the sensor among the many sensors in a vehicle that requires dynamic time registration. Time registration refers to fine-tuning the time of the sensor to be calibrated so that the position of the target object acquired by the sensor is aligned with a reference point (i.e., a baseline value). This reference point can come from other sensors used as a reference or from a high-precision map. "Dynamic" means that the time registration of the sensor to be calibrated can be performed in real time according to the vehicle's operating conditions while the vehicle is in motion.

[0021] It should be noted that vehicle sensors meeting the following conditions can be used as sensors to be calibrated: First, the collected data points have timestamps, meaning each data point (or data frame) must record a specific acquisition time; second, the sensor's own coordinate system changes dynamically with the vehicle, and the transformation matrix T between the sensor and the world coordinate system changes as the vehicle moves; third, there is a spatiotemporal reference point, such as a visual inspection box or high-precision map markers, which can provide a target for alignment. For example, millimeter-wave radar, lidar, cameras, ultrasonic sensors, and line-scan sensors can each be used as sensors to be calibrated.

[0022] The target object refers to an object in the vehicle's surrounding environment that is detected by the sensor to be calibrated. Examples include pedestrians, buildings, other vehicles, or obstacles such as non-motorized vehicles. Each target object also has its own reference point for time registration.

[0023] During implementation, points belonging to the target object can be selected from the data frames collected by the sensor to be calibrated and used as points to be optimized.

[0024] It should be noted that each target object can correspond to one point to be optimized. In this case, the center point of the target object can be preferentially selected as the point to be optimized. For example, for radar sensors, in addition to selecting the center point of the target object as the point to be optimized, the point with the highest RCS (Radar Cross Section) value of the target object can also be preferentially selected as the point to be optimized.

[0025] Of course, multiple points can be selected as optimization points for a single target object. Taking LiDAR as an example, it can collect and return a large amount of point cloud data. Through target detection algorithms, it can determine which points belong to the target object, and multiple points can be selected from these points as subsequent optimization points.

[0026] For example, when a target object is identified from a data frame to be processed based on a neural network, the confidence level of each point belonging to the target object can be determined, and points with confidence levels higher than the confidence level threshold can be selected as points to be optimized.

[0027] For example, points within a preset radius around the center point of the target object can be selected as points to be optimized.

[0028] In summary, the points to be optimized can include at least one point in the data frame to be processed.

[0029] S102, in the world coordinate system, minimize the spatial distance between the point to be optimized and the reference point of the target object to obtain the time compensation amount of the sensor to be calibrated; the spatial distance is determined based on the time compensation amount as a variable.

[0030] The world coordinate system is a unified, fixed coordinate system used to describe the relative positions of a vehicle and all objects in its surrounding environment. During implementation, data collected by the sensors to be calibrated can be transformed into the world coordinate system for processing to ensure data consistency and comparability. Transforming the points to be optimized collected by the sensors to the world coordinate system requires time compensation. Therefore, the spatial distance between the two is related to the amount of time compensation to be optimized.

[0031] A reference point is the position of a target object in a world coordinate system. In practice, the reference point can be determined by the position of the target object obtained through a vehicle's visual sensors or other means. For example, it can be the center position of the target object detected by a visual sensor, or the precise position of the target object determined by GPS and high-precision map information. A reference point can include one point or multiple points (such as points within a pre-defined neighborhood around the center position), and this disclosure does not limit this.

[0032] Understandably, when using the position of the target object detected by other sensors, such as a vision sensor, in the world coordinate system as a reference point, the embodiments of this disclosure can resolve the relative time deviation between different sensors. For example, after reducing or eliminating the time deviation between radar sensors and vision sensors through the embodiments of this disclosure, it is possible to adaptively compensate for the timestamp deviation of the sensor keys, reducing the registration error between different sensors. Therefore, a more accurate time reference can be provided for candidate velocity fusion.

[0033] The spatial distance between the point to be optimized and the reference point of the target object can be represented by Euclidean distance.

[0034] During implementation, the value of the time compensation can be continuously adjusted to minimize the spatial distance between the point to be optimized and the reference point. When the spatial distance is minimized, the corresponding optimized time compensation is the time compensation used for registering the sensor to be calibrated.

[0035] S103, corrects the time of the sensor to be calibrated based on the time compensation amount.

[0036] The determined time compensation amount is applied to the timestamp of the sensor to be calibrated to correct its time. For example, if the time compensation amount is positive, it indicates that the sensor is lagging behind, and the time compensation amount can be added to the original timestamp; if the time compensation amount is negative, it indicates that the sensor is ahead, and the time compensation amount can be subtracted from the original timestamp.

[0037] Multiple sensors on the vehicle can perform the above operations separately. Finally, through registration, the time of each of the multiple sensors can be fine-tuned, thereby improving the time consistency of the multiple sensors.

[0038] During implementation, sensor time registration can be completed in real time or on demand, depending on the actual situation, thereby enabling online dynamic fine-tuning of sensor time.

[0039] Therefore, in this embodiment of the disclosure, by acquiring the optimization point of the target object in the sensor to be calibrated, and determining the time compensation amount in the world coordinate system by minimizing the spatial distance between the optimization point and the reference point of the target object, the problem of inconsistent sampling time caused by various issues such as differences in hardware, working mechanisms, and data processing procedures among different vehicle sensors can be effectively solved. The determined time compensation amount can be used to correct the time of the sensor to be calibrated, so that the data collected by different sensors of the vehicle can be fine-tuned in the time dimension, improving the time alignment accuracy and thus improving the accuracy of multi-sensor data fusion. Through this scheme, the vehicle can perceive the surrounding environment more accurately, providing reliable data support for autonomous driving, assisted driving, and safety systems, reducing the safety risks caused by multi-sensor time inconsistency, and improving the safety and reliability of vehicle operation.

[0040] In this embodiment of the disclosure, during the optimization process, the time compensation amount can be referred to as the compensation amount to be optimized. For example... Figure 2 As shown, in the world coordinate system, minimizing the spatial distance between the point to be optimized and the reference point of the target object yields the time compensation amount of the sensor to be calibrated, which can be implemented as follows: S201, determine the time correction value based on the compensation amount to be optimized and the timestamp of the point to be optimized; the timestamp is the collection time of the point to be optimized.

[0041] That is, when the sensor to be calibrated acquires the data frame to be processed, each point has a corresponding acquisition time, which serves as the timestamp of the corresponding point.

[0042] Different sensors operate on different principles, resulting in different time representations for each point in the acquired data frame. For example, a unified timestamp for the entire frame means that all data points within the same frame share the same acquisition timestamp, suitable for scenarios with strong data synchronization and high acquisition speed; a row-by-row timestamp marks time on a row-by-row basis, with all data points within the same row using the same timestamp, and the timestamps of different rows increasing sequentially with the acquisition order, commonly seen in line-scan sensors; and a point-by-point timestamp, where each data point corresponds to an independent acquisition timestamp, accurately reflecting the actual acquisition time of a single point, often used in scenarios with extremely high time accuracy requirements.

[0043] During implementation, the amount of compensation to be optimized can be accumulated to the timestamp to obtain a time correction value. This enables a simple accumulation operation, improves the efficiency of time registration, and provides an accurate time reference for subsequent coordinate system transformation of the points to be optimized and optimization of time compensation, ensuring the consistency of data in time and space and the rationality of optimization.

[0044] S202, based on the vehicle's pose corresponding to the time correction value, transform the point to be optimized to the world coordinate system to obtain the projected point.

[0045] S203, by minimizing the spatial distance between the projection point and the reference point, optimize the compensation amount to be optimized, and obtain the time compensation amount.

[0046] In practice, minimizing the spatial distance between the point to be optimized and the reference point of the target object can be expressed by the following expression (1): (1) In expression (1), t i The timestamp of the i-th point among the points to be optimized is represented by δt; δt represents the compensation amount to be optimized, which is a variable; t i +δt represents the time correction value obtained by adding the compensation amount to be optimized to the timestamp; T radar→world (t) i +δt) represents the vehicle pose corresponding to the time correction value, T is the pose transformation matrix, which contains rotation and translation information, and radar→world indicates that the transformation relationship is from the radar coordinate system to the world coordinate system; T represents the coordinates of the i-th point to be optimized in the radar coordinate system; radar→world (t) i +δt) This represents the projected point obtained by transforming the i-th point among the points to be optimized to the world coordinate system; This represents the coordinates of the reference point corresponding to the i-th point among the points to be optimized in the world coordinate system; ||·|| 2 This represents the Euclidean distance, used to minimize the spatial distance between the projection point and the reference point; This represents the summation of the errors of N points to be optimized, used to synthesize the errors of all points; arg min δt This means finding the optimal compensation amount δt to be optimized, which minimizes the spatial distance between the projection point and the reference point; This is the final amount of time compensation obtained.

[0047] It is understandable that expression (1) is not limited to a single target object in the data frame to be processed; multiple target objects can be selected from the data frame to be processed, each with its own corresponding reference point. The optimization points and reference points of multiple target objects jointly participate in optimizing the time compensation amount. The processing method for each target object is the same.

[0048] It is also understandable that minimizing the aforementioned spatial distance is not limited to a single data frame to be processed. At least one data frame to be processed can be used together to optimize the compensation amount, thereby improving the accuracy of the optimization.

[0049] In this embodiment, a time correction value is determined based on the compensation amount to be optimized and the timestamp of the point to be optimized, providing an accurate time reference for subsequent coordinate transformation. By using the vehicle pose corresponding to this time correction value to transform the point to be optimized into the world coordinate system to obtain the projection point, the local data collected by the sensors can be unified into the global world coordinate system, facilitating subsequent spatial distance calculations. By minimizing the spatial distance between the projection point and the reference point to optimize the compensation amount to be optimized, the optimized time compensation amount is obtained, which can effectively eliminate time deviations between different sensors and improve the accuracy of time registration of vehicle sensors.

[0050] In multi-sensor time registration scenarios, considering that the compensation amount to be optimized for the sensor to be calibrated may drift slowly over time, in order to further improve the accuracy of time registration as needed, it is necessary to find a suitable compensation amount to be optimized to correct the time of the sensor to be calibrated.

[0051] In implementation, it can be assumed that this drift occurs slowly over time, and a linear variation model between the compensation amount to be optimized and time can be established to represent this variation. Based on this, a time correction value is determined based on the compensation amount to be optimized and the timestamp of the optimization point, such as... Figure 3 As shown, it includes the following: S301, Based on the linear change model, determine the expression for the compensation amount to be optimized; the linear change model is used to represent the linear change of the compensation amount to be optimized with time, and the linear change model includes the parameters to be optimized.

[0052] During implementation, the compensation amount to be optimized is optimized by optimizing the parameter to obtain the time compensation amount.

[0053] During implementation, the expression for the compensation amount to be optimized is determined based on the linear change model, and can be described by expression (2): (2) In expression (2), δt(t) i ) represents the compensation amount to be optimized corresponding to the i-th point among the points to be optimized; a and b represent the parameters to be optimized; where a can be represented as the linear drift rate of the compensation amount to be optimized over time; b can be represented as the initial compensation amount to be optimized.

[0054] S302, the expression of the compensation amount to be optimized is added to the timestamp to obtain the time correction value.

[0055] During implementation, the expression for the compensation amount to be optimized is accumulated to the timestamp, which can be described by expression (3): (3) In expression (3), t corr This indicates the obtained time correction value.

[0056] To further find a suitable compensation amount to correct the time of the sensor to be calibrated, the linear variation model of expression (2) can be used to replace the compensation amount in expression (1). By optimizing the parameters a and b, the time compensation amount can be solved. This optimization process can be described by expression (4): (4) In expression (4), J(a, b) represents the error function with parameters a and b to be optimized as variables; arg min （a，b） This represents the parameters to be optimized, a and b, and a·t, obtained after finding the most suitable compensation amount. i +b represents the expression of the compensation variable to be optimized; the meanings of the other parameters can be found in the previous text and will not be repeated here.

[0057] In this embodiment, the expression for the compensation amount to be optimized is determined based on a linear variation model. Considering the characteristic that the compensation amount to be optimized slowly drifts over time, and by introducing parameters to be optimized, the expression becomes flexible and adjustable, adapting to the actual situation of different sensor time deviations. The expression for the compensation amount to be optimized is accumulated to the timestamp to obtain the time correction value. This allows for the dynamic calculation of the time correction value corresponding to each data point using the linear variation model, thereby effectively improving the accuracy of time registration of vehicle sensors.

[0058] Among them, the linear change model has good applicability in short-term scenarios and can quickly output the optimal compensation amount to be optimized with extremely low complexity.

[0059] In real-world driving scenarios, vehicle movement often exhibits nonlinear and dynamic fluctuations, such as acceleration and deceleration. Continuing to use a linear model in such situations would overlook this dynamic nature, causing the error in the compensation quantity to be optimized to accumulate over time and eventually deviate completely from the true state. To address the dynamic nature of vehicle acceleration and deceleration, a state-space model can be used to estimate the compensation quantity to be optimized by introducing hidden states.

[0060] Based on this, in other embodiments, a time correction value is determined based on the compensation amount to be optimized and the timestamp of the point to be optimized, such as... Figure 4 As shown, it includes the following: S401, the compensation amount to be optimized is taken as a hidden state, and the compensation amount to be optimized is estimated by Kalman filtering.

[0061] Latent states refer to variables that cannot be directly measured but can be indirectly inferred from observational data.

[0062] Kalman filtering is an optimal recursive filter used to estimate the state of a system. It can optimally estimate the true state of the system based on a series of noisy measurements. Kalman filtering typically involves two main steps: prediction and correction.

[0063] In practice, the prediction step forecasts the amount of compensation to be optimized at the current time step; in the correction step, the predicted value is corrected using the observation data at the current time step to obtain a more accurate estimate. By continuously repeating these two steps, the optimal value of the compensation to be optimized is estimated.

[0064] S402, the amount of compensation to be optimized is added to the timestamp to obtain the time correction value.

[0065] In this embodiment, the compensation quantity to be optimized is treated as a hidden state, and its estimation is achieved through the powerful ability of Kalman filtering to handle uncertainties and dynamic changes. Kalman filtering can optimally estimate the hidden state (i.e., the compensation quantity to be optimized) based on the dynamic change model of the compensation quantity and measurement data, effectively reducing the impact of noise and interference, and obtaining a more accurate compensation quantity for long-term scenarios. The estimated compensation quantity is accumulated to a timestamp to obtain a time correction value, which accurately corrects the time of sensor data, further aligning the data from different sensors in time.

[0066] The Kalman filter is used to estimate the compensation amount to be optimized, which can be implemented as follows: Step A1: Construct key parameters; Key parameters include: Process noise covariance is used to simulate the random disturbance of the compensation quantity to be optimized due to changes in vehicle speed. The rate of change of the compensation amount to be optimized over time, where it is assumed that the rate of change is constant in the short term and is less than the duration threshold. Observe the noise covariance to quantify the uncertainty of each optimization result for the compensation quantity to be optimized; The state transition assumption is used to indicate that the effect of vehicle speed change on the rate of change of the compensation quantity to be optimized is constant over adjacent time intervals.

[0067] Other parameters required for Kalman filtering can be set according to actual business conditions, and are not limited here.

[0068] Step A2: Based on key parameters, and taking the minimization of the spatial distance between the point to be optimized and the reference point as the criterion for generating the observed values ​​of the compensation amount to be optimized, the compensation amount to be optimized is recursively updated using Kalman filtering.

[0069] The following example illustrates some of the execution processes of Kalman filtering: Step 1: Define state variables Modeling the time compensation as a dynamic system includes: δt: Represents the time difference that needs to be compensated, i.e., the amount to be optimized. The value range can be set to ±0.1 seconds for fine-tuning.

[0070] : Represents the trend of δt changing over time, and can also be called the rate of change.

[0071] Therefore, the state vector of the compensation quantity to be optimized ( As shown in expression (5): (5) Step 2: Kalman filter initialization Initial state, assuming no initial compensation, the state vector of the compensation amount to be optimized ( As shown in expression (6): (6) Initial covariance matrix ( When implementing, ( The error range is determined based on a preset error range for the fine-tuning time. For example, as shown in expression (7): (7) Expression (7) indicates that the initial error is within ±0.1 seconds, which is the preset error range.

[0072] Key parameters include: The observation matrix (H), as shown in expression (8), indicates that only δt can be directly measured: (8) Observation noise (R): can be set empirically, for example, a matrix with a fixed value can be set to describe the uncertainty of each optimization result.

[0073] Process noise intensity (q1): used to reflect the random influence of vehicle motion on the change of δt.

[0074] Step 3: Predicting Future State The state transition matrix (F) is shown in expression (9): (9) Where Δt represents the time interval Δt since the last optimization (which can be set based on empirical values).

[0075] The process noise covariance (Q) is shown in expression (10): (10) The process noise covariance can simulate the random disturbance of δt caused by vehicle acceleration and deceleration.

[0076] The prediction step is shown in expression (11): (11) in, This represents the predicted state value of the compensation quantity to be optimized at the current moment; This represents the state estimate of the compensation quantity to be optimized at the previous moment; This represents the prediction covariance matrix at the current moment, used for the propagation of uncertainty in the predicted state (inheriting the error from the previous moment). The superposition of Q can represent the additional uncertainty introduced by the noise covariance during the superposition process. This represents the state covariance matrix at the previous time step; This indicates the time deviation of the prediction at the current moment; Indicates the time deviation from the previous moment; This represents the rate of change of time deviation over time, used to reflect the dynamic trend of time deviation; the meanings of other parameters are as described above.

[0077] Spatiotemporal registration optimization generates observations ( This step involves using expression (11) Applying this to minimizing alignment error (i.e., minimizing the spatial distance between the point to be optimized and the reference point), generating As the observation input for Kalman filtering.

[0078] The chi-square test is used to achieve the... Outlier detection, in If the value is reliable, then proceed with the subsequent update steps if it is not an outlier.

[0079] The update step is used to fuse the predicted values. Compared with observed values Optimal output As shown in the formula below: Kalman gain ( The calculation is shown in the following expression (12): (12) Expression (12) is used to calculate the predicted covariance. With observation noise The weights are assigned based on the ratio. Here, H represents the observation matrix; This represents the transpose of the observation matrix; This represents the inverse matrix of the observation noise.

[0080] The state update is shown in the following expression (13): (13) in, This represents the updated state estimate of the compensation amount to be optimized. The meaning of KH is explained in the previous text and will not be repeated here.

[0081] Output after this iteration: Optimal time compensation amount .

[0082] The covariance matrix is ​​updated as shown in expression (14): (14) in, This represents the updated covariance matrix; This represents the identity matrix, used to ensure dimension matching in matrix operations. The meanings of K and H are explained above and will not be repeated here.

[0083] In this embodiment of the disclosure, by setting key parameters, the influence of vehicle speed on the time compensation amount to be optimized is simulated. Then, by using Kalman filtering, the estimation results are continuously corrected, which can effectively reduce the influence of noise interference and improve the accuracy and reliability of the estimation of the compensation amount to be optimized.

[0084] In this embodiment of the disclosure, the vehicle pose corresponding to the time compensation amount to be optimized can be determined based on the following steps: Step B1: Determine the interpolation range of the pose based on the candidate values ​​of the time compensation amount to be optimized; During the optimization process, the time compensation amount to be optimized is an intermediate candidate value when the optimization algorithm searches for the optimal compensation variable, rather than a predetermined fixed value.

[0085] In implementation, multiple candidate values ​​for the time compensation amount to be optimized can be generated first using any method such as grid search, gradient descent, or Bayesian optimization. Since the vehicle pose is a record of discrete time points (e.g., one pose is output every n ms), the time correction value corresponding to the time compensation amount to be optimized may fall between the time points corresponding to any two discrete poses. Therefore, it is necessary to first determine the time points corresponding to the two adjacent discrete poses to which the time correction value belongs; the range formed by these two time points is the interpolation interval of the pose. The interpolation interval is a time range within which the vehicle pose corresponding to the time correction value can be calculated using interpolation methods. For example, the time points corresponding to the two adjacent discrete poses to which the time correction value belongs might be t. i and t i+1 Then the interpolation interval is [t]. i， t i+1 ].

[0086] Step B2: Within the interpolation interval, determine the vehicle pose corresponding to the time compensation amount to be optimized.

[0087] After determining the interpolation interval, the vehicle pose can be calculated using interpolation based on the known vehicle pose data and the time compensation amount to be optimized, at that time compensation amount.

[0088] In this embodiment of the disclosure, by determining the interpolation range of the pose based on the candidate values ​​of the time compensation amount to be optimized, a suitable range can be defined for the time compensation amount to be optimized. By determining the vehicle pose corresponding to the time compensation amount to be optimized within this interpolation range, the accuracy and reliability of vehicle pose determination can be effectively improved.

[0089] During implementation, linear interpolation can be performed within the interpolation interval to obtain the vehicle pose corresponding to the time compensation amount to be optimized.

[0090] It should be noted that the vehicle's position T t Typically, it's a 4×4 homogeneous transformation matrix, consisting of two parts: a known rotation vector... and the known translation vector .

[0091] With the interpolation interval as [t] i， t i+1 For example, t i The vehicle's pose at a given time can be represented as: ;t i+1 The vehicle's pose at a given time can be represented as: And the time compensation amount t' to be optimized satisfies t.i <t’<t i+1 The vehicle pose corresponding to the time compensation amount to be optimized can then be obtained through linear interpolation, and can be described by expression (15): (15) In expression (15), This indicates that the translation vector is obtained through linear interpolation; The vehicle pose is obtained through linear interpolation; Indicates t i The translation vector at time; Indicates t i+1 The translation vector at time; λ represents the time weight, used to determine the time compensation amount t' to be optimized within the interpolation interval [t]. i， t i+1 The relative position within ]

[0092] In this embodiment of the disclosure, the vehicle pose corresponding to the time compensation amount to be optimized is obtained by linear interpolation within the interpolation interval. This allows for a reasonable estimation of the vehicle pose corresponding to the time compensation amount to be optimized using a linear relationship, avoiding the time consumption caused by complex calculations.

[0093] During vehicle movement, the vehicle's pose changes over time. When the vehicle's turning radius is large, simple linear interpolation may not accurately describe the changes in vehicle pose because the vehicle's rotation is complex. Therefore, it is necessary to perform different interpolation processes on the translation vector and rotation matrix of the vehicle pose to more accurately determine the vehicle pose corresponding to the time compensation amount to be optimized.

[0094] During implementation, within the interpolation interval, the vehicle pose corresponding to the time compensation amount to be optimized is determined, such as... Figure 5 As shown, it may include the following: S501, when the radius of the vehicle's turn is greater than the radius threshold, the known translation vector of the candidate value in the interpolation interval is linearly interpolated to obtain the translation matrix.

[0095] A known translation vector refers to a vector representing the vehicle's translation information at different points in time. These translation vectors record the vehicle's position change relative to a reference point at each time point.

[0096] In practice, the known translation vector of the candidate value in the interpolation interval can be linearly interpolated based on the expression (15) described above to obtain the translation matrix.

[0097] S502, rotate and interpolate the known rotation vectors of the candidate values ​​in the interpolation interval to obtain the rotation matrix.

[0098] In practice, the rotation matrix can be obtained based on the following steps: Step C1: Calculate the relative rotation matrix of the known rotation vectors in the interpolation interval; The relative rotation matrix can be expressed based on expression (16): (16) In expression (16), R rel Indicates the t-th i Time t i+1 The relative rotation matrix at time; Indicates the t-th i+1 The known rotation vector at time; Indicates the t-th i The known rotation vector at time t.

[0099] Step C2, R rel Decompose into the rotation axis v and the rotation angle corresponding to the rotation axis v. ; In practice, the Rodrigues Rotation Formula can be used for decomposition.

[0100] Step C3: Obtain the target rotation angle through rotation interpolation; The rotation angle is obtained and expressed by expression (17): (17) In expression (17), This indicates the obtained target rotation angle; Indicates the interpolation coefficients; This represents the rotation angle corresponding to the rotation axis v.

[0101] Step C4: Reconstruct the rotation matrix.

[0102] The reconstructed rotation matrix is ​​expressed by expression (18): (18) In expression (18), ( ) represents the exponential mapping from a rotation vector to a rotation matrix; This indicates that the rotation matrix is ​​obtained.

[0103] S503 determines the vehicle pose corresponding to the time compensation amount to be optimized based on the rotation matrix and translation matrix.

[0104] Based on the rotation and translation matrices, the vehicle's in-situ pose can be derived from T = The vehicle pose T corresponding to the time compensation amount to be optimized is transformed into the time compensation amount to be optimized. This ensures that the vehicle's motion interpolation conforms to both the linear law of positional change and the spherical law of rotational change, resulting in smooth changes in vehicle attitude.

[0105] In this embodiment of the disclosure, by acquiring image data collected by a vision sensor and converting the detection box of the target object in the image to the world coordinate system to obtain the reference point of the target object, the target object identified in the image can be accurately mapped to the real physical space, thereby improving the accuracy of time registration of the vehicle sensor.

[0106] In this embodiment of the disclosure, the reference point of the target object can be determined based on the following steps: Step D1: Acquire image data collected by the vision sensor; Visual sensors typically refer to devices such as cameras that can capture image information of the surrounding environment. Examples include obstacles such as pedestrians, buildings, other vehicles or non-motorized vehicles, and even stationary obstacles.

[0107] Step D2: Transform the detection bounding box of the target object in the image data to the world coordinate system to obtain the reference point of the target object.

[0108] In practice, target detection algorithms can be used to identify the location of target objects in image data, and then a detection box can be used to define the location. The detection box is usually represented by a rectangle and contains the positional information of the target object in the image coordinate system.

[0109] Using the pose transformation matrix described above, the bounding box of the target object can be transformed to the world coordinate system to obtain the reference point of the target object.

[0110] In this embodiment of the disclosure, by acquiring image data collected by a vision sensor and converting the detection box of the target object in the image to the world coordinate system to obtain the reference point of the target object, the target object identified in the image can be accurately mapped to the real physical space, thereby improving the accuracy of time registration of the vehicle sensor.

[0111] Based on the same technical concept, this disclosure also provides a time registration device 600 for a vehicle sensor, such as... Figure 6 As shown, it includes: The acquisition module 601 is used to acquire points belonging to the target object from the data frames to be processed collected by the sensors to be calibrated in the vehicle, as points to be optimized. The determination module 602 is used to minimize the spatial distance between the point to be optimized and the reference point of the target object in the world coordinate system to obtain the time compensation amount of the sensor to be calibrated; the spatial distance is determined based on the time compensation amount as a variable. Correction module 603 is used to correct the time of the sensor to be calibrated based on the time compensation amount.

[0112] In some embodiments, the determining module includes: The determination unit is used to determine the time correction value based on the compensation amount to be optimized and the timestamp of the point to be optimized; the timestamp is the acquisition time of the point to be optimized. The transformation unit is used to transform the point to be optimized to the world coordinate system based on the vehicle's pose corresponding to the time correction value, so as to obtain the projected point; The optimization unit is used to optimize the compensation amount to be optimized by minimizing the spatial distance between the projection point and the reference point, thereby obtaining the time compensation amount.

[0113] In some embodiments, the determining unit is configured to: The amount of compensation to be optimized is added to the timestamp to obtain the time correction value.

[0114] In some embodiments, the determining unit is configured to: Based on the linear change model, the expression for the compensation amount to be optimized is determined; the linear change model is used to represent that the compensation amount to be optimized changes linearly with time, and the linear change model includes the parameters to be optimized. The expression for the compensation amount to be optimized is added to the timestamp to obtain the time correction value.

[0115] In some embodiments, the determining unit is configured to: The amount of compensation to be optimized is taken as a hidden state, and Kalman filtering is used to estimate the amount of compensation to be optimized. The amount of compensation to be optimized is added to the timestamp to obtain the time correction value.

[0116] In some embodiments, the determining unit is configured to: Build key parameters; Based on key parameters, and taking the minimization of the spatial distance between the point to be optimized and the reference point as the criterion for generating the observed value of the compensation amount to be optimized, the compensation amount to be optimized is recursively updated by Kalman filtering. Key parameters include: Process noise covariance is used to simulate the random disturbance of the compensation quantity to be optimized due to changes in vehicle speed. The rate of change of the compensation amount to be optimized over time, where it is assumed that the rate of change is constant in the short term and is less than the duration threshold. Observe the noise covariance to quantify the uncertainty of each optimization result for the compensation quantity to be optimized; The state transition assumption is used to indicate that the effect of vehicle speed change on the rate of change of the compensation quantity to be optimized is constant over adjacent time intervals.

[0117] In some embodiments, a pose determination module is further included, for: Based on the candidate values ​​of the time compensation amount to be optimized, the interpolation range of the pose is determined; Within the interpolation interval, determine the vehicle pose corresponding to the time compensation amount to be optimized.

[0118] In some embodiments, the pose determination module is configured to: Linear interpolation is performed within the interpolation interval to obtain the vehicle pose corresponding to the time compensation amount to be optimized.

[0119] In some embodiments, the pose determination module is configured to: When the radius of the vehicle's turn is greater than the radius threshold, the known translation vector of the candidate value in the interpolation interval is linearly interpolated to obtain the translation matrix; The known rotation vectors of the candidate values ​​in the interpolation interval are rotated and interpolated to obtain the rotation matrix; Based on the rotation and translation matrices, the vehicle pose corresponding to the time compensation amount to be optimized is determined.

[0120] In some embodiments, a reference point for the target object is determined based on the following method: Acquire image data from a vision sensor; Transform the detection bounding box of the target object in the image data to the world coordinate system to obtain the reference point of the target object.

[0121] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0122] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0123] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0124] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0125] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded into random access memory (RAM) 703 from storage unit 708. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. Input / output (I / O) interface 705 is also connected to bus 704.

[0126] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0127] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the time registration method for vehicle sensors. For example, in some embodiments, the time registration method for vehicle sensors can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the time registration method for vehicle sensors described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the time registration method for vehicle sensors by any other suitable means (e.g., by means of firmware).

[0128] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0130] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0132] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0133] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0134] Based on the aforementioned electronic devices, this disclosure also provides a vehicle that may include electronic devices, and may also include communication components, a display screen for realizing a human-machine interface, and an information collection device for collecting information about the surrounding environment, etc., wherein the communication components, the display screen, the information collection device and the electronic devices are communicatively connected.

[0135] According to embodiments of this disclosure, the electronic device can be integrated with the communication component, display screen, and information acquisition device, or it can be separately configured with the communication component, display screen, and information acquisition device.

[0136] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0137] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for time registration of a sensor of a vehicle, comprising: acquiring, from a data frame to be processed collected by a sensor to be calibrated in the vehicle, a point belonging to a target object as a point to be optimized; minimizing, in a world coordinate system, a spatial distance between the point to be optimized and a reference point of the target object to obtain a time compensation amount of the sensor to be calibrated; the spatial distance being determined based on the time compensation amount as a variable; correcting, based on the time compensation amount, a time of the sensor to be calibrated.

2. The method of claim 1, wherein, The minimizing, in a world coordinate system, a spatial distance between the point to be optimized and a reference point of the target object to obtain a time compensation amount of the sensor to be calibrated, comprises: determining a time correction value based on the time compensation amount to be optimized and a time stamp of the point to be optimized; the time stamp being a collection time of the point to be optimized; converting the point to be optimized to the world coordinate system to obtain a projection point based on a pose of the vehicle corresponding to the time correction value; optimizing the time compensation amount to be optimized by minimizing a spatial distance between the projection point and the reference point.

3. The method of claim 2, wherein, The determining a time correction value based on the time compensation amount to be optimized and a time stamp of the point to be optimized, comprises: accumulating the time compensation amount to be optimized to the time stamp to obtain the time correction value.

4. The method of claim 2, wherein, The determining a time correction value based on the time compensation amount to be optimized and a time stamp of the point to be optimized, comprises: determining an expression of the time compensation amount to be optimized based on a linear change model; the linear change model being used to represent a linear change of the time compensation amount to be optimized with time, and the linear change model including an optimization parameter; accumulating the expression of the time compensation amount to be optimized to the time stamp to obtain the time correction value.

5. The method of claim 2, wherein, The determining a time correction value based on the time compensation amount to be optimized and a time stamp of the point to be optimized, comprises: estimating the time compensation amount to be optimized by using Kalman filtering with the time compensation amount to be optimized as a hidden state; accumulating the time compensation amount to be optimized to the time stamp to obtain the time correction value.

6. The method of claim 5, wherein, The estimating the time compensation amount to be optimized by using Kalman filtering, comprises: constructing a key parameter; recursively updating the time compensation amount to be optimized by Kalman filtering based on the key parameter, and generating an observation value of the time compensation amount to be optimized under a criterion of minimizing a spatial distance between the point to be optimized and the reference point; wherein, the key parameter comprises: a process noise covariance used to simulate random disturbance of the time compensation amount to be optimized caused by a change in vehicle speed; a change rate of the time compensation amount to be optimized with time, wherein it is assumed that the change rate is constant in a short time, and the short time is less than a time threshold; an observation noise covariance used to quantify uncertainty of each optimization result of the time compensation amount to be optimized; a state transition assumption used to represent that an influence of a change in vehicle speed on the change rate of the time compensation amount to be optimized is constant in adjacent time intervals. 7.The method of any one of claims 2-6, further comprising determining the pose of the vehicle based on a method comprising: determining an interpolation interval of the pose based on a candidate value of the time compensation amount to be optimized; In the interpolation interval, a pose of the vehicle corresponding to the to-be-optimized time compensation amount is determined.

8. The method of claim 7, wherein, The determination of the pose of the vehicle corresponding to the to-be-optimized time compensation amount in the interpolation interval comprises: In the interpolation interval, linear interpolation is performed to obtain the pose of the vehicle corresponding to the to-be-optimized time compensation amount.

9. The method of claim 7, wherein, The determination of the pose of the vehicle corresponding to the to-be-optimized time compensation amount in the interpolation interval comprises: In a case where the curvature of the turning of the vehicle is greater than a curvature threshold, linear interpolation is performed on the known translation vector of the candidate value in the interpolation interval to obtain a translation matrix; Rotational interpolation is performed on the known rotation vector of the candidate value in the interpolation interval to obtain a rotation matrix; Based on the rotation matrix and the translation matrix, the pose of the vehicle corresponding to the to-be-optimized time compensation amount is determined.

10. The method of any one of claims 1-9, further comprising determining a reference point of the target object based on the following method: obtaining image data collected by a vision sensor; converting a detection box of the target object in the image data to a world coordinate system to obtain a reference point of the target object.

11. A time registration device of a vehicle sensor, comprising: an obtaining module configured to obtain, from a to-be-processed data frame collected by a to-be-calibrated sensor in a vehicle, a point belonging to a target object as a to-be-optimized point; a determining module configured to minimize a spatial distance between the to-be-optimized point and a reference point of the target object in a world coordinate system to obtain a time compensation amount of the to-be-calibrated sensor; the spatial distance is determined based on the time compensation amount as a variable; a correcting module configured to correct a time of the to-be-calibrated sensor based on the time compensation amount.

12. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

13. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-10.

14. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-10.

15. An autonomous vehicle comprising the electronic device of claim 12.

Images