Automatic driving fusion perception automatic calibration method

Abstract

The invention provides an automatic driving fusion perception automatic calibration method. The method comprises an internal reference calibration step; a data acquisition step; a data preprocessing step; a feature extraction step; a data fusion step; a calibration algorithm step and a calibration result verification step; wherein a feature fusion method is adopted in the data fusion step; comprising the following steps: performing feature extraction on data of each sensor to obtain a feature vector; a step of splicing the feature vectors to form a joint feature vector; and processing and fusing the joint feature vector by using a deep learning model to obtain fused feature representation. In the data fusion step, a feature fusion method is adopted, so that automatic calibration of automatic driving fusion perception is convenient.

Description

TECHNICAL FIELD

[0001] The application relates to an automatic driving fusion perception automatic calibration method. BACKGROUND

[0002] In an automatic driving system, the accuracy of sensor calibration is directly related to the perception ability and decision level of the vehicle. Traditional calibration methods mainly rely on manual operation, using special places, special markers, measuring instruments and tools, etc. However, these methods not only consume time and effort, but are also easily affected by environmental interference, personnel experience and other factors, resulting in inaccurate calibration results.

[0003] With the continuous development of computer vision, lidar, millimeter wave radar and other multi-sensor fusion perception technologies, an automatic calibration method based on fusion perception is designed to achieve high-precision automatic calibration by fusing data from multiple sensors. This technology can significantly improve the calibration efficiency and reduce human error, providing strong support for automatic driving.

[0004] Invention authorization announcement No. CN 114755662B discloses a road-vehicle fusion perception laser radar and GPS calibration method and device, which solves the problem that existing automatic driving vehicles cannot provide accurate geographic environment data and target position information. It is based on the mapping of vehicle-mounted laser radar point cloud coordinates in the GPS global coordinate system and the mapping of roadside laser radar point cloud coordinates in the GPS global coordinate system, realizing the mapping of vehicle-end perception data and roadside perception data in the GPS global coordinate system, and realizing the calibration of road-vehicle fusion perception data. This is very complex. SUMMARY

[0005] The purpose of the present application is to provide a simple automatic driving fusion perception automatic calibration method.

[0006] The technical scheme adopted by the present application to achieve its technical purpose is: an automatic driving fusion perception automatic calibration method, comprising:

[0007] S1, an intrinsic calibration step;

[0008] S2, a data acquisition step;

[0009] S3, a data preprocessing step;

[0010] S4, a feature extraction step;

[0011] S5, a data fusion step; in this step, a feature fusion method is used; comprising:

[0012] S5-1, feature extraction is performed on the data of each sensor to obtain a feature vector;

[0013] S5-2, concatenating the feature vectors to form a joint feature vector;

[0014] S5-3, processing and fusing the joint feature vector using a deep learning model to obtain a fused feature representation;

[0015] S6, calibration algorithm step;

[0016] S7, calibration result verification step.

[0017] Further, in the above automatic driving fusion perception automatic calibration method: the S6, calibration algorithm step includes:

[0018] S6-1, point cloud denoising step, which dynamically removes outliers with insufficient neighborhood density based on the mean and standard deviation of neighborhood distance;

[0019] S6-2, image enhancement step;

[0020] S6-3, nonlinear iteration step.

[0021] Further, in the above automatic driving fusion perception automatic calibration method: in the S6-1, point cloud denoising step, by setting the search radius and the minimum point number threshold in the neighborhood, the threshold range is set according to the point cloud coordinate attribute, and the target region data is directly intercepted; if the average distance of a point in the neighborhood exceeds 1-3 times the global mean standard deviation, it is determined as an outlier.

[0022] Further, in the above automatic driving fusion perception automatic calibration method: the S6-2, image enhancement step includes:

[0023] S6-2-1, image defogging step; in this step: based on the atmospheric scattering model, the fog imaging process is:

[0024] I

x

x

x

x

[0025] In the formula: I

x

[0026] J

x

[0027] t

x

[0028] A: atmospheric light value;

[0029] The defogging algorithm estimates the transmittance t

x

x

[0030] Retinex algorithm: by separating the brightness and reflection components of the image, enhancing the local contrast, and indirectly eliminating the fog effect;

[0031] Dark channel prior: using the statistical regularity of low intensity values in at least one color channel of natural scenes to directly estimate the transmission and the airlight;

[0032] Physical model optimization: constructing an energy function by constraints to minimize the error and recover the clear image.

[0033] Further, in the automatic driving fusion perception automatic calibration method described above, the image enhancement step S6-2 further comprises:

[0034] S6-2-2 image rain removal step; this step establishes the following model:

[0035] I

x

x

x

[0036] I

x

[0037] B

x

[0038] R

x

[0039] The image enhancement method in this step comprises:

[0040] Frequency domain filtering method: this method uses the high frequency characteristics of rain streaks in the frequency domain to design a band-stop filter to suppress rain line signals;

[0041] Sparse representation method: this method is based on the difference in sparsity of rain streaks and background under function operation;

[0042] Motion prior method: this method is based on the trajectory characteristics of dynamic raindrops and combines the optical flow method to remove rain line motion blur.

[0043] Further, in the automatic driving fusion perception automatic calibration method described above, the nonlinear iteration step S6-3 comprises:

[0044] S6-3-1 calculates the Jacobian matrix of the error function;

[0045] S6-3-2 algorithm solution; this step updates the parameters by a linear equation set; the linear equation set is composed of the Jacobian matrix of the error function and the parameter update amount;

[0046] S6-3-3 algorithm evaluates the new error value and adjusts the iteration step according to the error reduction.

[0047] Further, in the automatic driving fusion perception automatic calibration method described above, in the step S1 intrinsic calibration: a white light 1 meter*1 meter calibration board is used in a 20 meter long and 10 meter wide calibration site, and each sensor is calibrated separately to obtain accurate reference parameters.

[0048] Further, in the automatic driving fusion perception automatic calibration method described above: in the S2 data acquisition step, the time synchronization and space synchronization between sensors need to be ensured; time synchronization is realized through unified time stamp, and space synchronization converts and unifies the coordinate systems of different sensors.

[0049] Further, in the automatic driving fusion perception automatic calibration method described above: in the S3 data preprocessing step, it includes:

[0050] S3-1 data denoising step; this step removes noise and outliers in the data;

[0051] S3-2 data filtering step; this step smoothes the data;

[0052] S3-3 data enhancement step; this step increases the amount and diversity of data to improve the adaptive ability of the model.

[0053] Further, in the automatic driving fusion perception automatic calibration method described above: the S4 feature extraction step includes:

[0054] Camera data extraction, using image processing and computer vision technology to extract edge, corner, texture and other features;

[0055] Laser radar data extraction, using point cloud data processing and three-dimensional vision technology to extract point cloud density, normal, curvature and other features;

[0056] Millimeter wave radar data extraction, using signal processing and target detection technology to extract speed, distance, direction and other features.

[0057] The feature fusion method adopted in the data fusion step of the present application makes the automatic driving fusion perception automatic calibration convenient.

[0058] The automatic calibration of the present application mainly includes the following two aspects:

[0059] 1. Multi-sensor fusion collaborative calibration

[0060] Common sensors include cameras, laser radars, millimeter wave radars, etc., and the system fuses and processes the data collected by multiple sensors. The sensors are accurately calibrated by placing calibration boards on site to ensure that the data output by the sensors can accurately reflect the true situation of the surrounding environment, and to realize comprehensive perception of the external environment.

[0061] 2. Automatic calibration fusion algorithm

[0062] The data fusion algorithm combines the data characteristics of different sensors, applies data acquisition, feature extraction, data fusion, point cloud denoising, image enhancement, nonlinear iteration and other technologies for real-time data processing, and realizes the spatial alignment and time synchronization of the coordinate systems of multiple sensors.

[0063] By designing the above two technical methods, the problems of low calibration accuracy, dependence on manual operation and environmental interference of automatic driving integrated perception calibration are solved, and high-precision automatic calibration of automatic driving is realized.

[0064] The application will be further described in combination with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0065] FIG. 1 is a flowchart of the present application; Figure 1

[0066] FIG. 2 is a flowchart of embodiment 1 of the present application; Figure 2

[0067] FIG. 3 is a schematic diagram of the internal parameter calibration scene used in embodiment 1 of the present application. Figure 3 DETAILED DESCRIPTION As shown in

[0068] and Figure 1 , this embodiment is an automatic driving integrated perception automatic calibration method. The integrated perception automatic calibration method realizes automatic calibration of sensors by fusing data of multiple sensors. The method mainly includes the following steps: S1 internal parameter calibration step, S2 data acquisition step, S3 data preprocessing step, S4 feature extraction step, S5 data fusion step, S6 calibration algorithm step and S7 calibration result verification step as shown in Figure 2 . Figure 1

[0069] Specifically as shown in Figure 2 :

[0070] 1. Internal parameter calibration step

[0071] The internal parameter calibration step is the basis for ensuring the accuracy of sensor data. A white light surface 1 meter * 1 meter calibration board is used in a 20 meter long and 10 meter wide calibration site. Each sensor needs to be calibrated individually to obtain accurate reference parameters such as the focal length, principal point and distortion coefficient of the camera, and the internal parameters such as the scanning frequency and angle resolution of the laser radar. As shown in Figure 3 .

[0072] 2. Data acquisition step

[0073] ​​​This step collects data on the surrounding environment through various sensors installed on the vehicle. To ensure the accuracy and consistency of the data, time synchronization and spatial synchronization between sensors are required. Time synchronization is achieved through a unified timestamp, while spatial synchronization involves converting and unifying the coordinate systems of different sensors.

[0074] 3. Data preprocessing step

[0075] This step processes the collected raw data, mainly including data denoising, data filtering and data enhancement, to improve the quality and reliability of the data. Data denoising removes noise and outliers in the data, improving data accuracy; data filtering smooths the data, reducing fluctuations and uncertainties; data enhancement increases the amount and diversity of data to improve the adaptive ability of the model.

[0076] 4. Feature extraction step

[0077] This step extracts useful feature information from raw data for subsequent data fusion and parameter solving. Different feature extraction methods are used for different sensor data. Camera data extraction uses image processing and computer vision techniques to extract edge, corner, texture and other features; laser radar data extraction uses point cloud data processing and three-dimensional vision techniques to extract point cloud density, normal, curvature and other features; millimeter wave radar data extraction uses signal processing and target detection techniques to extract speed, distance, orientation and other features.

[0078] 5. Data fusion step

[0079] This step is the key step of the embodiment, which uses feature fusion method. It includes the following steps: S5-1 extracts features from each sensor data to obtain feature vectors. Read various sensor data features and obtain various sensor feature vectors through mathematical modeling operations.

[0080] S5-2 splices these feature vectors to form a joint feature vector. The various sensor feature vectors obtained by mathematical modeling operations in the previous step are spliced to form a joint feature vector.

[0081] S5-3 uses a deep learning model to process and fuse the joint feature vector to obtain a fused feature representation. The joint feature vector obtained in the previous step is processed by dimensionality reduction to fuse into a high-dimensional feature vector that can be automatically recognized by the system.

[0082] The method of data fusion mainly includes:

[0083] 5.1 Internal and external parameter modeling

[0084] (1) Visual camera: distortion model (radial distortion + tangential distortion) and intrinsic matrix, including focal length fx, fy, principal point coordinates cx, cy, where (u, v) is the pixel coordinate, (X, Y, Z) is the three-dimensional point coordinate, distortion coefficients k1, k2, p1, p2;

[0085] The mapping relationship between the three-dimensional point and the imaging plane is established, and the mathematical expression is:

[0086]

[0087] D = [k1, k2, p1, p2]

[0088] (2) Laser radar: extrinsic matrix T = [R / t] Optimize the rotation matrix R and translation vector t.

[0089] 5.2 Cross-modal data alignment

[0090] (1) Time synchronization: Based on PTP protocol, the clock is aligned to the level of microsecond, and the timing deviation is controlled within ±5ms;

[0091] (2) Spatial alignment: The point cloud is mapped to the image plane, and the projection error is iteratively optimized by algorithm.

[0092] 5.3 Feature fusion

[0093] Before fusion, the features are reduced in dimension to reduce redundant information, and the features of different modalities and different sources of visual camera and laser radar are connected and combined to form a high-dimensional feature vector, while retaining as much original data variance as possible for subsequent model training or prediction.

[0094] 6. The calibration algorithm method mainly includes

[0095] S6-1 Point cloud denoising

[0096] In this embodiment, the point cloud denoising is based on the mean and standard deviation of the neighborhood distance, and the outlying points with insufficient neighborhood density are removed.

[0097] By setting the search radius and the minimum point number threshold in the neighborhood, the threshold range is set according to the point cloud coordinate attribute, and the target area data is directly intercepted. If the average distance of a point in the neighborhood exceeds 1-3 times the global mean standard deviation, it is determined as an outlier.

[0098] S6-2 Image enhancement mainly includes:

[0099] S6-2-1 Image defogging; The core goal of image defogging is to eliminate the interference of fog and haze on the image and restore the true color and details of the scene.

[0100] Based on the atmospheric scattering model, the model converts the foggy imaging process as:

[0101] I(x) = J(x)t(x) + A(1 - t(x))

[0102] I(x): Observed hazy image;

[0103] J(x): Clear image to be recovered;

[0104] t(x): Transmittance (degree of attenuation of light propagation in fog), negatively correlated with scene depth;

[0105] A: Atmospheric light (global intensity of background light).

[0106] Dehazing algorithms estimate transmittance t(x) and atmospheric light A to recover the original clear image J(x).

[0107] Image enhancement methods mainly include:

[0108] a) Retinex algorithm: Enhance local contrast by separating image brightness and reflection components, indirectly eliminate fog effect;

[0109] b) Dark channel prior: Use the statistical law of low brightness values in at least one color channel of natural scenes to directly estimate transmittance and atmospheric light;

[0110] c) Physical model optimization: Construct energy function by constraint conditions (such as transmittance smoothness, color consistency), minimize error to recover clear image.

[0111] S6-2-2 Image Rain Removal:

[0112] Image rain removal aims to eliminate the interference of rain on images and recover clear background content. The core is to separate rain streaks (raindrops, rain lines) from background signals and establish the following model:

[0113] I(x) = B(x) + R(x)

[0114] I(x): Rainy image;

[0115] B(x): Background image to be recovered;

[0116] R(x): Rain streak component (including raindrop scattered light, motion blur effect, etc.).

[0117] By estimating rain streak R(x), the background B(x) is separated from the observed image I(x), and the blur, noise and brightness distortion caused by rain streaks are suppressed.

[0118] Image enhancement methods mainly include:

[0119] a) Frequency domain filtering: Use the high frequency characteristics of rain streaks in the frequency domain to design a band-stop filter to suppress rain line signals.

[0120] b) Sparse representation: based on the sparsity difference of rain streaks and background under function operation.

[0121] c) Motion prior: for the trajectory characteristics of dynamic raindrops, combined with the optical flow method to remove rain line motion blur.

[0122] S6-3 Nonlinear iteration

[0123] Nonlinear least squares optimization algorithm, used to solve nonlinear minimization problems, gradually adjusts parameters through iterative method.

[0124] Iteration process: in each iteration, the algorithm first calculates the Jacobian matrix of the error function, then the algorithm solves a linear equation to update the parameters. This linear equation is composed of the Jacobian matrix of the error function and the parameter update amount. After updating the parameters, the algorithm evaluates the new error value, and adjusts the iteration step according to the error reduction.

[0125] S7. Calibration result verification

[0126] Automatic calibration needs to verify the calibration parameters obtained by the calibration algorithm. The verification method uses the extracted feature vectors for comparison test, and carries out field test in the actual scene. Through comparison test and field test, the accuracy and reliability of the calibration parameters are verified. If the calibration result is not accurate or reliable, the data acquisition, data preprocessing, feature extraction, data fusion and calibration algorithm steps need to be re-performed until the satisfactory calibration result is output.

[0127] The above calibration method flow is the flow of the initial calibration. When the calibration starts, automatic calibration initialization can also be performed.

[0128] If it is not the initial calibration, after initialization, the current abnormal state information is collected, and the current system abnormal state information is read. After the two steps, the feature extraction, data fusion and calibration algorithm steps are performed.

[0129] The above abnormal data item processing is to compare and analyze the abnormal data and the calibration result data.

[0130] Data acquisition and data preprocessing are performed for initial calibration. The collection of current abnormal state information and abnormal data item processing are performed when the system is put into use and abnormal conditions occur.

[0131] When non-initial calibration is executed, if the calibration result is not accurate or reliable, return to the two steps of collecting current abnormal state information and reading current system abnormal state information, and then perform the feature extraction step, data fusion step and calibration algorithm step, until the satisfactory calibration result is output.

Claims

1. An automatic calibration method for autonomous driving based on fused perception, characterized in that: include: S1. Internal parameter calibration steps; S2, Data Acquisition Steps; S3, Data preprocessing steps; S4, Feature extraction step; S5, Data Fusion Step; This step employs a feature fusion method; including: S5-1, extract features from the data of each sensor to obtain a feature vector; S5-2, concatenate these feature vectors to form a joint feature vector; S5-3 uses a deep learning model to process and fuse the joint feature vectors to obtain the fused feature representation; S6. Calibration algorithm steps; S7. Verification steps for calibration results.

2. The automatic calibration method for fusion perception in autonomous driving according to claim 1, characterized in that: The S6 calibration algorithm steps include: The S6-1 point cloud denoising step dynamically removes outliers with insufficient neighborhood density based on the mean and standard deviation of the neighborhood distance. S6-2 Image Enhancement Steps; S6-3 Nonlinear Iterative Steps.

3. The automatic calibration method for fusion perception in autonomous driving according to claim 2, characterized in that: In the S6-1 point cloud denoising step, by setting the search radius and the minimum number of points in the neighborhood threshold, and setting the threshold range according to the point cloud coordinate attributes, the target area data is directly extracted; if the average neighborhood distance of a point exceeds 1-3 times the global mean standard deviation, it is determined to be an outlier.

4. The automatic calibration method for fusion perception in autonomous driving according to claim 2, characterized in that: The S6-2 image enhancement step includes: S6-2-1 Image Dehazing Step; In this step: Based on the atmospheric scattering model, the foggy imaging process is as follows: I[x]=J[x]t[x]+A[1-t[x]]; In the formula: I[x] is the observed foggy image; J[x] represents the clear, fog-free image to be recovered; t[x] is the transmittance; A: Atmospheric light value; The dehazing algorithm estimates the transmittance t[x] and atmospheric light value A to inversely deduce the original haze-free image J[x]. The image enhancement methods used in this step include: Retinex algorithm: By separating the luminance and reflection components of an image, it enhances local contrast and indirectly eliminates the fog effect; Dark channel prior: Utilize the statistical regularity of low brightness values ​​in at least one color channel in a natural scene to directly estimate transmittance and atmospheric light; Physical model optimization: Construct an energy function through constraints to minimize errors and recover a clear image.

5. The automatic calibration method for fusion perception in autonomous driving according to claim 4, characterized in that: The S6-2 image enhancement step further includes: S6-2-2 Image Deraining Steps; This step establishes the following model: I[x]=B[x]+R[x]; I[x] refers to the image containing rain; B[x] refers to the background image to be recovered; R[x] refers to the rain streak component; The image enhancement methods used in this step include: Frequency domain filtering method: This method utilizes the high-frequency characteristics of rain streaks in the frequency domain to design a band-stop filter to suppress rain streak signals; Sparse representation method: This method is based on the sparsity difference between rain patterns and background under function operation; Motion Prior Method: This method targets the trajectory characteristics of dynamic raindrops and combines optical flow to remove motion blur of rain lines.

6. The automatic calibration method for fusion perception in autonomous driving according to claim 2, characterized in that: The S6-3 nonlinear iteration step includes: S6-3-1 Calculate the Jacobian matrix of the error function; The S6-3-2 algorithm is used to solve this problem; a system of linear equations is used to update the parameters in this step; this system of linear equations consists of the Jacobian matrix of the error function and the parameter update amount; The S6-3-3 algorithm evaluates the new error value and adjusts the iteration step size based on the reduction of the error.

7. The automatic calibration method for fusion perception in autonomous driving according to any one of claims 1 to 6, characterized in that: In step S1, the intrinsic parameter calibration is performed using a white, smooth 1m x 1m calibration board in a 20m long and 10m wide calibration area. Each sensor is individually calibrated to obtain accurate reference parameters.

8. The automatic calibration method for fusion perception in autonomous driving according to any one of claims 1 to 6, characterized in that: In the S2 data acquisition step, it is necessary to ensure time synchronization and spatial synchronization between sensors; time synchronization is achieved through a unified timestamp, while spatial synchronization involves transforming and unifying the coordinate systems of different sensors.

9. The automatic calibration method for fusion perception in autonomous driving according to any one of claims 1 to 6, characterized in that: The S3 data preprocessing step includes: S3-1 Data Denoising Step; This step removes noise and outliers from the data; S3-2 Data Filtering Step; This step smooths the data. S3-3 Data Augmentation Step; This step improves the model's adaptability by increasing the amount and diversity of data.

10. The automatic calibration method for fusion perception in autonomous driving according to any one of claims 1 to 6, characterized in that: The S4 feature extraction step includes: Camera data extraction uses image processing and computer vision techniques to extract features such as edges, corners, and textures. LiDAR data extraction uses point cloud data processing and 3D vision technology to extract features such as point cloud density, normals, and curvature. Millimeter-wave radar data extraction uses signal processing and target detection techniques to extract features such as velocity, distance, and azimuth.

Images