A Multi-Sensor Spatiotemporal Cooperative Calibration Method for Camera and Millimeter-Wave Radar Fusion Perception

Abstract

The present invention discloses a multi-sensor spatio-temporal collaborative calibration method for camera and millimeter-wave radar fusion perception. First, spatial calibration is performed on the millimeter-wave radar to determine its world coordinates. Then, the internal and external parameters of the roadside camera are determined, and according to the transitivity of continuous coordinate transformation, the conversion relationship between the millimeter-wave radar coordinate system and the pixel coordinate system is determined. Pulse signals are used to correct the time of sensors in the local area network. At the same time, according to the data confidence index, a method based on simple linear interpolation and Bayesian neural network is used to predict the equivalent data frame. This method takes into account the situation when pixel coordinates and image coordinates are irregularly arranged, greatly improving the conversion accuracy of target positions in different coordinate systems. At the same time, according to different data requirements, the reliability of the equivalent data frame is introduced, and the calibration process under different accuracy requirements is completed, providing a decision-making basis for the refined management and control of highways.

Description

Technical Field

[0001] The present invention relates to a multi-sensor spatio-temporal calibration method, and in particular to a multi-sensor spatio-temporal collaborative calibration method for camera and millimeter-wave radar fusion perception. Background Art

[0002] Video surveillance, as an important part of security, is the main front-end device of the Internet of Things. With the rapid development of social economy, intelligent transportation has become one of the fields with the greatest demand in the video surveillance market, and video devices provide guarantee for traffic safety. However, with the increasingly complex road traffic environment, the requirements for road traffic management are also getting higher and higher, and the existing highway management means can no longer meet the needs of intelligent management and diversified travel services. To achieve refined control of highways, there is an urgent need for multi-source traffic information holographic perception technology and high-quality traffic data extraction technology.

[0003] The time and space calibration of sensors is the basis for realizing multi-source data fusion. Radar information and image information need to be unified in the same spatio-temporal dimension for data fusion calculation. Usually, taking the vision system as the benchmark, the two-dimensional plane points of the detected targets in the millimeter-wave radar coordinate system can be converted to the corresponding pixel coordinate system of the camera through coordinate transformation to achieve the spatial synchronization of the two. At the same time, in the actual road environment, the clock sources and data sampling periods of the camera and the millimeter-wave radar are different, and uncontrollable time delays may occur in processes such as data transmission and exposure time within the perception system. It is necessary to convert the data information of the sensors at different times to the same moment to generate equivalent information to complete the time calibration. Summary of the Invention

[0004] Object of the Invention: The present invention proposes a multi-sensor spatio-temporal collaborative calibration method for camera and millimeter-wave radar fusion perception, which effectively solves the spatio-temporal asynchronization problem of detected targets in multi-source roadside devices.

[0005] Technical Solution: To achieve the object of the present invention, the technical solution adopted by the present invention is: A multi-sensor spatio-temporal collaborative calibration method for camera and millimeter-wave radar fusion perception, including two components: sensor spatial calibration and time calibration; fusing and comprehensively describing the information of the target in a unified reference frame;

[0006] The spatial calibration includes the conversion between the millimeter-wave radar coordinate system, the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system; the conversion relationships between each coordinate system include:

[0007] 1) Conversion between the millimeter-wave radar coordinate system and the world coordinate system; 2) Conversion between the world coordinate system and the camera coordinate system;

[0008] 3) Conversion between the camera coordinate system and the image coordinate system; 4) Conversion between the image coordinate system and the pixel coordinate system; 5) When the image is distorted, perform distortion correction on the image, including: radial distortion correction and tangential distortion correction of the image;

[0009] Time calibration includes time alignment of sensors and data frame synchronization prediction;

[0010] Unify the clock source and eliminate the clock drift to ensure the initial time alignment of sensors; Index the front and back data frames through a sliding time window. According to the final confidence requirement for data, use interpolation and build a Bayesian neural network to achieve the prediction of equivalent data frames.

[0011] Furthermore, the conversion relationships between the coordinate systems are specifically as follows:

[0012] The conversion relationship between the millimeter-wave radar coordinate system and the world coordinate system includes: spatial calibration of the millimeter-wave radar, that is, the conversion of a two-dimensional plane coordinate system, which is jointly affected by translation and rotation;

[0013] The conversion between the world coordinate system and the camera coordinate system includes: the transition from the world coordinate system to the camera coordinate system is a three-dimensional coordinate conversion. The rotation under different coordinate axes corresponds to different rotation matrices. The final rotation matrix is the dot product result of three rotation matrices. This rotation matrix and the translation matrix jointly form the external parameter matrix of the camera spatial calibration;

[0014] The conversion from the camera coordinate system to the image coordinate system is based on the projection principle;

[0015] The conversion between the image coordinate system and the pixel coordinate system includes: the unit of the image coordinate system is mm, and the unit of the pixel coordinate system is pixel; the origin of the image coordinate system is located at the center of the imaging plane, and the origin of the pixel coordinate system is located at the upper left corner of the imaging plane; at the same time, the pixels have regular rectangular and non-rectangular shapes, that is, there are two cases of an angle and no angle between the image coordinate system and the pixel coordinate system; the conversion from the image coordinate system to the pixel coordinate system is based on the conversion of physical unit lengths and the translation of coordinate systems;

[0016] The conversion between the camera coordinate system and the image coordinate system, and the conversion between the image coordinate system and the pixel coordinate system. All the parameters to be solved in the conversion process constitute the internal parameters of the camera spatial calibration.

[0017] Furthermore, the time alignment method is to unify the clock source through a pulse generator. All sensors are triggered by this pulse, and each trigger corrects its own clock to align the timestamps of each sensor.

[0018] Furthermore, the pulse generator calibrates the timestamp, uses the GPS clock as the reference clock source, and uses the sensor synchronization timing protocol to send the timestamp synchronization request to the camera and the millimeter-wave radar, achieving the time synchronization of all sensors in the local area network environment.

[0019] Furthermore, for the sensor data frame indexing method, since the sampling frequencies of the camera and the millimeter-wave radar are different, the remaining sensor data within the time window range of the reference sensor data acquisition moment is indexed.

[0020] Furthermore, for the sensor data frame prediction method, the sensor equivalent data frame is predicted according to the prediction accuracy requirement, including:

[0021] When the prediction accuracy requirement or the data dimension is less than a certain value, the linear interpolation calculation method is used to combine the indexed data; when the prediction accuracy requirement is not less than a certain value or a reliability evaluation needs to be performed on the equivalent data, the method of building a Bayesian neural network is used to combine probability modeling and neural network to obtain the equivalent data with confidence.

[0022] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0023] The present invention proposes a multi-sensor spatio-temporal collaborative calibration method for camera and millimeter-wave radar fusion perception, taking time calibration into account, and completing the conversion relationship between the image coordinate system and the pixel coordinate system, that is, not only considering the corresponding relationship between the image coordinate system and the pixel coordinate system under ordinary non-included angle conditions, but also considering the situation where there is an included angle between the image coordinate system and the pixel coordinate system, improving the accuracy of the spatial calibration of the roadside camera and ensuring the reliability of perception and positioning in the vehicle-road collaborative field. According to different data reliabilities, this method uses simple interpolation and Bayesian neural network methods to predict the sensor equivalent frame data respectively, completing the calibration process under different accuracy requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is the architecture diagram of the multi-sensor spatio-temporal collaborative calibration method of the present invention;

[0025] Figure 2 is the schematic diagram of the spatial conversion relationship of the coordinate system;

[0026] Figure 3 is the composition diagram of the multi-source sensor time calibration;

[0027] Figure 4 is the schematic diagram of the relative conversion relationship between the millimeter-wave radar coordinate system and the world coordinate system;

[0028] Figure 5It is a schematic diagram of the relative conversion relationship between the world coordinate system and the camera coordinate system;

[0029] Figure 6 It is a schematic diagram of the relative conversion relationship between the camera coordinate system and the image coordinate system;

[0030] Figure 7 It is a schematic diagram of the relative conversion relationship between the image coordinate system and the pixel coordinate system without an included angle;

[0031] Figure 8 It is a schematic diagram of the relative conversion relationship between the image coordinate system and the pixel coordinate system with an included angle;

[0032] Figure 9 It is a schematic diagram of the radial distortion and tangential distortion of the roadside camera;

[0033] Figure 10 It is a schematic diagram of the data transceiver process between the roadside camera and the millimeter-wave radar;

[0034] Figure 11 It is a schematic diagram of the performance of different data sampling frequencies of the roadside camera and the millimeter-wave radar. Detailed implementation method

[0035] In order to clearly discuss the technical means and requirements designed by the present invention, the present invention will be further elaborated in detail below in combination with the accompanying drawings and specific implementation methods:

[0036] The multi-sensor spatio-temporal collaborative calibration method for camera and millimeter-wave radar fusion perception described in the present invention, the space calibration step involves the conversion between the millimeter-wave radar coordinate system, the world coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system. The time calibration includes the time alignment of the sensors and the data frame synchronization prediction. Figure 2 Describes the conversion relationship between each coordinate system, including:

[0037] 1) Conversion between the millimeter-wave radar coordinate system and the world coordinate system;

[0038] 2) Conversion between the world coordinate system and the camera coordinate system;

[0039] 3) Conversion between the camera coordinate system and the image coordinate system;

[0040] 4) Conversion between the image coordinate system and the pixel coordinate system.

[0041] Among them, the conversion from the camera coordinate system to the pixel coordinate system belongs to the internal parameter calibration part of the camera, involving perspective transformation and translation transformation; the conversion between the camera coordinate system and the world coordinate system belongs to the external parameter calibration part of the camera, involving rigid body transformation; the conversion between the millimeter-wave radar coordinate system and the world coordinate system belongs to the radar calibration part, also involving rigid body transformation. At the same time, when the image is distorted, it is necessary to correct the distortion of the image.

[0042] Figure 3 It shows the time calibration part of the cooperation between the camera and the millimeter-wave radar. The unified clock source and the elimination of clock drift ensure the initial time alignment of the sensors; the front and back data frames are indexed through a sliding time window, and according to the final confidence requirement of the data, interpolation and the construction of a Bayesian neural network are used to achieve the prediction of equivalent data frames.

[0043] Figure 4 It is a schematic diagram of the spatial calibration of the millimeter-wave radar in the present invention, specifically referring to the conversion process between the millimeter-wave radar coordinate system and the world coordinate system.

[0044] The millimeter-wave radar can obtain the x and y coordinate information of the target, but there is no z-direction coordinate information. Therefore, the conversion of the millimeter-wave radar coordinate system to the world coordinate system can be equivalent to the conversion of a two-dimensional plane coordinate system. Let Figure 4 In it, O w is the origin of the world coordinate system, O m is the origin of the millimeter-wave radar coordinate system, L and d are the lateral offset and longitudinal offset between the two coordinate systems respectively, θ is the lateral angle between the two coordinate systems, P is any point in the real world, and the mutual conversion between the two coordinate systems can be achieved through translation and rotation. The specific conversion relationship is represented by formula (1):

[0045]

[0046] The conversion matrix in formula (1) consists of two parts, the translation matrix brought by the distance and the rotation matrix brought by the angle. When installing the millimeter-wave radar, it should be ensured that its horizontal angle and yaw angle are 0 as much as possible, otherwise the conversion matrix will become more complex.

[0047] Figure 5 It is a schematic diagram of the external parameter calibration of the camera in the present invention, specifically referring to the conversion process between the world coordinate system and the camera coordinate system. O c is the origin of the camera coordinate system, O w is the origin of the world coordinate system, R and T are the rotation matrix and translation matrix respectively, and P is any point in the real world.

[0048] Compared with Figure 4 the two-dimensional plane conversion between the millimeter-wave radar coordinate system and the world coordinate system in it, the conversion between the world coordinate system and the camera coordinate system belongs to a three-dimensional space conversion, that is, there are translation and rotation processes in all three directions of the coordinate system. When rotating around different coordinate axes, the corresponding rotation matrices are obtained.

[0049] When rotating θ around the Z axis, this process is consistent with the above Figure 3 conversion relationship, as shown in formula (2):

[0050]

[0051] When rotating about the X-axis the rotation matrix is as shown in Equation (3):

[0052]

[0053] When rotating about the Y-axis by ω, the rotation matrix is as shown in Equation (4):

[0054]

[0055] Multiplying the three rotation matrices gives the final rotation matrix, i.e., R = R1·R2·R3;

[0056] Considering the translation matrix at the same time, the transformation relationship from the world coordinate system to the camera coordinate system is obtained, specifically as shown in Equation (5):

[0057]

[0058] In Equation (5), R is a 3×3 rotation matrix and T is a 3×1 translation matrix. Representing Equation (5) in homogeneous coordinates is as shown in Equation (6), and M is the desired external camera parameter matrix.

[0059]

[0060] Figure 6 is a schematic diagram of the transformation relationship between the camera coordinate system and the image coordinate system in the present invention, mainly based on the projection principle. O c is the origin of the camera coordinate system, 8 is the origin of the image coordinate system, P is an arbitrary point in the real world, p is the projection point, and f is the focal length of the camera.

[0061] According to the perspective projection principle, the coordinate relationship formula between P and p can be obtained, as shown in Equation (7):

[0062]

[0063] Integrating Equation (7) into the matrix form of homogeneous coordinates is as shown in Equation (8):

[0064]

[0065] Figure 7 and Figure 8 are respectively schematic diagrams of the transformation relationship between the image coordinate system and the pixel coordinate system for the actual pixel rectangular arrangement and non-rectangular arrangement in the present invention. The pixel coordinate system and the image coordinate system are both on the imaging plane, only their respective origins and measurement units are different. The unit of the image coordinate system is mm, which belongs to a physical unit; while the unit of the pixel coordinate system is pixel. Figure 6 The origin O of the pixel coordinate systemuv It is located at the upper left corner of the image imaging plane. The u-axis and v-axis are parallel to the x-axis and y-axis of the image coordinate system respectively. The origin 8(u0, v0) of the image coordinate system is located at the midpoint of the imaging plane. dx and dy respectively represent the physical sizes of each pixel in the x and y directions of the image plane.

[0066] According to Figure 7 the geometric relationship shown, the conversion relationship between the pixel coordinate system and the image coordinate system is as shown in formula (9):

[0067]

[0068] Converted to homogeneous coordinates as shown in formula (10):

[0069]

[0070] Similarly, when the pixels are not arranged in a rectangle, that is, when there is an angle θ between the pixel coordinate system and the image coordinate system ( Figure 8 ), its conversion relationship is as shown in formula (11):

[0071]

[0072] Converted to homogeneous coordinates as shown in formula (12):

[0073]

[0074] Combining formula (6), formula (8), and formula (10), the conversion relationship between the pixel coordinate system and the world coordinate system under rectangular conditions can be obtained as shown in formula (13):

[0075]

[0076] Formula (13) can be simplified to an expression (14) containing the internal and external parameters of the camera. M1 is the internal parameter matrix of the camera at this time, and M2 is the external parameter matrix.

[0077]

[0078] Similarly, combining formula (6), formula (8), and formula (12), the conversion relationship formula (15) between the pixel coordinate system and the world coordinate system when there is a certain angle between the pixel coordinate system and the image coordinate system can be obtained. M1 is the internal parameter matrix of the camera at this time, and M2 is the external parameter matrix.

[0079]

[0080] After obtaining the conversion relationships between the millimeter-wave radar coordinate system and the world coordinate system, and between the world coordinate system and the pixel coordinate system respectively, through formula integration, the conversion relationship between the pixel coordinate system and the millimeter-wave radar coordinate system can be obtained as shown in formula (16):

[0081]

[0082] In addition, when the image captured by the camera is distorted, relevant distortion parameters need to be obtained through calibration, and the non-linear correction is performed on each image coordinate (x, y) obtained by the camera model by using the distortion parameters, so that the corrected image more conforms to the spatial image to be captured. Figure 9 FIG. is a schematic diagram of radial distortion and tangential distortion, where dr and dt are radial distortion and tangential distortion respectively.

[0083] The radial distortion correction can be represented by formula (17):

[0084]

[0085] x 2 + y 2 = r 2 , where j1, k2, and k3 are radial distortion parameters, x and y are ideal undistorted coordinates, and x kr , y kr are the coordinates after introducing the radial distortion parameters.

[0086] The tangential distortion correction can be represented by formula (18):

[0087]

[0088] x 2 + y 2 = r 2 , where p1 and p2 are tangential distortion parameters, x and y are ideal undistorted coordinates, and x kt , y kt are the coordinates after introducing the tangential distortion parameters.

[0089] By integrating the radial distortion correction and the tangential distortion correction, the final correction formula (19) acting on the real image can be obtained.

[0090]

[0091] Through the above conversions of each coordinate system and the distortion correction of the image, the spatial synchronization of the detection target in each sensor is achieved. Next, through the attached drawings and embodiments, the multi-sensor time calibration technical solution of the present invention will be further described in detail.

[0092] In the actual road environment, each sensor is based on its own clock source. The existence of clock drifts in different clock sources causes the time of each sensor to deviate. For a vehicle moving at high speed, the instantaneous time difference may lead to the inability to match target information. By using a pulse generator to unify the clock sources, all sensors are triggered by this pulse, and each sensor corrects its own clock every time it is triggered. In this way, the cumulative error of the clock source can be eliminated, so that the timestamps of each sensor are aligned. Since GPS comes with a second pulse generator, the GPS clock can be used as the reference clock source, and the TPSN sensor synchronization timing protocol can be used to send timestamp synchronization requests to the camera and millimeter-wave radar to achieve the time synchronization of all sensors in the local area network environment.

[0093] The time alignment at the hardware level only ensures that there is no cumulative drift in the time difference. However, the sampling moments and frequencies of each sensor are not the same. The existence of trigger delay, transmission delay, exposure delay, etc. results in the inability to match the data frames at the same moment. Figure 10 The receiving and transmitting processes of camera and millimeter-wave radar data are respectively shown. The camera belongs to passive data reception. The received data needs to be converted from optical and electrical signals to digital signals, and at the same time, the digital signals are processed by DSP and encoded into the image format and video format supported by the camera; while the millimeter-wave radar relies on the way of emitting electromagnetic waves and belongs to active reception to detect target information. The receiver processes the echo signal and mixes the high-frequency radio frequency signal into an intermediate-frequency signal; at the same time, the intermediate-frequency signal undergoes envelope detection to generate a video signal; finally, the signal processor processes the digital signal and displays the target information detected by the radar on the screen in a visual way.

[0094] Figure 11 The difference in the sampling frequencies of the camera and millimeter-wave radar is shown under the same time axis. Solving the synchronization problem of data frames consists of two steps: Based on the road environment data collected by the environmental sensing device, first, based on the time point detected by the sensor with a higher weight assignment, the data of the previous and next frames of the remaining sensors at this moment are indexed. The specific indexing method is the sliding time window method. Set an initial threshold to index the data of other sensors within the time window range of the reference sensor. The indexed data should be the data displayed after being processed inside each sensor. When the data of other sensors within the time window range are indexed, these data need to be combined to obtain the equivalent data at this moment.

[0095] When the requirement for time calibration accuracy is not high, the method of linear interpolation is used to simply combine the data of the previous and next frames. Assume that the data of the previous and next frames of the remaining sensors indexed within the time window range are y1 and y2 respectively, and the corresponding timestamps are t1 and t2, and the timestamp of the reference sensor is t c (t1 < t cIf <t2), the equivalent data y of the non-reference sensor at this moment can be obtained from the formula (20).

[0096]

[0097] On the contrary, when the time calibration accuracy requirement is high, the neural network algorithm is used to combine the data. Specifically, based on the road environment data collected by the environmental sensing device, by assigning dynamic weights to each sensor, the reliability of the data of each sensor is evaluated. The weights follow a Gaussian distribution with a mean of u and a variance of δ, and the weights of each sensor follow different Gaussian distributions. By building a Bayesian neural network (BNN), not only the probability modeling and the neural network are combined, but also the predicted value of the equivalent information and the confidence of the prediction result can be obtained. Suppose the network weight parameter of the BNN is w, specifically referring to the weights of the sensed data of each sensor, p(w) is the prior distribution of the parameters. Given the training set D=(X,Y), X refers to the sensed data of the remaining sensors except the reference sensor; Y refers to the detection data of the reference sensor, then the probability model is as follows:

[0098] P(Y * |X * ,D) = ∫P(Y * |X * ,w)P(w|D)dw (21)

[0099] However, the posterior distribution p(w,D) is difficult to calculate. Therefore, the variational method is used to approximate the true posterior distribution p(w,D) with a distribution q(w|θ) controlled by a set of parameters θ. The optimization problem of finding θ can be achieved by minimizing the KL divergence of the two distributions, written in the form of an objective function:

[0100] F(D,θ) = D KL [q(w|θ)||P(w)] - E q(w|θ) [logP(D|w)] (22)

[0101] The first term of the above objective function describes the degree of fit between the weights and the prior, and the second term describes the degree of fit to the samples. According to the Monte Carlo method, the objective function can be approximated as:

[0102]

[0103] where w (i) is the weight sampling when processing the i-th data point. Using mini-batch gradient descent to calculate the model average can obtain P(Y * |X * ,D). The final output layer obtains the equivalent data including the confidence, and combines the equivalent information with the reference sensor information to achieve the synchronization of the data frame.

[0104] The above are only some embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A multi-sensor spatio-temporal collaborative calibration method for camera and millimeter-wave radar fusion perception, characterized in that: The method includes two components: sensor spatial calibration and time calibration; fusing and comprehensively describing the information of the target in a unified reference system framework; Spatial calibration includes the conversion between the millimeter-wave radar coordinate system, the world coordinate system, the camera coordinate system, the image coordinate system, and the pixel coordinate system; The conversion relationships between each coordinate system include: 1) The conversion between the millimeter-wave radar coordinate system and the world coordinate system; 2) The conversion between the world coordinate system and the camera coordinate system; 3) The conversion between the camera coordinate system and the image coordinate system; 4) The conversion between the image coordinate system and the pixel coordinate system; 5) When the image is distorted, perform distortion correction on the image, including: radial distortion correction and tangential distortion correction of the image; Time calibration includes the time alignment of sensors and the prediction of data frame synchronization; A unified clock source and eliminating clock drift ensure the initial time alignment of sensors; Index the front and rear data frames through a sliding time window. According to whether the confidence requirement of the data is ultimately needed, use interpolation and build a Bayesian neural network to achieve the prediction of equivalent data frames. The process is as follows: Based on the road environment data collected by the environmental sensing device, first, based on the moment detected by the sensor with a weight assignment exceeding the threshold, index the data of the other sensors in the two frames before and after this moment. The specific indexing method is the sliding time window method; Set an initial threshold to index the data of other sensors within the time window of the reference sensor. The indexed data is the data displayed after being processed by each sensor internally; When indexing the data of other sensors within the time window, combine these data to obtain the equivalent data at this moment; Predict the sensor equivalent data frames according to the prediction accuracy requirements, including: (1) When the prediction accuracy requirement or the data dimension is less than the set threshold, use the linear interpolation calculation method to combine the indexed data; Let the front and back frame data of other sensors indexed within the time window be y1 and y2 respectively, with corresponding timestamps t1 and t2, and the timestamp of the reference sensor be t c , t1 < t c < t2, then the equivalent data y of the non-reference sensor at this moment is obtained from formula (20): (2) When the prediction accuracy requirement is not less than the set threshold or a reliability evaluation of the equivalent data is required, use the method of building a Bayesian neural network to combine probability modeling and neural network to obtain equivalent data containing confidence, that is, obtain the predicted value of the equivalent data and the confidence of the prediction result; Based on the road environment data collected by the environmental sensing device, perform a reliability evaluation on the data of each sensor by assigning dynamic weights to each sensor. The weights follow a Gaussian distribution with a mean of μ and a variance of δ, and the weights of each sensor follow different Gaussian distributions; The network weight parameters of the BNN are the weights of the sensed data of each sensor. The training set D of the BNN network = (X, Y), where X refers to the sensed data of other sensors except the reference sensor, and Y refers to the detection data of the reference sensor. The output layer of the BNN network obtains equivalent data containing confidence, and combines the equivalent information with the reference sensor information to achieve data frame synchronization.

2. The multi-sensor spatio-temporal collaborative calibration method according to claim 1, wherein: The conversion relationships between each coordinate system are specifically as follows: The conversion relationship between the millimeter-wave radar coordinate system and the world coordinate system includes: the spatial calibration of the millimeter-wave radar, that is, the conversion of the two-dimensional plane coordinate system, which is jointly affected by translation and rotation; The conversion between the world coordinate system and the camera coordinate system includes: the transition from the world coordinate system to the camera coordinate system is a three-dimensional coordinate conversion. The rotation under different coordinate axes corresponds to different rotation matrices, and the final rotation matrix is the dot product result of three rotation matrices. This rotation matrix and the translation matrix together form the external parameter matrix for camera space calibration; The conversion from the camera coordinate system to the image coordinate system is based on the projection principle; The conversion between the image coordinate system and the pixel coordinate system includes: the unit of the image coordinate system is mm, and the unit of the pixel coordinate system is pixel; the origin of the image coordinate system is located at the center of the imaging plane, and the origin of the pixel coordinate system is located at the upper left corner of the imaging plane; at the same time, the pixels have regular rectangular and non-rectangular shapes, that is, there are two cases of an angle and a non-angle between the image coordinate system and the pixel coordinate system; the conversion from the image coordinate system to the pixel coordinate system is based on the conversion of the physical unit length and the translation of the coordinate system; The conversion between the camera coordinate system and the image coordinate system, and the conversion between the image coordinate system and the pixel coordinate system. All the parameters to be solved during the conversion process constitute the internal parameters of camera space calibration.

3. The multi-sensor spatio-temporal collaborative calibration method according to claim 1, wherein: The time alignment method is to unify the clock source through a pulse generator. All sensors are triggered by the pulses generated by the pulse generator, and each trigger corrects its own clock to align the timestamps of each sensor.

4. The multi-sensor spatio-temporal collaborative calibration method according to claim 3, wherein: The pulse generator calibrates the timestamps. Taking the GPS clock as the reference clock source and using the sensor synchronization timing protocol, it sends timestamp synchronization requests to the camera and the millimeter-wave radar to achieve time synchronization of all sensors in the local area network environment.

5. The multi-sensor spatio-temporal collaborative calibration method according to any one of claims 1-4, characterized in that: The sensor data frame indexing method. Since the sampling frequencies of the camera and the millimeter-wave radar are different, the data of the remaining sensors within the time window of the reference sensor data acquisition moment are indexed.

Images