Joint calibration method, device and terminal equipment for millimeter wave radar and laser radar

Abstract

The application is suitable for the field of automatic driving technology, and provides a joint calibration method, device and terminal equipment of a millimeter wave radar and a laser radar. The joint calibration method comprises the following steps: acquiring an intermediate frequency signal received by the millimeter wave radar and a plurality of first echo signals received by the laser radar; determining position information of a corner reflector in a millimeter wave radar coordinate system according to the intermediate frequency signal; determining a distance between the corner reflector and the laser radar according to the plurality of first echo signals, and the distance between the corner reflector and the laser radar and an angle of the corner reflector in a visual angle of the laser radar are position information of the corner reflector in a laser radar coordinate system; and determining a relative pose between the millimeter wave radar coordinate system and the laser radar coordinate system according to the position information of the corner reflector in the millimeter wave radar coordinate system and the position information of the corner reflector in the laser radar coordinate system. Through the application, spatial synchronization of the millimeter wave radar and the laser radar can be realized.

Description

Technical Field

[0001] This application belongs to the field of autonomous driving technology, and in particular relates to a joint calibration method, apparatus and terminal equipment for millimeter-wave radar and lidar. Background Technology

[0002] In the field of autonomous driving, millimeter-wave radar and lidar play a crucial role as distance sensors, capable of detecting targets such as vehicles and pedestrians on the road and providing information such as the target's position and velocity within the radar's own coordinate system. Compared to a single radar, fusing millimeter-wave radar and lidar can achieve better perception results. However, before fusing millimeter-wave radar and lidar, spatial synchronization between the two sensors needs to be achieved. Summary of the Invention

[0003] This application provides a method, apparatus, and terminal device for joint calibration of millimeter-wave radar and lidar, so as to achieve spatial synchronization of millimeter-wave radar and lidar.

[0004] In a first aspect, embodiments of this application provide a joint calibration method for millimeter-wave radar and lidar, wherein corner reflectors are disposed within the sensing range of the millimeter-wave radar and the lidar, and the joint calibration method includes:

[0005] Acquire the intermediate frequency signal received by the millimeter-wave radar and the multiple first echo signals received by the lidar;

[0006] Based on the intermediate frequency signal, determine the position information of the corner reflector in the millimeter-wave radar coordinate system;

[0007] Based on the plurality of first echo signals, the distance between the corner reflector and the lidar is determined. The distance between the corner reflector and the lidar, as well as the angle of the corner reflector in the lidar's field of view, constitute the position information of the corner reflector in the lidar coordinate system.

[0008] Based on the position information of the corner reflector in the millimeter-wave radar coordinate system and the position information in the lidar coordinate system, the relative pose between the millimeter-wave radar coordinate system and the lidar coordinate system is determined.

[0009] Secondly, embodiments of this application provide a joint calibration device for millimeter-wave radar and lidar, wherein corner reflectors are disposed within the sensing range of the millimeter-wave radar and the lidar, and the joint calibration device includes:

[0010] The signal acquisition module is used to acquire the intermediate frequency signal received by the millimeter-wave radar and the multiple first echo signals received by the lidar.

[0011] The position determination module is used to determine the position information of the corner reflector in the millimeter-wave radar coordinate system based on the intermediate frequency signal;

[0012] The distance determination module is used to determine the distance between the corner reflector and the lidar based on the plurality of first echo signals. The distance between the corner reflector and the lidar, as well as the angle of the corner reflector in the view of the lidar, are the position information of the corner reflector in the lidar coordinate system.

[0013] The pose determination module is used to determine the relative pose between the millimeter-wave radar coordinate system and the lidar coordinate system based on the position information of the corner reflector in the millimeter-wave radar coordinate system and the position information of the lidar coordinate system, respectively.

[0014] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the joint calibration method described in the first aspect above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the joint calibration method described in the first aspect above.

[0016] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to perform the steps of the joint calibration method described in the first aspect above.

[0017] As can be seen from the above, this application, by placing corner reflectors within the sensing range of millimeter-wave radar and lidar, can acquire the intermediate frequency (IF) signal received by the millimeter-wave radar and multiple first echo signals received by the lidar. Based on the IF signal, the position information of the corner reflector in the millimeter-wave radar coordinate system is determined, and the distance between the corner reflector and the lidar is determined based on the multiple first echo signals. The distance between the corner reflector and the lidar, as well as the angle of the corner reflector in the lidar's viewpoint, constitute the position information of the corner reflector in the lidar coordinate system. Based on the position information of the corner reflector in both the millimeter-wave radar and lidar coordinate systems, the relative pose between the two coordinate systems can be determined, thereby achieving spatial synchronization between the millimeter-wave radar and lidar. This application identifies corner reflectors placed within the sensing range of millimeter-wave radar and lidar at the signal level, enabling the identification of IF signals and first echo signals returned by corner reflectors from stationary targets, thus removing the limitation that millimeter-wave radar point clouds only contain moving targets. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating the implementation process of the joint calibration method for millimeter-wave radar and lidar provided in Embodiment 1 of this application;

[0020] Figure 2 This is an example diagram of a corner reflector;

[0021] Figure 3 This is an example diagram showing the echo intensity of the two first echo signals;

[0022] Figure 4 This is a schematic diagram illustrating the implementation process of the joint calibration method for millimeter-wave radar and lidar provided in Embodiment 2 of this application;

[0023] Figure 5 This is a schematic diagram of the structure of the joint calibration device for millimeter-wave radar and lidar provided in Embodiment 3 of this application;

[0024] Figure 6 This is a schematic diagram of the terminal device provided in Embodiment 4 of this application. Detailed Implementation

[0025] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0026] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0027] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0028] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0029] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0030] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0031] When achieving spatial synchronization between millimeter-wave radar and lidar (e.g., spatial synchronization between vehicle-mounted millimeter-wave radar and vehicle-mounted lidar, or spatial synchronization between roadside millimeter-wave radar and roadside lidar), one existing solution is based on manual physical measurement. This method requires human intervention, is inefficient, and has poor accuracy. To address the aforementioned technical problems in existing solutions, this application provides a joint calibration method for millimeter-wave radar and lidar. By placing corner reflectors within the sensing range of both the millimeter-wave radar and lidar, the intermediate frequency (IF) signal received by the millimeter-wave radar and multiple first echo signals received by the lidar can be acquired. The position information of the corner reflector in the millimeter-wave radar coordinate system is determined based on the IF signal, and the distance between the corner reflector and the lidar is determined based on the multiple first echo signals. The distance between the corner reflector and the lidar, as well as the angle of the corner reflector in the lidar's field of view, constitute the position information of the corner reflector in the lidar coordinate system. Based on the position information of the corner reflector in both the millimeter-wave radar and lidar coordinate systems, the relative pose between the millimeter-wave radar and lidar coordinate systems can be determined, thereby achieving spatial synchronization between the millimeter-wave radar and lidar. This process requires no manual intervention, improving the efficiency and accuracy of spatial synchronization between the millimeter-wave radar and lidar.

[0032] It should be understood that the sequence number of each step in this embodiment does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0033] To illustrate the technical solution described in this application, specific embodiments are provided below.

[0034] See Figure 1 This is a schematic diagram illustrating the implementation process of the joint calibration method for millimeter-wave radar and lidar provided in Embodiment 1 of this application. This joint calibration method is applied to terminal devices. Figure 1 As shown, the joint calibration method may include the following steps:

[0035] Step 101: Acquire the intermediate frequency signal received by the millimeter-wave radar and multiple first echo signals received by the lidar.

[0036] In one embodiment, corner reflectors can be placed within the sensing range of millimeter-wave radar and lidar to achieve joint calibration of millimeter-wave radar and lidar based on corner reflectors.

[0037] It should be noted that the prerequisite for joint calibration of millimeter-wave radar and lidar is that the millimeter-wave radar and lidar are relatively stationary to ensure that joint calibration of millimeter-wave radar and lidar can be achieved.

[0038] Corner reflectors are radar wave reflectors made of metal sheets in various specifications for different applications. When radar electromagnetic waves scan a corner reflector, the electromagnetic waves are refracted and amplified at the metal corner, producing a very strong echo signal; that is, corner reflectors have extremely strong reflected echo characteristics. For example... Figure 2 The image shown is an example diagram of a corner reflector. Figure 2 The corner reflector in the design is made up of three right-angled triangular metal plates, which can reflect electromagnetic waves emitted at multiple angles back to the signal emission direction.

[0039] To reduce the impact of other targets with strong echo characteristics on the joint calibration of millimeter-wave radar and lidar, the millimeter-wave radar and lidar can be placed in environments with no strong echoes. For example, when the millimeter-wave radar and lidar are vehicle-mounted, the vehicle equipped with them can be driven into an open space without obvious metallic reflectors. When the millimeter-wave radar and lidar are roadside millimeter-wave radar and roadside lidar, efforts should be made to ensure that there are no metallic reflectors with strong echoes within their sensing range.

[0040] The intermediate frequency signal received by millimeter-wave radar refers to the frequency difference signal obtained by mixing the electromagnetic waves emitted by the millimeter-wave radar and the reflected echo signal received.

[0041] As an optional embodiment, the terminal device can simultaneously send signal acquisition commands to the millimeter-wave radar and the lidar respectively. After receiving the signal acquisition command, the millimeter-wave radar feeds back the received intermediate frequency signal to the terminal device, and the lidar feeds back the received multiple first echo signals to the terminal device after receiving the signal acquisition command.

[0042] As another optional embodiment, the millimeter-wave radar actively sends the intermediate frequency signal to the terminal device when it receives the intermediate frequency signal, and the lidar actively sends multiple first echo signals to the terminal device when it receives multiple first echo signals.

[0043] Step 102: Determine the position information of the corner reflector in the millimeter-wave radar coordinate system based on the intermediate frequency signal.

[0044] The millimeter-wave radar coordinate system can refer to a coordinate system established with the millimeter-wave radar as the origin.

[0045] The position information of the corner reflector in the millimeter-wave radar coordinate system includes the distance between the corner reflector and the millimeter-wave radar, as well as the angle of the corner reflector in the millimeter-wave radar's field of view.

[0046] Optionally, determining the position information of the corner reflector in the millimeter-wave radar coordinate system based on the intermediate frequency signal includes:

[0047] Perform a distance Fourier transform on the intermediate frequency signal to obtain the first Fourier transform result;

[0048] Based on the first Fourier transform result, the frequency of the second echo signal returned by the corner reflector is determined. The second echo signal is the echo received by the millimeter-wave radar.

[0049] The distance between the corner reflector and the millimeter-wave radar is determined based on the frequency of the second echo signal returned by the corner reflector.

[0050] Perform a Doppler Fourier transform on the intermediate frequency signal to obtain the second Fourier transform result;

[0051] The Doppler frequency shift of the intermediate frequency signal is determined based on the results of the second Fourier transform.

[0052] The angle of the corner reflector in the millimeter-wave radar's field of view is determined based on the Doppler frequency shift of the intermediate frequency signal.

[0053] In the intermediate frequency (IF) signals received by millimeter-wave radar, different IF signals of different frequencies can be obtained based on the distance between different targets and the millimeter-wave radar within the sensing range. Therefore, by performing a range Fourier transform on the IF signals received by the millimeter-wave radar, and based on the extremely strong reflection echo characteristics of the corner reflector, the distance between the corner reflector and the millimeter-wave radar can be determined.

[0054] The formula for calculating the distance between the corner reflector and the millimeter-wave radar is as follows:

[0055]

[0056] Where c represents the speed of light in a vacuum, f represents the frequency of the first echo signal of the corner reflector, and s represents the frequency modulation slope of the frequency-modulated signal transmitted by the millimeter-wave radar.

[0057] The formula for calculating the angle of a corner reflector in the field of view of a millimeter-wave radar is as follows:

[0058]

[0059] Where, d a λ represents the physical distance between the two receiving antennas in a millimeter-wave radar, λ represents the wavelength of the intermediate frequency signal, and ω represents the Doppler shift of the intermediate frequency signal.

[0060] Optionally, the Fourier transform in this application may refer to the Fast Fourier Transform to improve computation speed.

[0061] Optionally, the frequency of the second echo signal returned by the corner reflector is determined based on the first Fourier transform result, including:

[0062] Find the frequency of the spike in the first Fourier transform result and determine that the frequency of the spike is the frequency of the second echo signal returned by the corner reflector;

[0063] Based on the results of the second Fourier transform, the Doppler frequency shift of the intermediate frequency signal is determined to include:

[0064] Find the frequency shift to which the peak belongs from the second Fourier transform result, and determine that the frequency shift to which the peak belongs is the Doppler frequency shift of the intermediate frequency signal.

[0065] Because corner reflectors have extremely strong echo characteristics, their reflectivity is much higher than that of other targets within the sensing range. Therefore, it can be determined that the frequency of the peak in the first Fourier transform result is the frequency of the second echo signal returned by the corner reflector, and the frequency shift of the peak in the second Fourier transform result is the Doppler frequency shift of the intermediate frequency signal.

[0066] Step 103: Based on multiple first echo signals, determine the distance between the corner reflector and the lidar. The distance between the corner reflector and the lidar, as well as the angle of the corner reflector in the lidar's field of view, are the position information of the corner reflector in the lidar coordinate system.

[0067] The lidar coordinate system can refer to a coordinate system established with the lidar as the origin.

[0068] Since the angle of the corner reflector in the lidar's field of view is fixed during transmission, the angle of the corner reflector in the lidar's field of view is a fixed value and does not require calculation.

[0069] Optionally, determining the distance between the corner reflector and the lidar based on multiple first echo signals includes:

[0070] From multiple first echo signals, determine the first echo signal returned by the corner reflector;

[0071] The distance between the corner reflector and the lidar is determined based on the time difference between the lidar's transmitted signal and the first echo signal received from the corner reflector.

[0072] The formula for calculating the distance between the corner reflector and the lidar is as follows:

[0073]

[0074] Where Δt represents the time difference between the laser radar transmitting the signal and receiving the first echo signal returned by the corner reflector.

[0075] Optionally, determining the first echo signal returned by the corner reflector from a plurality of first echo signals includes:

[0076] Obtain the saturation level of the echo intensity of multiple first echo signals;

[0077] The first echo signal with the highest saturation level is determined as the first echo signal returned by the corner reflector.

[0078] In the multiple first echo signals received by the lidar, different echo intensities can produce waveforms with different degrees of saturation, such as... Figure 3 The image shows example graphs of the echo intensity of two first echo signals. The saturation level of a first echo signal can be understood as the area enclosed by the echo intensity curve of that first echo signal and the X-axis and Y-axis. Figure 3 It can be seen that the saturation level of the first echo signal A is higher than that of the first echo signal B.

[0079] Because the reflectivity of the corner reflector is much higher than that of other targets within its sensing range, the saturation level of the first echo signal returned by the corner reflector is much higher than that of other targets. The first echo signal with the highest saturation level among multiple first echo signals is the first echo signal returned by the corner reflector.

[0080] Step 104: Determine the relative pose between the millimeter-wave radar coordinate system and the lidar coordinate system based on the position information of the corner reflector in the millimeter-wave radar coordinate system and the position information in the lidar coordinate system, respectively.

[0081] In this embodiment, the corner reflector is used as a reference point. Based on the position information of the corner reflector in the millimeter-wave radar coordinate system and the position information in the lidar coordinate system, the coordinate transformation relationship between the millimeter-wave radar and the lidar can be calculated. This coordinate transformation relationship is the relative pose between the millimeter-wave radar coordinate system and the lidar coordinate system.

[0082] This embodiment of the application sets up a corner reflector within the sensing range of a millimeter-wave radar and a lidar, which can acquire the intermediate frequency signal received by the millimeter-wave radar and multiple first echo signals received by the lidar. The position information of the corner reflector in the millimeter-wave radar coordinate system is determined based on the intermediate frequency signal, and the distance between the corner reflector and the lidar is determined based on the multiple first echo signals. The distance between the corner reflector and the lidar, as well as the angle of the corner reflector in the lidar's field of view, constitute the position information of the corner reflector in the lidar coordinate system. Based on the position information of the corner reflector in the millimeter-wave radar coordinate system and the lidar coordinate system, the relative pose between the millimeter-wave radar coordinate system and the lidar coordinate system can be determined, thereby achieving spatial synchronization between the millimeter-wave radar and the lidar.

[0083] See Figure 4 This is a schematic diagram illustrating the implementation process of the joint calibration method for millimeter-wave radar and lidar provided in Embodiment 2 of this application. This joint calibration method is applied to a terminal device. Figure 4 As shown, the joint calibration method may include the following steps:

[0084] Step 401: Acquire the intermediate frequency signal received by the millimeter-wave radar and multiple first echo signals received by the lidar.

[0085] This step is the same as step 101. For details, please refer to the relevant description of step 101. It will not be repeated here.

[0086] Step 402: Determine the position information of the corner reflector in the millimeter-wave radar coordinate system based on the intermediate frequency signal.

[0087] This step is the same as step 102, and you can refer to the relevant description of step 102 for details, which will not be repeated here.

[0088] Step 403: Based on multiple first echo signals, determine the distance between the corner reflector and the lidar. The distance between the corner reflector and the lidar, as well as the angle of the corner reflector in the lidar's field of view, are the position information of the corner reflector in the lidar coordinate system.

[0089] This step is the same as step 103. For details, please refer to the relevant description of step 103. It will not be repeated here.

[0090] Step 404: Calculate the residuals of each of the N first position information groups based on the N first position information groups and the first optimization parameters.

[0091] The first position information group includes the position information of the corner reflector in the millimeter-wave radar coordinate system and the position information in the lidar coordinate system when the corner reflector is in one position. The corner reflector is set in N different positions in sequence, and the N different positions correspond to N first position information groups, where N is an integer greater than 1.

[0092] Keeping the positions of the millimeter-wave radar and lidar unchanged, corner reflectors can be placed sequentially at N different locations. By repeating steps 401 to 403, first position information groups corresponding to the N different locations can be obtained. The corner reflectors can be placed evenly, with a fixed interval between adjacent placements, to ensure the accuracy of subsequent relative pose calculations.

[0093] For a first location information group, the residual of the first location information group can be expressed as follows:

[0094] p LIDAR -(R·p RADAR +T)

[0095] Where, p LIDAR p represents the position information of the corner reflector in the lidar coordinate system. RADAR This represents the position information of the corner reflector in the millimeter-wave radar coordinate system, and R and T represent the first optimization parameters.

[0096] Step 405: Sum the residuals of the N first position information groups to obtain the first sum.

[0097] The first cumulative sum can be represented as follows:

[0098] e * =∑p LIDAR -(R·p RADAR +T)

[0099] Step 406: Adjust the first optimization parameter to minimize the first cumulative sum. The optimization parameter when the first cumulative sum is minimized is the relative pose between the millimeter-wave radar coordinate system and the lidar coordinate system.

[0100] Among them, e * The optimal parameters R and T, when minimized, represent the relative pose between the millimeter-wave radar coordinate system and the lidar coordinate system.

[0101] Optionally, this embodiment also includes:

[0102] Obtain the absolute position information of the corner reflector;

[0103] The absolute position information of the corner reflector is transformed into the ENU coordinate system to obtain the position information of the corner reflector in the ENU coordinate system. The ENU coordinate system is determined based on any position within the sensing range of the millimeter-wave radar and lidar.

[0104] Based on the position information of the corner reflector in the ENU coordinate system and the position information in the millimeter-wave radar coordinate system, the pose of the millimeter-wave radar relative to the ENU coordinate system is determined.

[0105] The absolute pose of the millimeter-wave radar is determined based on its pose relative to the ENU coordinate system.

[0106] Based on the position information of the corner reflector in the ENU coordinate system and the position information of the lidar coordinate system, the pose of the lidar relative to the ENU coordinate system is determined.

[0107] The absolute pose of the lidar is determined based on the pose of the lidar sensor relative to the ENU coordinate system.

[0108] The absolute position information of the corner reflector refers to the absolute coordinates of the corner reflector relative to the Earth. The absolute position information of the corner reflector can be represented by the Universal Transverse Mercator Grid System (UTM) coordinates or latitude and longitude coordinates.

[0109] In one embodiment, the absolute position information of the corner reflector can be measured using a Real-Time Kinematic (RTK) carrier phase differential device. The RTK carrier phase differential device can send the measured absolute position information of the corner reflector to a terminal device, or the user can input the measured absolute position information of the corner reflector into the terminal device; this is not limited to this. The RTK carrier phase differential device is a high-precision satellite navigation positioning device that can measure the latitude and longitude coordinates (Lon, Lat), altitude h, etc., of the corner reflector and record these data.

[0110] Taking the latitude and longitude coordinates of a corner reflector as an example, the origin can be established based on the location information (Lon0, Lat0) of any point within the sensing range, with the positive x-axis pointing east, the positive y-axis pointing north, and the positive z-axis pointing upwards. The formula for converting the measured latitude and longitude coordinates of the corner reflector to the ENU coordinate system is as follows:

[0111]

[0112] Among them, R M R N These are the radii of curvature of the Earth's meridian and trochanter corresponding to the location of the corner reflector, respectively, and their calculation formulas are as follows:

[0113]

[0114] Among them, R e R p , respectively, are the radii of the Earth's semi-major and semi-minor axes, and e is the eccentricity of the Earth's rotating ellipsoid. The radii of the Earth's semi-major and semi-minor axes are based on parameters from the World Geodetic System-1984 (WGS-84).

[0115] In one embodiment, a second position information group includes the position information of the corner reflector in the ENU coordinate system and the position information in the millimeter-wave radar coordinate system when it is in one position. A third position information group includes the position information of the corner reflector in the ENU coordinate system and the position information in the lidar coordinate system when it is in one position. The corner reflector is sequentially set in N different positions, and the N different positions correspond to N second position information groups and N third position information groups, where N is an integer greater than 1. Determining the pose of the millimeter-wave radar relative to the ENU coordinate system based on the position information of the corner reflector in the ENU coordinate system and the position information in the millimeter-wave radar coordinate system includes:

[0116] Calculate the residuals of each of the N second position information groups based on the N second position information groups and the second optimization parameters;

[0117] The residuals of the N second position information groups are summed to obtain the second sum.

[0118] The second optimization parameter is adjusted to minimize the second cumulative sum. The second optimization parameter when the second cumulative sum is minimized is the pose of the millimeter-wave radar relative to the ENU coordinate system.

[0119] Based on the position information of the corner reflector in the ENU coordinate system and the position information in the lidar coordinate system, the pose of the lidar relative to the ENU coordinate system is determined as follows:

[0120] Calculate the residuals of each of the N third location information groups based on the N third optimization parameters;

[0121] The residuals of the N third position information groups are summed to obtain the third summation;

[0122] Adjust the third optimization parameter to minimize the third cumulative sum. The third optimization parameter that minimizes the third cumulative sum is the pose of the lidar relative to the ENU coordinate system.

[0123] The residual of the second location information can be represented as follows:

[0124] ||p ENU -(R RADAR ·p RADAR +T RADAR )||

[0125] Where, p ENU This indicates the position information of the corner reflector in the ENU coordinate system, R RADAR and T RADAR This represents the second optimization parameter.

[0126] The second cumulative sum can be represented as follows:

[0127] e RADAR =∑||p ENU -(R RADAR ·p RADAR +T RADAR )||

[0128] Make e RADAR The optimization function R that reaches its minimum value RADAR and T RADAR This refers to the pose of the millimeter-wave radar relative to the ENU coordinate system.

[0129] For the third location information, the residual of this third location information can be represented as follows:

[0130] ||p ENU -(R LIDAR ·p LIDAR +T LIDAR )||

[0131] Among them, R LIDAR and T LIDAR This represents the second optimization parameter.

[0132] The third cumulative sum can be represented as follows:

[0133] e LIDAR =∑||p ENU -(R LIDAR ·pLIDAR +T LIDAR )||

[0134] Make e LIDAR The optimization function R that reaches its minimum value LIDAR and T LIDAR This refers to the pose of the millimeter-wave radar relative to the ENU coordinate system.

[0135] This application embodiment controls the movement of a corner reflector within the sensing range of millimeter-wave radar and lidar, allowing measurement of the corner reflector's position in the millimeter-wave radar coordinate system and lidar coordinate system at different locations. Residual optimization of the position information from multiple measurements improves the accuracy of joint calibration of millimeter-wave radar and lidar.

[0136] See Figure 5 This is a schematic diagram of the combined calibration device for millimeter-wave radar and lidar provided in Embodiment 3 of this application. Corner reflectors are provided within the sensing range of the millimeter-wave radar and lidar. For ease of explanation, only the parts relevant to the embodiments of this application are shown.

[0137] The aforementioned joint calibration device includes:

[0138] The signal acquisition module 51 is used to acquire the intermediate frequency signal received by the millimeter-wave radar and the multiple first echo signals received by the lidar.

[0139] The position determination module 52 is used to determine the position information of the corner reflector in the millimeter-wave radar coordinate system based on the intermediate frequency signal.

[0140] The distance determination module 53 is used to determine the distance between the corner reflector and the lidar based on the plurality of first echo signals. The distance between the corner reflector and the lidar and the angle of the corner reflector in the view of the lidar are the position information of the corner reflector in the lidar coordinate system.

[0141] The pose determination module 54 is used to determine the relative pose between the millimeter-wave radar coordinate system and the lidar coordinate system based on the position information of the corner reflector in the millimeter-wave radar coordinate system and the position information in the lidar coordinate system, respectively.

[0142] Optionally, the position information of the corner reflector in the millimeter-wave radar coordinate system includes the distance between the corner reflector and the millimeter-wave radar, and the angle of the corner reflector in the viewing angle of the millimeter-wave radar; the position determination module 52 includes:

[0143] The first transformation unit is used to perform a distance Fourier transform on the intermediate frequency signal to obtain a first Fourier transform result;

[0144] The first determining unit is used to determine the frequency of the second echo signal returned by the corner reflector based on the first Fourier transform result, wherein the second echo signal is the echo received by the millimeter-wave radar.

[0145] The second determining unit is used to determine the distance between the corner reflector and the millimeter-wave radar based on the frequency of the second echo signal returned by the corner reflector.

[0146] The second transformation unit is used to perform a Doppler Fourier transform on the intermediate frequency signal to obtain a second Fourier transform result.

[0147] The third determining unit is used to determine the Doppler frequency shift of the intermediate frequency signal based on the second Fourier transform result;

[0148] The fourth determining unit is used to determine the angle of the corner reflector in the field of view of the millimeter-wave radar based on the Doppler frequency shift of the intermediate frequency signal.

[0149] Optionally, the first determining unit described above is specifically used for:

[0150] Find the frequency of the peak in the first Fourier transform result and determine that the frequency of the peak is the frequency of the second echo signal returned by the corner reflector;

[0151] The aforementioned third determining unit is specifically used for:

[0152] Find the frequency shift to which the peak belongs from the second Fourier transform result, and determine that the frequency shift to which the peak belongs is the Doppler frequency shift of the intermediate frequency signal.

[0153] Optionally, the distance determination module 53 includes:

[0154] An echo determination unit is used to determine the first echo signal returned by the corner reflector from the plurality of first echo signals;

[0155] The distance determination unit is used to determine the distance between the corner reflector and the lidar based on the time difference between the lidar's transmitted signal and the first echo signal received from the corner reflector.

[0156] Optionally, the above-mentioned echo determination unit is specifically used for:

[0157] Obtain the saturation level of the echo intensity of the plurality of first echo signals;

[0158] The first echo signal with the highest saturation level of echo intensity is determined as the first echo signal returned by the corner reflector.

[0159] Optionally, a first position information group includes the position information of the corner reflector in the millimeter-wave radar coordinate system and the position information of the lidar coordinate system when the corner reflector is in one position. The corner reflector is sequentially set in N different positions, and the N different positions correspond to N first position information groups, where N is an integer greater than 1. The pose determination module 54 is specifically used for:

[0160] Based on the N first location information groups and the first optimization parameters, calculate the residuals of each of the N first location information groups;

[0161] The residuals of the N first position information groups are summed to obtain the first summation;

[0162] The first optimization parameter is adjusted to minimize the first cumulative sum. The optimization parameter when the first cumulative sum is minimized is the relative pose between the millimeter-wave radar coordinate system and the lidar coordinate system.

[0163] Optionally, the aforementioned joint calibration device further includes:

[0164] A position acquisition module is used to acquire the absolute position information of the corner reflector;

[0165] The position conversion module is used to convert the absolute position information of the corner reflector to the ENU coordinate system to obtain the position information of the corner reflector in the ENU coordinate system, wherein the ENU coordinate system is determined based on any position within the sensing range of the millimeter-wave radar and the lidar.

[0166] The first determining module is used to determine the pose of the millimeter-wave radar relative to the ENU coordinate system based on the position information of the corner reflector in the ENU coordinate system and the position information of the millimeter-wave radar in the millimeter-wave radar coordinate system, respectively.

[0167] The second determining module is used to determine the absolute pose of the millimeter-wave radar based on the pose of the millimeter-wave radar relative to the ENU coordinate system.

[0168] The third determining module is used to determine the pose of the lidar relative to the ENU coordinate system based on the position information of the corner reflector in the ENU coordinate system and the position information of the lidar coordinate system.

[0169] The fourth determining module is used to determine the absolute pose of the lidar based on the pose of the lidar sensor relative to the ENU coordinate system.

[0170] Optionally, a second position information group includes the position information of the corner reflector in the ENU coordinate system and the position information of the millimeter-wave radar coordinate system when the corner reflector is in one position, and a third position information group includes the position information of the corner reflector in the ENU coordinate system and the position information of the lidar coordinate system when the corner reflector is in one position. The corner reflector is sequentially set in N different positions, and the N different positions correspond to N second position information groups and N third position information groups, where N is an integer greater than 1; the aforementioned first determining module is specifically used for:

[0171] Based on the N second location information groups and the second optimization parameters, calculate the residuals of each of the N second location information groups;

[0172] The residuals of the N second position information groups are summed to obtain the second sum.

[0173] The second optimization parameter is adjusted to minimize the second cumulative sum. The second optimization parameter when the second cumulative sum is minimized is the pose of the millimeter-wave radar relative to the ENU coordinate system.

[0174] The third determining module mentioned above is specifically used for:

[0175] Based on the N third location information groups and the third optimization parameters, calculate the residuals of each of the N third location information groups;

[0176] The residuals of the N third position information groups are summed to obtain the third summation.

[0177] The third optimization parameter is adjusted to minimize the third cumulative sum. The third optimization parameter when the third cumulative sum is minimized is the pose of the lidar relative to the ENU coordinate system.

[0178] The joint calibration device provided in this application embodiment can be applied in the foregoing method embodiments. For details, please refer to the description of the above method embodiments, which will not be repeated here.

[0179] Figure 6 This is a schematic diagram of the terminal device provided in Embodiment 4 of this application. Figure 6 As shown, the terminal device 6 of this embodiment includes: one or more processors 60 (only one is shown in the figure), a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60. When the processor 60 executes the computer program 62, it implements the steps in the various joint calibration method embodiments described above.

[0180] The terminal device 6 can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that... Figure 6 This is merely an example of terminal device 6 and does not constitute a limitation on terminal device 6. It may include more or fewer components than shown, or combine certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0181] The processor 60 may be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0182] The memory 61 can be an internal storage unit of the terminal device 6, such as a hard disk or memory of the terminal device 6. The memory 61 can also be an external storage device of the terminal device 6, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device 6. Furthermore, the memory 61 can include both internal and external storage units of the terminal device 6. The memory 61 is used to store the computer program and other programs and data required by the terminal device. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0183] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0184] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0185] This application also provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the above-described method embodiments.

[0186] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0187] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0188] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0191] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A joint calibration method for millimeter-wave radar and lidar, characterized in that, An angle reflector is provided within the sensing range of the millimeter-wave radar and the lidar, and the joint calibration method includes: Acquire the intermediate frequency signal received by the millimeter-wave radar and the multiple first echo signals received by the lidar; Based on the intermediate frequency signal, determine the position information of the corner reflector in the millimeter-wave radar coordinate system; Based on the plurality of first echo signals, the distance between the corner reflector and the lidar is determined. The distance between the corner reflector and the lidar, as well as the angle of the corner reflector in the lidar's field of view, constitute the position information of the corner reflector in the lidar coordinate system. Based on the position information of the corner reflector in the millimeter-wave radar coordinate system and the position information in the lidar coordinate system, the relative pose between the millimeter-wave radar coordinate system and the lidar coordinate system is determined. The position information of the corner reflector in the millimeter-wave radar coordinate system includes the distance between the corner reflector and the millimeter-wave radar, and the angle of the corner reflector in the viewing angle of the millimeter-wave radar; determining the position information of the corner reflector in the millimeter-wave radar coordinate system based on the intermediate frequency signal includes: Perform a distance Fourier transform on the intermediate frequency signal to obtain the first Fourier transform result; Based on the first Fourier transform result, the frequency of the second echo signal returned by the corner reflector is determined, and the second echo signal is the echo received by the millimeter-wave radar; The distance between the corner reflector and the millimeter-wave radar is determined based on the frequency of the second echo signal returned by the corner reflector. Perform a Doppler Fourier transform on the intermediate frequency signal to obtain a second Fourier transform result; The Doppler frequency shift of the intermediate frequency signal is determined based on the second Fourier transform result; The angle of the corner reflector in the field of view of the millimeter-wave radar is determined based on the Doppler frequency shift of the intermediate frequency signal. A first position information group includes the position information of the corner reflector in the millimeter-wave radar coordinate system and the position information of the lidar coordinate system when the corner reflector is in one position. The corner reflector is sequentially set in N different positions, and the N different positions correspond to N first position information groups, where N is an integer greater than 1. Determining the relative pose between the millimeter-wave radar coordinate system and the lidar coordinate system based on the position information of the corner reflector in the millimeter-wave radar coordinate system and the position information of the lidar coordinate system includes: Based on the N first location information groups and the first optimization parameters, calculate the residuals of each of the N first location information groups; The residuals of the N first position information groups are summed to obtain the first summation; The first optimization parameter is adjusted to minimize the first cumulative sum. The optimization parameter when the first cumulative sum is minimized is the relative pose between the millimeter-wave radar coordinate system and the lidar coordinate system.

2. The joint calibration method as described in claim 1, characterized in that, Determining the frequency of the second echo signal returned by the corner reflector based on the first Fourier transform result includes: Find the frequency of the peak in the first Fourier transform result and determine that the frequency of the peak is the frequency of the second echo signal returned by the corner reflector; The step of determining the Doppler frequency shift of the intermediate frequency signal based on the second Fourier transform result includes: Find the frequency shift to which the peak belongs from the second Fourier transform result, and determine that the frequency shift to which the peak belongs is the Doppler frequency shift of the intermediate frequency signal.

3. The joint calibration method as described in claim 1, characterized in that, Determining the distance between the corner reflector and the lidar based on the plurality of first echo signals includes: From the plurality of first echo signals, determine the first echo signal returned by the corner reflector; The distance between the corner reflector and the lidar is determined based on the time difference between the lidar's transmitted signal and the first echo signal received from the corner reflector.

4. The joint calibration method as described in claim 3, characterized in that, Determining the first echo signal returned by the corner reflector from the plurality of first echo signals includes: Obtain the saturation level of the echo intensity of the plurality of first echo signals; The first echo signal with the highest saturation level of echo intensity is determined as the first echo signal returned by the corner reflector.

5. The joint calibration method according to any one of claims 1 to 4, characterized in that, The joint calibration method further includes: Obtain the absolute position information of the corner reflector; The absolute position information of the corner reflector is converted to the ENU coordinate system to obtain the position information of the corner reflector in the ENU coordinate system, which is determined based on any position within the sensing range of the millimeter-wave radar and the lidar. Based on the position information of the corner reflector in the ENU coordinate system and the position information of the millimeter-wave radar in the millimeter-wave radar coordinate system, the pose of the millimeter-wave radar relative to the ENU coordinate system is determined. The absolute pose of the millimeter-wave radar is determined based on its pose relative to the ENU coordinate system. Based on the position information of the corner reflector in the ENU coordinate system and the position information of the lidar coordinate system, the pose of the lidar relative to the ENU coordinate system is determined. The absolute pose of the lidar is determined based on the pose of the lidar sensor relative to the ENU coordinate system.

6. The joint calibration method as described in claim 5, characterized in that, A second position information group includes the position information of the corner reflector in the ENU coordinate system and the position information of the millimeter-wave radar coordinate system when the corner reflector is in one position. A third position information group includes the position information of the corner reflector in the ENU coordinate system and the position information of the lidar coordinate system when the corner reflector is in one position. The corner reflector is sequentially set in N different positions, and the N different positions correspond to N second position information groups and N third position information groups, where N is an integer greater than 1. Determining the pose of the millimeter-wave radar relative to the ENU coordinate system based on the position information of the corner reflector in the ENU coordinate system and the position information of the millimeter-wave radar coordinate system includes: Based on the N second location information groups and the second optimization parameters, calculate the residuals of each of the N second location information groups; The residuals of the N second position information groups are summed to obtain the second sum. The second optimization parameter is adjusted to minimize the second cumulative sum. The second optimization parameter when the second cumulative sum is minimized is the pose of the millimeter-wave radar relative to the ENU coordinate system. The step of determining the pose of the lidar relative to the ENU coordinate system based on the position information of the corner reflector in the ENU coordinate system and the position information of the lidar coordinate system includes: Based on the N third location information groups and the third optimization parameters, calculate the residuals of each of the N third location information groups; The residuals of the N third position information groups are summed to obtain the third summation. The third optimization parameter is adjusted to minimize the third cumulative sum. The third optimization parameter when the third cumulative sum is minimized is the pose of the lidar relative to the ENU coordinate system.

7. A combined calibration device for millimeter-wave radar and lidar, characterized in that, An angle reflector is provided within the sensing range of the millimeter-wave radar and the lidar. The joint calibration device includes: The signal acquisition module is used to acquire the intermediate frequency signal received by the millimeter-wave radar and the multiple first echo signals received by the lidar. The position determination module is used to determine the position information of the corner reflector in the millimeter-wave radar coordinate system based on the intermediate frequency signal; The distance determination module is used to determine the distance between the corner reflector and the lidar based on the plurality of first echo signals. The distance between the corner reflector and the lidar, as well as the angle of the corner reflector in the view of the lidar, are the position information of the corner reflector in the lidar coordinate system. The pose determination module is used to determine the relative pose between the millimeter-wave radar coordinate system and the lidar coordinate system based on the position information of the corner reflector in the millimeter-wave radar coordinate system and the position information of the lidar coordinate system, respectively. The position information of the corner reflector in the millimeter-wave radar coordinate system includes the distance between the corner reflector and the millimeter-wave radar, and the angle of the corner reflector in the field of view of the millimeter-wave radar; the position determination module includes: The first transformation unit is used to perform a distance Fourier transform on the intermediate frequency signal to obtain a first Fourier transform result; The first determining unit is used to determine the frequency of the second echo signal returned by the corner reflector based on the first Fourier transform result, wherein the second echo signal is the echo received by the millimeter-wave radar. The second determining unit is used to determine the distance between the corner reflector and the millimeter-wave radar based on the frequency of the second echo signal returned by the corner reflector. The second transformation unit is used to perform a Doppler Fourier transform on the intermediate frequency signal to obtain a second Fourier transform result. The third determining unit is used to determine the Doppler frequency shift of the intermediate frequency signal based on the second Fourier transform result; The fourth determining unit is used to determine the angle of the corner reflector in the field of view of the millimeter-wave radar based on the Doppler frequency shift of the intermediate frequency signal. A first position information group includes the position information of the corner reflector in the millimeter-wave radar coordinate system and the position information in the lidar coordinate system when the corner reflector is in one position. The corner reflector is sequentially set in N different positions, and the N different positions correspond to N first position information groups, where N is an integer greater than 1; the pose determination module is specifically used for: Based on the N first location information groups and the first optimization parameters, calculate the residuals of each of the N first location information groups; The residuals of the N first position information groups are summed to obtain the first summation; The first optimization parameter is adjusted to minimize the first cumulative sum. The optimization parameter when the first cumulative sum is minimized is the relative pose between the millimeter-wave radar coordinate system and the lidar coordinate system.

8. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the joint calibration method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the joint calibration method as described in any one of claims 1 to 6.

Images