A joint calibration method, device and medium for radar and camera

Abstract

The present application discloses a method, device and medium for joint calibration of a radar and a camera, relating to the field of target calibration. The main problem solved is that the existing radar-camera calibration method requires the camera and the millimeter-wave radar to be arranged at the same position so that their longitude and latitude coordinates are consistent. The method provided by the present application obtains the radar UTM coordinates of the radar GPS coordinates in the global UTM coordinate system, that is, the offset distance of the radar relative to the global UTM coordinate system, and the rotation angle of the radar coordinate system relative to the global UTM coordinate system, that is, the mapping relationship between the global UTM coordinate system and the radar coordinate system. This method does not require the radar and the camera to be set at the same longitude and latitude. In addition, the GPS motion data of the calibration vehicle and the radar measurement data are used as the data sources for screening calibration point traces, greatly improving the accuracy and speed of calibration.

Description

Technical Field

[0001] This application relates to the field of target calibration, and in particular, to a method, device, and medium for joint calibration of a radar and a camera. Background Art

[0002] In the field of intelligent transportation, the current target detection and tracking system mainly relies on video detection. However, cameras are vulnerable to external environments such as changing lighting and bad weather, making it difficult to perform stable and highly accurate all-weather and full-time-domain detection of targets. In recent years, millimeter-wave radars have received extensive attention from researchers due to their low price and excellent characteristics of being less affected by weather factors such as lighting, soot, rain, and snow. More and more researchers use the complementarity of millimeter-wave radars and cameras to fuse the two sensors as a stable and cost-effective environmental perception solution. However, the premise of fusing a millimeter-wave radar and a camera is to perform joint calibration to obtain a unified expression of the same target position by the two sensors. Currently, by converting the pixel coordinates and radar coordinates of all calibration vehicles into world coordinates respectively, the mapping relationship between the pixel coordinates and radar coordinates is solved based on a global optimal matching algorithm.

[0003] Through the solution of converting the pixel coordinates and radar coordinates of all calibration vehicles into world coordinates respectively, the camera and the millimeter-wave radar need to be installed at the same location so that their longitude and latitude coordinates are consistent, which is not applicable to traffic scenarios where the camera and the millimeter-wave radar are installed at different locations. In addition, the existing solution consumes a large amount of time in complex traffic scenarios, that is, when there are many vehicle targets, and cannot well meet the requirements of rapid calibration in traffic scenarios. When there are few vehicle targets, it is difficult to ensure the calibration accuracy. Moreover, during this process, the calibration result of the camera is used to convert the vehicle pixel coordinates and world coordinates, but this mapping relationship is calculated from a point within the detection frame of the calibration vehicle and the positioning data of the calibration vehicle, which obviously has a large error. Directly applying this coordinate conversion to this process, the accumulated error will further reduce the calibration accuracy and slow down the calibration speed.

[0004] Therefore, it is a technical problem urgently to be solved by those skilled in the art to provide a method for joint calibration of a radar and a camera that does not limit the installation positions of the camera and the radar and has high calibration accuracy. Summary of the Invention

[0005] The purpose of this application is to provide a method for joint calibration of a radar and a camera that does not limit the installation positions of the camera and the radar.

[0006] To solve the above technical problems, this application provides a method for joint calibration of a radar and a camera, including:

[0007] Obtain the video data captured by the camera, the radar measurement data monitored by the radar, the GPS motion data uploaded by the calibrated vehicle, the camera GPS coordinates, and the radar GPS coordinates;

[0008] Establish a global UTM coordinate system with the camera as the coordinate origin;

[0009] Convert the radar GPS coordinates to the global UTM coordinate system to obtain the radar UTM coordinates;

[0010] Construct an ROI region, and the monitoring points of the radar are located in the ROI region;

[0011] According to the GPS motion data, obtain the GPS traces of the calibrated vehicle located in the ROI region and meeting the continuous segmented preset distance threshold;

[0012] According to the radar measurement data, obtain the radar traces of the calibrated vehicle corresponding to the time period of the GPS traces, and the radar traces meet the continuous segmented preset distance threshold and the length preset threshold;

[0013] Take the straight-line traces in the radar traces as the calibration traces;

[0014] According to the calibration traces, obtain the rotation angle of the radar coordinate system of the radar relative to the global UTM coordinate system.

[0015] Preferably, for the joint calibration method of the radar and the camera, the constructing of the ROI region, where the monitoring points of the radar are located in the ROI region, includes:

[0016] Select 3 or more of the radars;

[0017] Connect the positions of the radars to form a polygon region, and take the polygon region as the ROI region. Among them, the sum of the areas enclosed by the connections between the monitoring points of the radar and any two adjacent vertices of the ROI region is equal to the area of the ROI region.

[0018] Preferably, for the joint calibration method of the radar and the camera, the obtaining of the rotation angle of the radar coordinate system of the radar relative to the global UTM coordinate system according to the calibration traces includes:

[0019] Establish a local UTM coordinate system with the radar as the origin;

[0020] Obtain the first coordinate of the calibrated vehicle in the local UTM coordinate system, and the first coordinate is denoted as (X t , Y t );

[0021] Obtain the rectangular coordinates of the marked dot points, and mark the rectangular coordinates as (X r , Y r );

[0022] Convert the rectangular coordinates to the local UTM coordinate system according to the first formula to obtain the second coordinates (X' r , Y' r );

[0023] The first formula is: X' r = X r cos(R θ ) - Y r sin(R θ ), Y' r = X r sin(R θ ) + Y r cos(R θ );

[0024] Among them, R θ is the rotation angle, and initialize R θ to 0;

[0025] Through the optimization function, iteratively obtain the optimal solution of the rotation angle by the gradient descent method;

[0026] The optimization function is: L(R θ ) = [X t - X' r , Y t - Y' r ;

[0027] Among them, L(R θ ) represents the value set of R θ .

[0028] Preferably, for the joint calibration method of the radar and the camera, after obtaining the rotation angle of the radar coordinate system of the radar relative to the global UTM coordinate system according to the marked dot points, it further includes:

[0029] Calibrate the coordinates of the calibrated vehicle detected from the video data, and establish a first mapping relationship between the pixel coordinates of the calibrated vehicle and the global UTM coordinate system;

[0030] Obtain the radar coordinates of the target vehicle in the radar coordinate system;

[0031] Obtain the target pixel coordinates of the target vehicle;

[0032] According to the second formula, obtain the theoretical pixel coordinates of the target vehicle;

[0033] The second formula is as follows:

[0034]

[0035] wherein, X represents the abscissa of the theoretical pixel coordinates, Y represents the ordinate of the theoretical pixel coordinates, T x represents the abscissa of the radar UTM coordinates, T y represents the ordinate of the radar UTM coordinates, X r represents the abscissa of the radar coordinates of the target vehicle, Y r represents the ordinate of the radar coordinates of the target vehicle, λ represents the coordinate adjustment factor, and the coordinate adjustment factor is initialized to 1, and H represents the first mapping relationship;

[0036] Determine whether the cosine distance between the theoretical pixel coordinates and the target pixel coordinates is greater than a preset distance;

[0037] If so, use the binary search algorithm to obtain the coordinate adjustment factor that satisfies the preset distance.

[0038] Preferably, for the joint calibration method of the radar and the camera, calibrating the coordinates of the calibrated vehicle detected from the video data and establishing a first mapping relationship between the pixel coordinates of the calibrated vehicle and the global UTM coordinate system, includes:

[0039] Obtain the detection frame coordinates of the calibrated vehicle in the video data, and use the center point coordinates of the detection frame as the pixel coordinates of the calibrated vehicle;

[0040] Convert the GPS coordinates of the calibrated vehicle to the global UTM coordinate system to obtain the global UTM coordinate system;

[0041] Obtain the homography matrix from the global UTM coordinate system to the pixel coordinates, and the homography matrix is the first mapping relationship.

[0042] Preferably, for the joint calibration method of the radar and the camera, obtaining the detection frame coordinates of the calibrated vehicle in the video data and using the center point coordinates of the detection frame as the pixel coordinates of the calibrated vehicle, includes:

[0043] Identify the calibrated vehicle in the video data based on a pre-trained deep learning vehicle re-identification algorithm;

[0044] Obtain the detection frame coordinates of the calibrated vehicle, and use the center point coordinates of the detection frame as the pixel coordinates of the calibrated vehicle.

[0045] Preferably, in the joint calibration method of the radar and the camera, the calibrated point tracks include target distance, angle, latitude, longitude, and altitude data information.

[0046] To solve the above technical problems, the present application also provides a joint calibration device for a radar and a camera, including:

[0047] An acquisition module, configured to acquire video data captured by the camera, radar measurement data monitored by the radar, camera GPS coordinates, radar GPS coordinates, and calibrated vehicle GPS coordinates;

[0048] A establishment module, configured to establish a global UTM coordinate system with the camera as the coordinate origin;

[0049] A radar UTM coordinate determination module, configured to convert the radar GPS coordinates to the global UTM coordinate system to obtain radar UTM coordinates;

[0050] A construction module, configured to construct a ROI region, and the monitoring points of the radar are located in the ROI region;

[0051] A GPS point track acquisition module, configured to acquire the GPS point tracks of the calibrated vehicle that are located in the ROI region and satisfy the continuous segmented preset distance threshold according to the GPS motion data;

[0052] A radar point track acquisition module, configured to acquire the radar point tracks of the calibrated vehicle corresponding to the time period of the GPS point tracks according to the radar measurement data, and the radar point tracks satisfy the continuous segmented preset distance threshold and the length preset threshold;

[0053] A determination module, configured to use the straight-line point tracks in the radar point tracks as calibrated point tracks;

[0054] A calculation module, configured to obtain the rotation angle of the radar coordinate system of the radar relative to the global UTM coordinate system according to the calibrated point tracks.

[0055] To solve the above technical problems, the present application also provides a joint calibration device for a radar and a camera, including:

[0056] A memory, configured to store a computer program;

[0057] A processor, configured to implement the steps of the joint calibration method of the radar and the camera when executing the computer program.

[0058] To solve the above technical problems, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the joint calibration method of the radar and the camera are implemented.

[0059] The joint calibration method of radar and camera provided by this application obtains the video data captured by the camera, the radar measurement data monitored by the radar, the GPS motion data uploaded by the calibration vehicle, the camera GPS coordinates, and the radar GPS coordinates. A global UTM coordinate system is established with the camera as the coordinate origin. The radar GPS coordinates are converted to the global UTM coordinate system, and the radar UTM coordinates can be obtained, that is, the offset distance of the radar relative to the global UTM coordinate system. An ROI area is constructed, and the monitoring points of the radar are located in the ROI area. According to the GPS motion data, the GPS traces of the calibration vehicle located in the ROI area and satisfying the continuous segmented preset distance threshold are obtained. According to the radar measurement data, the radar traces of the calibration vehicle corresponding to the time period of the GPS traces are obtained, and the radar traces satisfy the continuous segmented preset distance threshold and the length preset threshold. The straight traces in the radar traces are used as the calibration traces. According to the calibration traces, the rotation angle of the radar coordinate system relative to the global UTM coordinate system is obtained. According to the joint calibration method of radar and camera provided by this embodiment, it is not necessary for the radar and the camera to be set at the same longitude and latitude. The GPS motion data and radar measurement data of the calibration vehicle are used as the data sources for screening the calibration traces, and the rotation angle between the radar coordinate system and the global UTM coordinate system is calculated, that is, the mapping relationship between the global UTM coordinate system and the radar coordinate system, so as to obtain a unified expression of the same target position.

[0060] In addition, this application also provides a device and a medium, corresponding to the above method, with the same effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of this application, the drawings required for the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0062] Figure 1 It is a flowchart of a joint calibration method of radar and camera provided by an embodiment of this application;

[0063] Figure 2 It is a schematic diagram of a joint calibration device of radar and camera provided by an embodiment of this application;

[0064] Figure 3 It is a schematic diagram of another joint calibration device of radar and camera provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0065] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the protection scope of the present application.

[0066] The core of the present application is to provide a joint calibration method for a radar and a camera that does not limit the installation positions of the camera and the radar and can perform online self-calibration with high precision and speed.

[0067] In order to enable those skilled in the art to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0068] In the existing solution of converting the pixel coordinates and radar coordinates of all calibration vehicles into world coordinates respectively and solving the mapping relationship between the pixel coordinates and radar coordinates based on the global optimal matching algorithm, the camera and the millimeter-wave radar need to be installed at the same position so that their longitude and latitude coordinates are consistent, which is not applicable to traffic scenarios where the camera and the millimeter-wave radar are installed at different positions. In addition, in the existing solution, all calibration vehicles in the measurement area need to be calibrated. This process consumes a large amount of time in complex traffic scenarios, that is, when there are many vehicle targets, and cannot meet the requirements of rapid calibration in traffic scenarios. When there are few vehicle targets, it is difficult to ensure the calibration accuracy. And during this process, the camera calibration result is used to convert the vehicle pixel coordinates and world coordinates, but this mapping relationship is calculated from the coordinates of a point in the connected vehicle detection frame and the positioning data of the connected vehicle. It is obvious that there is a large error in using the coordinates of a point in the detection frame as the coordinates of the calibration vehicle. Applying this coordinate directly to the coordinate conversion, the accumulated error will further reduce the calibration accuracy and slow down the calibration speed.

[0069] Figure 1 The following is a flowchart of a joint calibration method for a radar and a camera provided in an embodiment of the present application. As Figure 1 shown, the joint calibration method for a radar and a camera includes:

[0070] S11: Obtain the video data captured by the camera, the radar measurement data monitored by the radar, the camera GPS coordinates, the radar GPS coordinates, and the calibration vehicle GPS coordinates;

[0071] It should be noted that this embodiment is applied to an environment with at least one camera, at least one radar, and at least one calibration vehicle. The calibration vehicle mentioned in this embodiment refers to a vehicle equipped with a Real-time Kinematic (RTK) survey instrument or a Global Positioning System (GPS) survey instrument, which is used to upload the GPS coordinate data of the calibration vehicle. When performing joint calibration in the embodiments of this application, only the measurement data of at least one calibration vehicle is required, and it is not necessary to calibrate the data of all calibration vehicles, thereby improving the calibration rate.

[0072] The video data captured by the camera in this embodiment includes the movement data of the calibration vehicle for subsequent use. In practical applications, the data captured by the camera may include calibration vehicles and non-calibration vehicles. Algorithms can be used to distinguish all the video data captured by the camera to obtain the movement data of the required calibration vehicle; the radar measurement data monitored by the radar refers to the moving point tracks of the calibration vehicle monitored by the radar. In practical applications, the data monitored by the radar may include calibration vehicles and non-calibration vehicles. Algorithms can be used to distinguish all the radar measurement data monitored by the radar to obtain the moving point tracks of the required calibration vehicle.

[0073] The GPS coordinates of the camera and the GPS coordinates of the radar can be detected when the camera or radar is installed, or can be uploaded in real time by the camera or radar. This embodiment does not make any restrictions.

[0074] In addition, the radar mentioned in this embodiment is a millimeter-wave radar.

[0075] S12: Establish a global UTM coordinate system with the camera as the coordinate origin;

[0076] S13: Convert the radar GPS coordinates to the global UTM coordinate system to obtain the radar UTM coordinates;

[0077] The horizontal and vertical coordinates of the radar UTM coordinates obtained in this embodiment represent the offset distances of the radar relative to the global UTM coordinate system.

[0078] S14: Construct a ROI region where the monitoring points of the radar are located in the ROI region;

[0079] The Region of Interest (ROI) region mentioned in this embodiment refers to a pre-set region, and the data detected in the ROI region is used as the basis for calibration calculation. Among them, the monitoring points of the radar need to be located within the ROI region.

[0080] Preferably, when the number of radars is greater than 3, select 3 or more radars; taking the calibration process of one radar and a camera as an example, the calibration of the remaining preset radars and the camera repeats this process.

[0081] Connect the positions of the radars to form a polygon area, and use the polygon area as the ROI area. Among them, the sum of the areas enclosed by the connection lines between the monitoring points of the radars and any two adjacent vertices of the ROI area is equal to the area of the ROI area.

[0082] The polygon area formed by the radars mentioned in this embodiment is the area with the largest area that the radars can enclose. If the sum of the areas enclosed by the connection lines between any radar monitoring point and any two adjacent vertices of the polygon area is equal to the area of the ROI area, it means that the radar monitoring point is located within the ROI area.

[0083] S15: According to the GPS movement data, obtain the GPS traces of the calibration vehicle located in the ROI area and meeting the continuous segmented preset distance threshold;

[0084] S16: According to the radar measurement data, obtain the radar traces of the calibration vehicle corresponding to the time period of the GPS traces, and the radar traces meet the continuous segmented preset distance threshold and the length preset threshold;

[0085] Among the radar measurement data monitored by the radar, fields such as timestamp, target ID, distance, and angle are included. Among the GPS movement data uploaded by the calibration vehicle, fields such as timestamp, latitude, longitude, and altitude are included.

[0086] In step S15, for the GPS traces in the ROI area, the GPS traces of the calibration vehicle are segmented according to whether the distance between adjacent points meets the continuous segmented preset distance threshold, and the GPS traces of the calibration vehicle that meet the requirements are obtained;

[0087] According to the radar measurement data, in step S16, according to the GPS traces of the calibration vehicle obtained in step S15, the radar traces corresponding to the corresponding time period are extracted, and the radar traces are divided according to the ID of the calibration vehicle to obtain the radar traces of the calibration vehicle in the corresponding time period, and then it is judged whether the obtained radar traces at this time meet the continuous segmented preset distance threshold and the length preset threshold to obtain the radar traces of the calibration vehicle that meet the requirements.

[0088] Through the screening of step S15 and step S16, the GPS traces and radar traces of the target vehicle that meet the preset distance threshold and are in the same time period are obtained.

[0089] S17: Take the straight - line traces in the radar traces as the calibration traces;

[0090] For the convenience of calculation, the straight track points in the radar track points are selected as the calibration track points for subsequent calculation. Preferably, the calibration track points include target distance, angle, latitude, longitude, and altitude data information.

[0091] S18: Obtain the rotation angle of the radar coordinate system of the radar relative to the global UTM coordinate system according to the calibration track points.

[0092] When the GPS track points and the radar track points satisfy the preset distance threshold and are in the same time period, and in combination with the global UTM coordinate system, the rotation angle of the radar coordinate system relative to the global UTM coordinate system can be calculated.

[0093] Specifically, obtain the video data captured by the camera, the radar measurement data monitored by the radar, the GPS motion data uploaded by the calibrated vehicle, the camera GPS coordinates, the radar GPS coordinates, and establish a global UTM coordinate system with the camera as the coordinate origin, construct a ROI area, the monitoring points of the radar are located in the ROI area, according to the GPS motion data, obtain the GPS track points of the calibrated vehicle located in the ROI area and satisfying the continuous segmented preset distance threshold, according to the radar measurement data, obtain the radar track points of the calibrated vehicle corresponding to the time period of the GPS track points, and the radar track points satisfy the continuous segmented preset distance threshold and the length preset threshold, take the straight track points in the radar track points as the calibration track points, according to the calibration track points, obtain the rotation angle of the radar coordinate system of the radar relative to the global UTM coordinate system. In addition, convert the radar GPS coordinates to the global UTM coordinate system, and the radar UTM coordinates can be obtained, that is, the offset distance of the radar relative to the global UTM coordinate system. According to the joint calibration method of the radar and the camera provided in this embodiment, it is not necessary for the radar and the camera to be set at the same longitude and latitude. The GPS motion data and the radar measurement data of the calibrated vehicle are used as the data sources for screening the calibration track points, and the rotation angle between the radar coordinate system and the global UTM coordinate system is calculated, that is, the mapping relationship between the global UTM coordinate system and the radar coordinate system, so as to obtain a unified expression of the same target position.

[0094] According to the above embodiment, this embodiment provides a preferred scheme for calculating the rotation angle. Step S18 includes obtaining the rotation angle of the radar coordinate system of the radar relative to the global UTM coordinate system according to the calibration track points, including:

[0095] Establish a local UTM coordinate system with the radar as the origin;

[0096] Obtain the first coordinates of the calibrated vehicle in the local UTM coordinate system, and the first coordinates are denoted as (X t , Y t );

[0097] Obtain the rectangular coordinates of the calibration track points, and the rectangular coordinates are marked as (X r , Y r );

[0098] Convert the rectangular coordinates to the local UTM coordinate system according to the first formula to obtain the second coordinates (X′ r , Y′ r );

[0099] The first formula is: X′ r = X r cos(R θ ) - Y r sin(R θ ), Y′ r = X r sin(R θ ) + Y r cos(R θ );

[0100] Among them, R θ is the rotation angle, and R θ is initialized to 0;

[0101] Through the optimization function, the optimal solution of the rotation angle is obtained by iterative gradient descent method;

[0102] The optimization function is: L(R θ ) = [X t - X′ r , Y t - Y′ r ;

[0103] Among them, L(R θ ) represents the value set of R θ .

[0104] It should be noted that obtaining the first coordinates of the calibration vehicle in the local UTM coordinate system mentioned in this embodiment refers to converting the GPS coordinates of the calibration vehicle to the UTM coordinate system.

[0105] The rectangular coordinates of the calibration point track mentioned in this embodiment refer to converting the polar coordinates of the calibration point track to rectangular coordinates for unified calculation; further, the second coordinates of the calibration point track in the local UTM coordinate system are obtained through the first formula.

[0106] Through iterative gradient descent method, the optimal solution of the rotation angle R θ is obtained, that is, the minimum value of the rotation angle R θ .

[0107] Through the calculation method provided in this embodiment, the rotation angle between the radar coordinate system and the global UTM coordinate system is calculated to obtain a unified expression of the same target position.

[0108] In the existing technical solutions, when the camera and the millimeter-wave radar have slight vibration offsets or rotations, the entire calibration step needs to be repeated for recalibration, which requires the connected vehicle to provide positioning data again. This undoubtedly increases the cost of calibration and is not conducive to large-scale applications in traffic scenarios. This embodiment provides a solution that does not require recalibration and realizes the correction of the rotation angle. After obtaining the rotation angle of the radar coordinate system of the radar relative to the global UTM coordinate system according to the calibration point traces, it further includes:

[0109] Calibrate the coordinates of the calibrated vehicle detected from the video data, and establish a first mapping relationship between the pixel coordinates of the calibrated vehicle and the global UTM coordinate system;

[0110] Obtain the radar coordinates of the target vehicle in the radar coordinate system;

[0111] Obtain the target pixel coordinates of the target vehicle;

[0112] According to the second formula, obtain the theoretical pixel coordinates of the target vehicle;

[0113] The second formula is:

[0114]

[0115] Where X represents the abscissa of the theoretical pixel coordinates, Y represents the ordinate of the theoretical pixel coordinates, T x represents the abscissa of the radar UTM coordinates, T y represents the ordinate of the radar UTM coordinates, X r represents the abscissa of the radar coordinates of the target vehicle, Y r represents the ordinate of the radar coordinates of the target vehicle, λ represents the coordinate adjustment factor, the coordinate adjustment factor is initialized to 1, and H represents the first mapping relationship;

[0116] Judge whether the cosine distance between the theoretical pixel coordinates and the target pixel coordinates is greater than the preset distance;

[0117] If so, use the binary search algorithm to obtain the coordinate adjustment factor that meets the preset distance.

[0118] It should be noted that the pixel coordinates mentioned in this embodiment refer to the pixel coordinates of the calibrated vehicle obtained from the video data captured by the camera. The global UTM coordinate system mentioned in this embodiment refers to converting the GPS coordinates of the calibrated vehicle to the global UTM coordinate system.

[0119] The target vehicle mentioned in this embodiment is different from the calibrated vehicle, and there is no need to set a device for uploading GPS positioning data on the target vehicle. The pixel coordinates of the target vehicle refer to the pixel coordinates of the target vehicle obtained from the video data captured by the camera.

[0120] The radar UTM coordinates mentioned in this embodiment are the coordinates obtained by converting the radar GPS coordinates to the global UTM coordinate system, and the horizontal and vertical coordinates represent the offset distances of the local UTM coordinate system relative to the global UTM coordinate system.

[0121] If the rotation angle does not change due to the positions of the camera or the radar, when the coordinate adjustment factor is 1, the theoretical pixel coordinates obtained according to the second formula are the same as the target pixel coordinates.

[0122] When the theoretical pixel coordinates are not the same as the target pixel coordinates, calculate whether the cosine distance between the theoretical pixel coordinates and the target pixel coordinates is greater than the preset distance, that is, whether the current offset is within the expected range. If the cosine distance exceeds the preset distance, the rotation angle needs to be corrected, and the coordinate adjustment factor that meets the preset distance is obtained through the binary search algorithm, that is, through the value of the coordinate adjustment factor in this table, so that the cosine distance between the theoretical pixel coordinates and the target pixel coordinates is less than the preset distance.

[0123] Through the solution provided in this embodiment, after calculating the rotation angle, there is no need to repeat the calibration process again, and the rotation angle is corrected through the coordinate adjustment factor to reduce the calibration cost.

[0124] According to the above embodiment, this embodiment provides a preferred solution for calculating pixel coordinates, calibrates the coordinates of the calibrated vehicle detected from the video data, and establishes a first mapping relationship between the pixel coordinates of the calibrated vehicle and the global UTM coordinate system, including:

[0125] Obtain the detection box coordinates of the calibrated vehicle in the video data, and use the center point coordinates of the detection box as the pixel coordinates of the calibrated vehicle;

[0126] Convert the GPS coordinates of the calibrated vehicle to the global UTM coordinate system to obtain the global UTM coordinate system;

[0127] Obtain the homography matrix from the global UTM coordinate system to the pixel coordinates, and the homography matrix is the first mapping relationship.

[0128] By using the center point of the detection box as the pixel coordinates of the calibrated vehicle, the calibration accuracy is improved. In addition, the pixel coordinates of the target vehicle can also be obtained by selecting the center point of the detection box as the pixel coordinates of the target vehicle to improve the accuracy of calculating the coordinate adjustment factor.

[0129] According to the above embodiment, this embodiment provides a preferred solution to obtain the detection box coordinates of the calibrated vehicle in the video data and use the center point coordinates of the detection box as the pixel coordinates of the calibrated vehicle, including:

[0130] Identify the calibrated vehicle in the video data based on the pre-trained deep learning vehicle re-identification algorithm;

[0131] Obtain the detection box coordinates of the calibration vehicle, and use the coordinates of the center point of the detection box as the pixel coordinates of the calibration vehicle.

[0132] The deep learning vehicle re-identification algorithm based on pre-training mentioned in this embodiment obtains a trained deep learning vehicle re-identification algorithm that can be used to identify the calibration vehicle through pre-training, so as to better identify the calibration vehicle for subsequent calibration calculations.

[0133] In the above embodiment, the joint calibration method of the radar and the camera is described in detail. The present application also provides an embodiment corresponding to the joint calibration device of the radar and the camera. It should be noted that the present application describes the embodiments of the device part from two perspectives, one is from the perspective of functional modules, and the other is from the perspective of hardware.

[0134] From the perspective of functional modules, Figure 2 is a schematic diagram of a joint calibration device for a radar and a camera provided by an embodiment of the present application, as Figure 2 shown, including:

[0135] An acquisition module 21, configured to acquire video data captured by the camera, radar measurement data monitored by the radar, GPS motion data uploaded by the calibration vehicle, camera GPS coordinates, and radar GPS coordinates;

[0136] A building module 22, configured to establish a global UTM coordinate system with the camera as the coordinate origin;

[0137] A radar UTM coordinate determination module 23, configured to convert the radar GPS coordinates to the global UTM coordinate system to obtain radar UTM coordinates;

[0138] A construction module 24, configured to construct an ROI area, and the monitoring points of the radar are located in the ROI area;

[0139] A GPS trace acquisition module 25, configured to acquire the GPS traces of the calibration vehicle located in the ROI area and satisfying the continuous segmented preset distance threshold according to the GPS motion data;

[0140] A radar trace acquisition module 26, configured to acquire the radar traces of the calibration vehicle corresponding to the time period of the GPS traces according to the radar measurement data, and the radar traces satisfy the continuous segmented preset distance threshold and the length preset threshold;

[0141] A determination module 27, configured to use the straight-line traces in the radar traces as calibration traces;

[0142] A calculation module 28, configured to obtain the rotation angle of the radar coordinate system of the radar relative to the global UTM coordinate system according to the calibration traces.

[0143] Since the embodiments in the device part correspond to the embodiments in the method part, please refer to the description of the embodiments in the method part for the embodiments in the device part, which will not be elaborated here.

[0144] Specifically, the video data captured by the camera, the radar measurement data monitored by the radar, the GPS motion data uploaded by the calibrated vehicle, the camera GPS coordinates, and the radar GPS coordinates are obtained through the acquisition module 21. The global UTM coordinate system is established with the camera as the coordinate origin through the establishment module 22. The radar UTM coordinate module 23 converts the radar GPS coordinates to the global UTM coordinate system to obtain the radar UTM coordinates; the construction module 24 constructs the ROI area, and the monitoring points of the radar are located in the ROI area; the GPS track module 25 obtains the GPS track of the calibrated vehicle located in the ROI area and satisfying the continuous segmented preset distance threshold according to the GPS motion data obtained by the acquisition module 21; the radar track module 26 obtains the radar track of the calibrated vehicle corresponding to the time period of the GPS track according to the radar measurement data obtained by the acquisition module 21, and the radar track satisfies the continuous segmented preset distance threshold and the length preset threshold; the determination module 27 takes the straight track in the radar track obtained by the radar track acquisition module 25 as the calibration track; the calculation module 28 obtains the rotation angle of the radar coordinate system relative to the global UTM coordinate system according to the calibration track. This embodiment provides a combined calibration device for a radar and a camera. It is not necessary for the radar and the camera to be set at the same longitude and latitude. The GPS motion data and radar measurement data of the calibrated vehicle are used as the data sources for screening the calibration tracks, and the rotation angle between the radar coordinate system and the global UTM coordinate system, that is, the mapping relationship between the global UTM coordinate system and the radar coordinate system, is calculated to obtain a unified expression of the same target position.

[0145] Figure 3 The following is a structural diagram of another combined calibration device for a radar and a camera provided by an embodiment of the present application, as Figure 3 shown, the combined calibration device for a radar and a camera includes: a memory 30 for storing a computer program;

[0146] a processor 31 for implementing the steps of the combined calibration method for a radar and a camera in the above embodiment when executing the computer program.

[0147] The combined calibration device for a radar and a camera provided in this embodiment may include, but is not limited to, a smart phone, a tablet computer, a notebook computer, or a desktop computer, etc.

[0148] Among them, the processor 31 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 31 may be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 31 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 31 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 31 may further include an artificial intelligence (AI) processor, and the AI processor is used to process computational operations related to machine learning.

[0149] The memory 30 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 30 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In this embodiment, the memory 30 is at least used to store the following computer program 301. After the computer program is loaded and executed by the processor 31, it can implement the relevant steps of the joint calibration method of the radar and the camera disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 30 may further include an operating system 302 and data 303, etc., and the storage method may be transient storage or permanent storage. Among them, the operating system 302 may include Windows, Unix, Linux, etc. The data 303 may include, but is not limited to, data related to implementing the joint calibration method of the radar and the camera.

[0150] In some embodiments, the joint calibration device of the radar and the camera may further include a display screen 32, an input / output interface 33, a communication interface 34, a power supply 35, and a communication bus 36.

[0151] Those skilled in the art can understand that Figure 3 the structure shown in

[0152] The joint calibration device for radar and camera provided by the embodiment of the present application includes a memory and a processor. When the processor executes the program stored in the memory, the following method can be implemented: the joint calibration method for radar and camera, which does not require the radar and the camera to be set at the same longitude and latitude. The GPS motion data of the calibration vehicle and the radar measurement data are used as the data sources for screening calibration point traces, and the rotation angle between the radar coordinate system and the global UTM coordinate system is calculated. Then, the mapping relationship between the global UTM coordinate system and the radar coordinate system is established to obtain a unified expression of the same target position.

[0153] Finally, the present application also provides an embodiment corresponding to a computer-readable storage medium. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps recorded in the joint calibration method embodiment of the radar and camera as described above are implemented.

[0154] It can be understood that if the method in the above embodiments is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store program codes.

[0155] The computer-readable storage medium provided by this embodiment stores a computer program. When the processor executes this program, the following method can be implemented: the joint calibration method for radar and camera, which does not require the radar and the camera to be set at the same longitude and latitude. The GPS motion data of the calibration vehicle and the radar measurement data are used as the data sources for screening calibration point traces, and the rotation angle between the radar coordinate system and the global UTM coordinate system is calculated. Then, the mapping relationship between the global UTM coordinate system and the radar coordinate system is established to obtain a unified expression of the same target position.

[0156] The above has introduced in detail the method, device and medium for joint calibration of radar and camera provided by this application. Each embodiment in the specification is described in a progressive manner. The key point of each embodiment is the difference from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple. For the relevant parts, reference can be made to the description in the method part. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0157] It should also be noted that in this specification, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of another identical element in the process, method, article or device including the element.

Claims

1. A joint calibration method for a radar and a camera, characterized in that Including: Obtaining video data captured by a camera, radar measurement data monitored by a radar, GPS motion data uploaded by a calibrated vehicle, camera GPS coordinates, and radar GPS coordinates; the radar and the camera are set at different latitudes and longitudes; Establishing a global UTM coordinate system with the camera as the coordinate origin; Converting the radar GPS coordinates to the global UTM coordinate system to obtain radar UTM coordinates; Constructing an ROI region, where the monitoring points of the radar are located in the ROI region; According to the GPS motion data, obtaining the GPS traces of the calibrated vehicle that are located in the ROI region and satisfy the continuous segmented preset distance threshold; According to the radar measurement data, obtaining the radar traces of the calibrated vehicle corresponding to the time period of the GPS traces, and the radar traces satisfy the continuous segmented preset distance threshold and the length preset threshold; Taking the straight-line traces in the radar traces as calibration traces; According to the calibration traces, obtaining the rotation angle of the radar coordinate system of the radar relative to the global UTM coordinate system; Wherein, after obtaining the rotation angle of the radar coordinate system of the radar relative to the global UTM coordinate system according to the calibration traces, it further includes: Calibrating the coordinates of the calibrated vehicle detected in the video data, and establishing a first mapping relationship between the pixel coordinates of the calibrated vehicle and the global UTM coordinate system; Obtaining the target vehicle radar coordinates of the target vehicle in the radar coordinate system; Obtaining the target pixel coordinates of the target vehicle; Judging whether the cosine distance between the theoretical pixel coordinates and the target pixel coordinates of the target vehicle is greater than a preset distance; If so, obtaining a coordinate adjustment factor that satisfies the preset distance by means of a binary search algorithm; Correcting the rotation angle through the coordinate adjustment factor.

2. The joint calibration method of the radar and the camera according to claim 1, wherein The constructing the ROI region, where the monitoring points of the radar are located in the ROI region, includes: Selecting 3 or more of the radars; Connecting the positions of the radars to enclose a polygon region, and taking the polygon region as the ROI region, wherein the sum of the areas enclosed by the connections between the monitoring points of the radar and any two adjacent vertices of the ROI region is equal to the area of the ROI region.

3. The joint calibration method of the radar and the camera according to claim 1, characterized in that The obtaining the rotation angle of the radar coordinate system of the radar relative to the global UTM coordinate system according to the calibration traces includes: Establishing a local UTM coordinate system with the radar as the origin; Obtain the first coordinate of the calibrated vehicle in the local UTM coordinate system, and denote the first coordinate as ; Obtain the rectangular coordinates of the marked dot points, and mark the rectangular coordinates as ; Convert the rectangular coordinates to the local UTM coordinate system according to the first formula to obtain the second coordinates ; The first formula is as follows: , ; Wherein, is the rotation angle and is initialized to be 0; Through an optimization function, iteratively obtaining the optimal solution of the rotation angle by the gradient descent method; The optimization function is as follows: ; Among them, denotes the set of values taken.

4. The joint calibration method of the radar and the camera according to claim 3, wherein Before judging whether the cosine distance between the theoretical pixel coordinates and the target pixel coordinates of the target vehicle is greater than a preset distance, it further includes: According to a second formula, obtaining the theoretical pixel coordinates of the target vehicle; The second formula is: ； Among them, represents the abscissa of the theoretical pixel coordinates, represents the ordinate of the theoretical pixel coordinates, represents the abscissa of the radar UTM coordinates, represents the ordinate of the radar UTM coordinates, represents the abscissa of the radar coordinates of the target vehicle, represents the ordinate of the radar coordinates of the target vehicle, represents the coordinate adjustment factor, and the coordinate adjustment factor is initialized to 1, represents the first mapping relationship.

5. The joint calibration method of the radar and the camera according to claim 4, characterized in that, The calibrating the coordinates of the calibrated vehicle detected in the video data and establishing a first mapping relationship between the pixel coordinates of the calibrated vehicle and the global UTM coordinate system includes: Obtaining the detection frame coordinates of the calibrated vehicle in the video data, and taking the center point coordinates of the detection frame as the pixel coordinates of the calibrated vehicle; Convert the GPS coordinates of the calibrated vehicle to the global UTM coordinate system to obtain the global UTM coordinate system; Obtain the homography matrix from the global UTM coordinate system to the pixel coordinates, and the homography matrix is the first mapping relationship.

6. The joint calibration method of the radar and the camera according to claim 5, wherein Obtain the detection box coordinates of the calibrated vehicle in the video data, and use the center point coordinates of the detection box as the pixel coordinates of the calibrated vehicle, including: Identify the calibrated vehicle in the video data based on a pre-trained deep learning vehicle re-identification algorithm; Obtain the detection box coordinates of the calibrated vehicle, and use the center point coordinates of the detection box as the pixel coordinates of the calibrated vehicle.

7. The joint calibration method of the radar and the camera according to claim 1, characterized in that, The calibrated dot tracks include target distance, angle, latitude, longitude, and altitude data information.

8. A combined calibration device for a radar and a camera, characterized in that Include: An acquisition module for acquiring video data captured by a camera, radar measurement data monitored by a radar, camera GPS coordinates, radar GPS coordinates, and calibrated vehicle GPS coordinates; the radar and the camera are set at different latitudes and longitudes; A establishment module for establishing a global UTM coordinate system with the camera as the coordinate origin; A radar UTM coordinate determination module for converting the radar GPS coordinates to the global UTM coordinate system to obtain radar UTM coordinates; A construction module for constructing an ROI area where the monitoring points of the radar are located; A GPS dot track acquisition module for acquiring the GPS dot tracks of the calibrated vehicle located in the ROI area and satisfying the continuous segmented preset distance threshold according to the GPS motion data; A radar dot track acquisition module for acquiring the radar dot tracks of the calibrated vehicle corresponding to the time period of the GPS dot tracks according to the radar measurement data, and the radar dot tracks satisfy the continuous segmented preset distance threshold and the length preset threshold; A determination module for using the straight-line dot tracks in the radar dot tracks as calibrated dot tracks; A calculation module for obtaining the rotation angle of the radar coordinate system of the radar relative to the global UTM coordinate system according to the calibrated dot tracks; Wherein, after obtaining the rotation angle of the radar coordinate system of the radar relative to the global UTM coordinate system according to the calibrated dot tracks, it further includes: Calibrate the coordinates of the calibrated vehicle detected in the video data, and establish a first mapping relationship between the pixel coordinates of the calibrated vehicle and the global UTM coordinate system; Obtain the target vehicle radar coordinates of the target vehicle in the radar coordinate system; Obtain the target pixel coordinates of the target vehicle; Judge whether the cosine distance between the theoretical pixel coordinates and the target pixel coordinates of the target vehicle is greater than a preset distance; If so, obtain a coordinate adjustment factor that satisfies the preset distance by a binary search algorithm; Correct the rotation angle through the coordinate adjustment factor.

9. A combined calibration device for a radar and a camera, characterized in that, Include: A memory for storing a computer program; A processor for implementing the steps of the joint calibration method of the radar and the camera according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method for jointly calibrating a radar and a camera according to any one of claims 1 to 7 are implemented.

Images