Millimeter wave radar and camera calibration method and device, storage medium and electronic equipment

Abstract

The invention discloses a millimeter wave radar and camera calibration method and device, a storage medium and electronic equipment, and the method comprises the steps: carrying out the detection in a millimeter wave radar frame view and a camera frame image, obtaining a radar target and a camera target, and carrying out the screening, and obtaining an effective target; matching the effective radar target with the effective camera target, and determining N matching pairs; determining matching pairs participating in calibration in all the matching pairs; calibrating the millimeter-wave radar and the camera by using the matching pairs participating in calibration to obtain a homography matrix and a calibration error of the calibration; and determining a matching pair participating in calibration next time based on the calibration error, and executing processing of determining the homography matrix and the calibration error until the calibration error meeting a preset quality requirement is determined, and outputting the calibrated homography matrix corresponding to the calibration error meeting the quality requirement as a calibration result of the millimeter wave radar and the camera. By applying the method, the accuracy and the calibration efficiency of the calibration relation can be greatly improved.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to a calibration method, device, storage medium and electronic device for millimeter wave radar and camera. Background Art

[0002] With the advancement of image processing technology, image processing technology has been increasingly widely used in traffic control systems.

[0003] In traffic scenarios, vehicle traffic jams and traffic queues often occur. In order to reduce the occurrence of these situations, it is often necessary to count the traffic volume and speed at traffic intersections and road sections. In traffic scenarios, cameras mainly provide image information, and can analyze the outline, color, license plate and other information of traffic targets through algorithms, but cannot obtain information such as the distance and speed of traffic targets. At the same time, under severe weather conditions (such as rain, snow, fog, etc.), the image information of the camera is distorted, and it is difficult to provide accurate traffic target information. Millimeter-wave radar can provide accurate distance and speed information of traffic targets, and can work all-weather, unaffected by severe weather, which makes up for the disadvantages of the camera. Therefore, in order to integrate the advantages of the two sensors, the data obtained by the two sensors are usually fused and applied to traffic scenarios around the clock.

[0004] The output data of the millimeter wave radar and camera sensors are the results of their respective coordinate systems. Therefore, in order to fuse the output data of the two sensors, it is first necessary to calibrate the coordinate systems of the two sensors to realize the mutual conversion of the two sensor data.

[0005] The calibration relationship obtained by the existing calibration methods of millimeter-wave radar and camera is not very accurate. Summary of the invention

[0006] The present application provides a millimeter-wave radar and camera calibration method, device, storage medium and electronic device, which can greatly improve the accuracy of the calibration relationship.

[0007] To achieve the above objectives, this application adopts the following technical solutions:

[0008] A millimeter wave radar and camera calibration method, comprising:

[0009] The radar target and the camera target are detected in the frame view of the millimeter wave radar and the frame view of the camera respectively;

[0010] The radar target and the camera target are screened respectively to obtain a valid radar target and a valid camera target;

[0011] Matching the valid radar target with the valid camera target to determine N matching pairs;

[0012] Determining a matching pair participating in calibration among all the matching pairs;

[0013] Using the matching pairs involved in the calibration, the millimeter-wave radar and the camera are calibrated to obtain a homography matrix of this calibration, which is used to characterize the mapping relationship between the millimeter-wave radar view and the camera view;

[0014] Based on the homography matrix, all the valid camera targets are converted to the millimeter-wave radar view to obtain auxiliary camera targets. Taking targets as units, bipartite graph matching is performed based on a first queue consisting of all the auxiliary camera targets under multiple frames and a second queue consisting of all the valid radar targets under the multiple frames to obtain a matching relationship with the minimum error, and the error sum of all errors under the matching relationship is calculated as the calibration error corresponding to this calibration and the matching pairs participating in the calibration;

[0015] Based on the calibration error, determining a matching pair to participate in the next calibration, and performing a process of determining a homography matrix and a calibration error until a calibration error that meets a preset quality requirement is determined, and outputting a homography matrix calibrated corresponding to the calibration error that meets the quality requirement as a calibration result of the millimeter wave radar and the camera;

[0016] Wherein, N is a natural number greater than 1.

[0017] Preferably, in the first N calibrations, each calibration is performed using a matching pair, and the matching pairs involved in each calibration are different, and the homography matrix and calibration error corresponding to the first N calibrations are obtained;

[0018] In the calibration after the Nth time, the number of matching pairs participating in the calibration is greater than 1. When determining the matching pairs participating in the next calibration based on the calibration error, determine whether the calibration error corresponding to this calibration is less than all the calibration errors obtained. If so, retain the matching pairs newly introduced in this calibration to participate in the subsequent calibration; otherwise, the matching pairs newly introduced in this calibration no longer participate in the subsequent calibration; determine that the matching pairs participating in the next calibration include: the matching pairs that have been retained and a matching pair that has not participated in the calibration after the Nth time.

[0019] Preferably, after the Nth calibration is completed, determining the matching pair to participate in the next calibration includes:

[0020] Select the smallest calibration error from all the calibration errors obtained, take the matching pair corresponding to the smallest calibration error as the selected matching pair set, and take the smallest calibration error as the current calibration error; select a matching pair from the remaining matching pairs as the current test matching pair to be added to the selected matching pair set, take all matching pairs in the selected matching pair set as the matching pairs participating in the calibration for the N+1th time; and take all matching pairs except the matching pairs participating in the calibration for the N+1th time to form a test matching pair set;

[0021] After the nth calibration is completed, determining the matching pair to participate in the next calibration includes:

[0022] Determine whether the calibration error determined after the nth calibration is less than the current calibration error; if so, keep the selected matching pair set unchanged, and update the current calibration error to the calibration error determined after the nth calibration; otherwise, delete the current test matching pair from the selected matching pair set, and keep the current calibration error unchanged;

[0023] Update the current test matching pair to a matching pair in the test matching pair set, add the current test matching pair to the selected matching pair set, use all matching pairs in the selected matching pair set as matching pairs participating in the calibration for the n+1th time, and delete the current test matching pair from the test matching pair set;

[0024] Wherein, n is an integer greater than N.

[0025] Preferably, when selecting and updating the current test matching pair, the matching pair corresponding to the minimum calibration error in the selected matching pair set is used as the current test matching pair.

[0026] Preferably, the number of times a matching pair is deleted from the selected matching pair set is counted, and when the count value reaches a set threshold, it is determined that the current calibration error meets the quality requirement; or,

[0027] When the test matching pair set is 0, the last calibration is completed, and the smaller value between the calibration error determined after the last calibration and the current calibration error is used as the calibration error that meets the quality requirement;

[0028] or,

[0029] When the value of the current calibration error is less than the set error threshold, it is determined that the current calibration error meets the quality requirement.

[0030] Preferably, the screening of the radar target and the camera target respectively includes:

[0031] Determine the motion trajectory of the radar target, take the radar target moving straight as the valid radar target, and take the radar target moving non-straight as the invalid radar target;

[0032] The motion trajectory of the camera target is determined, and the camera target moving straight is used as the valid radar target, and the camera target moving non-straight is used as the invalid camera target.

[0033] Preferably, matching the valid radar target with the valid camera target includes:

[0034] For each to-be-matched pair consisting of two of the valid radar targets and the valid camera targets, obtaining an overlapping survival time based on the survival times of the two;

[0035] The to-be-matched pairs whose overlapping survival time is greater than the matching threshold are determined as the matching pairs.

[0036] Preferably, the calibrating the millimeter wave radar and the camera using the matching pair involved in the calibration includes:

[0037] For the matching pair involved in calibration, the coordinate values ​​of the two targets in the matching pair are sampled in the frame image of the millimeter wave radar and the frame image of the camera respectively; wherein, during sampling, the targets are uniformly sampled in different set areas.

[0038] Preferably, the first queue composed of all the auxiliary camera targets and the second queue composed of all the valid radar targets are subjected to bipartite graph matching processing, including:

[0039] The targets in the first queue and the second queue are taken as vertices of a bipartite graph, and the error between the two vertices is taken as an edge. The Hungarian matching method is performed for matching processing, and after each search for the targets in the first queue or the second queue, if no augmenting path is found, the search record array is not cleared.

[0040] A millimeter wave and camera calibration device, comprising: a target detection unit, a target screening unit, a matching pair determination unit, a calibration relationship acquisition unit and a calibration error checking module;

[0041] The target detection unit is used to detect the radar target and the camera target in the frame image of the millimeter wave radar and the frame image of the camera respectively;

[0042] The target screening unit is used to screen the radar target and the camera target respectively to obtain a valid radar target and a valid camera target;

[0043] The matching pair determination unit is used to match the valid radar target with the valid camera target to determine N matching pairs;

[0044] The calibration relationship acquisition unit is used to determine the matching pairs participating in each calibration from among all the matching pairs, and use the matching pairs participating in the calibration to calibrate the millimeter-wave radar and the camera to obtain a corresponding homography matrix for characterizing the mapping relationship between the millimeter-wave radar plane and the camera plane, until a calibration error that meets a preset quality requirement is determined from the calibration error fed back by the calibration error verification unit, and the homography matrix corresponding to the calibration error that meets the quality requirement is output as the calibration result of the millimeter-wave radar and the camera;

[0045] The calibration error checking unit is used to convert all the valid camera targets into the millimeter wave radar view based on the homography matrix to obtain auxiliary camera targets, perform bipartite graph matching processing based on the first queue composed of all the auxiliary camera targets and the second queue composed of all the valid radar targets to obtain a matching relationship with the minimum error, and calculate the error sum of all errors under the matching relationship as the calibration error corresponding to this calibration and the matching pairs participating in the calibration, and feed it back to the calibration relationship acquisition unit;

[0046] Wherein, N is a natural number greater than 1.

[0047] Preferably, in the calibration relationship acquisition unit and the calibration verification unit,

[0048] In the first N calibrations, each calibration is performed using a matching pair, and the matching pairs involved in each calibration are different, and the homography matrix and calibration error corresponding to the first N calibrations are obtained;

[0049] In the calibration after the Nth time, the number of matching pairs participating in the calibration is greater than 1. When determining the matching pairs participating in the next calibration based on the calibration error, determine whether the calibration error corresponding to this calibration is less than all the calibration errors obtained. If so, retain the matching pairs newly introduced in this calibration to participate in the subsequent calibration; otherwise, the matching pairs newly introduced in this calibration no longer participate in the subsequent calibration; determine that the matching pairs participating in the next calibration include: the matching pairs that have been retained and a matching pair that has not participated in the calibration after the Nth time.

[0050] Preferably, in the calibration relationship acquisition unit,

[0051] After the Nth calibration is completed, the processing of the matching pairs participating in the next calibration is determined, including:

[0052] Select the smallest calibration error from all the calibration errors obtained, take the matching pair corresponding to the smallest calibration error as the selected matching pair set, and take the smallest calibration error as the current calibration error; select a matching pair from the remaining matching pairs as the current test matching pair to be added to the selected matching pair set, take all matching pairs in the selected matching pair set as the matching pairs participating in the calibration for the N+1th time; and take all matching pairs except the matching pairs participating in the calibration for the N+1th time to form a test matching pair set;

[0053] After the nth calibration is completed, determining the matching pair to participate in the next calibration includes:

[0054] Determine whether the calibration error determined after the nth calibration is less than the current calibration error; if so, keep the selected matching pair set unchanged, and update the current calibration error to the calibration error determined after the nth calibration; otherwise, delete the current test matching pair from the selected matching pair set, and keep the current calibration error unchanged;

[0055] Update the current test matching pair to a matching pair in the test matching pair set, add the current test matching pair to the selected matching pair set, use all matching pairs in the selected matching pair set as matching pairs participating in the calibration for the n+1th time, and delete the current test matching pair from the test matching pair set;

[0056] Wherein, n is an integer greater than N.

[0057] Preferably, in the calibration relationship acquisition unit, when selecting and updating the current test matching pair, the matching pair corresponding to the minimum calibration error in the selected matching pair set is used as the current test matching pair.

[0058] Preferably, the calibration relationship acquisition unit is further used to count the number of times a matching pair is deleted from the selected matching pair set, and when the count value reaches a set threshold, determine that the current calibration error meets the quality requirement; or,

[0059] In the calibration relationship acquisition unit, when the test matching pair set is 0, the last calibration is completed, and the smaller error value between the calibration error determined after the last calibration and the current calibration error is used as the calibration error that meets the quality requirement;

[0060] or,

[0061] In the calibration relationship acquisition unit, when the value of the current calibration error is less than a set error threshold, it is determined that the current calibration error meets the quality requirement.

[0062] Preferably, in the target screening unit, screening the radar target and the camera target respectively includes:

[0063] Determine the motion trajectory of the radar target, take the radar target moving straight as the valid radar target, and take the radar target moving non-straight as the invalid radar target;

[0064] The motion trajectory of the camera target is determined, and the camera target moving straight is used as the valid radar target, and the camera target moving non-straight is used as the invalid camera target.

[0065] Preferably, in the matching pair determination unit, matching the valid radar target with the valid camera target includes:

[0066] For each to-be-matched pair consisting of two of the valid radar targets and the valid camera targets, obtaining an overlapping survival time based on the survival times of the two;

[0067] The to-be-matched pairs whose overlapping survival time is greater than the matching threshold are determined as the matching pairs.

[0068] Preferably, in the calibration relationship acquisition unit, the calibrating of the millimeter wave radar and the camera using the matching pair involved in the calibration includes:

[0069] For the matching pair involved in calibration, the coordinate values ​​of the two targets in the matching pair are sampled in the frame image of the millimeter wave radar and the frame image of the camera respectively; wherein, during sampling, the targets are uniformly sampled in different set areas.

[0070] Preferably, in the calibration relationship acquisition unit, the first queue composed of all the auxiliary camera targets and the second queue composed of all the valid radar targets are subjected to bipartite graph matching processing, including:

[0071] The targets in the first queue and the second queue are taken as vertices of a bipartite graph, and the error between the two vertices is taken as an edge. The Hungarian matching method is performed for matching processing, and after each search for the targets in the first queue or the second queue, if no augmenting path is found, the search record array is not cleared.

[0072] A computer-readable storage medium stores computer instructions, which, when executed by a processor, can implement any of the above-mentioned methods for calibrating a millimeter-wave radar and a camera.

[0073] An electronic device comprising at least a computer-readable storage medium and a processor;

[0074] The processor is used to read executable instructions from the storage medium readable by the computer and execute the instructions to implement any of the above-mentioned millimeter-wave radar and camera calibration methods.

[0075] As can be seen from the above technical solution, in this application, first, target detection and effective target screening are performed on the frame image of the millimeter wave radar and the frame image of the camera to obtain effective radar targets and effective camera targets, which represent the target objects under the radar view and the target objects under the camera view, respectively. Next, the effective targets under the two views are matched to obtain N groups of matching pairs for calibration of the millimeter wave radar and the camera. The calibration is performed multiple times in a loop. Before each calibration, the matching pairs participating in the calibration are selected and calibrated to obtain the corresponding homography matrix. Based on the homography matrix, the valid targets under the camera view are converted to the radar view, and bipartite graph matching is performed with all valid targets under the radar view to obtain a matching relationship with the minimum error. Under the matching relationship, the error between the target converted to the radar view and the actual target under the radar view is calculated. All errors are summed to obtain the error sum corresponding to this calibration, which is used to measure the accuracy of this calibration, and then the matching pairs participating in the next calibration are adjusted based on the error sum. The calibration is performed multiple times in a loop in the above manner until the error sum that meets the preset quality requirements is determined, and the homography matrix corresponding to the error sum for that calibration is output as the calibration result. In the above manner, after the calibration is completed, the calibration result is verified to determine the error sum used to characterize the accuracy of the calibration, and based on the error sum, the matching pairs participating in the next calibration are adjusted to finally determine the error sum that meets the preset quality requirements, that is, to determine which calibration accuracy meets the quality requirements, and use the homography matrix of this calibration as the final calibration result, thereby effectively improving the calibration accuracy of the millimeter-wave radar and the camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 A schematic diagram of the basic flow of the millimeter wave radar and camera calibration method in this application;

[0077] Figure 2a and Figure 2b They are respectively image examples corresponding to two frames in the target screening method of this application;

[0078] Figure 3 This is a schematic diagram of a specific process of a calibration method for a millimeter wave radar and a camera in a specific embodiment of the present application;

[0079] Figure 4 This is an example diagram of optimizing the Hungarian algorithm in the embodiments of this application;

[0080] Figure 5 This is a basic structural diagram of the calibration device for millimeter wave radar and camera of this application;

[0081] Figure 6 It is a schematic diagram of the basic structure of the electronic device in this application. DETAILED DESCRIPTION

[0082] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0083] In the existing calibration methods of millimeter wave radar and camera, usually after obtaining matching targets for multiple consecutive frames, the calibration relationship between the two sensors is directly calculated based on the matching pairs, and the calculated calibration relationship is output as the calibration result. In this processing method, the calculated calibration relationship is not verified, and the accuracy of the calibration result cannot be guaranteed.

[0084] The basic idea of ​​this application is to verify the calibration results and continuously improve the selection of matching pairs involved in the calibration in a data-driven manner, so as to finally determine a calibration relationship that meets the quality requirements.

[0085] Figure 1 Schematic diagram of the basic flow of the calibration method of the millimeter wave radar and camera in this application. Figure 1 As shown, the method includes:

[0086] Step 101 : Detect and obtain a radar target and a camera target in a frame view of a millimeter wave radar and a frame view of a camera respectively.

[0087] In this step, the radar target is detected in the frame view of the millimeter wave radar, and the camera target is detected in the frame view of the camera. Detecting the target in the frame view means detecting it in the view image in frames, and the detection method can adopt various existing methods, which will not be repeated here.

[0088] Step 102 , screening radar targets and camera targets respectively to obtain valid radar targets and valid camera targets.

[0089] Among the targets detected in step 101, some may not be the desired targets, so it is necessary to screen targets to select valid targets for subsequent processing. The method of screening valid targets can adopt existing methods, such as based on information such as lane lines in a set area.

[0090] The present application provides a new target screening method, which does not require the use of additional information such as lane lines, target category information, etc., and only screens the information of millimeter-wave radar targets and camera targets themselves. In traffic scenarios, sensors are installed to aim at straight roads. Therefore, when screening targets in the present application, the motion trajectory of the target is first determined, and then the straight-moving targets are regarded as valid targets, and the non-straight-moving targets are regarded as invalid targets. In specific implementation, multiple frames of targets can be taken continuously to determine whether the motion trajectory of the target under the two sensors is straight. If so, it is screened as a valid target, otherwise, it is screened as an invalid target. As shown in Figure 2, an example of the target screening method provided by the present application, the left view is the camera view, and the right view is the millimeter-wave radar view, where Figure 2a is the first frame image, Figure 2b is the sixth frame image. Figure 2a and 2b In the figure, the target circled by the gray circle has turned around in the sixth frame. This non-straight target is an invalid target. When screening targets in the above manner, there is no need to set a fixed area. All targets within the sensor's field of view can be screened, covering all areas in the corresponding traffic scene, which is more adaptable.

[0091] Step 103: Match the valid radar targets with the valid camera targets to determine N matching pairs.

[0092] Wherein, N is a natural number greater than 1. Based on the valid target determined in step 102, matching is performed in this step to obtain a plurality of matching pairs. Wherein, the matching method can adopt the existing method. This application also provides a new matching method for determining matching pairs. This matching method is simple to implement and can obtain a preliminary matching pair for subsequent calibration of the two sensors. Specifically, the matching methods provided in this application include:

[0093] Determine the survival time of each valid target, specifically including the time when a valid target is first found in the frame image (i.e., the start time of the survival of the valid target) and the time when a valid target is last found (i.e., the end time of the survival of the valid target);

[0094] For each pair of valid radar targets and valid camera targets to be matched, the overlapping survival time is obtained based on the survival time of the two.

[0095] The to-be-matched pairs whose overlapping survival time is greater than the matching threshold are determined as matching pairs.

[0096] in, are the start time and end time of the effective camera target survival, are the start time and end time of the survival of the valid radar target respectively. Every two valid radar targets and valid camera targets mentioned here refer to one valid radar target and one valid camera target.

[0097] Step 104: determining the matching pairs involved in calibration among all matching pairs.

[0098] There can be one or more matching pairs involved in calibration.

[0099] Step 105 , calibrate the millimeter wave radar and the camera using the matching pairs involved in the calibration, and obtain a corresponding homography matrix for characterizing the mapping relationship between the millimeter wave radar plane and the camera plane.

[0100] In this step, the millimeter-wave radar and the camera are calibrated once. The specific calibration method can be performed using the existing method to obtain the corresponding homography matrix, that is, the projection matrix of the millimeter-wave radar view plane and the camera view plane.

[0101] Among them, for each matching pair involved in the calibration, the coordinates in multiple frames are obtained for the above calibration. Since there are many traffic light intersections in traffic scenes, there will be a lot of effective target stop time, which will result in more coordinate points of the corresponding effective targets in the stop area. If the coordinate points are collected by frame, the position of the calibration relationship will usually focus on the stop area. The data collected in other areas is limited, which may lead to inaccurate calibration relationships. Taking the above problems into consideration, in the present application, when sampling the coordinates of the matching pairs, it is optional to perform uniform sampling in different areas, that is, for each effective target, the number of sampled coordinate points in each area is guaranteed to be the same.

[0102] Step 106: Based on the homography matrix, all valid camera targets are converted to the millimeter-wave radar view to obtain auxiliary camera targets. Based on the first queue consisting of all auxiliary camera targets under multiple frames and the second queue consisting of all valid radar targets under multiple frames, bipartite graph matching is performed on a target-by-target basis to obtain a matching relationship with the minimum error. The error sum of all errors under the matching relationship is calculated as the calibration error corresponding to this calibration and the matching pairs participating in the calibration.

[0103] According to the calibration relationship determined in step 105, all valid targets under the camera view are converted to the millimeter wave radar view (hereinafter referred to as the radar view) to obtain the auxiliary camera target. In this way, the error of the corresponding target pair under this calibration relationship can be obtained by comparing the auxiliary camera target and the effective radar target. Among them, the comparison and determination of the error is carried out in units of targets, using the target pairs composed of the auxiliary camera target and the effective radar target under the multi-frame view. By using the bipartite graph matching process, the matching relationship with the minimum error can be obtained. The so-called matching relationship here refers to the matching of all auxiliary camera targets and all effective radar targets, wherein each auxiliary camera target can only be matched with one effective radar target, and each effective radar target can only be matched with one auxiliary camera target. The error of the two matched targets is calculated. In this way, the matching relationship with the minimum error can be obtained by using the bipartite graph matching process.

[0104] After determining the matching relationship with the smallest error, under this matching relationship, the sum of the errors of each pair of matched targets is calculated, which is called the calibration error corresponding to this calibration or the calibration error corresponding to the homography matrix.

[0105] Step 107, based on the calibration error, determine the matching pair to be involved in the next calibration, and return to step 105, until the calibration error that meets the preset quality requirement is determined.

[0106] In this application, multiple calibrations of millimeter radar waves and cameras are performed, so that a calibration relationship with better accuracy can be obtained. In order to select a more suitable matching pair to participate in the calibration and improve the accuracy of the calibration relationship, before the calibration error that meets the preset quality requirements is determined, the matching pairs that participate in the subsequent calibration are determined based on the obtained calibration error, and the process returns to execute step 105 and its subsequent steps, and the calibration operation is performed to obtain the homography matrix and the corresponding calibration error, and the cycle is repeated until the calibration error that meets the preset quality requirements is determined, and the cycle ends.

[0107] Specifically, the processing of the matching pairs involved in the subsequent calibration is determined based on the obtained calibration error, and can be designed according to actual needs and processing resource conditions. This application provides a design method, which specifically includes:

[0108] In the first N calibrations, each calibration is performed using a matching pair, and the matching pairs involved in each calibration are different, and the calibration errors corresponding to the first N calibrations are obtained; thus, the error status when the matching pair is used for calibration is obtained corresponding to each matching pair;

[0109] In the calibration after the Nth time, the number of matching pairs participating in the calibration is greater than 1. Each time a matching pair participating in the calibration is introduced, a matching pair that has not participated in the calibration after the Nth time is introduced. After the calibration is completed, it is determined whether the calibration error corresponding to the current calibration is less than all the calibration errors obtained. If so, it indicates that the calibration error is reduced after the introduction of the new matching pair, that is, the accuracy of the calibration relationship is improved. Therefore, the newly added matching pair is retained and continues to participate in subsequent calibrations. Otherwise, it indicates that the calibration error is not reduced after the introduction of the new matching pair, that is, the accuracy of the calibration relationship is not improved. Therefore, the newly added matching pair is deleted and no longer participates in subsequent calibrations.

[0110] The above method can gradually optimize the matching pairs participating in subsequent calibration and continuously improve the accuracy of the calibration relationship.

[0111] Step 108: Output the homography matrix corresponding to the calibration error that meets the quality requirements as the calibration relationship between the millimeter wave radar and the camera.

[0112] Once a calibration error that meets the quality requirements is determined, that is, the accuracy of the corresponding calibration relationship meets the requirements, the homography matrix under the calibration relationship can be output as the calibration result of the millimeter wave radar and the camera.

[0113] So far, Figure 1 The method flow shown ends. In the above-mentioned calibration method of the millimeter-wave radar and the camera, after the calibration relationship is determined using the data of the matching pair, the calibration relationship is verified, the calibration error under the calibration relationship is determined, and then the matching pairs involved in the subsequent calibration are optimized based on the calibration error to improve the accuracy of the corresponding calibration relationship until the calibration error that meets the quality requirements is determined, and the calibration relationship corresponding to the calibration error is used as the final calibration result. Therefore, the calibration using the above method can effectively improve the accuracy of the calibration relationship.

[0114] The specific implementation of the calibration method of the present application is described below through specific embodiments. Figure 3 FIG. 1 is a schematic diagram of a specific flow chart of a method for calibrating a millimeter wave radar and a camera in a specific embodiment. Figure 3 As shown, the method includes:

[0115] Step 301: Acquire a frame view of a millimeter wave radar and a frame view of a camera to perform target detection, and obtain a radar target and a camera target.

[0116] The processing of this step is the same as step 101, and will not be repeated here.

[0117] Step 302 , screening radar targets and camera targets to obtain valid radar targets and valid camera targets.

[0118] In this embodiment, the target screening adopts the screening method provided in this application, that is, for all traffic areas covered by the frame image, the straight-moving targets are screened out as valid targets using the previous and next multiple frames of information.

[0119] Step 303: Match the valid radar targets with the valid camera targets to determine N matching pairs.

[0120] The so-called matching pair means that a certain valid radar target and a certain valid camera target are considered to be the same target in the actual scene. In this embodiment, the matching method given in this application is adopted, that is, the overlapping survival time of the valid target is used to match the valid radar target and the valid camera target. When the overlapping survival time is greater than the matching threshold, it is determined to be a match, and when the overlapping survival time is not greater than the matching threshold, it is determined to be a mismatch. All valid radar targets and valid camera targets are traversed in the above manner to determine N matching pairs. Here, a simple method is used for rough matching, and the calibration relationship verification is used later to further optimize the matching pairs involved in the calibration.

[0121] Step 304: for the first N calibrations, one of the N matching pairs is selected for calibration, and the matching pair selected for each calibration is different.

[0122] In the first N calibrations, each time a matching pair is used to complete the calibration, and a calibration relationship corresponding to the matching pair is obtained. In this way, the calibration relationships corresponding to the N matching pairs can be obtained.

[0123] Next, through the processing of steps 305 to 306, a calibration relationship is determined and a calibration relationship is verified to determine the calibration relationship and the corresponding calibration error.

[0124] Step 305 : For each calibration, obtain the coordinates of the corresponding matching pair, and calculate the homography matrix for the transformation between the millimeter wave radar view and the camera view.

[0125] In this application, a homography matrix is ​​used to characterize the position mapping relationship between the millimeter-wave radar view coordinate system and the camera view coordinate system, and also includes the intrinsic and extrinsic parameter information of the camera.

[0126] Specifically, a point pair from two different perspectives can be expressed by a projective transformation, which is a homography matrix. The corresponding projective transformation can be expressed by formula (1), where (x r ,y r ) is the point coordinate under the millimeter wave radar view, (x c ,y c ) is the corresponding point coordinate under the camera view, H 3×3 is the homography matrix.

[0127]

[0128] From formula (1), we can get:

[0129]

[0130]

[0131] Written in the form of matrix AX=0:

[0132]

[0133] Formula (3) is in the form of a homogeneous equation, with m equations solving n unknowns. For this application, the number of coordinates of the matching pairs is greater than the number of features, that is, the number of constraints is greater than the number of unknowns, which is an overdetermined problem, that is, there are too many conditions to be satisfied and it is impossible to satisfy them at the same time. Therefore, this application can use the least squares method to fit the optimal solution, that is, the optimal homography matrix. Among them, (x r ,y r ) and (x c ,y c ) are the coordinates of the effective radar target and the effective camera target in the matching pair, respectively. The corresponding coordinates are obtained by sampling the same target at different frame images and positions. To ensure the accuracy of the homography matrix, in this embodiment, when sampling the coordinates of the matching pair, uniform sampling of the target is performed in different set areas.

[0134] Step 306, calculating the calibration error of the homography matrix in units of valid targets.

[0135] This step is used to calculate the calibration error of this calibration, that is, the calibration error of the homography matrix. The existing calibration method is performed in frames, which easily causes different targets of the same sensor in the previous and next frames to match the same target of another sensor. Due to this mismatching problem, the calibration relationship fed back by the homography matrix is ​​wrong. In this application, the error of the calibration relationship is calculated in units of targets, which avoids the problem of target mismatching and better reflects the accuracy of the calibration relationship expressed by the homography matrix. Among them, considering that the number of millimeter-wave radars is much larger than the number of cameras, the workload of converting the coordinate points under the camera view to the millimeter-wave radar view is much smaller. Based on this, in this application, the error calculation is performed in the millimeter-wave coordinate system, and various existing error measurement methods can be selected, such as Euclidean distance, as the error value.

[0136] The specific calculation method of the calibration error may specifically include: through the homography matrix, all valid camera targets in the preset multiple frames are converted to the millimeter-wave radar view, and the converted coordinates are called auxiliary camera targets; the auxiliary camera targets in the millimeter-wave radar view constitute a first queue, and all valid radar targets in the preset multiple frames constitute a second queue, so that the error between an auxiliary camera target in the first queue and a valid radar target in the second queue can be obtained, and the valid targets in the two queues are subjected to bipartite graph matching to obtain a matching relationship with the minimum error. Under this matching relationship, the error value between each two matched auxiliary camera targets and the valid radar target can be determined, and the sum of all error values ​​is taken as the calibration error of this calibration, that is, the calibration error corresponding to the homography matrix, which can also be called the calibration error corresponding to the matching pair participating in this calibration.

[0137] Among them, when performing bipartite graph matching processing, various existing methods can be used, such as the Hungarian algorithm. In this embodiment, the Hungarian algorithm is improved to determine the matching relationship with the smallest error. Specifically, in the method of this embodiment, the targets in the first queue are regarded as one type of vertices, and the targets in the second queue are regarded as another type of vertices. The two types of vertices are matched, and the error between the two types of vertices is regarded as an edge, and the Hungarian algorithm is executed for matching processing.

[0138] The essence of the Hungarian matching algorithm is the maximum matching of a bipartite graph, that is, in a bipartite graph, if and only if there is no augmenting path in the maximum matching M. Figure 4 As shown, the error under a certain matching relationship is obtained by the first queue and the second queue to form an m*n error matrix. The specific method is to enumerate each unmatched point r in the second queue set, and then enumerate each unmatched point c in the first queue set. If r can be added to the interlaced path where c is located, it is augmented, otherwise, it is not augmented. The time complexity of the Hungarian matching algorithm is O(E*V), where E is the number of edges and V is the number of nodes. In the Hungarian algorithm, each time a point in a set is searched, the search record array is cleared; and considering the characteristics of the sensor, the number of millimeter-wave radars is much larger than the number of cameras, resulting in the number of points in the second queue set being much larger than the number of points in the first queue set. If the search record array is cleared after each search of a set, then all vertices need to be re-searched and calculated next time. Based on this, in this embodiment, after each search of the points in the first queue set or the second queue set, if the augmented path is not found, the search record array is not cleared. After such processing, the time complexity of the Hungarian algorithm can be reduced to O(M*(E+V)), where M is the maximum number of matches. This complexity mainly lies in the difference between M and V. When the number of points in the two sensor sets is very different, the efficiency of the algorithm can be significantly improved.

[0139] Through the processing of the aforementioned steps 304-306, for the first N calibrations, the corresponding homography matrix and calibration error are calculated for each matching pair. The processing of calculating the homography matrix and calibration error for these N calibrations can be performed sequentially or in parallel, and this application does not limit this.

[0140] Step 307, sort the calibration errors of the previous N calibrations by size, take the smallest calibration error as the current calibration error err', form the matching pairs corresponding to the smallest calibration error into a selected matching pair set, and take the matching pairs corresponding to the remaining calibration errors as a test matching pair set.

[0141] In the subsequent process of determining and verifying the calibration relationship, two sets are involved, namely the selected matching pair set and the test matching pair set. The first N calibration processes all use one matching pair to determine the calibration relationship and the corresponding calibration error; after the Nth calibration, calibration processing will be performed using multiple matching pairs. In the successive calibrations using multiple matching pairs, matching pairs that can improve the accuracy of the calibration relationship are selected. These matching pairs form the selected matching pair set and participate in all subsequent calibration processes; matching pairs that have not participated in the calibration process of multiple matching pairs form the test matching pair set. In each calibration, one matching pair is selected from the test matching pair set to temporarily join the selected matching pair set to participate in the next calibration to determine whether the newly selected matching pair from the test matching pair set can be retained in the selected matching pair set.

[0142] After the Nth calibration, the selected matching pair set and the test matching pair set are initialized. The minimum calibration error among the calibration errors of the first N calibrations is determined, and the matching pairs corresponding to the calibration error are formed into the selected matching pair set, and the matching pairs corresponding to the remaining calibration errors are formed into the test matching pair set.

[0143] Step 308 , select the matching pair with the smallest calibration error from the test matching pair set as the current test matching pair, temporarily add it to the selected matching pair set, and determine that the matching pairs participating in the next calibration are all the matching pairs in the selected matching pair set.

[0144] Each time when multiple matching pairs participating in the next calibration are determined, the matching pair with the smallest calibration error is selected from the test matching pair set as the current test matching pair, temporarily added to the selected matching pair set, and the matching pairs participating in the next calibration are determined to be all the matching pairs in the selected matching pair set. Hereinafter, the matching pair with the smallest calibration error in the test matching pair set (that is, the matching pair temporarily added to the selected matching pair set) is referred to as the current test matching pair.

[0145] Next, the next calibration process and verification is carried out. When introducing it in detail, this calibration is referred to as this calibration.

[0146] Step 309 , obtaining the coordinates of the matching pairs involved in the calibration, calculating the homography matrix for the conversion between the millimeter wave radar view and the camera view, and calculating the calibration error of the homography matrix in units of valid targets.

[0147] The processing of this step is the same as steps 305 to 306, and will not be repeated here.

[0148] Step 310, determining the calibration error err calculated in step 309 n Is it less than the current calibration error err'? If so, execute step 311; otherwise, execute step 314.

[0149] This step is based on the calibration error err n The calibration error err' is compared with the current calibration error to determine whether the calibration relationship of this calibration has improved accuracy. The current calibration error is used to represent the minimum calibration error up to the last calibration, and the calibration relationship corresponding to the minimum calibration error is also the calibration relationship with the highest accuracy up to the last calibration.

[0150] If the calibration error of this calibration is less than the current calibration error, it indicates that the calibration relationship obtained this time has improved accuracy compared with all calibration relationships obtained previously, otherwise it indicates that the calibration relationship obtained this time has not improved accuracy.

[0151] Step 311: retain the current test match pair in the selected match pair set, and update the current calibration error err' to the current calibration error err n , return to step 308.

[0152] The error of this calibration is smaller than the current calibration error, which means that the introduction of the current test matching pair can improve the accuracy of the calibration relationship. Therefore, the current test matching pair is retained in the selected matching pair set and participates in all subsequent calibration processes. At the same time, the error of this calibration is smaller than the current calibration error. Therefore, the current calibration error is updated and the error of this calibration is used as the new current calibration error, thereby ensuring that the current calibration error is still the smallest calibration error.

[0153] Step 312, delete the current test matching pair from the selected matching pair set, keep the current calibration error unchanged, and increase the number of deletions by 1.

[0154] The calibration error of this calibration is not less than the current calibration error, indicating that the introduction of the current test match pair fails to improve the accuracy of the calibration relationship, so the current test match pair is deleted from the selected match pair set and no longer participates in all subsequent calibration processes. At the same time, in this embodiment, the deletion operation of the selected match pair set is counted, and the number of deletions is used as a standard to determine whether the preset quality requirements are met.

[0155] Step 313, determine whether the number of deletions reaches the set deletion threshold, if so, execute step 314, otherwise return to step 308.

[0156] When the number of deletions reaches the set deletion threshold, it means that the accuracy of the calibration relationship cannot be reduced for many times, and it is considered that the calibration relationship corresponding to the current calibration error has met the quality requirements, and step 314 is executed to output the calibration result. If the number of deletions does not reach the set deletion threshold, it means that the quality requirements have not been met, and step 308 is executed to continue the next calibration.

[0157] In this embodiment, the quality judgment is performed by the number of deletion operations of the selected matching pair set. In fact, other methods can also be used for quality judgment, which can be set according to the actual environment and needs. For example, all matching pair combinations can be calibrated and the calibration error can be calculated, and the minimum value of all calibration errors is selected, that is, the matching pair combination with the highest accuracy of the corresponding calibration relationship. For the processing of this embodiment, that is, when the test matching pair set is 0, the last calibration is completed, and the smaller value of the calibration error determined after the last calibration and the current calibration error is used as the calibration error that meets the quality requirements. Or, for another example, after step 311, it can be determined whether the value of the current calibration error is less than the set error threshold. If so, it is determined that the current calibration error meets the quality requirements, otherwise it returns to step 308 for the next calibration.

[0158] Step 314: output the homography matrix corresponding to the current calibration error as the calibration result of the millimeter wave radar and the camera.

[0159] By this step, a calibration relationship that meets the quality requirements has been found, and the homography matrix used to characterize the calibration relationship is output as the calibration result. Since various calibration relationships are verified through the above-mentioned processing, the output calibration relationship can achieve the required accuracy, which can effectively improve the accuracy compared with the existing calibration relationship directly obtained.

[0160] So far, Figure 3 The method flow in ends. In the processing of the above embodiment, by screening out the straight-moving targets in the entire traffic intersection, the homography matrix calculated using these targets can be applied to the entire traffic intersection; at the same time, the calibration relationship between the two sensors can be accurately expressed through the homography matrix, and the time complexity is low, the calculation speed is fast, and compared with the existing method of calculating the calibration error in units of timestamps, the calibration error is calculated in units of targets, and by optimizing the Hungarian matching algorithm, mismatching is avoided and the accuracy and speed are higher; further, the matching pairs are iterated through the data calibration error, and the homography matrix is ​​continuously optimized, and finally the optimal calibration relationship is obtained. Compared with the existing method, the accuracy and efficiency of the calibration relationship can be significantly improved.

[0161] The above is a specific implementation of the calibration method of the millimeter wave radar and camera in this application. This application also provides a calibration device for a millimeter wave radar and camera, which can be used to implement the above calibration method. Figure 5 This is a basic structural diagram of the calibration device for the millimeter wave radar and camera of this application. Figure 5 As shown, the device includes: a target detection unit, a target screening unit, a matching pair determination unit, a calibration relationship acquisition unit and a calibration error checking module.

[0162] The target detection unit is used to detect the radar target and the camera target in the frame image of the millimeter wave radar and the frame image of the camera respectively;

[0163] A target screening unit, used to screen radar targets and camera targets respectively to obtain valid radar targets and valid camera targets;

[0164] A matching pair determination unit, used to match valid radar targets with valid camera targets to determine N matching pairs; wherein N is a natural number greater than 1;

[0165] a calibration relationship acquisition unit, used to determine the matching pairs participating in each calibration from among all matching pairs, calibrate the millimeter-wave radar and the camera using the matching pairs participating in the calibration, and obtain a corresponding homography matrix for characterizing the mapping relationship between the millimeter-wave radar plane and the camera plane, until a calibration error that meets a preset quality requirement is determined from the calibration error fed back by the calibration error verification unit, and the homography matrix corresponding to the calibration error that meets the quality requirement is output as the calibration result of the millimeter-wave radar and the camera;

[0166] The calibration error checking unit is used to convert all valid camera targets into the millimeter-wave radar view based on the homography matrix to obtain auxiliary camera targets, perform bipartite graph matching processing based on the first queue composed of all auxiliary camera targets and the second queue composed of all valid radar targets to obtain a matching relationship with the minimum error, and calculate the error sum of all errors under the matching relationship as the calibration error corresponding to this calibration and the matching pairs participating in this calibration, and feed it back to the calibration relationship acquisition unit.

[0167] In order to better perform calibration and verification to obtain a calibration relationship with better accuracy, optionally, in the calibration relationship acquisition unit and the calibration verification unit,

[0168] In the first N calibrations, each calibration is performed using a matching pair, and the matching pairs involved in each calibration are different, and the homography matrix and calibration error corresponding to the first N calibrations are obtained;

[0169] In the calibration after the Nth time, the number of matching pairs participating in the calibration is greater than 1. When determining the matching pairs participating in the next calibration based on the calibration error, determine whether the calibration error corresponding to this calibration is less than all the calibration errors obtained. If so, retain the matching pairs newly introduced in this calibration to participate in the subsequent calibration; otherwise, the matching pairs newly introduced in this calibration no longer participate in the subsequent calibration; determine that the matching pairs participating in the next calibration include: the matching pairs that have been retained and a matching pair that has not participated in the calibration after the Nth time.

[0170] Optionally, in the calibration relationship acquisition unit,

[0171] After the Nth calibration is completed, the processing of the matching pairs participating in the next calibration is determined, including:

[0172] Select the smallest calibration error from all the calibration errors obtained, take the matching pair corresponding to the smallest calibration error as the selected matching pair set, and take the smallest calibration error as the current calibration error; select one matching pair from the remaining matching pairs as the current test matching pair to be added to the selected matching pair set, take all matching pairs in the selected matching pair set as the matching pairs participating in the calibration for the N+1th time; and take all matching pairs except the matching pairs participating in the calibration for the N+1th time to form the test matching pair set;

[0173] After the nth calibration is completed, the matching pairs for the next calibration are determined, including:

[0174] Determine whether the calibration error determined after the nth calibration is less than the current calibration error. If so, keep the selected matching pair set unchanged and update the current calibration error to the calibration error determined after the nth calibration; otherwise, delete the current test matching pair from the selected matching pair set and keep the current calibration error unchanged;

[0175] Update the current test matching pair to a matching pair in the test matching pair set, add the current test matching pair to the selected matching pair set, and use all matching pairs in the selected matching pair set as matching pairs participating in the calibration for the n+1th time, and delete the current test matching pair from the test matching pair set; wherein n is an integer greater than N.

[0176] Optionally, in the calibration relationship acquisition unit, when selecting and updating the current test matching pair, the matching pair corresponding to the minimum calibration error in the selected matching pair set is used as the current test matching pair.

[0177] Optionally, the calibration relationship acquisition unit is further used to count the number of times the matching pair is deleted from the selected matching pair set, and when the count value reaches a set threshold, determine that the current calibration error meets the quality requirement; or,

[0178] In the calibration relationship acquisition unit, when the test matching pair set is 0, the last calibration is completed, and the smaller error value between the calibration error determined after the last calibration and the current calibration error is used as the calibration error that meets the quality requirements;

[0179] or,

[0180] In the calibration relationship acquisition unit, when the value of the current calibration error is less than the set error threshold, it is determined that the current calibration error meets the quality requirement.

[0181] Optionally, in the target screening unit, screening the radar target and the camera target respectively includes:

[0182] Determine the motion trajectory of the radar target, treat the straight-moving radar target as a valid radar target, and treat the non-straight-moving radar target as an invalid radar target;

[0183] Determine the motion trajectory of the camera target, treat the straight-moving camera target as a valid radar target, and treat the non-straight-moving camera target as an invalid camera target.

[0184] Optionally, in the matching pair determination unit, matching the valid radar target with the valid camera target includes:

[0185] For each to-be-matched pair consisting of two valid radar targets and valid camera targets, the overlapping survival time is obtained based on the survival time of the two;

[0186] The to-be-matched pairs whose overlapping survival time is greater than the matching threshold are determined as matching pairs.

[0187] Optionally, in the calibration relationship acquisition unit, the millimeter wave radar and the camera are calibrated using the matching pair involved in the calibration, including:

[0188] For the matching pair involved in calibration, the coordinate values ​​of the two targets in the matching pair are sampled in the frame image of the millimeter wave radar and the frame image of the camera respectively; wherein, during sampling, the targets are uniformly sampled in different set areas.

[0189] Optionally, in the calibration relationship acquisition unit, a first queue consisting of all auxiliary camera targets and a second queue consisting of all valid radar targets are subjected to bipartite graph matching processing, including:

[0190] The targets in the first queue and the second queue are taken as vertices of the bipartite graph, and the error between the two vertices is taken as an edge. The Hungarian matching method is performed for matching. After each search for the targets in the first queue or the second queue, if no augmenting path is found, the search record array is not cleared.

[0191] The present application also provides a computer-readable storage medium, which stores instructions, and when the instructions are executed by a processor, the steps in the method for calibrating a millimeter-wave radar and a camera as described above can be executed. In practical applications, the computer-readable medium can be included in each device / apparatus / system in the above-mentioned embodiments, or it can exist independently without being assembled into the device / apparatus / system. Instructions are stored in a computer-readable storage medium, and when the instructions stored therein are executed by a processor, the steps in the method for calibrating a millimeter-wave radar and a camera as described above can be executed.

[0192] According to the embodiments disclosed in the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof, but it is not used to limit the scope of protection of the present application. In the embodiments disclosed in the present application, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0193] Figure 6 This application also provides an electronic device. Figure 6 As shown, it shows a schematic diagram of the structure of the electronic device involved in the embodiment of the present application, specifically:

[0194] The electronic device may include a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When executing the program in the memory 602, a calibration method for a millimeter-wave radar and a camera may be implemented.

[0195] Specifically, in practical applications, the electronic device may further include components such as a power supply 603, an input / output unit 604, etc. Those skilled in the art may understand that Figure 6 The structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0196] The processor 601 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the server and processes data by running or executing software programs and / or modules stored in the memory 602 and calling data stored in the memory 602, thereby monitoring the electronic device as a whole.

[0197] The memory 602 can be used to store software programs and modules, that is, the above-mentioned computer-readable storage medium. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

[0198] The electronic device also includes a power supply 603 for supplying power to various components, which can be logically connected to the processor 601 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 603 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0199] The electronic device may also include an input and output unit 604, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, and optical signal inputs related to user settings and function control. The input unit output 604 may also be used to display information input by the user or information provided to the user and various graphical user interfaces, which may be composed of graphics, text, icons, videos, and any combination thereof.

[0200] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for calibrating a millimeter wave radar and a camera, characterized in that: include: The radar target and the camera target are detected in the frame view of the millimeter wave radar and the frame view of the camera respectively; The radar target and the camera target are screened respectively to obtain a valid radar target and a valid camera target; Matching the valid radar target with the valid camera target to determine N matching pairs; Determining a matching pair participating in calibration among all the matching pairs; Using the matching pairs involved in the calibration, the millimeter-wave radar and the camera are calibrated to obtain a homography matrix of this calibration, which is used to characterize the mapping relationship between the millimeter-wave radar view and the camera view; Based on the homography matrix, all the valid camera targets are converted to the millimeter-wave radar view to obtain auxiliary camera targets. Taking targets as units, bipartite graph matching is performed based on a first queue consisting of all the auxiliary camera targets under multiple frames and a second queue consisting of all the valid radar targets under the multiple frames to obtain a matching relationship with the minimum error, and the error sum of all errors under the matching relationship is calculated as the calibration error corresponding to this calibration and the matching pairs participating in the calibration; Based on the calibration error, determining a matching pair to participate in the next calibration, and performing a process of determining a homography matrix and a calibration error until a calibration error that meets a preset quality requirement is determined, and outputting a homography matrix calibrated corresponding to the calibration error that meets the quality requirement as a calibration result of the millimeter wave radar and the camera; Wherein, N is a natural number greater than 1.

2. The method according to claim 1, characterized in that In the first N calibrations, each calibration is performed using a matching pair, and the matching pairs involved in each calibration are different, and the homography matrix and calibration error corresponding to the first N calibrations are obtained; In the calibration after the Nth time, the number of matching pairs participating in the calibration is greater than 1. When determining the matching pairs participating in the next calibration based on the calibration error, it is determined whether the calibration error corresponding to this calibration is less than all the calibration errors obtained. If so, the matching pairs newly introduced in this calibration are retained to participate in the subsequent calibration. Otherwise, the matching pairs newly introduced in this calibration are no longer involved in the subsequent calibration. The matching pairs that are determined to participate in the next calibration include: the reserved matching pairs and a matching pair that has not participated in the calibration after the Nth time.

3. The method according to claim 2, characterized in that After the Nth calibration is completed, determining the matching pair to participate in the next calibration includes: Select the smallest calibration error from all the calibration errors obtained, take the matching pair corresponding to the smallest calibration error as the selected matching pair set, and take the smallest calibration error as the current calibration error; select a matching pair from the remaining matching pairs as the current test matching pair to be added to the selected matching pair set, take all matching pairs in the selected matching pair set as the matching pairs participating in the calibration for the N+1th time; and take all matching pairs except the matching pairs participating in the calibration for the N+1th time to form a test matching pair set; After the nth calibration is completed, determining the matching pair to participate in the next calibration includes: Determine whether the calibration error determined after the nth calibration is less than the current calibration error; if so, keep the selected matching pair set unchanged, and update the current calibration error to the calibration error determined after the nth calibration; otherwise, delete the current test matching pair from the selected matching pair set, and keep the current calibration error unchanged; Update the current test matching pair to a matching pair in the test matching pair set, add the current test matching pair to the selected matching pair set, use all matching pairs in the selected matching pair set as matching pairs participating in the calibration for the n+1th time, and delete the current test matching pair from the test matching pair set; Wherein, n is an integer greater than N.

4. The method according to claim 3, characterized in that When selecting and updating the current test matching pair, the matching pair corresponding to the minimum calibration error in the selected matching pair set is used as the current test matching pair.

5. The method according to claim 2, 3 or 4, characterized in that: Counting the number of times a matching pair is deleted from the selected matching pair set, and when the count value reaches a set threshold, determining that the current calibration error meets the quality requirement; or, When the test matching pair set is 0, the last calibration is completed, and the smaller value between the calibration error determined after the last calibration and the current calibration error is used as the calibration error that meets the quality requirement; or, When the value of the current calibration error is less than the set error threshold, it is determined that the current calibration error meets the quality requirement.

6. The method according to claim 1, characterized in that The screening of the radar target and the camera target respectively includes: Determine the motion trajectory of the radar target, take the radar target moving straight as the valid radar target, and take the radar target moving non-straight as the invalid radar target; The motion trajectory of the camera target is determined, and the camera target moving straight is used as the valid radar target, and the camera target moving non-straight is used as the invalid camera target.

7. The method according to claim 1, characterized in that The matching of the valid radar target and the valid camera target includes: For each to-be-matched pair consisting of two of the valid radar targets and the valid camera targets, obtaining an overlapping survival time based on the survival times of the two; The to-be-matched pairs whose overlapping survival time is greater than the matching threshold are determined as the matching pairs.

8. The method according to claim 1, characterized in that The calibrating of the millimeter wave radar and the camera using the matching pair involved in the calibration includes: For the matching pair involved in calibration, the coordinate values ​​of the two targets in the matching pair are sampled in the frame image of the millimeter wave radar and the frame image of the camera respectively; wherein, during sampling, the targets are uniformly sampled in different set areas.

9. The method according to claim 1, characterized in that: The first queue formed by all the auxiliary camera targets and the second queue formed by all the valid radar targets are subjected to bipartite graph matching processing, including: The targets in the first queue and the second queue are taken as vertices of a bipartite graph, and the error between the two vertices is taken as an edge. The Hungarian matching method is performed for matching processing, and after each search for the targets in the first queue or the second queue, if no augmenting path is found, the search record array is not cleared.

10. A millimeter wave and camera calibration device, characterized in that: include: Target detection unit, target screening unit, matching pair determination unit, calibration relationship acquisition unit and calibration error checking module; The target detection unit is used to detect the radar target and the camera target in the frame image of the millimeter wave radar and the frame image of the camera respectively; The target screening unit is used to screen the radar target and the camera target respectively to obtain a valid radar target and a valid camera target; The matching pair determination unit is used to match the valid radar target with the valid camera target to determine N matching pairs; The calibration relationship acquisition unit is used to determine the matching pairs participating in each calibration from among all the matching pairs, and use the matching pairs participating in the calibration to calibrate the millimeter-wave radar and the camera to obtain a corresponding homography matrix for characterizing the mapping relationship between the millimeter-wave radar plane and the camera plane, until a calibration error that meets a preset quality requirement is determined from the calibration error fed back by the calibration error verification unit, and the homography matrix corresponding to the calibration error that meets the quality requirement is output as the calibration result of the millimeter-wave radar and the camera; The calibration error checking unit is used to convert all the valid camera targets into the millimeter wave radar view based on the homography matrix to obtain auxiliary camera targets, perform bipartite graph matching processing based on the first queue composed of all the auxiliary camera targets and the second queue composed of all the valid radar targets to obtain a matching relationship with the minimum error, and calculate the error sum of all errors under the matching relationship as the calibration error corresponding to this calibration and the matching pairs participating in the calibration, and feed it back to the calibration relationship acquisition unit; Wherein, N is a natural number greater than 1.

11. The device according to claim 10, characterized in that In the calibration relationship acquisition unit and the calibration verification unit, In the first N calibrations, each calibration is performed using a matching pair, and the matching pairs involved in each calibration are different, and the homography matrix and calibration error corresponding to the first N calibrations are obtained; In the calibration after the Nth time, the number of matching pairs participating in the calibration is greater than 1. When determining the matching pairs participating in the next calibration based on the calibration error, it is determined whether the calibration error corresponding to this calibration is less than all the calibration errors obtained. If so, the matching pairs newly introduced in this calibration are retained to participate in the subsequent calibration. Otherwise, the matching pairs newly introduced in this calibration are no longer involved in the subsequent calibration. The matching pairs that are determined to participate in the next calibration include: the reserved matching pairs and a matching pair that has not participated in the calibration after the Nth time.

12. The device according to claim 11, characterized in that In the calibration relationship acquisition unit, After the Nth calibration is completed, the processing of the matching pairs participating in the next calibration is determined, including: Select the smallest calibration error from all the calibration errors obtained, take the matching pair corresponding to the smallest calibration error as the selected matching pair set, and take the smallest calibration error as the current calibration error; select a matching pair from the remaining matching pairs as the current test matching pair to be added to the selected matching pair set, take all matching pairs in the selected matching pair set as the matching pairs participating in the calibration for the N+1th time; and take all matching pairs except the matching pairs participating in the calibration for the N+1th time to form a test matching pair set; After the nth calibration is completed, determining the matching pair to participate in the next calibration includes: Determine whether the calibration error determined after the nth calibration is less than the current calibration error; if so, keep the selected matching pair set unchanged, and update the current calibration error to the calibration error determined after the nth calibration; otherwise, delete the current test matching pair from the selected matching pair set, and keep the current calibration error unchanged; Update the current test matching pair to a matching pair in the test matching pair set, add the current test matching pair to the selected matching pair set, use all matching pairs in the selected matching pair set as matching pairs participating in the calibration for the n+1th time, and delete the current test matching pair from the test matching pair set; Wherein, n is an integer greater than N.

13. The device according to claim 12, characterized in that In the calibration relationship acquisition unit, when selecting and updating the current test matching pair, the matching pair corresponding to the minimum calibration error in the selected matching pair set is used as the current test matching pair.

14. The device according to claim 11, 12 or 13, characterized in that The calibration relationship acquisition unit is further used to count the number of times a matching pair is deleted from the selected matching pair set, and when the count value reaches a set threshold, determine that the current calibration error meets the quality requirement; or, In the calibration relationship acquisition unit, when the test matching pair set is 0, the last calibration is completed, and the smaller error value between the calibration error determined after the last calibration and the current calibration error is used as the calibration error that meets the quality requirement; or, In the calibration relationship acquisition unit, when the value of the current calibration error is less than a set error threshold, it is determined that the current calibration error meets the quality requirement.

15. The device according to claim 10, characterized in that In the target screening unit, screening the radar target and the camera target respectively includes: Determine the motion trajectory of the radar target, take the radar target moving straight as the valid radar target, and take the radar target moving non-straight as the invalid radar target; The motion trajectory of the camera target is determined, and the camera target moving straight is used as the valid radar target, and the camera target moving non-straight is used as the invalid camera target.

16. The device according to claim 10, characterized in that In the matching pair determination unit, matching the valid radar target with the valid camera target includes: For each to-be-matched pair consisting of two of the valid radar targets and the valid camera targets, obtaining an overlapping survival time based on the survival times of the two; The to-be-matched pairs whose overlapping survival time is greater than the matching threshold are determined as the matching pairs.

17. The device according to claim 10, characterized in that In the calibration relationship acquisition unit, the calibrating of the millimeter wave radar and the camera using the matching pair involved in the calibration includes: For the matching pair involved in calibration, the coordinate values ​​of the two targets in the matching pair are sampled in the frame image of the millimeter wave radar and the frame image of the camera respectively; wherein, during sampling, the targets are uniformly sampled in different set areas.

18. The device according to claim 10, characterized in that In the calibration relationship acquisition unit, the first queue composed of all the auxiliary camera targets and the second queue composed of all the valid radar targets are subjected to bipartite graph matching processing, including: The targets in the first queue and the second queue are taken as vertices of a bipartite graph, and the error between the two vertices is taken as an edge. The Hungarian matching method is performed for matching processing, and after each search for the targets in the first queue or the second queue, if no augmenting path is found, the search record array is not cleared.

19. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by the processor, the millimeter wave radar and camera calibration method described in any one of claims 1 to 9 can be implemented.

20. An electronic device, characterized in that: The electronic device includes at least a computer-readable storage medium and also includes a processor; The processor is used to read executable instructions from the computer-readable storage medium and execute the instructions to implement the millimeter-wave radar and camera calibration method described in any one of claims 1 to 9.

Images