Automatic calibration method of all-in-one machine based on scene features

Abstract

The invention discloses a scene feature-based automatic calibration method for a thunder-vision all-in-one machine, and the method comprises the following steps: S1, initialization and reference information acquisition: obtaining initial external parameters and internal parameters, selecting a scene open moment to obtain a radar reference point, and projecting the radar reference point as an image reference point; s2, collecting data in real time and calculating errors, collecting and projecting real-time radar points, and calculating weighted errors after removing outer points; and S3, deflection detection and calibration: if an error exceeds a threshold value, judging deflection, and calculating new external parameters through an optimization algorithm to complete calibration. The method can dynamically monitor equipment deflection without manual intervention, and solves the problem of insufficient real-time performance of a traditional method; the dual-mode data are mutually redundant, so that the environmental interference influence is reduced; extra reference objects or hardware are not needed, and the cost is reduced; and the precision is improved by an outer point elimination and optimization algorithm.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent transportation, and specifically relates to a radar-camera integrated machine automatic calibration method based on scene features. BACKGROUND

[0002] In an intelligent transportation system, a road-side radar-camera integrated machine as a key device can collect real-time traffic data such as vehicle position, speed and trajectory by fusing the high-precision distance detection capability of radar and the image recognition capability of camera, thereby providing an important basis for traffic management decision.

[0003] However, during the daily operation and maintenance of the radar-camera integrated machine, the device is prone to angular deflection due to vehicle collision, strong wind, vibration and other operations or external factors. Such deflection can cause deviation between the radar detection angle and the camera shooting angle, directly leading to inaccurate collected traffic data and seriously affecting the normal operation of the intelligent transportation system and the scientific nature of the traffic management decision. Therefore, how to quickly and accurately detect device deflection and perform effective correction has become a key problem to ensure the reliability of the data of the radar-camera integrated machine.

[0004] In the prior art, the schemes for calibrating the radar-camera integrated machine mainly include the following types: IMU inertial detection scheme: the device built-in gyroscope is used to monitor the angle change to determine whether the device is deflected. However, the high-precision IMU module is costly, and it is difficult to be widely applied due to poor economy in large-scale road-side deployment.

[0005] Manual calibration method: the device angle offset is checked by manual inspection, and the sensor coordinate system is calibrated by using a calibration board. This method needs to consume a large amount of labor cost, and the periodic inspection cannot timely find the sudden deflection, leading to continuous accumulation of sensing error and poor real-time performance.

[0006] Visual detection based on fixed reference: a two-dimensional code or other markers are arranged in the field of view of the device, and the calibration is realized by calculating the coordinate system mapping relationship. However, the fixed markers need additional infrastructure investment, and are easily blocked by passing vehicles and pedestrians, leading to calibration interruption or failure.

[0007] Radar-assisted posture monitoring: the position change of static targets in the radar point cloud is used to infer the device deflection. This method only relies on radar single-mode data, and is easily disturbed by factors such as dynamic vehicle shielding and radar noise, leading to deflection misjudgment and low reliability.

[0008] In summary, the prior art has defects such as poor real-time performance, dependence on external reference, high cost and low reliability of single-mode data, and cannot meet the efficient, accurate and low-cost calibration requirements of the road-side radar-camera integrated machine. SUMMARY

[0009] The present application aims to provide a scene feature-based radar-camera integrated machine automatic calibration method to solve the real-time deficiency, dependence on external reference, high cost and low reliability of single modal data in the prior art.

[0010] To solve the above technical problems, the technical solution adopted by the present application is: A scene feature-based radar-camera integrated machine automatic calibration method, comprising the following steps: S1, initialization and reference information acquisition: S101, obtaining radar-camera initial extrinsic parameters and camera intrinsic parameters of the radar-camera integrated machine by using a calibration method; S102, selecting a scene open moment to obtain a radar reference point P r and projecting it to the camera image plane to obtain an image reference point Pref, the radar reference point P r is the three-dimensional coordinates of a fixed target in the scene detected by the radar, and the image reference point Pref is obtained based on the radar-camera initial extrinsic parameters and camera intrinsic parameters obtained in step S101; S2, real-time data acquisition and error calculation: S201, real-time acquisition of the three-dimensional coordinates of the target detected by the radar as real-time radar points P rn ; S202, projecting the real-time radar points P rn to the camera image plane based on the radar-camera initial extrinsic parameters and camera intrinsic parameters obtained in step S101 to obtain real-time projection points P in ; S203, performing outlier rejection on the real-time projection points P in and the real-time radar points P rn ; S204, calculating a weighted error d e based on the remaining points after the outlier rejection; S3, deflection detection and automatic calibration: S301, comparing the weighted error d e with a preset threshold T err , if d e >T err , it is determined that the radar-camera integrated machine has deflected; S302, when it is determined that the deflection has occurred, a new radar-camera extrinsic parameter is solved by an optimization algorithm to replace the initial extrinsic parameter to complete the calibration.

[0011] According to the above technical solution, in step S101, the process of obtaining the initial extrinsic parameters of the radar-camera calibration method includes: arranging multiple calibration objects at known positions in the calibration scene, obtaining the three-dimensional coordinates of the calibration objects through the radar, obtaining the two-dimensional image coordinates of the calibration objects through the camera, and obtaining the rotation matrix R and translation vector t through target point matching and optimization algorithms.

[0012] According to the above technical solution, in step S101, the initial extrinsic parameters of the radar-camera include the rotation matrix R and the translation vector t, and the intrinsic parameters of the camera include the focal length fx, fy, the principal point coordinates cx, cy, and the distortion coefficient.

[0013] According to the above technical solution, in step S101, the camera intrinsic parameters are obtained using the Zhang Zhengyou calibration method, and the camera intrinsic parameters K are recorded.

[0014] According to the above technical solution, in step S102, the time when the scene is empty is the time when there are no dynamic obstacles such as vehicles or pedestrians in the scene.

[0015] According to the above technical solution, in open conditions, the radar detects targets in the scene and records the three-dimensional coordinates P of all radar target points. r =[xr,yr,zr] T , as a radar reference point.

[0016] According to the above technical solution, the specific process of removing external points in step S203 includes: Calculate the real-time projection point P in Euclidean distance d to the nearest image reference point Pref i Calculate real-time radar point P rn To the nearest radar reference point P r Euclidean distance d r ; Calculate weighted distance , where α is the proportionality coefficient. w i w is the image resolution width. r This represents the actual width of the main observation area of ​​the image; If d > preset distance threshold T d Then the corresponding real-time radar point P rn and real-time projection point P in It is identified as an outlier and removed.

[0017] According to the above technical solution, in step S204, the weighted error d e The calculation process includes: For the remaining real-time projected points after removing outliers Calculate the average pixel error d between it and the corresponding image reference point Pref. m ; The remaining real-time radar points after the outliers are removed , and the average distance error d r of the real-time projection point P n and the corresponding radar reference point P m is calculated. The weighted error d e is calculated based on d m and d n , d e = β·d m + (1-β)·d n , wherein β is a weight coefficient.

[0018] According to the above technical solution, in step S302, the optimization algorithm is Levenberg-Marquardt algorithm, and a new rotation matrix R new and translation vector t new are obtained by constructing a target function with the objective of minimizing the pixel error between the real-time projection point and the image reference point.

[0019] According to the above technical solution, the target function is defined as: wherein, ||*|| represents the modulus of the vector, and E is the error symbol.

[0020] Compared with the prior art, the present application has the following beneficial effects: The method in the present application can dynamically monitor the deflection state of the equipment through real-time fusion analysis of radar and visual data, without manual intervention, and solves the problem of insufficient real-time performance of the traditional method.

[0021] The radar and visual dual-mode data are redundant to each other, and when a single sensor is blocked or disturbed by noise, the other mode can ensure that the calibration process continues, reducing the impact of environmental interference on reliability. Without the need to arrange fixed reference objects or high-precision IMU modules, the calibration is realized through the inherent characteristics of the scene, reducing the additional infrastructure and hardware costs, and avoiding the labor input of manual inspection.

[0022] Through outlier rejection and a nonlinear optimization algorithm, the interference of noise and dynamic targets can be effectively reduced, and the accuracy of deflection detection and parameter correction is improved. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The present application is a calibration flowchart. DETAILED DESCRIPTION

[0024] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0025] Embodiment one The embodiment provides a radar-camera integrated machine automatic calibration method based on scene features, comprising the following steps: Step 1, initialization and reference information acquisition, which aims to establish the reference of calibration, and provide comparison basis for subsequent real-time detection and correction, specifically comprising: Step 101, obtaining initial parameters by traditional calibration, after the installation of the radar-camera integrated machine, the radar-camera extrinsic parameters and camera intrinsic parameters are obtained by using the traditional calibration method: Radar-camera extrinsic calibration: a plurality of calibration objects with known three-dimensional coordinates (such as cubes with reflective markers) are arranged in a specific calibration scene, the radar obtains the three-dimensional space coordinates (distance, angle, etc.) of the calibration objects through electromagnetic wave detection, and the camera synchronously captures the image of the calibration objects and extracts the two-dimensional image coordinates. Through a target point matching algorithm (such as SIFT feature matching), the radar and camera collected calibration object data are associated, and then a nonlinear optimization algorithm (such as the LM algorithm) is used to solve the rotation matrix R and the translation vector t between the radar coordinate system and the camera coordinate system, which together constitute the initial extrinsic parameters of the radar-camera.

[0026] Camera intrinsic calibration: Zhang Zhengyou calibration method is used, different angle checkerboard images are captured by the camera, and the corresponding relationship of the checkerboard corner points in the world coordinate system and the image coordinate system is used to calculate the camera intrinsic K, specifically: The camera intrinsic K includes focal length fx, fy, principal point coordinates cx, cy and distortion coefficients (radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2), which are used to describe the optical characteristics and imaging model of the camera.

[0027] Step 102, selecting a reference point at a vacant time, selecting a vacant time (such as 3-5 am) in the scene without vehicles, pedestrians and other dynamic obstacles, recording the fixed features of the scene as a reference benchmark, specifically comprising: Recording the radar reference point: the radar scans the scene and collects the three-dimensional coordinates P of the fixed targets such as road markings, kerbs and lamp posts r =[xr,yr,zr] T to form a radar reference point set.

[0028] Projection generation image reference point: using the radar-camera extrinsic parameters (R, t) and camera intrinsic parameters K obtained in step 101, the radar reference point P r is projected onto the camera image plane to obtain the image reference point P ref , and the projection formula is as follows: Where [u, v] T are the pixel coordinates of the image reference point P ref , [x r , y r , z r ] T are the three-dimensional coordinates of the radar reference point P r , R is the rotation matrix, t is the translation vector, and K is the camera intrinsic matrix.

[0029] Step 2, real-time data acquisition and error calculation: this step quantifies the error caused by the device deflection by continuously collecting device data and comparing it with the reference information, which includes the following steps: Step 201, real-time acquisition of radar target points: during the operation of the device, the radar real-time detects the scene target, and collects the three-dimensional coordinates P rn =[x rn , y rn , z rn ] T of dynamic vehicles and fixed targets.

[0030] Step 202, projection to generate real-time image points: using the same projection method as step 102, the real-time radar target point P rn is projected onto the image plane through the initial extrinsic parameters (R, t) and camera intrinsic parameters K to obtain the real-time projection point P in =[x in , y in ] T .

[0031] Step 203, remove outliers: to eliminate dynamic targets, noise and other interference, the real-time data is subjected to outlier removal, and the specific process is as follows: Calculate the Euclidean distance d i of the real-time projection point P in to the nearest image reference point P ref : Calculate the Euclidean distance d r of the real-time radar target point P rn to the nearest radar reference point P r : Calculate the weighted distance d: Where α is the proportionality coefficient, used to unify d i (pixel unit) and d r The order of magnitude (of physical units) w i The width of the camera image resolution (in pixels), w r This represents the actual width (in meters) of the main observation area of ​​the image.

[0032] If the weighted distance d is greater than the preset threshold T d (Based on scene characteristics, such as 50 pixels per meter), then the point is determined to be an outlier, and the corresponding real-time radar target point P is removed. rn and real-time projection point P in .

[0033] Step 204: Calculate the weighted error. Based on the valid data after removing outliers, calculate the comprehensive error caused by equipment deflection. For the remaining real-time projection points With the corresponding image reference point P ref Matching point pairs are formed based on the closest Euclidean distance as the matching criterion, and the average pixel error d is calculated. m : Where N1 is the number of image matching point pairs, ( )for pixel coordinates, (x ref ,y ref ) is P ref The pixel coordinates.

[0034] For the remaining real-time radar target points With the corresponding radar reference point P r Matching point pairs are formed based on the closest Euclidean distance as the matching criterion, and the average distance error d is calculated. n : Where N2 is the number of radar matching point pairs, ( )for The three-dimensional coordinates, (x r ,y r ,z r ) is P r The three-dimensional coordinates.

[0035] Calculate the weighted error d e : Wherein, β is a weighting coefficient (which can be dynamically adapted according to the real-time performance of the radar and camera, such as β being 0.3 when the radar data is stable and β being 0.7 when the camera data is clear).

[0036] Step 3, deflection detection and automatic calibration, which determines whether the device is deflected according to the error judging device, and automatically corrects when needed: Step 301, threshold judgment, compare the weighted error d e with the preset threshold T err (accuracy requirement according to application scenario, such as 30 pixels / m) : if d e ≤T err , it is determined that the device is not deflected and no calibration is needed; if d e >T err , it is determined that the device is deflected and the calibration process is triggered.

[0037] Step 302, optimization of new calibration matrix, when the device is determined to be deflected, the new radar-camera external parameter is solved by optimization algorithm: Construct the objective function: the objective function is defined to minimize the pixel error between the real-time projection point and the image reference point: Where R new is the new rotation matrix, t new is the new translation vector, ||・|| is the L2 norm of the vector, and (u new ,v new ) is the image coordinate of the real-time radar target point projected by the new external parameter.

[0038] Optimization solution: the Levenberg-Marquardt nonlinear optimization algorithm is used to iteratively solve the objective function to obtain the new rotation matrix R new and translation vector t new , which constitute the new radar-camera external parameter.

[0039] Parameter update: replace the initial external parameter (R new , t new ) with the new external parameter (R, t) to complete the device calibration, and subsequent data acquisition and processing are based on the new parameters.

[0040] Example two This embodiment takes the calibration of a roadside radar-camera integrated machine on urban roads as an example to illustrate the specific implementation process of the present application.

[0041] Step 1, initialization and reference information acquisition: within a range of 20 meters around the installation position of the radar-camera integrated machine, 5 calibration objects with known three-dimensional coordinates (coordinate accuracy ± 5 cm) are arranged, the calibration objects are cubes with a side length of 30 cm, and high reflectivity materials are pasted on the surface.

[0042] A radar (range 100 meters, angular resolution 0.1°) collects the three-dimensional coordinates of the calibration object, and a camera (resolution 1920x1080) takes an image containing the calibration object, and extracts the two-dimensional coordinates of the calibration object in the image.

[0043] The radar and camera data are associated by the SIFT feature matching algorithm, and the LM algorithm is used to solve the initial extrinsic parameters of the radar-camera: the rotation matrix R and the translation vector t (t = [0.5m, 0.3m, 0.2m] T ).

[0044] The Zhang Zhengyou calibration method is used, and the camera takes 10 images of the checkerboard at different angles, and the camera intrinsic parameters K are calculated: fx = 1200 pixels, fy = 1200 pixels, cx = 960 pixels, cy = 540 pixels, distortion coefficients k1 = -0.03, k2 = 0.01, p1 = p2 = 0.

[0045] Reference point acquisition: select 4am (no vehicles, pedestrians) as the empty time, and scan the scene with the radar to collect the three-dimensional coordinates P r of 100 fixed targets such as kerb stones, zebra crossing corners, and street lamp bases, forming a radar reference point set.

[0046] P r is projected to the camera image by the projection formula to generate 100 image reference points P ref , which are stored in the local database of the device.

[0047] Step 2: Real-time data acquisition and error calculation Real-time data acquisition: during the device operation period (e.g. 7-9am during the morning rush hour), the radar collects scene data every 100ms, obtaining real-time radar target points P rn .

[0048] Projection and outlier rejection: project P rn to the image plane to obtain P in , calculate d i (pixels), d r (meters), where α = 1920 pixels / 30 meters = 64 pixels / meter.

[0049] Set T d = 50 pixels·m, reject outliers (mainly radar points of passing vehicles and corresponding projection points) with d > 50, and retain 30-50 valid fixed target points.

[0050] Error calculation: calculate d m = 8 pixels, d n = 0.5 meters, β = 0.5 for the valid points, and obtain d e= 0.5 x 8 + (1 - 0.5) x 0.5 = 4.25 pixel m (the unit is unified by α, and the actual value is the integrated error value).

[0051] Step 3: deflection detection and calibration Set T err = 10 pixel m, since d e = 4.25 < 10, it is determined that the device is not deflected, and no calibration is required.

[0052] Example Three This example simulates the calibration process after the device is slightly deflected.

[0053] Step 1-2: initialization and real-time data acquisition As in Example Two, the device is deflected due to strong wind during operation, and after processing the real-time data: effective point d m = 15 pixels, d n = 1.2 meters, β = 0.5, d e = 0.5 x 15 + 0.5 x 1.2 = 8.1 pixel m (still less than T err , no calibration is triggered).

[0054] Step 3: calibration after deflection intensifies, the device deflection angle increases, and after processing the real-time data: d m = 25 pixels, d n = 2.0 meters, d e = 0.5 x 25 + 0.5 x 2.0 = 13.5 pixel m > 10 pixel m, calibration is triggered.

[0055] Construct the objective function, and solve for new external parameters R new , t new by iterating 100 times using the Levenberg-Marquardt algorithm.

[0056] After updating the external parameters of the device, the data is reacquired and calculated d e = 3.8 pixel m, and the normal accuracy is restored.

[0057] It should be noted that in this paper, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device.

[0058] Finally, it should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art will appreciate that modifications can be made to the technical solutions described in the foregoing embodiments, or some of the technical features thereof can be replaced by equivalent features. Any modifications, equivalent replacements, improvements, and the like made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for automatic calibration of a radar and visual integrated machine based on scene features, characterized in that: Comprising the following steps: S1, initialization and reference information acquisition: S101, obtaining radar-camera initial extrinsic parameters of the radar-camera integrated machine and camera intrinsic parameters by a calibration method; S102, acquire the radar reference point P at a vacant moment of the scene r and project it to the camera image plane to obtain the image reference point Pref, the radar reference point P r is the three-dimensional coordinate of a fixed target in the scene detected by the radar, and the image reference point Pref is obtained based on the radar-camera initial extrinsic parameters acquired in step S101 and the camera intrinsic parameters. S2, real-time data acquisition and error calculation: S201, collect the three-dimensional coordinates of the target detected by the radar in real time as a real-time radar point P rn ; S202, projecting the real-time radar point P rn Based on the radar-camera initial extrinsic parameters and the camera intrinsic parameters obtained in step S101, the real-time projection point P in ; S203, performing outlier rejection on the real-time projection point P in and the real-time radar point P rn ; S204, calculate the weighted error d based on the remaining points after the outliers are removed e ; S3, deflection detection and automatic calibration: S301, compare the weighted error d e with a preset threshold T err , if d e >T err , it is determined that the radar and visual integrated machine is deflected. S302, when it is determined that deflection occurs, a new radar-camera extrinsic parameter is solved by an optimization algorithm to replace the initial extrinsic parameter to complete calibration.

2. The method according to claim 1, wherein the method is characterized in that: In step S101, the process of obtaining radar-camera initial extrinsic parameters by the calibration method includes: arranging a plurality of calibration objects with known positions in a calibration scene, obtaining three-dimensional coordinates of the calibration objects by a radar, obtaining two-dimensional image coordinates of the calibration objects by a camera, and obtaining a rotation matrix R and a translation vector t by target point matching and an optimization algorithm.

3. The method according to claim 2, wherein the method further comprises: In step S101, the radar-camera initial extrinsic parameters include the rotation matrix R and the translation vector t, and the camera intrinsic parameters include focal lengths fx, fy, principal point coordinates cx, cy, and distortion coefficients.

4. The method according to claim 3, characterized in that: In step S101, the camera intrinsic parameters are obtained by using Zhang Zhengyou calibration method, and the camera intrinsic parameters K are recorded.

5. The method according to claim 4, wherein the method further comprises: In step S102, the scene is open when there is no dynamic obstacle such as vehicle or pedestrian in the scene.

6. The method according to claim 5, wherein the method further comprises: At the moment of emptiness, the radar detects the target in the scene, and records the three-dimensional coordinates P of all radar target points r = [xr, yr, zr] T , as the radar reference point.

7. The method according to claim 1, characterized in that: In step S203, the specific process of the outlier rejection includes: Calculate the real-time projection point P in Euclidean distance d to the nearest image reference point Pref i Calculate real-time radar point P rn To the nearest radar reference point P r Euclidean distance d r ; Computing the weighted distance where a is a proportionality coefficient, , w i is the image resolution width, w r is the actual width of the main observation area of the image; If d > preset distance threshold T d , the corresponding real-time radar point P rn and real-time projection point P in are determined as outliers and eliminated.

8. The method according to claim 1, characterized in that: In step S204, the weighted error d e The calculation process includes: remaining real-time projection points after outliers are removed , calculate the average pixel error d ref of the corresponding image reference point P m ; remaining real-time radar points after outliers are removed , the average distance error d r of each real-time radar point P n with the corresponding radar reference point P ref Based on d m And d n Calculate the weighted error d e , d e = β・d m +(1-β)・d n , where β is the weight coefficient.

9. The method according to claim 1, wherein the method further comprises: In step S302, the optimization algorithm is Levenberg-Marquardt algorithm, by constructing the objective function with the goal of minimizing the pixel error between the real-time projection points and the image reference points, the new rotation matrix R is obtained new and translation vector t new .

10. The method according to claim 9, wherein the method further comprises: The objective function is defined as: Wherein, wherein ||*|| represents the modulus of the vector, and E is an error symbol.

Images