Intelligent traffic thunder-vision linkage method, device and application

Abstract

The invention discloses an intelligent traffic thunder-vision linkage method and device and application, and belongs to the technical field of intelligent traffic systems. Comprising the steps of radar global scanning and event detection, outputting target point cloud data in real time and judging an abnormal event; coordinate mapping and space synchronization are carried out, radar detection coordinates are converted into a camera holder rotation angle through a four-stage coordinate conversion system (a radar coordinate system-a world coordinate system-a camera coordinate system-an image coordinate system), and a space-time deviation is eliminated by adopting a space synchronization mechanism combining static initial calibration and dynamic real-time calibration; a PTZ control instruction is generated and issued; and camera linkage and zoom confirmation are carried out to form complete alarm information. According to the method, the advantages of the millimeter-wave radar and the high-definition camera are deeply fused, efficient cooperation of radar wide-area detection and video accurate recognition is achieved, the problems that a traditional method is low in coordinate mapping precision, lacks dynamic calibration and is poor in environmental adaptability are solved, and the method can be widely applied to intelligent traffic monitoring and security perimeter protection.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation system technology, and in particular to an intelligent transportation radar-video linkage method, device and application based on the deep integration of millimeter-wave radar and high-definition camera, which is used to automatically trigger and guide the linkage video to a precise location after an abnormal event is detected on a highway, and to perform high-definition video review, confirmation and recording. Background Technology

[0002] In existing intelligent transportation systems, radar and video sensors typically operate independently or employ simple data overlay and fusion. The main problem lies in their reliance on fixed coordinate mappings for target association, making them ill-suited to variations in weather, lighting conditions, or slight equipment displacement, resulting in high matching error rates. Furthermore, the initial system calibration is complex and requires frequent maintenance and recalibration, lacking automation and reliability. This prevents the perception data from being stably and accurately complementary in complex scenarios, limiting overall performance improvement.

[0003] Specifically, the existing technology has the following shortcomings:

[0004] 1. Low coordinate mapping accuracy: Traditional methods rely on simple two-dimensional projection or linear transformation, without considering factors such as lens distortion and changes in device posture, resulting in large positioning errors for distant targets and poor linkage accuracy.

[0005] 2. Lack of dynamic calibration mechanism: Existing systems mostly use one-time calibration, which cannot cope with the failure of calibration parameters caused by factors such as equipment vibration, temperature drift, and installation offset, and requires frequent manual recalibration.

[0006] 3. Poor environmental adaptability: The advantages of radar and camera are not fully complementary. In harsh environments such as rain, snow, and night, the failure of a single sensor leads to a decline in the overall system performance.

[0007] 4. Discontinuous target tracking: The lack of effective motion state prediction and multi-sensor data fusion mechanism makes it easy to lose the target at the edge of the field of view or when it is occluded.

[0008] 5. Large linkage response delay: The entire process from radar detection to camera rotation into position lacks optimization, and the linkage delay usually exceeds 200 milliseconds, making it impossible to accurately capture fast-moving targets.

[0009] For example, Chinese patent CN109920185A discloses a method for detecting moving targets based on the fusion of millimeter-wave radar and video data. This method performs perimeter intrusion detection through feature extraction and data fusion. However, this method only performs simple spatiotemporal registration and does not involve a precise four-level coordinate transformation system and a dynamic real-time calibration mechanism, so it cannot guarantee long-term stable operation.

[0010] Therefore, there is a need for an intelligent transportation radar-camera linkage method that can achieve high-precision coordinate mapping between radar and cameras, has dynamic calibration capabilities, strong environmental adaptability, and rapid linkage response. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an intelligent transportation radar-visual linkage method, device and application. By deeply integrating the advantages of millimeter-wave radar and high-definition camera, it achieves efficient collaboration of "wide-area radar detection and accurate video recognition", solves the pain points of traditional video surveillance such as "blind scanning, low efficiency and great influence from the environment", and realizes a leap from "passive monitoring" to "active early warning".

[0012] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0013] A method for intelligent traffic radar-visual linkage includes the following steps:

[0014] Step 1: Radar full-domain scanning and event detection. The millimeter-wave radar continuously scans the covered road sections and outputs point cloud data of the detected targets in real time, including the target's position coordinates, speed, and direction of movement. The radar algorithm analyzes the data stream in real time and judges whether there are any abnormal events according to preset rules. Once a suspected event is detected, the event type and its geographical coordinates are output.

[0015] Step 2: Coordinate mapping and spatial synchronization. Through a four-level coordinate transformation system, the coordinates of the event points detected by the radar are converted into the horizontal and pitch angles that the camera pan-tilt unit needs to rotate. The four-level coordinate transformation system includes: rigid transformation from radar coordinate system to world coordinate system, rigid transformation from world coordinate system to camera coordinate system, and perspective projection transformation from camera coordinate system to image coordinate system.

[0016] Step 3: Generate and issue control commands. Based on the calculated pan-tilt rotation angle, generate standard PTZ control commands and send them to the camera's pan-tilt control system via the network.

[0017] Step 4: Camera linkage and zoom confirmation. After receiving the command, the camera pan-tilt unit automatically rotates to the specified angle, binding the radar event information with the video footage to form complete alarm information including time, location, event type, speed information, and high-definition images or videos.

[0018] Preferably, the fourth-level coordinate transformation in step two specifically includes:

[0019] Step 1: Radar coordinate system to world coordinate system transformation, using homogeneous coordinate transformation matrix for translation and rotation:

[0020] P w =T rw ×Pr ,

[0021] Among them, P r P represents the homogeneous coordinates of the target detected by the radar. w T represents the homogeneous coordinates in the world coordinate system. rw It is a 4×4 homogeneous transformation matrix containing the translation vector T. rw and rotation matrix R rw ;

[0022] Step 2: Transformation from World Coordinate System to Camera Coordinate System

[0023] P c =T wc ×P w ,

[0024] Among them, P c Let T be the homogeneous coordinates of the camera coordinate system. wc This is the transformation matrix from the world coordinate system to the camera coordinate system;

[0025] Step 3: Camera coordinate system to image coordinate system transformation. First, distortion correction is performed, and then perspective projection is performed using the camera intrinsic parameter matrix.

[0026]

[0027] Where K is the camera intrinsic parameter matrix, (x′′′, y′′′) are the normalized coordinates after distortion correction, and (U i V i () represents the pixel coordinates of the image.

[0028] Preferably, step two further includes a spatial synchronization mechanism, comprising static initial calibration and dynamic real-time calibration:

[0029] Static initial calibration is performed during system deployment. The target calibration method is used to obtain the fixed parameters required for coordinate mapping, including relative translation, relative rotation angle, camera intrinsic parameters, and distortion coefficients.

[0030] Dynamic real-time calibration is performed during system operation, including dynamic calibration based on fixed reference points or dynamic calibration based on dynamic target association, and continuously corrects the mapping parameters through feature matching and error feedback.

[0031] Preferably, the dynamic calibration based on a fixed reference point specifically includes:

[0032] A database is established by pre-recording the world coordinates of reference points and image pixel coordinates;

[0033] Real-time detection of the current world coordinates and real-time pixel coordinates of the reference point;

[0034] Calculate the mapping error. When the error exceeds a preset threshold, fine-tune the rotation or translation parameters using the gradient descent method until the error meets the standard.

[0035] Preferably, the dynamic calibration based on dynamic target association specifically includes:

[0036] Select the same target detected by radar and camera that meets the conditions of time correlation, motion correlation and shape correlation;

[0037] For targets that are successfully associated, calculate the error between the mapped pixel coordinates and the actual pixel coordinates;

[0038] Kalman filtering or extended Kalman filtering is used, with multi-target error as the observation value, to smoothly correct key parameters.

[0039] Preferably, step two further includes dynamic fitting, specifically:

[0040] Based on the linear motion model, the motion state equation of the target in the world coordinate system is constructed, and the state vector is defined to include position, velocity and acceleration.

[0041] Establish a state transition equation to describe the change in the target's motion state from the previous observation time to the current time;

[0042] Establish radar observation equations and camera observation equations, and map the observation data to a unified world coordinate system;

[0043] The target's motion status is updated by fusing radar and camera observation data through Kalman filtering.

[0044] A smart traffic radar-visual linkage device includes:

[0045] The millimeter-wave radar module is used to perform continuous scanning and detection 24 / 7 and output target point cloud data.

[0046] High-definition camera module, equipped with pan-tilt or dome camera, for high-definition video verification, confirmation and recording;

[0047] Intelligent algorithm platform for executing the radar-visual linkage method according to any one of claims 1-6, including coordinate mapping, spatial synchronization, dynamic fitting and PTZ control command generation;

[0048] The communication module is used to transmit data and control commands between radar, cameras, and intelligent algorithm platforms.

[0049] Preferably, the millimeter-wave radar module has a detection range of 5-200 meters, a detection angle of ±60 degrees horizontally, and a frame rate of 10-20Hz; the high-definition camera module has a resolution of 1080P or higher, supports optical zoom of more than 20x, and has a PTZ control delay of no more than 100 milliseconds.

[0050] Preferably, the intelligent algorithm platform further includes:

[0051] The calibration module is used to perform static initial calibration and dynamic real-time calibration;

[0052] The event detection module is used to determine abnormal events based on preset rules;

[0053] The coordinate transformation module is used to perform four-level coordinate transformations;

[0054] The fusion tracking module is used to perform motion fitting and target tracking.

[0055] The application of intelligent transportation radar-visual linkage method or intelligent transportation radar-visual linkage device in intelligent transportation system, wherein the intelligent transportation system includes highway abnormal event monitoring, urban road traffic violation monitoring, intersection vehicle behavior monitoring or security perimeter protection system.

[0056] The beneficial effects of this invention are:

[0057] Significantly improved environmental adaptability: The accuracy rate remains at 90.5% in rainy weather, an improvement of 32.3% compared to single devices; the accuracy rate reaches 92.1% at night, with 88.6% of the area obscured, achieving "all-weather, all-area monitoring without blind spots".

[0058] Target attribute accuracy has been greatly improved: the speed measurement error has been reduced to 1.8% (compared to 15.2% for a single camera); the category recognition misclassification rate is only 0.3% (compared to 8.3% for a single radar), achieving accurate classification that is "unambiguous and does not cause confusion".

[0059] The alarm accuracy is near perfect: the false alarm rate is reduced to 0.5%-0.8% (18.5%-25.3% for a single radar); the missed alarm rate is only 0.4%-0.7% (22.1%-28.4% for a single camera), meeting the core requirement of "zero missed alarms" in security.

[0060] Rapid linkage response: The average linkage response latency is 142-156ms, far below the excellent threshold of 200ms, ensuring that "the target is not lost".

[0061] High long-term stability: Through a dynamic real-time calibration mechanism, the mapping error change is ≤2 pixels in the temperature range of -20℃ to 60℃, and the error change is ≤1.5 pixels in a vibration environment, eliminating the need for frequent manual maintenance. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating the radar detection and camera linkage process.

[0063] Figure 2 This is the algorithm logic framework diagram. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0065] The present invention provides a method for intelligent traffic radar-visual linkage, comprising the following steps:

[0066] Step 1: Radar Full-Domain Scanning and Event Detection. Millimeter-wave radar continuously scans the covered road segments, outputting real-time point cloud data of detected targets, including their position coordinates (X_radar, Y_radar), speed, and direction of movement. The radar algorithm analyzes this data stream in real time, determining whether any abnormal events have occurred based on preset rules (e.g., sudden drop in speed to 0 at the same location → parking event; sudden reverse movement of a target → reverse driving event; sharp increase in target density and decrease in average speed in an area → congestion event). Once a suspected event is detected, the algorithm immediately outputs the event type and its precise geographic coordinates.

[0067] Step Two: Coordinate Mapping and Spatial Synchronization. Since the radar and camera are heterogeneous sensors with completely different coordinate systems (radar uses polar coordinates, camera uses image pixel coordinates), a linkage algorithm is needed for coordinate mapping. During system initialization, an accurate spatial transformation model from the radar coordinate system to the camera pixel coordinate system has been established using joint calibration technology. When the radar provides the coordinates (X_radar, Y_radar) of an event point, the algorithm uses this transformation model to immediately calculate the corresponding pixel coordinates (U, V) of that point in the camera's view, as well as the horizontal (Pan) and pitch (Tilt) angles that the pan-tilt unit needs to rotate to center the point in the image.

[0068] This coordinate mapping is achieved through a four-level coordinate transformation system:

[0069] Level 1: Radar Coordinate System → World Coordinate System

[0070] A rigid transformation is performed using a homogeneous coordinate transformation matrix (4×4), involving only translation and rotation:

[0071] P w =T rw ×P r

[0072] in:

[0073] P r =[X r ,Y r Zr ,1] T Let X be the homogeneous coordinates of the radar-detected target, where X is the coordinates of the target. r ,Y r Z r The target's three-dimensional position in the radar coordinate system, in meters.

[0074] P w =[X w ,Y w Z w ,1] T Here are the homogeneous coordinates of the target in the world coordinate system, in meters.

[0075] T rw It is a 4×4 homogeneous transformation matrix, in the form of:

[0076]

[0077] in:

[0078] R rw This is a 3×3 rotation matrix describing the rotation relationship between the radar coordinate system and the world coordinate system, r. ij For each element of the rotation matrix

[0079] t rw =[T xrw ,T yrw ,T zrw ] T This is a translation vector, representing the position of the radar origin in the world coordinate system, in meters.

[0080] Rotation matrix R rw Derived from the Euler angles (heading angle α, pitch angle β, roll angle γ) of the radar and the world coordinate system:

[0081] R rw =R z (α)×R x (β)×R y (γ)

[0082] in:

[0083] α is the heading angle, representing the rotation angle about the Z-axis, in degrees.

[0084] β is the pitch angle, representing the rotation angle about the X-axis, in degrees.

[0085] γ is the roll angle, representing the rotation angle about the Y-axis, in degrees.

[0086] R z (α),R x (β),R y(γ) are the standard rotation matrices for rotation about each axis.

[0087] Level 2: World Coordinate System → Camera Coordinate System

[0088] Consistent with the principle of the first stage, a homogeneous transformation matrix T is used. wc Perform a rigid transformation:

[0089] P c =T wc ×P w

[0090] in:

[0091] P c =[X c ,Y c Z c ,1] T Let X be the homogeneous coordinates of the target in the camera coordinate system, where X... c ,Y c Z c The target's three-dimensional position in the camera coordinate system, in meters.

[0092] T wc R is a 4×4 homogeneous transformation matrix from the world coordinate system to the camera coordinate system, containing the camera's rotation matrix. wc Translation vector t wc This information is obtained through camera calibration (such as the Zhang Zhengyou calibration method).

[0093] Level 3: Camera Coordinate System → Image Coordinate System

[0094] This step achieves the "3D to 2D" conversion. It requires first correcting the lens optical distortion, and then mapping it to the image plane through a perspective projection matrix.

[0095] (1) Distortion correction:

[0096] Normalization: (x′,y′)=(X c / Z c ,Y c / Z c )

[0097] Where x′, y′ are normalized coordinates, representing the projected coordinates of the target on the camera's normalized plane.

[0098] Radial distortion correction:

[0099] x′′=x′×(1+k1r 2 +k2r 4 +k3r 6 )

[0100] y′′=y′×(1+k1r 2 +k2r 4 +k3r 6 )

[0101] in:

[0102] x′′, y′′ are the coordinates after radial distortion correction.

[0103] r 2 =x′ 2 +y′ 2 The squared distance from the normalized coordinates to the optical center.

[0104] k1, k2, and k3 are radial distortion coefficients, obtained through camera calibration, and are dimensionless.

[0105] Tangential distortion correction:

[0106] x′′′=x′′+[2p1x′y′+p2(r 2 +2x′ 2 )]

[0107] y′′′=y′′+[p1(r 2 +2y′ 2 )+2p2x′y′]

[0108] in:

[0109] x′′′, y′′′ are the final normalized coordinates after tangential distortion correction.

[0110] p1 and p2 are tangential distortion coefficients, obtained through camera calibration, and are dimensionless.

[0111] (2) Perspective projection:

[0112] The corrected normalized coordinates are mapped to the image coordinate system using the camera intrinsic parameter matrix K:

[0113]

[0114] in:

[0115] Ui,Vi are the pixel coordinates of the target in the image, in pixels.

[0116] The intrinsic parameter matrix K is in the form of:

[0117]

[0118] in:

[0119] f x=f / dx is the pixel focal length in the X direction, in pixels, where f is the physical focal length of the lens (in millimeters), and dx is the physical size of a single pixel in the X direction (in millimeters per pixel).

[0120] f y =f / dy is the pixel focal length in the Y direction, in pixels, where dy is the physical size of a single pixel in the Y direction (in millimeters per pixel).

[0121] u0, v0 are the pixel coordinates of the principal point of the image (the intersection of the optical axis and the image plane), in pixels.

[0122] s is the pixel skew coefficient, which describes the degree to which the coordinate axes of the image sensor are not orthogonal. It is usually 0 and dimensionless.

[0123] Spatial synchronization: a dynamic calibration mechanism to eliminate spatiotemporal deviations

[0124] After the initial coordinate mapping is completed, the radar and camera will have registration errors due to hardware delays, environmental disturbances, target motion blur, etc. Spatial synchronization needs to be achieved through "static initial calibration + dynamic real-time calibration".

[0125] (1) Static initial calibration (executed during deployment):

[0126] All fixed parameters are obtained using a target calibration method. After target deployment, the coordinates of the target marker points detected by radar and the pixel coordinates of the marker points in the camera image are read, and the solution is obtained using a nonlinear least squares method.

[0127]

[0128] in:

[0129] θ is the set of parameters to be solved, including rotation matrix, translation vector, intrinsic parameter matrix, distortion coefficients, etc.

[0130] n is the number of marked points on the calibration target.

[0131] , Let be the actual measured pixel coordinates of the i-th marker point in the camera image, in pixels.

[0132] , These are the predicted pixel coordinates of the i-th marker point, calculated using coordinate mapping based on parameter θ, in pixels.

[0133] The objective function is to minimize the sum of squared Euclidean distances between the predicted and actual coordinates of all marked points.

[0134] (2) Dynamic real-time calibration (executed during operation):

[0135] Dynamic calibration based on fixed reference points: The world coordinates and image pixel coordinates of reference points (such as traffic light poles and ground markings) are pre-recorded, the current coordinates of the reference points are detected in real time, and the mapping error is calculated. If the error exceeds a threshold (such as 3 pixels), the rotation / translation parameters are fine-tuned using the gradient descent method until the error meets the standard.

[0136] Mapping error calculation formula:

[0137] in:

[0138] e represents the mapping error, expressed in pixels.

[0139] , The pixel coordinates are detected in real time for the reference point, in pixels.

[0140] , Predicted pixel coordinates for reference points calculated based on current calibration parameters, in pixels.

[0141] Dynamic calibration based on dynamic target association: Selecting "same target" requires meeting the following criteria: ① Time association (radar detection and camera acquisition time difference ≤ 50ms); ② Motion association (radar target velocity matches camera target optical flow velocity); ③ Shape association (camera target category matches radar RCS). For successfully associated targets, the error between mapped pixel coordinates and actual pixel coordinates is calculated. Kalman filtering (KF) or extended Kalman filtering (EKF) is used, with multi-target error as the observed value, to smoothly correct key parameters.

[0142] Time synchronization:

[0143] A unified timestamp (error ≤ 1ms) is provided through a GPS timing module (such as PPS pulse signal), or the timestamp is aligned through a system clock (such as Linux NTP service) (error 10-50ms), and the position offset is compensated by the target speed.

[0144] Position offset compensation formula: ΔX=V×Δt

[0145] in:

[0146] ΔX represents the target position offset due to time asynchrony, in meters.

[0147] V represents the target velocity, measured in meters per second.

[0148] Δt is the time difference between radar and camera data acquisition, in seconds.

[0149] Motion fit:

[0150] The motion state equation of the target in the world coordinate system is constructed based on the linear motion model (uniform velocity / uniform acceleration).

[0151] State vector definition:

[0152] Uniform velocity model (6-dimensional): x(t)=[X w (t),Y w (t),Z w (t),V x (t),V y (t),V z (t)] T

[0153] Where: X w (t),Y w (t),Z w (t) represents the three-dimensional position of the target in the world coordinate system at time t (unit: meters); V x (t),V y (t),V z (t) represents the target's velocity along the corresponding axis (unit: meters per second).

[0154] Uniform acceleration model (9 dimensions): x(t) = [X w (t),Y w (t),Z w (t),V x (t),V y (t),V z (t),a x (t),a y (t),a z (t)] T

[0155] Among them: the first 6 elements are the same as those in the uniform velocity model; a x (t),a y (t),a z (t) represents the acceleration of the target along the corresponding axis (unit: m / s²).

[0156] State transition equation:

[0157] x(t k+1 )=F⋅x(t k )+w(t k )

[0158] in:

[0159] F is the state transition matrix, which describes the evolution of the target state over time.

[0160] w(t kLet w(t) be the process noise vector, which follows a Gaussian distribution. k )∼N(0,Q),

[0161] Q is the process noise covariance matrix, which describes the uncertainty of the motion model.

[0162] Observation equation:

[0163] Radar observation: z r (t k )=H r ⋅x(t k )+v r (t k Where: z r (t k )=[X r,w (t k ),Y r,w (t k ),Z r,w (t k )] T For radar at t k The observation vector at time (position in world coordinate system, unit: meters); H r For the radar observation matrix, only the position term in the state vector is extracted; v r (t k ) represents radar observation noise, which follows a Gaussian distribution. r (t k )∼N(0,R r ); R r The radar observation noise covariance matrix,

[0164] Camera observation: z c (t m )=H c ⋅x(t m )+v c (t m Where: z c (t m )=[X c,w (t m ),Y c,w (t m ),Z c,w (t m )] T H represents the camera's observation vector at time tm (the position in the world coordinate system after pixel coordinate transformation, in meters); c For camera observation matrix; v c (t m The noise observed by the camera follows a Gaussian distribution. c (t m )∼N(0,Rc ); R c The noise covariance matrix for camera observation.

[0165] Step 3: Generate and issue control commands. Based on the calculated pan-tilt rotation angle (Pan, Tilt), the algorithm platform generates standard PTZ control commands (such as national standard protocols or ONVIF protocols), which are then sent to the corresponding camera's pan-tilt control system via the network.

[0166] Step 4: Camera Linkage and Zoom Confirmation. The camera's pan-tilt unit receives the command and quickly and automatically rotates to the specified angle. The system binds the "radar event information" with the "video footage" to form a complete alarm message containing time, location, event type, speed information, and high-definition images / videos.

[0167] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0168] Example 1: Traffic monitoring on urban main roads

[0169] like Figure 1 and Figure 2 As shown, this embodiment provides an intelligent traffic radar-visual linkage system applied to urban main roads, including:

[0170] 1. Hardware Configuration

[0171] Millimeter-wave radar: Digital Intelligence Radar, detection range 5-200m, detection angle horizontal ±60°, frame rate 15Hz, speed measurement range -40~120km / h.

[0172] High-definition camera: Hikvision, 4K resolution (3840×2160), 30fps frame rate, 20x optical zoom, PTZ control latency ≤100ms.

[0173] Intelligent Algorithm Platform: Self-developed radar-camera linkage fusion algorithm, supporting time synchronization (PTP protocol, error ≤1ms), spatial registration (calibration error ≤0.5 pixels), and target association accuracy ≥95%.

[0174] 2. Deployment Scenarios

[0175] This is a six-lane, two-way urban arterial road with a monitoring range of 150 meters and a speed limit of 60 km / h. Monitoring targets include: speeding vehicles, pedestrians running red lights, and vehicles changing lanes illegally.

[0176] 3. Implementation Process

[0177] (1) Relative pose setting of radar and camera

[0178] Installation height: Both the radar and the camera are installed on poles along the road, at a height of H=4.5m.

[0179] Relative translation: T rc =[0.3m,0.2m,0m], meaning the radar and camera are offset by 0.3m in the X-axis direction (along the road extension direction), offset by 0.2m in the Y-axis direction (perpendicular to the road direction), and have the same height in the Z-axis direction.

[0180] Relative rotation angle: A rc =[0°,8°,0°], meaning the heading angle is the same, the camera's pitch angle is 8° greater than the radar's (covering the road surface), and the roll angle is horizontal.

[0181] (2) Static initial calibration

[0182] Three calibration targets of different sizes (small target 0.5m×0.5m, medium target 1m×1m, and large target 2m×2m) were placed at positions of 3m, 50m, and 150m respectively.

[0183] The translation was measured using a laser rangefinder (accuracy ±1 mm), and the rotation angle was measured using an electronic level (accuracy ±0.1°) and an IMU (accuracy ±0.05°).

[0184] Obtaining camera intrinsic parameters using Zhang Zhengyou's calibration method:

[0185] Focal length: f x =2800 pixels, f y =2800 pixels

[0186] • Principal points: u0 = 1920 pixels, v0 = 1080 pixels

[0187] • Distortion coefficients: k1=−0.15, k2=0.08, k3=−0.01, p1=0.001, p2=−0.001

[0188] The rotation matrix and translation vector are solved using the nonlinear least squares method, and the mapping error after calibration is ≤2 pixels.

[0189] (3) Radar global scanning and event detection

[0190] The radar scans the road at a frequency of 15Hz and outputs target point cloud data in real time. At a certain moment, it detects a vehicle suddenly decelerating, with its speed dropping from 80km / h to 0km / h, lasting for more than 5 seconds, which is determined to be a "stopping event".

[0191] Event coordinates: (X) radar =120m,Y radar =5m,Z radar =0.5m)(Radar coordinate system)

[0192] (4) Coordinate mapping and spatial synchronization

[0193] First-level transformation: Radar coordinate system → World coordinate system

[0194] Translation vector: T rw =[0m,0m,4.5m] T (Radar installation height)

[0195] Rotation matrix: R rw =I (Radar is aligned with the world coordinate system)

[0196] Conversion result: (X) w =120m,Y w =5m,Z w =5m)

[0197] Second-level transformation: World coordinate system → Camera coordinate system

[0198] Translation vector: T wc =[0.3m,0.2m,4.5m] T (Camera offset relative to radar)

[0199] Rotation matrix: Camera pitch angle 8°, R wc Corresponding rotation matrix

[0200] Conversion result: (X) c =119.7m,Y c =4.8m,Z c =5m)(Camera coordinate system)

[0201] Third-level transformation: Camera coordinate system → Image coordinate system

[0202] Normalization: (x′=X) c / Z c =23.94, y′=Y c / Z c =0.96)

[0203] Distortion correction: r 2 =x′ 2 +y′ 2 =574.3

[0204] x′′=23.94×(1−0.15×574.3+0.08×574.3 2 )=23.85 y′′=0.96×(1−0.15×574.3+0.08×574.3 2 )=0.95

[0205] Tangential distortion correction:

[0206] x′′′=23.85+[2×0.001×23.94×0.96+(−0.001)×(574.3+2×23.94 2 )]=23.20y′′′=0.95+[0.001×(574.3+2×0.96 2 )+2×(−0.001)×23.94×0.96]=0.52

[0207] Perspective projection:

[0208] U i =2800×23.20+1920=66880 pixels

[0209] V i =2800×0.52+1080=2536 pixels

[0210] Because the image resolution is 3840×2160, (U i V i If it exceeds the range, the gimbal needs to be rotated.

[0211] Calculate the gimbal rotation angle:

[0212] Pan = arctan(X) c / Z c )=87.6°

[0213]

[0214] Dynamic real-time calibration:

[0215] The system pre-registers traffic light poles along the road as fixed reference points, in world coordinates (X). wf =100m,Y wf =10m,Z wf =6m), initial pixel coordinates (U if =2500,V if =1500).

[0216] Real-time detection of the current pixel coordinates of the reference point Calculate the mapping error:

[0217]

[0218] If the error exceeds the threshold of 3 pixels, dynamic calibration is initiated. The pitch angle is fine-tuned using the gradient descent method, adjusting it from 8° to 7.95°. The mapped coordinates are then recalculated, reducing the error to 2.1 pixels, which meets the requirements.

[0219] (5) Generate and issue PTZ control commands

[0220] Generate ONVIF protocol PTZ control commands:

[0221] xml

[0222] <tptz:absolutemove>

[0223] <tptz:profiletoken> Profile_1< / tptz:profiletoken>

[0224] <tptz:position>

[0225] <tt:pantilt x="87.6" y="2.3" / >

[0226] <tt:zoom x="10.0" / >

[0227] < / tptz:position>

[0228] <tptz:speed>

[0229] <tt:pantilt x="1.0" y="1.0" / >

[0230] < / tptz:speed>

[0231] < / tptz:absolutemove>

[0232] The data is sent to the camera's pan-tilt control system via a TCP / IP network with a latency of ≤10ms.

[0233] (6) Camera linkage and zoom confirmation

[0234] After receiving the command, the camera pan-tilt unit rotates to the specified angle (Pan=87.6°, Tilt=2.3°) within 90ms and adjusts the zoom to 10x to center the parked vehicle in the frame.

[0235] The system automatically captures high-definition images, records the license plate number, vehicle type (sedan), parking time (5 seconds), and parking location (120m from the radar), forming complete alarm information, which is then uploaded to the traffic management platform.

[0236] Test results:

[0237] The test was conducted continuously for 72 hours (3 days) and repeated 3 times under four scenarios: morning rush hour (7:30-9:00, traffic flow of 800 vehicles / hour), evening rush hour (17:30-19:00), nighttime (22:00-24:00), and rainy weather (moderate rain, visibility 50-80m).

[0238] Test metrics:

[0239] Evaluation indicators Single radar Single camera Linkage Algorithm Advantage enhancement Target detection accuracy (occlusion) 52.3% 49.7% 88.6% +36.3% Target detection accuracy (nighttime) 78.5% 49.7% 92.1% +42.4% Distance measurement error rate - 18.6% 1.2% -17.4% Category recognition misclassification rate 10.2% - 0.6% -9.6% Event alert accuracy 68.7% 65.2% 99.1% +30.4% False alarm rate 25.3% - 0.5% -24.8% Missed alarm rate - 28.4% 0.4% -28.0% Linkage response delay - - 142ms Satisfying ≤200ms

[0240] Example 2: Perimeter Security Protection

[0241] This embodiment provides a radar-visual linkage system for use on the perimeter of an industrial park.

[0242] 1. Deployment Scenario

[0243] A closed perimeter of 1 km in circumference, including lawns, walls, streetlights (10-20 lux illuminance at night), and areas obscured by trees. Monitoring targets: unauthorized personnel (climbing over walls, crawling through fences), and unauthorized vehicles (entering restricted areas).

[0244] 2. Key Technologies

[0245] (1) Dynamic calibration based on dynamic target association:

[0246] Since there are no fixed reference points in the perimeter environment, dynamic target association calibration is used. At a certain moment, the radar detects a pedestrian with coordinates (X... r =50m,Yr =2m,Z r =1.7m), velocity (V) x =1.2m / s,V y =0m), RCS=0.5m².

[0247] The camera simultaneously detected the pedestrian, pixel coordinates (U=1500, V=1200), category "pedestrian", optical flow velocity (V). u =15 pixels / s, V v =0).

[0248] Association determination:

[0249] • Time correlation: Timestamp difference ≤ 20ms ≤ 50ms, satisfied.

[0250] • Motion correlation: Camera optical flow velocity converted to actual velocity V c =15×0.08=1.2m / s (0.08 is the conversion factor from pixel to meter), which is consistent with the radar speed of 1.2m / s, thus satisfying the condition.

[0251] • Shape correlation: The camera detects the category "pedestrian," and the radar RCS = 0.5m² falls within the pedestrian range (0.3-1.0m²), thus satisfying the condition.

[0252] The association was successful. The error between the mapped pixel coordinates (Umap, Vmap) and the actual pixel coordinates (U, V) was calculated, and the Euler angles in the extrinsic parameter matrix were corrected by Kalman filtering.

[0253] (2) Adaptation to harsh environments:

[0254] Nighttime scenes: Switch the camera to "Night Vision Mode," turn on the infrared fill light (fill light distance ≥ 50m), and adjust the exposure time (1 / 50s). Rely on radar for primary detection, with the camera assisting in confirming the target category (pedestrian / vehicle).

[0255] In occlusion scenarios: When trees obscure 30% of the target area, the radar activates a "penetrate occlusion" mode, extending the point cloud integration time (from 50ms to 100ms) to improve the detection rate of occluded targets. The camera employs "multi-frame fusion" detection (3-5 frames superimposed) to identify partially visible targets.

[0256] 3. Test Results

[0257] Evaluation indicators Single radar Single camera Linkage Algorithm Advantage enhancement Target detection accuracy (sunny day) 82.3% 88.7% 96.8% +8.1% Target detection accuracy (rain / night) 65.1% 58.2% 90.5% +32.3% Speed ​​measurement error rate - 15.2% 1.8% -13.4% Category recognition misclassification rate 8.3% - 0.3% -8.0% Event alert accuracy 75.2% 70.3% 98.5% +23.3% False alarm rate 18.5% - 0.8% -17.7% Missed alarm rate - 22.1% 0.7% -21.4% Linkage response delay - - 156ms Satisfying ≤200ms

[0258] Example 3: Monitoring of Abnormal Events on Highways

[0259] This embodiment provides a radar-visual linkage system for use on highways to monitor abnormal events such as traffic accidents, congestion, parking, and wrong-way driving.

[0260] 1. Key Technologies

[0261] (1) Multi-target tracking and trajectory prediction

[0262] A uniformly accelerated motion model is adopted, with a 9-dimensional state vector: x(t)=[X w (t),Y w (t),Z w (t),Vx(t),V y (t),V z (t),a x (t),a y (t),a z (t)] T

[0263] State transition matrix:

[0264]

[0265] Radar observation equation: z r (t k )=H r ⋅x(t k )+vr(t k )

[0266] Camera observation equation: z c (t m )=H c ⋅x(t m )+v c (t m )

[0267] By employing extended Kalman filtering (EKF) to fuse radar and camera observation data, the target's position can be predicted one second in advance, guiding the camera to rotate in advance and ensuring that fast-moving targets (speed 120km / h) are not lost.

[0268] (2) Temperature compensation and vibration compensation

[0269] The ambient temperature on highways varies greatly (-20℃ to 60℃) and the vibration is strong (10-500Hz).

[0270] Temperature compensation: Deploy temperature sensors (±0.5℃ accuracy) near the radar and cameras, and establish a "temperature-error" fitting model: Δe(T)=0.002T 2 -0.05T+1.5

[0271] When the current temperature T=35℃, the compensation error Δe(35)=1.275 pixels, and the mapping coordinates are corrected.

[0272] Vibration Compensation: Install a vibration sensor (accuracy of ±0.01g) on the bracket and establish a "vibration - rotation angle offset" model: Δα = 0.15×∫a(t)dt

[0273] Adjust the rotation matrix in real time, and the mapping error change in the vibration environment is reduced from 3 pixels to 1.5 pixels.

[0274] 2. Application Effects

[0275] Continuous monitoring for 72 hours detected:

[0276] · 5 traffic accidents, with an accuracy rate of 100% and an average linkage response time of 148ms

[0277] · 32 vehicle parking events, with an accuracy rate of 98.5% and 1 false alarm (misjudgment of a tree)

[0278] · 8 congestion events, with an accuracy rate of 100%

[0279] · 0逆行 events (none occurred)

[0280] The accuracy rate of the system remains 91.2% in rainy days (moderate rain, visibility of 80m) and 93.5% at night (no street lights, illuminance ≤ 5 lux), demonstrating the excellent performance of the present invention in harsh environments.

[0281] The intelligent traffic radar - vision linkage method, device and application provided by the present invention can be applied in the intelligent traffic system of cities, highway monitoring systems, and industrial park security systems, and can achieve significant economic and social benefits:

[0282] 1. Reduce deployment and maintenance costs: The radar has a long detection distance (5 - 200m), and the camera has a large detection angle (horizontal ±60°). By linkage, the number of installed devices is reduced, saving 40% of the deployment cost. The dynamic calibration mechanism reduces the frequency of manual maintenance, saving 60% of the maintenance cost.

[0283] 2. Improve monitoring efficiency: From "passive monitoring" to "active warning", the radar automatically guides the camera to track after detecting anomalies, eliminating the need for manual blind scanning, and the monitoring efficiency is increased by 80%.

[0284] 3. Enhance public safety: High accuracy rate (≥96.8%), low false alarm rate (≤0.8%), low missed alarm rate (≤0.7%), effectively preventing and promptly handling security events such as traffic accidents and perimeter intrusions, and ensuring the safety of people's lives and property.

[0285] 4. Adapt to complex environments: The accuracy rate remains ≥88.6% in harsh environments such as rain, snow, night, and occlusion, truly achieving "all - weather, full - area, no - dead - angle" intelligent monitoring.

[0286] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for intelligent traffic radar-visual linkage, characterized in that, Includes the following steps: Step 1: Radar full-domain scanning and event detection. The millimeter-wave radar continuously scans the covered road sections and outputs point cloud data of the detected targets in real time, including the target's position coordinates, speed and direction of movement. The radar algorithm analyzes the data stream in real time and determines whether any abnormal events have occurred based on preset rules. Once a suspected event is detected, it outputs the event type and its geographic coordinates. Step 2: Coordinate mapping and spatial synchronization. Through a four-level coordinate transformation system, the coordinates of the event points detected by the radar are converted into the horizontal and pitch angles that the camera pan-tilt unit needs to rotate. The four-level coordinate transformation system includes: rigid transformation from radar coordinate system to world coordinate system, rigid transformation from world coordinate system to camera coordinate system, and perspective projection transformation from camera coordinate system to image coordinate system. Step 3: Generate and issue control commands. Based on the calculated pan-tilt rotation angle, generate standard PTZ control commands and send them to the camera's pan-tilt control system via the network. Step 4: Camera linkage and zoom confirmation. After receiving the command, the camera pan-tilt unit automatically rotates to the specified angle, binding the radar event information with the video footage to form complete alarm information including time, location, event type, speed information, and high-definition images or videos.

2. The intelligent transportation radar-visual linkage method according to claim 1, characterized in that, The fourth-level coordinate transformation in step two specifically includes: Step 1: Radar coordinate system to world coordinate system transformation, using homogeneous coordinate transformation matrix for translation and rotation: P w =T rw ×P r , Among them, P r P represents the homogeneous coordinates of the target detected by the radar. w T represents the homogeneous coordinates in the world coordinate system. rw It is a 4×4 homogeneous transformation matrix containing the translation vector T. rw and rotation matrix R rw ; Step 2: Transformation from World Coordinate System to Camera Coordinate System P c =T wc ×P w , Among them, P c Let T be the homogeneous coordinates of the camera coordinate system. wc This is the transformation matrix from the world coordinate system to the camera coordinate system; Step 3: Camera coordinate system to image coordinate system transformation. First, distortion correction is performed, and then perspective projection is performed using the camera intrinsic parameter matrix. ； Where K is the camera intrinsic parameter matrix, (x′′′, y′′′) are the normalized coordinates after distortion correction, and (U i V i () represents the pixel coordinates of the image.

3. The intelligent transportation radar-visual linkage method according to claim 1, characterized in that, Step two also includes a spatial synchronization mechanism, comprising static initial calibration and dynamic real-time calibration: Static initial calibration is performed during system deployment. The target calibration method is used to obtain the fixed parameters required for coordinate mapping, including relative translation, relative rotation angle, camera intrinsic parameters, and distortion coefficients. Dynamic real-time calibration is performed during system operation, including dynamic calibration based on fixed reference points or dynamic calibration based on dynamic target association, and continuously corrects the mapping parameters through feature matching and error feedback.

4. The intelligent transportation radar-visual linkage method according to claim 3, characterized in that, The dynamic calibration based on a fixed reference point specifically includes: A database is established by pre-recording the world coordinates of reference points and image pixel coordinates; Real-time detection of the current world coordinates and real-time pixel coordinates of the reference point; Calculate the mapping error. When the error exceeds a preset threshold, fine-tune the rotation or translation parameters using the gradient descent method until the error meets the standard.

5. The intelligent transportation radar-visual linkage method according to claim 3, characterized in that, The dynamic calibration based on dynamic target association specifically includes: Select the same target detected by radar and camera that meets the conditions of time correlation, motion correlation and shape correlation; For targets that are successfully associated, calculate the error between the mapped pixel coordinates and the actual pixel coordinates; Kalman filtering or extended Kalman filtering is used, with multi-target error as the observation value, to smoothly correct key parameters.

6. The intelligent transportation radar-visual linkage method according to claim 1, characterized in that, Step two also includes dynamic fitting, specifically: Based on the linear motion model, the motion state equation of the target in the world coordinate system is constructed, and the state vector is defined to include position, velocity and acceleration. Establish a state transition equation to describe the change in the target's motion state from the previous observation time to the current time; Establish radar observation equations and camera observation equations, and map the observation data to a unified world coordinate system; The target's motion status is updated by fusing radar and camera observation data through Kalman filtering.

7. An intelligent traffic radar-visual linkage device, characterized in that, include: The millimeter-wave radar module is used to perform continuous scanning and detection 24 / 7 and output target point cloud data. High-definition camera module, equipped with pan-tilt or dome camera, for high-definition video verification, confirmation and recording; Intelligent algorithm platform for executing the radar-visual linkage method according to any one of claims 1-6, including coordinate mapping, spatial synchronization, dynamic fitting and PTZ control command generation; The communication module is used to transmit data and control commands between radar, cameras, and intelligent algorithm platforms.

8. The intelligent traffic radar-visual linkage device according to claim 7, characterized in that, The millimeter-wave radar module has a detection range of 5-200 meters, a detection angle of ±60 degrees horizontally, and a frame rate of 10-20Hz; the high-definition camera module has a resolution of 1080P or higher, supports optical zoom of more than 20x, and has a PTZ control delay of no more than 100 milliseconds.

9. The intelligent traffic radar-visual linkage device according to claim 7, characterized in that, The intelligent algorithm platform also includes: The calibration module is used to perform static initial calibration and dynamic real-time calibration; The event detection module is used to determine abnormal events based on preset rules; The coordinate transformation module is used to perform four-level coordinate transformations; The fusion tracking module is used to perform motion fitting and target tracking.

10. The application of the intelligent transportation radar-visual linkage method according to any one of claims 1-6 or the intelligent transportation radar-visual linkage device according to any one of claims 7-9 in an intelligent transportation system, wherein the intelligent transportation system includes highway abnormal event monitoring, urban road traffic violation monitoring, intersection vehicle behavior monitoring or security perimeter protection system.

Images