Method for lane-level recognition of over-limit vehicle based on fusion of millimeter wave radar and vision

Abstract

The application discloses a super-limit vehicle lane-level identification method based on millimeter wave radar and vision fusion, and relates to the technical field of intelligent transportation Internet of Things, and comprises the following steps: step 1, deploying a cascaded millimeter wave radar array and a camera at a super-limit monitoring section, and collecting a structured point table; performing distortion correction on an image frame to obtain a corrected image frame; step 2, projecting each radar target in the structured point table to an image coordinate system by using a calibration parameter to obtain a projected pixel coordinate, fusing radar observation and vision observation to output a fusion state quantity, and concatenating the fusion state quantity to obtain a continuous fusion track; and step 3, performing inverse perspective mapping on lane markings in the corrected image frame to a road surface coordinate system, polynomial fitting to generate a lane center line, and outputting a super-limit vehicle lane-level identification result. The application solves the problem of adjacent lane target confusion caused by the insufficient angular resolution of a traditional single-chip radar.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation Internet of Things technology, and more particularly to the field of computer vision technology, specifically a lane-level identification method for overloaded vehicles based on millimeter-wave radar and vision fusion. Background Technology

[0002] The management of overloaded vehicles is a crucial aspect of highway infrastructure protection and traffic safety management. Current overload monitoring systems primarily rely on road-embedded detection devices such as geomagnetic coils, piezoelectric sensors, or laser light curtains to count vehicles and measure axle loads, supplemented by video checkpoint cameras for license plate recognition and image evidence collection. While these systems are relatively mature in vehicle detection, they suffer from significant shortcomings in lane-level positioning accuracy. In recent years, millimeter-wave radar has seen increasing applications in intelligent transportation. Millimeter-wave radar offers all-weather operation, directly measuring the distance, speed, and angle of targets, unaffected by lighting conditions. However, the angular resolution of a single millimeter-wave radar is limited. Traditional 3-transmitter, 4-receiver single-chip radar typically has an azimuth resolution of 10 to 15 degrees in the 77 GHz band, corresponding to a lateral resolution of over 5 meters at a 30-meter detection distance, making it impossible to distinguish between different vehicles in adjacent lanes. Furthermore, when vehicles in adjacent lanes are at similar radial distances and speeds, radar echoes are prone to aliasing on the range-Doppler spectrum, leading to incorrect identification as a single target and causing lane-level misjudgments.

[0003] Most existing radar-vision fusion solutions employ a method of detecting vehicles across the entire image and then matching them with radar targets. This approach incurs significant computational overhead, making it difficult to meet real-time requirements on roadside edge computing devices. Furthermore, the background false alarms introduced by full-image detection increase ambiguity in the matching process. Simultaneously, existing fusion solutions typically rely on fixed lane division areas for lane assignment determination, failing to fully utilize the actual geometry of lane markings. This can easily lead to misjudgments near lane boundaries on curved or gradually changing road sections. Therefore, achieving close collaboration between radar and vision while improving radar angular resolution, and completing accurate lane-level identification of overloaded vehicles with low computational cost, remains a key challenge for current overload monitoring technology. Summary of the Invention

[0004] The purpose of this invention is to provide a lane-level identification method for overloaded vehicles based on the fusion of millimeter-wave radar and vision, which solves the problem of target confusion between adjacent lanes caused by insufficient angular resolution of traditional single-chip radar.

[0005] To address the aforementioned technical problems, this invention provides a lane-level identification method for overloaded vehicles based on millimeter-wave radar and vision fusion, comprising the following steps: Step 1: Deploy a cascaded millimeter-wave radar array and camera at the overload monitoring section, and synchronously trigger to align the radar frame with the image frame; perform channel calibration, range-dimensional and Doppler-dimensional frequency domain transformation, and constant false alarm rate detection on the radar intermediate frequency signal; extract azimuth angles from candidate detection units to form candidate detection points; obtain radar candidate targets through spatial clustering; perform azimuth angle spectrum multi-peak verification on the radar candidate targets and split radar candidate targets containing multiple effective angle spectrum peaks into independent radar target entries, which are then summarized into a structured point trace table; perform distortion correction on the image frame to obtain the corrected image frame; Step 2: Project each radar target in the structured point table onto the image coordinate system using calibration parameters to obtain projected pixel coordinates. Crop the region of interest window with the projected pixel coordinates as the center and pass it through the target detection network to obtain the vehicle detection box. Construct a cost matrix based on the projected pixel coordinates and the vehicle detection box coordinates to solve for the optimal allocation and complete the frame-by-frame pairing association. Fuse radar observations and visual observations to output fused state variables and concatenate them into a continuous fused trajectory. Step 3: After mapping the lane markings in the corrected image frame to the road surface coordinate system through inverse perspective, multinomial fitting is used to generate the lane centerline. The continuous fusion trajectory is compared with the lateral distance of each lane centerline point by point, and a stable lane assignment result is obtained through voting. The vehicle outline size is estimated based on the vehicle detection box and fusion state quantity and compared with the limit value to determine the over-limit mark. The lane-level recognition result of the over-limit vehicle is output.

[0006] Furthermore, the cascaded millimeter-wave radar array is composed of four 77GHz millimeter-wave radar chips cascaded together. Each chip has three transmit channels and four receive channels. After cascading, the four chips form a total of 12 transmit channels and 16 receive channels. The 12 transmit channels transmit linear frequency modulated continuous waves in a time-division multiplexing manner, while the 16 receive channels receive them synchronously, forming 192 virtual receive channels. The camera is a global shutter industrial camera. Synchronous triggering includes the field programmable gate array simultaneously outputting a linear frequency modulated start signal and an exposure trigger pulse at the beginning of each radar frame period, so that the intermediate frequency signal acquisition center time of each radar frame is aligned with the exposure center time of the corresponding image frame.

[0007] Furthermore, channel calibration includes cascaded chip-to-channel amplitude and phase calibration and time-division multiplexing motion phase compensation. The specific method of cascaded chip-to-channel amplitude and phase calibration is as follows: using the first of the four millimeter-wave radar chips as a reference chip, the complex beat frequency sampling values ​​of the virtual receiving channels of the second, third, and fourth chips are multiplied by the amplitude correction coefficient and phase correction coefficient that have been measured offline and stored in the on-chip memory, so that the amplitude response and phase baseline of all 192 virtual receiving channels are aligned with the reference chip. The specific method of time-division multiplexing motion phase compensation is as follows: using the phase progression relationship of each transmission channel after completing the range dimension fast Fourier transform on the same range unit, the radial displacement of the target in one frame period is estimated, and the radial displacement is proportionally converted into a channel-by-channel phase compensation amount according to the transmission time sequence of each transmission channel. The corresponding channel-by-channel phase compensation amount is applied to the complex beat frequency sampling value of each virtual receiving channel.

[0008] Furthermore, the specific methods for range-dimensional and Doppler-dimensional frequency domain transformations and constant false alarm rate (CFAR) detection are as follows: For each virtual receiving channel, a range-dimensional fast Fourier transform is performed on the calibrated beat frequency signal sampling sequence within each linear frequency modulation cycle to obtain a range-pulse matrix. A Doppler-dimensional fast Fourier transform is then performed on the range-pulse matrix along the pulse dimension to obtain a range-Doppler spectrum. Ordered statistical CFAR detection is then performed on the range-Doppler spectrum, and spectral units exceeding the detection threshold are extracted as a candidate range-Doppler unit set. The specific method for extracting the azimuth angle is as follows: For each candidate unit in the candidate range-Doppler unit set, 192 complex spectral values ​​at the corresponding position are extracted from the range-Doppler spectra of the 192 virtual receiving channels. These values ​​are then arranged in the spatial order of the virtual receiving channels on the equivalent linear array to form a 192-dimensional complex response vector. An azimuth-dimensional fast Fourier transform is then performed on the 192-dimensional complex response vector to obtain the azimuth angle spectrum. The angle corresponding to the maximum amplitude peak in the azimuth angle spectrum is extracted as the estimated azimuth angle value.

[0009] Furthermore, the specific method of spatial clustering is as follows: all candidate detection points with angular attributes are subjected to density-based spatial clustering in a 3D parameter space composed of radial distance, radial velocity, and azimuth angle. Each cluster is regarded as a radar candidate target. The mean radial distance, mean radial velocity, mean azimuth angle, and radar cross section accumulated value of all candidate detection points in the cluster are taken as the radial distance, radial velocity, azimuth angle, and radar cross section of the radar candidate target, respectively.

[0010] Furthermore, the specific method for multi-peak verification of the azimuth angle spectrum is as follows: Returning to the range cell and Doppler cell corresponding to the radar candidate target, extracting the 192-dimensional complex response vector at the corresponding position from the 192 virtual receiving channels and performing a fast Fourier transform in the azimuth dimension to obtain the complete azimuth angle spectrum, searching for all local maxima points in the complete azimuth angle spectrum, and marking the local maxima points whose amplitude is higher than the preset proportional threshold of the maximum value of the complete azimuth angle spectrum as effective azimuth angle peaks; if there is only one effective azimuth angle peak, the cluster is retained unchanged; if there are two effective azimuth angle peaks, the median of the angle values ​​corresponding to the two effective azimuth angle peaks is calculated as the angle boundary value, and candidate detection points with azimuth angle estimates less than the angle boundary value are assigned to the first sub-cluster, and candidate detection points with azimuth angle estimates greater than or equal to the angle boundary value are assigned to the second sub-cluster. The radial distance mean, radial velocity mean, azimuth angle mean, and radar cross section accumulation of the candidate detection points in the first and second sub-clusters are respectively taken to form two independent radar target entries to replace the original entries.

[0011] Furthermore, the cropping method for the region of interest (ROI) window is as follows: The corresponding window width and height are retrieved from a pre-stored distance-window size mapping table based on the radial distance of the radar target. A rectangular region is cropped on the corrected image frame, centered on the projected pixel coordinates. All ROI windows are uniformly scaled to the input size required by the target detection network and then concatenated into a batch input tensor, which is then fed into the target detection network after 8-bit integer quantization. The vehicle detection box coordinates are obtained by taking the coordinates of the midpoint of the bottom edge of the vehicle detection box in the full-image coordinate system. Specifically, the bottom edge ordinate is determined by the sum of the ordinate of the top-left corner of the vehicle detection box and the box height; the midpoint of the bottom edge is determined by the sum of the x-coordinate of the top-left corner of the box and half the box width; and the offset of the top-left corner coordinate of the ROI window in the full image is added to obtain the full-image x-coordinate and ordinate of the midpoint of the bottom edge.

[0012] Furthermore, the optimal allocation is solved using the Hungarian algorithm; the fusion of radar and visual observations is achieved using an extended Kalman filter. The state variables of the extended Kalman filter include the road surface lateral coordinates, road surface longitudinal coordinates, lateral velocity, and longitudinal velocity. In the prediction stage, a uniform linear motion model is used to extrapolate the state variables of the previous frame based on the inter-frame time interval to obtain the predicted state variables and the predicted covariance matrix. In the update stage, the radial distance, azimuth angle, and radial velocity are used as radar observations. The visual road surface coordinates obtained by back-projecting the horizontal coordinates of the bottom midpoint of the full map and the vertical coordinates of the bottom midpoint of the full map to the road surface rectangular coordinate system under the constraint that the road surface is a known horizontal plane are used as visual observations. The radar and visual observations are then fed into the extended Kalman filter to perform measurement updates.

[0013] Furthermore, the specific method of polynomial fitting is as follows: extract lane marking pixels from the corrected image frame, transform the lane marking pixels from the image coordinate system to the road surface rectangular coordinate system through inverse perspective mapping to obtain lane marking scatter points, and perform cubic polynomial least squares fitting on the lane marking scatter points of each lane marking with the road surface longitudinal coordinate as the independent variable and the road surface lateral coordinate as the dependent variable to obtain a cubic polynomial lane line model; the lane center line is generated by the arithmetic mean of the road surface lateral coordinates at the same road surface longitudinal coordinate of two adjacent cubic polynomial lane line models; the specific method of voting is as follows: perform majority voting on the point-by-point lane assignment sequence within a fixed-length time window, and take the lane number that appears most frequently as the stable lane assignment result.

[0014] Furthermore, the estimation method for the vehicle's external dimensions is as follows: Take the vehicle detection box of the frame corresponding to the trajectory point closest to the camera in the continuous fusion trajectory. Use the road longitudinal coordinate in the fusion state quantity of the trajectory point closest to the camera as the actual longitudinal distance. Combine the focal length parameter in the camera intrinsic parameter matrix to convert the width and height of the vehicle detection box into estimated values ​​for the actual lateral width and actual height, respectively. Take the absolute value of the difference between the road longitudinal coordinate of the trajectory point where the vehicle first enters the monitoring section and the road longitudinal coordinate of the trajectory point where the vehicle last leaves the monitoring section. Subtract the geometric correction amount for the vehicle's longitudinal projection to obtain the estimated value for the actual length. Compare the estimated values ​​for the actual lateral width, actual height, and actual length with the preset vehicle width limit, vehicle height limit, and vehicle length limit, respectively. If any estimated value exceeds the corresponding limit, the vehicle is marked as an over-limit vehicle.

[0015] The lane-level recognition method for overloaded vehicles based on millimeter-wave radar and vision fusion of the present invention has the following beneficial effects: This invention significantly improves the azimuth resolution of radar by cascading multiple millimeter-wave radar chips to form a large-aperture virtual channel array. This allows for direct differentiation of different vehicles in adjacent lanes at typical detection distances, fundamentally solving the problem of target confusion in adjacent lanes caused by insufficient azimuth resolution of traditional single-chip radar systems. Furthermore, this invention introduces a multi-peak verification and composite target decomposition mechanism into the radar signal processing pipeline. This mechanism automatically identifies vehicle echoes from adjacent lanes that are incorrectly clustered due to similar distance and speed during the clustering stage, and correctly decomposes them into independent targets, effectively reducing the lane-level misjudgment rate in dense traffic scenarios.

[0016] This invention employs a radar projection-guided visual local detection strategy. It uses the projected position of the radar target as the center to crop an adaptively sized region of interest window, performing target detection network inference only on a local area within the window. Compared to traditional full-image detection methods, this significantly reduces computational load and false alarm interference introduced by background regions, enabling the entire fusion processing chain to meet the real-time frame rate requirements of roadside edge computing devices. This invention fuses radar and visual observations using extended Kalman filtering within a unified road coordinate system, fully leveraging the high precision of radar in ranging and velocity measurement and the rich information of vision in target classification and contour feature extraction. The output continuous fused trajectory possesses both positional smoothness and identity continuity.

[0017] This invention further utilizes cubic polynomial lane line fitting and a time window majority voting mechanism to match the point-by-point lateral coordinates of the fused trajectory with the actual lane geometry. This enables lane assignment determination for curves and gradually changing road sections, eliminating the systematic bias of fixed-region division methods on non-straight road sections. Furthermore, this invention achieves vehicle width, height, and length estimation through joint conversion of vehicle detection frames and fused state quantities, integrating lane-level positioning and overload determination into the same processing flow. This provides complete evidence information, including lane assignment, driving speed, continuous trajectory, and overall dimensions, for dynamic overload enforcement. Attached Figure Description

[0018] Figure 1 A schematic diagram illustrating the formation principle of a cascaded MIMO virtual channel array provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the principle of azimuth angle spectrum bimodal verification and composite target decomposition provided in an embodiment of the present invention. Figure 2 (a) shows the azimuth angle spectrum of a single vehicle target. Figure 2 (b) shows the composite target azimuth angle spectrum and its breakdown. Figure 3 This is a schematic diagram illustrating the principle of radar projection-guided region of interest clipping and vehicle detection box extraction provided in an embodiment of the present invention. Detailed Implementation

[0019] A lane-level identification method for overloaded vehicles based on millimeter-wave radar and vision fusion includes the following steps: Step 1: Deploy a cascaded millimeter-wave radar array and camera at the overload monitoring section, and synchronously trigger to align the radar frame with the image frame; perform channel calibration, range-dimensional and Doppler-dimensional frequency domain transformation, and constant false alarm rate detection on the radar intermediate frequency signal; extract azimuth angles from candidate detection units to form candidate detection points; obtain radar candidate targets through spatial clustering; perform azimuth angle spectrum multi-peak verification on the radar candidate targets and split radar candidate targets containing multiple effective angle spectrum peaks into independent radar target entries, which are then summarized into a structured point trace table; perform distortion correction on the image frame to obtain the corrected image frame; Step 2: Project each radar target in the structured point table onto the image coordinate system using calibration parameters to obtain projected pixel coordinates. Crop the region of interest window with the projected pixel coordinates as the center and pass it through the target detection network to obtain the vehicle detection box. Construct a cost matrix based on the projected pixel coordinates and the vehicle detection box coordinates to solve for the optimal allocation and complete the frame-by-frame pairing association. Fuse radar observations and visual observations to output fused state variables and concatenate them into a continuous fused trajectory. Step 3: After mapping the lane markings in the corrected image frame to the road surface coordinate system through inverse perspective, multinomial fitting is used to generate the lane centerline. The continuous fusion trajectory is compared with the lateral distance of each lane centerline point by point, and a stable lane assignment result is obtained through voting. The vehicle outline size is estimated based on the vehicle detection box and fusion state quantity and compared with the limit value to determine the over-limit mark. The lane-level recognition result of the over-limit vehicle is output.

[0020] In this embodiment, the overload monitoring section is selected at a fixed cross-section location at the entrance of a highway toll station or on the main line of the expressway, with the road width covering all lanes. A cascaded millimeter-wave radar array, consisting of four 77GHz millimeter-wave radar chips, is installed in the central area of ​​the gantry beam above the section. The antenna normal of the radar array points towards the direction of traffic, enabling the radar beam to cover a depth range of approximately 10 to 200 meters in front of the section and an azimuth range spanning all lanes. Adjacent to the cascaded millimeter-wave radar array on the gantry beam, a camera is fixedly installed, with its lens optical axis also pointing towards the direction of traffic. A wide-angle lens capable of covering all lanes is selected. The installation baseline distance between the radar array and the camera is controlled within 0.3 meters to reduce parallax between the two sensors.

[0021] The cascaded millimeter-wave radar array consists of four millimeter-wave radar chips connected on the same RF printed circuit board via a high-speed, low-voltage differential signal interface. The four chips share the same local oscillator signal source to ensure frequency synchronization. Each millimeter-wave radar chip has three transmit channels and four receive channels, resulting in a total of 12 transmit channels and 16 receive channels after cascading. It employs a time-division multiplexing (TDM) multiple-input multiple-output (MIMO) operating mode, where the 12 transmit channels sequentially transmit linear frequency modulated (FM) continuous waves in a time-division manner, with only one transmit channel active at a time, while the other 11 remain silent. Simultaneously, the 16 receive channels maintain synchronous reception. Since each transmit channel and each receive channel form an independent transmit / receive pair, the 12 transmit channels and 16 receive channels are combined in pairs to form 192 virtual receive channels. These 192 virtual receive channels spatially form an equivalent large-aperture linear array with equal spacing, the distance between adjacent virtual channels being equal to half the carrier wavelength, approximately 1.9 mm. Since the angular resolution of the array is proportional to the aperture length, the equivalent aperture formed by 192 virtual receiving channels is approximately 365 mm, achieving an azimuth resolution better than 1 degree at a 77 GHz carrier frequency. At a typical detection distance of 30 meters, 1 degree of angular resolution corresponds to a lateral resolution of approximately 0.52 meters, which is sufficient to distinguish different vehicles within a standard lane width of 3.75 meters.

[0022] The camera used is a global shutter industrial camera. Its image sensor employs a global shutter exposure method, where all pixels begin and end integration simultaneously at the moment of exposure, avoiding the image tilt distortion caused by rolling shutters when shooting high-speed moving vehicles. The camera resolution is selected to be above 5 megapixels, with a frame rate of no less than 25 frames per second, to ensure that the vehicle displacement between two adjacent frames does not exceed 1.3 meters at a speed of 120 kilometers per hour, meeting the continuity requirements of target tracking.

[0023] The Field Programmable Gate Array (FPGA) serves as the core of the entire sensor system's synchronization control and signal preprocessing engine, simultaneously connecting the digital interface of the cascaded millimeter-wave radar array and the camera's external trigger port. At the beginning of each radar frame cycle, the FPGA simultaneously sends a linear frequency modulation (LFM) start signal to the cascaded millimeter-wave radar array and an exposure trigger pulse to the camera. The LFM start signal drives the first transmit channel to begin transmitting the first LFM continuous wave, while the exposure trigger pulse drives the camera to immediately initiate global exposure. Hardware synchronization triggering is used instead of software timestamp alignment because software methods suffer from uncertain delays due to operating system scheduling, typically fluctuating on the order of milliseconds, while hardware triggering jitter can be controlled within the order of microseconds. For a vehicle traveling at 120 km / h, a 1-millisecond time deviation will result in a positional deviation of approximately 3.3 centimeters. Although the absolute value is small, in lane-level judgment scenarios, accumulated deviations may cause vehicles near the lane boundary to be misjudged into adjacent lanes; therefore, hardware-level synchronization is indispensable. Each radar frame contains 12 LFM cycles, corresponding to the 12 transmit channels transmitting sequentially, with the entire radar frame lasting approximately 40 milliseconds. The camera's exposure time is set to be within the range of 1 to 5 milliseconds, and the deviation between the exposure center time and the intermediate frequency signal acquisition center time of the radar frame does not exceed 100 microseconds.

[0024] In one alternative implementation, if the camera frame rate is higher than the radar frame rate, an external trigger mode can be used to trigger camera acquisition only at the beginning of a radar frame, discarding redundant image frames to maintain a strict one-to-one frame synchronization relationship. In another alternative implementation, a free-running mode can be used at the camera end, recording the precise exposure timestamp of each image frame and the intermediate frequency signal acquisition timestamp of each radar frame via a field-programmable gate array. The backend software then searches for and matches the radar-image frame pair with the closest timestamps; however, the time alignment accuracy of this method is slightly inferior to the hardware trigger method.

[0025] After receiving the intermediate frequency beat signals from 192 virtual receive channels, the field-programmable gate array (FPGA) first performs channel calibration before entering the frequency domain processing pipeline. Channel calibration is divided into two stages.

[0026] The first stage is the amplitude and phase calibration of the cascaded chips. Although the four millimeter-wave radar chips share the same local oscillator, the manufacturing processes of the mixer gain, filter passband characteristics, DC bias, and gain of the analog-to-digital converter within each chip differ. This results in different amplitude responses and phase shifts in the virtual receiving channels of different chips for the same target echo. If azimuth processing is performed directly without calibration, it is equivalent to introducing non-uniform amplitude and phase errors into the large-aperture linear array, severely degrading the sidelobe characteristics of the angular spectrum, producing false angular peaks, and consequently causing azimuth angle estimation errors or the inability to correctly separate composite targets. The calibration process uses the first of the four millimeter-wave radar chips as a reference chip. During the system's factory calibration phase, a corner reflector at a known location is placed approximately 10 meters in front of the radar array, causing all 192 virtual receiving channels to simultaneously receive the echo signal from the corner reflector. For the virtual receiving channel belonging to the reference chip, its complex beat frequency sampling value is recorded as a baseline. For each of the virtual receiving channels belonging to the 2nd, 3rd, and 4th chips, the amplitude ratio and phase difference between their complex beat frequency sample values ​​and the reference value at the corresponding channel position on the reference chip are calculated. The reciprocal of the amplitude ratio is used as the amplitude correction coefficient, and the negative value of the phase difference is used as the phase correction coefficient. These correction coefficients are stored in the on-chip memory of the field-programmable gate array (FPGA). In actual operation, after each new intermediate frequency beat frequency signal is received, the complex beat frequency sample values ​​of the virtual receiving channels belonging to the 2nd, 3rd, and 4th chips are multiplied by the corresponding amplitude correction coefficient and phase correction coefficient, respectively, to align the amplitude response and phase baseline of all 192 virtual receiving channels with the reference chip. The calibrated inter-channel amplitude consistency should be better than 0.5 dB, and the phase consistency should be better than 5 degrees.

[0027] The second stage is time-division multiplexing (TDML) motion phase compensation. An inherent problem with TDML's multiple-input multiple-output (MIMO) mode is that the 12 transmission channels are not transmitted simultaneously but sequentially, with a linear frequency modulation period (LFM) time interval between adjacent channels. If the target undergoes radial motion within the radar frame duration, the echoes received by different transmission channels will contain an additional phase proportional to the velocity. This additional phase is superimposed on the spatial phase of the virtual array, effectively altering the target's true incident angle and causing a system shift in the azimuth angle estimate. The compensation process uses the results of the range-dimensional Fast Fourier Transform (FFT) of each transmission channel in the current frame to estimate the phase introduced by target motion. Specifically, the range-Doppler position of a high-energy target within the same range cell is selected, and the complex spectral values ​​corresponding to the 1st to 12th transmission channels are extracted. The phase of these 12 complex values ​​is observed to change progressively with the channel number. If the target is stationary, the phase of the 12 complex values ​​should only reflect the array phase difference caused by spatial arrangement and should not include the motion-related additional term. If the target exhibits radial motion, the 12 complex phase values, in addition to the spatial term, include a linearly increasing motion-addition term, the slope of which is proportional to the radial displacement of the target within one frame period. This radial displacement is proportionally converted to the transmission timing of each transmission channel to obtain the motion-addition phase value corresponding to each transmission channel, and its negative value is used as the channel-by-channel phase compensation amount. After applying the corresponding channel-by-channel phase compensation amount to the complex beat frequency sample value of each virtual receiving channel, the array phase distortion introduced by motion is eliminated. In an optional implementation, if the vehicle speed range within the cross-section is known, a lookup table between radial velocity and compensation phase can be pre-established, and the channel-by-channel phase compensation amount can be directly obtained from the table based on the coarse velocity estimate obtained by the Doppler Fast Fourier Transform, reducing the amount of online computation.

[0028] After channel calibration, the field-programmable gate array (FPGA) enters the frequency domain processing pipeline. The first step involves performing a range-dimensional Fast Fourier Transform (FFT) on the calibrated beat frequency signal sampling sequence for each virtual receiving channel within each linear frequency modulation (LFM) cycle. The beat frequency signal acquired within each LFM cycle is a time-domain sequence, where the target echo at different distances is represented by sinusoidal components of different frequencies. After performing an FFT on this time-domain sequence, the frequencies of each sinusoidal component are separated into different range cells, each corresponding to a specific target radial distance. The number of points in the range-dimensional FFT determines the number of range cells, typically set to 256 or 512 points. The range-dimensional FFT results for each virtual receiving channel over all 12 LFM cycles are arranged row-wise to form a matrix, where the row index corresponds to the range cell and the column index corresponds to the LFM cycle number; this is the range-pulse matrix.

[0029] The second step involves performing a Doppler-dimensional Fast Fourier Transform (FFT) along the pulse dimension of the range-pulse matrix. The radial motion of the target causes a fixed phase increase in the echo signal between adjacent linear frequency modulation (LFM) cycles, which is proportional to the target's radial velocity. Performing the FFT along the pulse dimension transforms this phase increase into energy concentration on the Doppler frequency axis, with each Doppler element corresponding to a specific radial velocity value. After the transformation, a range-Doppler spectrum is obtained, where the amplitude value at each position reflects the presence of a target echo at a specific range and radial velocity. Since there are only 12 LFM cycles within a radar frame, the number of points for the Doppler-dimensional FFT is limited; typically, 16 or 32 points are chosen and filled with zeros to obtain a smoother velocity spectrum. The Doppler-dimensional velocity resolution is determined by the duration of the radar frame; a 40-millisecond frame period corresponds to a velocity resolution of approximately 0.05 meters per second.

[0030] The third step involves performing ordered statistical constant false alarm rate (CFAR) detection on the range-Doppler spectrum. The purpose of CFAR detection is to adaptively set the detection threshold amidst noise and clutter backgrounds, maintaining a constant false alarm probability. Ordered statistical CFAR detection sets a reference window around each cell to be detected. The sampled values ​​within the reference window are sorted in ascending order of amplitude, and the sorted value is then selected. Each sampled value is used as an estimate of the clutter power, where To ensure ordered statistical order, approximately three-quarters of the reference window length is typically chosen. The reason for selecting ordered statistics over the mean method is that in overload control scenarios, multiple vehicles may be closely arranged in the distance or Doppler dimension. Mean-based constant false alarm rate (CFAR) detection would include strong echoes from adjacent vehicles in the mean calculation of the reference window, raising the detection threshold and causing weak targets to be overwhelmed and missed. Ordered statistics, through sorting and truncation, excludes large-amplitude sampling of interfering targets within the reference window, ensuring that clutter power estimation is based solely on the noise floor. This results in significantly better detection performance than the mean method in multi-target scenarios. The threshold factor is set to... Its value is determined by the expected false alarm probability. The length of the reference window determines the time when Set as When the reference window length is 32 units, The typical value is approximately 4.5. If the amplitude value of the unit to be detected exceeds... Multiply by the first The product of these ordered statistics is used to identify candidate detection units that exceed the detection threshold; otherwise, they are identified as background noise. All distance-Doppler spectral units that exceed the detection threshold are extracted to form a candidate distance-Doppler unit set.

[0031] refer to Figure 1 , Figure 1The upper part of the diagram shows the cascaded arrangement of four 77GHz millimeter-wave radar chips. The four chips are arranged sequentially from left to right on the same RF printed circuit board, with the first chip designated as the reference chip, used as a reference for phase and amplitude during subsequent channel amplitude and phase calibration. Each chip contains three transmit channels and four receive channels, labeled Tx1 to Tx12 and Rx1 to Rx16 respectively. The four chips share the same local oscillator signal source and employ time-division multiplexing during operation. That is, the 12 transmit channels transmit linear frequency modulated continuous waves sequentially according to their numbering, with only one transmit channel transmitting at a time, while the 16 receive channels maintain synchronous reception. Figure 1 The central section showcases a virtual receive channel array formed by time-division multiplexing. Each transmit channel and each receive channel form an independent transmit / receive channel pair. The 12 transmit channels and 16 receive channels are paired to generate 192 virtual receive channels. These 192 virtual receive channels are arranged in a linear array with equal spacing between adjacent virtual receive channels. Equal to half the carrier wavelength, at a carrier frequency of 77 GHz Approximately 1.9 millimeters. From Figure 1 As can be seen, each chip contributes 48 virtual receiving channels, and the 48 channels of the four chips are arranged in sequence to form a complete 192-channel equivalent linear array. Figure 1 The middle section also uses bidirectional arrows to indicate that the equivalent aperture length corresponding to the 192 virtual receiving channels is approximately 365 mm. Figure 1 The lower half of the diagram presents two key performance indicators of the cascaded array in textual annotation form: an azimuth resolution better than 1 degree, and a lateral resolution distance of approximately 0.52 meters at a typical detection distance of 30 meters. This lateral resolution distance is much smaller than the standard lane width of 3.75 meters, indicating that the cascaded array has the ability to distinguish vehicles in different lateral positions within a single lane. Figure 1 The dashed arrows in the diagram illustrate the correspondence between the transceiver channels of each chip and the virtual receiver channels they contribute, visually demonstrating the physical process of virtual aperture expansion under the cascaded MIMO system.

[0032] Next, the azimuth angle is extracted for each candidate cell in the candidate range-Doppler cell set. For a given range cell index and Doppler cell index, 192 complex spectral values ​​at the corresponding positions are extracted from the range-Doppler spectra of the 192 virtual receiving channels. These 192 complex spectral values ​​reflect the amplitude and phase response of the echo from the same target on each channel of the equivalent linear array, where the phase variation with the spatial position of the channel contains the incident angle information of the target. The 192 complex spectral values ​​are arranged sequentially according to the spatial arrangement of the virtual receiving channels on the equivalent linear array to form a 192-dimensional complex response vector. An azimuth-dimensional Fast Fourier Transform is performed on this 192-dimensional complex response vector. The number of transformation points can be selected as 256 or 512 points to improve the refinement of the azimuth spectrum. After the transformation, the azimuth angle spectrum is obtained. The horizontal axis of the angle spectrum corresponds to the azimuth angle range from -90 degrees to +90 degrees, and the vertical axis corresponds to the echo energy at each angle. The true azimuth angle of the target corresponds to the energy peak position on the angle spectrum. The angle corresponding to the maximum peak value in the azimuth angle spectrum is extracted as the azimuth angle estimate. The distance value, radial velocity value and azimuth angle estimate of each candidate range-Doppler cell are combined to form a candidate detection point with angle attribute.

[0033] After all candidate detection points are formed, density-based spatial clustering is performed in the 3D parameter space consisting of radial distance, radial velocity, and azimuth angle. The core idea of ​​density-based spatial clustering is to group mutually adjacent points with sufficiently high density in the parameter space into the same cluster, while isolated low-density points are considered noise and eliminated. For overload control cross-section scenarios, a vehicle typically generates multiple scattering centers in the radar field of view. The metal parts of the front, roof, and rear of the vehicle reflect echoes, forming a group of adjacent but not completely overlapping candidate detection points in the range-Doppler-angle space. The clustering algorithm groups these candidate detection points belonging to the same vehicle into one cluster, and each cluster serves as a radar candidate target. The two key parameters required for clustering are the neighborhood radius and the minimum number of points. The neighborhood radius is set in three dimensions: 2 meters for distance (matching the typical longitudinal length of a vehicle); 1 meter per second for velocity (matching the velocity measurement deviation between different scattering centers of the same vehicle); and 2 degrees for angle (matching the azimuth deviation between different scattering centers of the same vehicle). A minimum of 3 points is set, meaning a cluster must contain at least 3 candidate detection points to be considered a valid target. For each cluster, the average radial distance of all candidate detection points within the cluster is taken as the radial distance of the radar candidate target, the average radial velocity as the radial velocity, and the average azimuth angle as the azimuth angle. Simultaneously, the cumulative sum of the radar cross sections of all candidate detection points within the cluster is taken as the radar cross section of the radar candidate target. The reason for using the cumulative value rather than the average for the radar cross section is that the total radar cross section of a vehicle is the sum of the contributions from all its scattering centers; the cumulative value more accurately reflects the overall electromagnetic scattering intensity of the vehicle and can be used to subsequently assist in determining the vehicle's size and type.

[0034] After clustering, azimuth angle spectrum multi-peak verification is performed on each radar candidate target. The purpose of multi-peak verification is to identify and separate composite targets. When two vehicles in adjacent lanes are at approximately the same radial distance and have similar radial velocities, they may fall into the same or adjacent spectral units on the range-Doppler spectrum, and are incorrectly grouped into one radar candidate target after clustering. However, these two vehicles must differ in azimuth angle because they are in different lanes with a lateral distance of at least 2 meters. Using the high angular resolution of 192 virtual receiving channels, the energy peaks of these two vehicles can be distinguished on the azimuth angle spectrum.

[0035] The specific verification process is as follows: Returning to the range and Doppler cells corresponding to the radar candidate target, the 192-dimensional complex response vector at the corresponding position is re-extracted from the 192 virtual receiving channels. An azimuth-dimensional Fast Fourier Transform is then performed on this vector to obtain the complete azimuth angle spectrum. All local maxima are searched within the complete azimuth angle spectrum. A local maximum is defined as the angular position where its amplitude is simultaneously greater than the amplitudes of both the left and right adjacent angular cells. Not all local maxima correspond to real targets; sidelobe effects can also produce local maxima. Therefore, a preset proportional threshold is set, marking only local maxima whose amplitude exceeds the maximum value of the complete azimuth angle spectrum multiplied by the preset threshold as valid angular spectrum peaks. The preset proportional threshold value is between 0.3 and 0.5. A value of 0.4 means that only when the amplitude of a local maximum reaches more than 40% of the peak amplitude is it considered a real target rather than a sidelobe. If there is only one valid angular spectrum peak, it indicates that the radar candidate target indeed corresponds to a single vehicle, and the cluster remains unchanged. If there are two effective angular spectral peaks, it indicates that the echoes of two vehicles are being merged, requiring separation. The separation method is as follows: the angle values ​​corresponding to the two effective angular spectral peaks are denoted as the first peak angle and the second peak angle, respectively, and their median is used as the angle boundary value. The physical meaning of the angle boundary value is the dividing line between the two vehicles on the azimuth angle spectrum; candidate detection points located on either side of this line belong to different vehicles. Candidate detection points within a cluster whose azimuth angle estimate is less than the angle boundary value are assigned to the first sub-cluster, and those with an azimuth angle estimate greater than or equal to the angle boundary value are assigned to the second sub-cluster. The mean radial distance, mean radial velocity, mean azimuth angle, and accumulated radar cross section value are recalculated for the candidate detection points in both the first and second sub-clusters, forming two independent radar target entries to replace the original entries.

[0036] In one optional implementation, if the number of effective angular peaks in the complete azimuth angle spectrum exceeds two, angle boundary values ​​can be set sequentially between adjacent effective angular peaks in ascending order of angle. This splits the cluster into sub-clusters equal in number of effective angular peaks, forming independent radar target entries. This scenario corresponds to an extreme congestion situation where three or more vehicles are exactly at the same distance and speed. Although the probability is low, it cannot be completely ruled out in actual deployment. In another optional implementation, azimuth dimension processing can use a super-resolution algorithm based on subspace decomposition instead of the fast Fourier transform to further improve the angle separation capability between two adjacent targets in scenarios with insufficient angular resolution. The trade-off is a significant increase in computational complexity.

[0037] refer to Figure 2 ,Include Figure 2 (a) and Figure 2 (b) Two sub-graphs. Figure 2Figure (a) shows the azimuth angle spectrum corresponding to a single vehicle target. The horizontal axis represents the azimuth angle in degrees, ranging from -30 degrees to +30 degrees. The vertical axis represents the normalized amplitude. A main peak and several side lobes can be observed in the azimuth angle spectrum. The main peak is located at approximately 3.5 degrees, with a significantly higher amplitude than the rest, corresponding to the true azimuth angle of a single vehicle. The side lobes are distributed at relatively far angles on either side of the main peak, with lower amplitudes. The position of the preset proportional threshold is marked with a horizontal dashed line in the figure. The preset proportional threshold is 0.4 times the maximum value of the complete azimuth angle spectrum. The main peak's amplitude exceeds the preset proportional threshold and is therefore marked as a valid azimuth peak; the side lobe amplitudes are all below the preset proportional threshold and are therefore not marked. Figure 2 In (a), the number of effective angular spectral peaks is 1, and the radar candidate target is determined to be a single vehicle target, and the cluster remains unchanged. Figure 2 Figure (b) shows the azimuth spectrum corresponding to the composite target. At the same range and Doppler cell locations, the azimuth spectrum exhibits two concentrated peaks, located at approximately -5.5 degrees and +6.0 degrees respectively, corresponding to the actual azimuth angles of two vehicles in adjacent lanes. Both peaks exceed a preset proportional threshold and are therefore marked as effective azimuth peaks. With two effective azimuth peaks, the radar candidate target is determined to be a composite target. The position of the angle boundary value is marked with a dashed line in the figure. The angle boundary value is the median of the first and second peak angles, approximately 0.25 degrees. The angle boundary value divides the azimuth spectrum into two regions, the left region corresponding to the first sub-cluster and the right region corresponding to the second sub-cluster. Figure 2 In (b), the first and second sub-clusters are distinguished by different filling textures. Candidate detection points with azimuth angle estimates less than the angle boundary value are assigned to the first sub-cluster, and candidate detection points with azimuth angle estimates greater than or equal to the angle boundary value are assigned to the second sub-cluster. The mean radial distance, mean radial velocity, mean azimuth angle, and radar cross section accumulation of each sub-cluster are calculated to form two independent radar target entries to replace the original single composite target entry. Figure 2 It intuitively demonstrates the complete judgment logic of azimuth angle spectrum multi-peak verification from single-peak retention to double-peak splitting.

[0038] All radar target entries that have undergone multi-peak verification of the azimuth angle spectrum are compiled to form a structured point trace table for the current radar frame. Each record in the structured point trace table contains four fields for a radar target: radial range, radial velocity, azimuth angle, and radar cross section, as well as a timestamp field for the current radar frame.

[0039] In parallel with radar signal processing, a field-programmable gate array (FPGA) performs distortion correction on the raw image frames output by the camera. Camera lenses exhibit radial and tangential distortion, especially when using a wide-angle lens to cover the entire lane; distortion in the image edge regions can reach tens of pixels. Without correction, this will cause significant deviations when subsequent radar targets are projected onto the image coordinate system. Distortion correction, based on a pre-calibrated camera intrinsic parameter matrix and distortion coefficients, reverse-maps the coordinates of each pixel in the raw image frame, removing the effects of radial and tangential distortion, and outputting the corrected image frame. The camera intrinsic parameter matrix includes focal length parameters. and These are the equivalent focal lengths (in pixels) in the horizontal and vertical directions, respectively, and the coordinates of the optical center. and These represent the horizontal and vertical offsets (in pixels) of the image coordinate system origin relative to the top-left corner of the image sensor. Distortion coefficients include radial distortion coefficients. , , and tangential distortion coefficient , These coefficients are obtained during the factory calibration stage by photographing a calibration board with a known geometric pattern and solving it using a nonlinear optimization method. Distortion correction can be implemented in a field-programmable gate array (FPGA) using a lookup table. That is, the coordinates of the source pixels in the original image frame corresponding to each pixel in the corrected image frame are pre-calculated and stored as a mapping lookup table. In actual operation, only table lookup and bilinear interpolation operations need to be performed, with constant computation and controllable latency.

[0040] After correction, the image frame and the structured point trace table are appended with a unified timestamp and written together into a shared buffer for subsequent radar-guided visual detection and fusion tracking steps. The shared buffer employs a dual-buffering mechanism: while the field-programmable gate array writes the current frame data to one buffer, the embedded graphics processing unit reads the previous frame data from the other buffer for processing, avoiding data inconsistency caused by read-write conflicts.

[0041] After the embedded graphics processing unit reads the structured point table and the corrected image frame from the shared buffer, it first transforms the spatial position of each radar target in the structured point table from the radar coordinate system to the image coordinate system, obtaining its projected pixel coordinates on the corrected image frame. This transformation process relies on the pre-calibrated radar-camera extrinsic and intrinsic parameter matrices. The radar-camera extrinsic parameter matrix describes the rigid body transformation relationship between the cascaded millimeter-wave radar array coordinate system and the camera coordinate system, and contains a 3x3 rotation matrix. and a 3x1 translation vector , where the rotation matrix This represents the projection relationship of the three axes of the radar coordinate system onto the camera coordinate system, and the translation vector. This represents the three-dimensional coordinate offset of the radar coordinate system origin in the camera coordinate system, expressed in meters. The calibration process is completed during system installation. Typically, several corner reflectors at known locations are placed on the road surface at the monitoring section. Simultaneously, radar echoes and camera images are acquired. Utilizing the correspondence between the three-dimensional coordinates of the corner reflectors in the radar coordinate system and their pixel coordinates in the image, a nonlinear optimization method minimizing reprojection error is used to jointly solve for the rotation matrix and translation vector.

[0042] For each radar target in the structured trace table, its radial distance is denoted as... The azimuth angle is denoted as Both use a cascaded millimeter-wave radar array as their origin. First, the polar coordinates are converted to three-dimensional Cartesian coordinates in the radar coordinate system, where the horizontal coordinates are... The vertical axis is The vertical coordinates are approximated to 0 based on the road surface height, meaning it's assumed the vehicle's reflection center is on the road surface. In reality, the main reflection center of large vehicles is often higher than the road surface. This approximation introduces a certain projection deviation, but since the camera's downward mounting angle is typically between 15 and 30 degrees, the vertical deviation has a limited impact on the image after perspective projection, generally not exceeding 10 pixels. The three-dimensional coordinate vector in the radar coordinate system is then multiplied by a rotation matrix on the left. Add translation vector This yields the 3D coordinates in the camera coordinate system. Subsequently, the 3D coordinates in the camera coordinate system are projected onto the image pixel coordinate system using the camera intrinsic parameter matrix. The horizontal focal length in the intrinsic parameter matrix... and vertical focal length Mapping angular relationships in three-dimensional space to pixel distances, optical center coordinates and Determine the position of the origin of the image coordinate system. The resulting two-dimensional pixel coordinates are the projected pixel coordinates, representing the expected location of the radar target in the corrected image frame.

[0043] After obtaining the projected pixel coordinates, a region of interest window is cropped from the corrected image frame using these coordinates as the center. The reason for not performing vehicle detection on the entire image but only on a local area near the radar projection position is twofold. First, the computational cost of feeding the entire 5-megapixel image into a deep learning detection network is far greater than the batch inference of several small windows. Given the real-time constraints of roadside edge computing devices, the frame rate for full-image detection is insufficient to meet the requirement of 25 frames per second, while local window detection can control the inference time per frame to within 15 milliseconds. Second, the background area within the local window is significantly reduced, excluding objects easily misidentified as vehicles, such as guardrails, streetlights, and signs, from the detection range, effectively reducing the false alarm rate.

[0044] The size of the region of interest (ROI) window is not a fixed value, but rather adaptively adjusts based on the radial distance of the radar target. Vehicles farther from the camera project smaller sizes onto the image, allowing for smaller windows; vehicles closer to the camera project larger sizes, requiring correspondingly larger windows. To achieve this adaptation, a distance-window size mapping table is pre-established and stored in the system. The mapping table is constructed as follows: during the calibration phase or initial system operation, standard-sized test vehicles are placed at different distances, and their projected pixel width and height on the image are measured. The window width and height at the corresponding distance are then taken as 1.5 times the projected size, ensuring the vehicle detection frame is completely contained within the window with sufficient margins. Typical mapping relationships are: at 30 meters, the window width is approximately 320 pixels and the window height is approximately 280 pixels; at 80 meters, the window width is approximately 140 pixels and the window height is approximately 120 pixels; and at 150 meters, the window width is approximately 80 pixels and the window height is approximately 70 pixels. During actual cropping, the corresponding window width and height are retrieved from the mapping table according to the radial distance of the radar target. A rectangular area is then cropped from the corrected image frame, centered on the projected pixel coordinates. If the window boundary exceeds the image boundary, the excess portion is truncated to the image boundary.

[0045] After all regions of interest (ROI) windows are cropped, they need to be uniformly scaled to the fixed input size required by the target detection network due to the different pixel sizes of each window. In this embodiment, the input size of the target detection network is 640 pixels by 640 pixels, and bilinear interpolation is used for scaling. The scaled windows are concatenated into a batch input tensor by batch dimension and fed into the target detection network for inference at once. The target detection network undergoes 8-bit integer quantization, which compresses the weights and activation values ​​originally stored as 32-bit floating-point numbers in the network into 8-bit integer representations. This increases the inference speed by about 3 times and reduces the memory usage by about 4 times without losing detection accuracy, enabling real-time performance on roadside embedded graphics processing units. The network is trained on vehicle targets, and the output categories include small passenger cars, large passenger cars, light trucks, heavy trucks, and special vehicles. The inference result within each ROI window is zero or more vehicle detection boxes. Each vehicle detection box includes the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the width, and the height of the box. These four values ​​are all relative to the local coordinate system of the ROI window. In one alternative implementation, the target detection network may also employ 16-bit half-precision floating-point quantization or mixed-precision quantization to achieve different balances between accuracy and speed.

[0046] To use a unified coordinate reference across the entire image in subsequent pairing and fusion tracking, the local coordinates of the vehicle detection boxes need to be transformed to the global coordinate system. The vehicle detection box coordinates are taken from the midpoint of the bottom edge of the vehicle detection box in the global coordinate system, rather than the box center. This choice is based on geometric considerations: the midpoint of the bottom edge of the vehicle detection box approximately corresponds to the contact area between the vehicle tire and the road surface. This point lies on the road surface plane, and when subsequently back-projecting from the image coordinates to the road surface coordinate system, the road surface plane constraint can be directly used for solving, resulting in smaller errors and independence from vehicle type. In contrast, the box center corresponds to a spatial location in the middle of the vehicle body, and its height varies significantly with vehicle type. The box center height of a small sedan is approximately 0.7 meters, while that of a heavy truck is approximately 1.8 meters. Back-projecting the box center onto the road surface would introduce a systematic bias proportional to the vehicle height.

[0047] The specific calculation method for the coordinates of the bottom edge midpoint across the entire image is as follows: the bottom edge ordinate is determined by the sum of the ordinate of the top-left corner of the frame and the frame height, and the bottom edge midpoint ordinate is determined by the sum of the x-coordinate of the top-left corner of the frame and half the frame width. Then, the x-coordinates and ordinates of the bottom edge midpoint are respectively added to the offset of the top-left corner coordinate of the region of interest window in the entire image to obtain the total coordinates of the bottom edge midpoint across the entire image.

[0048] After calculating the projected pixel coordinates of radar targets and extracting the vehicle detection box coordinates, the frame-by-frame pairing and association process begins. The task of pairing and association is to determine which vehicle detection box corresponds to each radar target and the same physical vehicle. A cost matrix is ​​constructed using the Euclidean distance between the projected pixel coordinates of each radar target and the x-coordinate and y-coordinate of the midpoint of the bottom edge of each vehicle detection box on the full image. The number of rows in the cost matrix equals the number of radar targets in the structured point table of the current frame, and the number of columns equals the total number of vehicle detection boxes detected within all regions of interest windows in the current frame. The matrix... Line number Column elements equal to the The projected pixel coordinates of the first radar target and the second target The Euclidean distance between the midpoints of the bottom edges of each vehicle detection frame and the coordinates of the entire map, where The unit is pixels. After the cost matrix is ​​constructed, the Hungarian algorithm is used to solve for the optimal allocation, that is, to find the pairing scheme that minimizes the sum of the selected elements in the cost matrix among all possible one-to-one pairing schemes. The time complexity of the Hungarian algorithm is O(n log n). ,in The value is the larger of the number of radar targets and the number of vehicle detection frames. In a typical overload control section scenario, the number of vehicles present at the same time is usually no more than 20, and the calculation time is negligible.

[0049] In the pairing results, if the Euclidean distance of a pair exceeds a preset association threshold (set to 100 pixels in this embodiment), the pairing is deemed invalid and discarded. Unpaired radar targets may correspond to vehicles that have just entered the monitoring range and have not yet been visually detected, and unpaired vehicle detection boxes may correspond to targets with extremely small radar cross-sections that have not been detected by radar. These unpaired targets can be re-associated in subsequent frames through track initiation and track continuation mechanisms. In an optional implementation, the cost matrix can also incorporate additional information items such as appearance feature similarity or speed consistency, using the weighted sum of Euclidean distance and appearance feature distance as the cost to improve the association accuracy in dense traffic scenarios.

[0050] For each successfully paired target, an extended Kalman filter is used to perform fusion state estimation in a road surface rectangular coordinate system. The road surface rectangular coordinate system has a fixed reference point on the road surface as its origin, with the horizontal axis parallel to the lane markings (i.e., perpendicular to the driving direction) and the vertical axis parallel to the driving direction, forming an orthogonal coordinate system within the road surface horizontal plane. The state variable of the extended Kalman filter is a 4-dimensional vector, containing the road surface lateral coordinates, road surface longitudinal coordinates, lateral velocity, and longitudinal velocity, respectively.

[0051] In the prediction phase, the state variables are extrapolated based on the motion model before each frame arrives. This embodiment uses a uniform linear motion model, assuming the vehicle travels at a constant speed along a straight line between adjacent frames. Let the inter-frame time interval be... The unit is seconds, for a system of 25 frames per second. The time is 0.04 seconds. The predicted state is calculated by adding the lateral coordinates of the road surface from the previous frame to the lateral velocity and multiplying by... The longitudinal coordinate of the road surface plus the longitudinal velocity multiplied by It was found that the lateral and longitudinal velocities remain constant under the uniform velocity model. The prediction covariance matrix is ​​obtained by propagating the covariance matrix of the previous frame through the state transition matrix and superimposing it with the process noise covariance matrix. The process noise covariance matrix reflects the degree of deviation between the actual vehicle motion and the uniform velocity assumption. For the straight road segment scenario of the overload control section, the standard deviation of the longitudinal process noise is set to 0.5 m / s², and the standard deviation of the lateral process noise is set to 0.2 m / s². The lateral value is smaller because the lateral movement amplitude of a normally traveling vehicle is much smaller than the longitudinal velocity change amplitude.

[0052] During the update phase, both radar and visual observations are simultaneously fed into an extended Kalman filter for measurement updates. Radar observations are derived from successfully paired radar target entries in a structured point table, comprising three components: radial range, azimuth angle, and radial velocity. These three components are observations in polar coordinates, exhibiting a nonlinear relationship with the Cartesian coordinate system of the state variables. This is precisely why an extended Kalman filter is used instead of a standard Kalman filter. The extended Kalman filter performs a first-order Taylor expansion of the nonlinear observation equations at the predicted state variables, yielding a linearized Jacobian matrix. ,in The matrix is ​​3x4, with each row corresponding to the partial derivative of one radar observation component with respect to the four state variables. The radar observation noise covariance matrix is ​​a 3x3 diagonal matrix, with the diagonal elements representing the variances of radial range observation noise, azimuth angle observation noise, and radial velocity observation noise, respectively. Under typical parameters of a 77 GHz millimeter-wave radar, the standard deviation of radial range observation noise is approximately 0.1 meters, the standard deviation of azimuth angle observation noise is approximately 0.5 degrees (approximately 0.0087 radians), and the standard deviation of radial velocity observation noise is approximately 0.05 meters per second.

[0053] The visual observations are derived from the horizontal and vertical coordinates of the midpoint of the bottom edge of the successfully paired vehicle detection boxes. These two values ​​are image pixel coordinates and need to be back-projected to the road surface Cartesian coordinate system for direct comparison with the state variables. The back-projection process utilizes the geometric constraint that the road surface is a known horizontal plane: given the camera intrinsic matrix, radar-camera extrinsic matrix, and the road surface height in the camera coordinate system, the two-dimensional pixel coordinates on the image are back-projected into three-dimensional coordinates on the road surface plane. The vertical coordinates are determined by the road surface height constraint, and the two horizontal coordinates are the horizontal and vertical coordinates in the road surface Cartesian coordinate system. The visual road surface coordinates obtained by back-projection are the visual observations, which contain two components. The visual observation equation is also nonlinear, and its Jacobian matrix... It is a 2x4 matrix. The visual observation noise covariance matrix is ​​a 2x2 diagonal matrix, with the diagonal elements being the observation noise variances of the visual road surface's lateral and longitudinal coordinates, respectively. Their values ​​depend on the positioning accuracy of the vehicle detection frame and the geometric amplification effect of back projection. At a detection distance of 30 meters, the standard deviations of the lateral and longitudinal observation noise are approximately 0.3 meters, and at 100 meters, they are approximately 0.8 meters.

[0054] When both radar and visual observations are fed into an extended Kalman filter for measurement updates, a sequential update approach can be used. This involves first updating the state variables and covariance matrix using radar observations, and then updating them again using visual observations. The mathematical result of these two sequential updates is equivalent to concatenating the five observation components into a single joint observation vector and updating it all at once. However, sequential updates offer better numerical stability and are simpler to implement. The fused state variables output after the update represent the optimal position and velocity estimate of the vehicle in the road surface Cartesian coordinate system for the current frame.

[0055] The fused state variables output in each frame are concatenated according to the target identifier to form a continuous fused trajectory. The target identifier is assigned a globally unique integer number when the target first appears, and this number is maintained through pairing and association in subsequent frames. The continuous fused trajectory records the road surface lateral coordinates, road surface longitudinal coordinates, lateral velocity, and longitudinal velocity for each frame throughout the entire process of a vehicle entering and leaving the monitoring section, forming a time series. In an optional implementation, the extended Kalman filter can be replaced with an unscented Kalman filter. The latter uses deterministic sampling points instead of the Jacobian matrix to approximate the statistical properties of the nonlinear transformation, resulting in higher estimation accuracy in highly nonlinear scenarios, but its computational cost is approximately 2 to 3 times that of the extended Kalman filter.

[0056] After continuous trajectory fusion is generated, the process proceeds to lane attribution determination and over-limit identification. First, lane markings in the corrected image frame are detected and modeled. Lane marking detection can employ semantic segmentation-based methods or traditional methods based on edge detection and Hough transform, extracting the set of pixels belonging to the lane markings from the corrected image frame to obtain the lane marking pixels. In this embodiment, inverse perspective mapping is performed on the detected lane marking pixels, transforming them from the image coordinate system to the road surface rectangular coordinate system. The inverse perspective mapping is based on the constraint that the road surface is a known horizontal plane, using the same geometric relationship as the aforementioned back-projection process of visual observations. The image pixel coordinates are inversely calculated into road surface coordinates point by point using the camera intrinsic matrix and the radar-camera extrinsic matrix. After transformation, lane marking points are obtained in the road surface coordinate system, each point having two components: a road surface lateral coordinate and a road surface longitudinal coordinate.

[0057] Polynomial fitting was performed on the scattered points of each lane marking. Using the longitudinal coordinates of the road surface as the independent variable and the lateral coordinates as the dependent variable, a cubic polynomial least squares fitting was performed to obtain a cubic polynomial lane line model. The form of the cubic polynomial is... ,in Let be the lateral coordinates of the road surface. For the longitudinal coordinates of the road surface, , , , These are the fitting coefficients. A cubic polynomial was chosen instead of a linear or quadratic polynomial because actual roads may have slight curves or gradual changes in alignment before and after the monitoring section. A cubic polynomial can fit straight segments, circular segments, and transition curves, and has sufficient expressive power for most highway alignments. The goal of least-squares fitting is to minimize the sum of the squares of the lateral distances from all lane marking points to the fitted curve. The analytical solution for the four fitting coefficients can be obtained by solving the system of equations. If the road segment at the monitoring section is a standard straight segment, the fitted coefficients will be... and When the value is close to 0, the model naturally degenerates into a straight line.

[0058] After fitting, lane centerlines are generated based on two adjacent cubic polynomial lane line models. Specifically, within the effective range of the road surface's longitudinal coordinates, several longitudinal coordinate values ​​are selected at 0.5-meter intervals. At each longitudinal coordinate value, the lateral coordinates of the road surface are calculated using the adjacent two cubic polynomial lane line models. The arithmetic mean of these two values ​​is taken as the lateral coordinate of the lane centerline at that longitudinal coordinate. Connecting all sampled points yields one lane centerline. This process is repeated for all lanes within the cross-section, generating a set of lane centerlines equal to the number of lanes. Each lane centerline is identified by a lane number, with the numbers increasing sequentially from the leftmost lane on the road surface.

[0059] In one alternative implementation, if the road markings at the monitoring section are clear and are straight segments, the calculation can be simplified to a first-order polynomial or even directly represented by fixed lateral coordinate values, reducing the computational load. In another alternative implementation, when road markings are difficult to detect stably due to wear or snow accumulation, the median of multiple frames of lane marking detection results can be used for time filtering, or the geometric parameters of the road markings can be pre-stored as prior knowledge, with fine-tuning only required during runtime.

[0060] Subsequently, each trajectory point in the continuously fused trajectory is matched with the lane centerline to determine lane assignment. For each trajectory point in the continuously fused trajectory, its lateral and longitudinal road coordinates are obtained. At the longitudinal road coordinate of the trajectory point, the lateral road coordinate value corresponding to each lane centerline is calculated by substituting it into the cubic polynomial lane line model of each lane centerline. The absolute value of the difference between the lateral road coordinate of the trajectory point and the lateral road coordinate of each lane centerline is taken to obtain the lateral distance from the trajectory point to each lane centerline. The lane number corresponding to the lane centerline with the smallest lateral distance is assigned to the trajectory point, indicating which lane's center position the vehicle is closest to at that moment. The above operation is performed point by point for all trajectory points in the continuously fused trajectory to obtain a point-by-point lane assignment sequence, where each element in the sequence is a lane number.

[0061] Because the fusion state variables of a single frame exhibit random fluctuations, especially when a vehicle is traveling near lane lines, trajectory points from adjacent frames may be alternately assigned to two adjacent lanes, resulting in unstable jitter. To eliminate this jitter, majority voting is performed on the point-to-point lane assignment sequence within a fixed-length time window. The time window length is set to 15 to 25 consecutive frames, corresponding to a time span of approximately 0.6 to 1.0 seconds for a system with 25 frames per second. Within the time window, the frequency of each lane number is counted, and the lane number with the highest frequency is taken as the stable lane assignment result. The effect of majority voting is to smooth out brief random jumps, retaining only the lane assignment results that occupy a majority of frames within the time window. If a vehicle is indeed performing a lane change, its trajectory points will gradually transition from the centerline of the original lane to the centerline of the target lane during the lane change process, and the majority vote result will naturally switch to the new lane number after the lane change is completed, without missing any actual lane changes.

[0062] After determining lane ownership, the external dimensions of each vehicle are estimated to determine whether it exceeds the limits. The vehicle's external dimensions include three items: the estimated actual lateral width, the estimated actual height, and the estimated actual length.

[0063] The estimation of actual lateral width and actual height is based on the ratio between the actual size of the object and the projected size of the image in the pinhole camera imaging model. The vehicle detection box in the frame corresponding to the trajectory point closest to the camera in the continuous fusion trajectory is selected. The reason for choosing the closest frame is that the projected size of the vehicle on the image is the largest at this time, and the pixel width and pixel height of the detection box have the highest measurement resolution, resulting in the highest estimation accuracy. The longitudinal coordinate of the road surface in the fusion state variables of the trajectory point closest to the camera is used as the actual longitudinal distance, denoted as . The unit is meters, and this value represents the distance between the vehicle and the camera mounting position in the longitudinal direction of the road surface. Combined with the focal length parameter in the camera intrinsic parameter matrix, the width of the vehicle detection box is determined. (Unit: pixels) Converted to estimated actual horizontal width ,in This is the equivalent focal length in the horizontal direction, expressed in pixels. Similarly, the height of the vehicle detection bounding box... Converted to actual height estimate ,in The equivalent focal length in the vertical direction is used. The geometric basis for this conversion is: in the pinhole model, the image projection size of an object is equal to its true size multiplied by the focal length and then divided by the depth distance. Transforming the equation, we get the true size as equal to the image projection size multiplied by the depth distance and then divided by the focal length. Here, the longitudinal coordinates of the road surface are approximated as the depth distance. Strictly speaking, the two are not completely equal when the camera has a pitch angle. However, since the pitch angle is usually no more than 30 degrees and the cosine correction factor is no less than 0.87, the relative error introduced by the estimation of width and height is within 13%, which is acceptable in the coarse screening stage of exceeding the limit judgment. In an optional implementation, the depth distance can be accurately calculated using the camera's pitch angle instead of directly taking the longitudinal coordinates of the road surface, further improving the estimation accuracy.

[0064] refer to Figure 3 , Figure 3Using the corrected image frame as the background coordinate system, the horizontal axis represents the image's horizontal coordinate in pixels, and the vertical axis represents the image's vertical coordinate in pixels. Five perspective convergence lines, representing five lane markings, are drawn from top to bottom in the image. These lane markings are wider at the bottom of the image and gradually narrow towards the vanishing point at the top, reflecting the geometric relationship of the road surface under the camera's perspective projection. Two adjacent lane markings form one lane, labeled from left to right as lane 1 to lane 4. The image shows the projection positions and detection results of four radar targets, corresponding to four vehicles at different distances. Each radar target is first projected onto the image coordinate system via the radar-camera extrinsic and intrinsic parameter matrices to obtain projected pixel coordinates, indicated by crosshairs in the image. Centered on the projected pixel coordinates, the window width and height are retrieved from the distance-window size mapping table based on the radial distance of the radar target, and then cropped to obtain the region of interest (ROI) window, represented by a dashed rectangle in the image. As can be seen from the figure, the region of interest (ROI) window size is larger for closer targets (e.g., target C, radial distance 35 meters), and smaller for farther targets (e.g., target D, radial distance 120 meters). This adaptive adjustment ensures that vehicle projections at different distances are completely contained within the window. After performing object detection network inference within each ROI window, vehicle detection boxes are obtained, represented by solid-line rectangles in the figure. The midpoint of the bottom edge of each vehicle detection box is marked with a square. Adding the coordinates of this midpoint to the coordinates of the top-left corner of the ROI window in the full image yields the horizontal and vertical coordinates of the midpoint in the full image. The figure connects the projected pixel coordinates with the corresponding midpoint coordinates in the full image using dotted lines; the Euclidean distance between these two coordinates represents the value of the corresponding element in the cost matrix. Figure 3 The legend area in the upper left corner provides annotations for the cross markers, square markers, dashed rectangles, and solid rectangles.

[0065] The estimation of the actual length utilizes the longitudinal span of the continuous fusion trajectory. When a vehicle enters a monitoring section and completely passes through it, its front end appears first within the detection range of the radar and camera, and its rear end leaves the detection range last. The absolute value of the difference between the longitudinal coordinates of the trajectory point where the vehicle first enters the monitoring section and the longitudinal coordinates of the trajectory point where it last leaves the monitoring section is taken along the continuous fusion trajectory. This value reflects the longitudinal distance the vehicle traverses from its entry to its exit. However, this longitudinal span is not equal to the vehicle length; it also includes the displacement of the vehicle during this time. Therefore, a longitudinal projection geometric correction for the vehicle body needs to be subtracted. The longitudinal projection geometric correction for the vehicle body is calculated as follows: the longitudinal velocity in the fusion state variables of each frame during the time interval from the first entry to the last exit is taken, multiplied by the inter-frame time interval, and accumulated frame by frame to obtain the total longitudinal displacement of the vehicle during this time. The absolute value of the difference between the longitudinal coordinates of the first and last trajectory points is subtracted from the total longitudinal displacement; the remaining difference is the projected length of the vehicle body in the longitudinal direction, i.e., the estimated actual length. This estimation method essentially uses continuous tracking trajectory to separate the vehicle's motion components, thereby extracting static vehicle length information.

[0066] In one alternative implementation, if the monitoring section is equipped with two sets of laser light curtains or geomagnetic coils with known intervals, the vehicle length can also be obtained directly by multiplying the vehicle speed by the time difference between the two sets of sensors being blocked, thus verifying the estimation method based on the fusion trajectory described above.

[0067] After obtaining the three estimated external dimensions, the actual estimated lateral width, actual height, and actual length are compared with preset vehicle width, height, and length limits, respectively. These preset limits are set according to national standards and local regulations governing overloading; for example, the vehicle width limit is 2.55 meters, the height limit is 4.0 meters, and the length limit is 18.1 meters. If any of the three estimated values ​​exceeds the corresponding limit, the vehicle is marked as overloaded. In practical applications, considering the measurement uncertainty in external dimension estimation, a tolerance margin can be set. For example, an overload flag can be triggered only when the estimated value exceeds the limit plus 0.1 meters, thus reducing the misjudgment rate caused by measurement errors.

[0068] Finally, the stable lane assignment result, longitudinal velocity, continuous fusion trajectory, and estimated actual lateral width, actual height, and actual length of the vehicle marked as overweight are combined to output the lane-level identification result for overweight vehicles. The output information can be transmitted to the overweight vehicle management platform for recording, archiving, and law enforcement evidence collection. The stable lane assignment result specifies the exact lane where the overweight vehicle is located, the longitudinal velocity provides the vehicle's speed when passing through the monitoring section, the continuous fusion trajectory provides the vehicle's complete travel path, and the three estimated external dimensions serve as the basis for determining overweight status.

[0069] The present invention has been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make various improvements and modifications to the present invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. A lane-level identification method for overloaded vehicles based on millimeter-wave radar and vision fusion, characterized in that, Includes the following steps: Step 1: Deploy a cascaded millimeter-wave radar array and camera at the overload monitoring section, and synchronously trigger to align the radar frame with the image frame; Channel calibration, range-dimensional and Doppler-dimensional frequency domain transformation and constant false alarm rate detection are performed on the radar intermediate frequency signal. The azimuth angle is extracted from the candidate detection unit to form candidate detection points. The radar candidate targets are obtained by spatial clustering. The azimuth angle spectrum multi-peak verification is performed on the radar candidate targets and the radar candidate targets containing multiple effective angle spectrum peaks are split into independent radar target entries and summarized into a structured point trace table. Perform distortion correction on the image frame to obtain the corrected image frame; Step 2: Project each radar target in the structured point table onto the image coordinate system using calibration parameters to obtain projected pixel coordinates. Crop the region of interest window with the projected pixel coordinates as the center and pass it through the target detection network to obtain the vehicle detection box. Construct a cost matrix based on the projected pixel coordinates and the vehicle detection box coordinates to solve for the optimal allocation and complete the frame-by-frame pairing association. Fuse radar observations and visual observations to output fused state variables and concatenate them into a continuous fused trajectory. Step 3: After mapping the lane markings in the corrected image frame to the road surface coordinate system through inverse perspective, multinomial fitting is used to generate the lane centerline. The continuous fusion trajectory is compared with the lateral distance of each lane centerline point by point, and a stable lane assignment result is obtained through voting. The vehicle outline size is estimated based on the vehicle detection box and fusion state quantity and compared with the limit value to determine the over-limit mark. The lane-level recognition result of the over-limit vehicle is output.

2. The method according to claim 1, characterized in that, The cascaded millimeter-wave radar array is composed of four 77GHz millimeter-wave radar chips cascaded together. Each chip has three transmit channels and four receive channels. After the four chips are cascaded together, a total of 12 transmit channels and 16 receive channels are formed. The 12 transmit channels transmit linear frequency modulated continuous waves in a time-division multiplexing manner, while the 16 receive channels receive them synchronously, thus forming 192 virtual receive channels. The camera is a global shutter industrial camera. Synchronous triggering includes the simultaneous output of a linear frequency modulation start signal and an exposure trigger pulse by a field-programmable gate array at the beginning of each radar frame cycle, so that the center time of the intermediate frequency signal acquisition of each radar frame is aligned with the center time of the exposure of the corresponding image frame.

3. The method according to claim 2, characterized in that, Channel calibration includes cascaded chip-to-channel amplitude and phase calibration and time-division multiplexing (TDM) motion phase compensation. The specific method for cascaded chip-to-channel amplitude and phase calibration is as follows: using the first of the four millimeter-wave radar chips as a reference chip, the complex beat frequency sampling values ​​of the virtual receiving channels belonging to the second, third, and fourth chips are multiplied by the amplitude correction coefficient and phase correction coefficient, which are pre-determined offline and stored in the on-chip memory, to align the amplitude response and phase baseline of all 192 virtual receiving channels with the reference chip. The specific method for time-division multiplexing motion phase compensation is as follows: using the phase progression relationship in the same range unit after each transmitting channel has completed a range-dimensional fast Fourier transform, the radial displacement of the target within one frame period is estimated. The radial displacement is proportionally converted into a channel-by-channel phase compensation amount according to the transmission timing of each transmitting channel, and the corresponding channel-by-channel phase compensation amount is applied to the complex beat frequency sampling value of each virtual receiving channel.

4. The method according to claim 2, characterized in that, The specific methods for range-dimensional and Doppler-dimensional frequency domain transformation and constant false alarm rate (CFAR) detection are as follows: For each virtual receiving channel, a range-dimensional fast Fourier transform is performed on the calibrated beat frequency signal sampling sequence within each linear frequency modulation cycle to obtain a range-pulse matrix. A Doppler-dimensional fast Fourier transform is then performed on the range-pulse matrix along the pulse dimension to obtain a range-Doppler spectrum. Ordered statistical CFAR detection is then performed on the range-Doppler spectrum, and spectral units exceeding the detection threshold are extracted as a candidate range-Doppler unit set. The specific method for extracting the azimuth angle is as follows: For each candidate unit in the candidate range-Doppler unit set, 192 complex spectral values ​​at the corresponding position are extracted from the range-Doppler spectrum of the 192 virtual receiving channels. These values ​​are then arranged in the spatial order of the virtual receiving channels on the equivalent linear array to form a 192-dimensional complex response vector. An azimuth-dimensional fast Fourier transform is then performed on the 192-dimensional complex response vector to obtain the azimuth angle spectrum. The angle corresponding to the maximum amplitude peak in the azimuth angle spectrum is extracted as the estimated azimuth angle value.

5. The method according to claim 4, characterized in that, The specific method of spatial clustering is as follows: all candidate detection points with angle attributes are subjected to density-based spatial clustering in a 3-dimensional parameter space composed of radial distance, radial velocity and azimuth angle. Each cluster is regarded as a radar candidate target. The mean radial distance, mean radial velocity, mean azimuth angle and the cumulative value of radar cross section of all candidate detection points in the cluster are taken as the radial distance, radial velocity, azimuth angle and radar cross section of the radar candidate target, respectively.

6. The method according to claim 5, characterized in that, The specific method for multi-peak verification of azimuth angle spectrum is as follows: Return to the range cell and Doppler cell corresponding to the radar candidate target, extract the 192-dimensional complex response vector at the corresponding position from the 192 virtual receiving channels and perform a fast Fourier transform in the azimuth dimension to obtain the complete azimuth angle spectrum. Search for all local maxima points in the complete azimuth angle spectrum, and mark the local maxima points whose amplitude is higher than the maximum value of the complete azimuth angle spectrum as effective azimuth angle peaks. If there is only one effective azimuth angle peak, the cluster remains unchanged. If there are two effective azimuth angle peaks, calculate the median of the angle values ​​corresponding to the two effective azimuth angle peaks as the angle boundary value. Candidate detection points with azimuth angle estimates less than the angle boundary value are assigned to the first sub-cluster, and candidate detection points with azimuth angle estimates greater than or equal to the angle boundary value are assigned to the second sub-cluster. Take the radial distance mean, radial velocity mean, azimuth angle mean, and radar cross section accumulation of the candidate detection points in the first and second sub-clusters respectively to form two independent radar target entries to replace the original entries.

7. The method according to claim 1, characterized in that, The cropping method for the region of interest (ROI) window is as follows: Based on the radial distance of the radar target, the corresponding window width and height are retrieved from a pre-stored distance-window size mapping table. A rectangular region is then cropped on the corrected image frame, centered on the projected pixel coordinates. All ROI windows are uniformly scaled to the input size required by the target detection network and then concatenated into a batch input tensor, which is then fed into the target detection network after 8-bit integer quantization. The vehicle detection box coordinates are obtained by taking the coordinates of the midpoint of the bottom edge of the vehicle detection box in the full-image coordinate system. Specifically, the bottom edge ordinate is determined by the sum of the ordinate of the top-left corner of the vehicle detection box and the box height; the midpoint of the bottom edge is determined by the sum of the x-coordinate of the top-left corner of the box and half the box width; and the offset of the top-left corner coordinate of the ROI window in the full image is added to obtain the full-image x-coordinate and ordinate of the midpoint of the bottom edge.

8. The method according to claim 7, characterized in that, The optimal allocation is solved using the Hungarian algorithm. The fusion of radar and visual observations is achieved using an extended Kalman filter. The state variables of the extended Kalman filter include the road surface lateral coordinates, road surface longitudinal coordinates, lateral velocity, and longitudinal velocity. In the prediction stage, a uniform linear motion model is used to extrapolate the state variables of the previous frame based on the inter-frame time interval to obtain the predicted state variables and the predicted covariance matrix. In the update stage, the radial distance, azimuth angle, and radial velocity are used as radar observations. The visual road surface coordinates obtained by back-projecting the horizontal coordinates of the bottom midpoint of the full map and the vertical coordinates of the bottom midpoint of the full map to the road surface rectangular coordinate system under the constraint that the road surface is a known horizontal plane are used as visual observations. The radar and visual observations are then fed into the extended Kalman filter to perform measurement updates.

9. The method according to claim 1, characterized in that, The specific method of polynomial fitting is as follows: extract lane marking pixels from the corrected image frame, transform the lane marking pixels from the image coordinate system to the road surface rectangular coordinate system through inverse perspective mapping to obtain lane marking scatter points, and perform cubic polynomial least squares fitting on the lane marking scatter points of each lane marking with the road surface longitudinal coordinate as the independent variable and the road surface transverse coordinate as the dependent variable to obtain a cubic polynomial lane line model; the lane center line is generated by the arithmetic mean of the road surface transverse coordinates at the same road surface longitudinal coordinate of two adjacent cubic polynomial lane line models; the specific method of voting is as follows: perform majority voting on the point-by-point lane assignment sequence within a fixed-length time window, and take the lane number that appears most frequently as the stable lane assignment result.

10. The method according to claim 1, characterized in that, The method for estimating the vehicle's external dimensions is as follows: Take the vehicle detection box of the frame corresponding to the trajectory point closest to the camera in the continuous fusion trajectory, use the road longitudinal coordinate in the fusion state quantity of the trajectory point closest to the camera as the actual longitudinal distance, and combine the focal length parameter in the camera intrinsic parameter matrix to convert the width and height of the vehicle detection box into estimated values ​​of the actual lateral width and actual height, respectively. The absolute value of the difference between the longitudinal coordinate of the trajectory point where the vehicle first enters the monitoring section and the longitudinal coordinate of the trajectory point where the vehicle last leaves the monitoring section is taken along the continuous fusion trajectory. The actual length estimate is obtained by subtracting the geometric correction amount of the vehicle's longitudinal projection. The actual lateral width estimate, actual height estimate, and actual length estimate are compared with the preset vehicle width limit, vehicle height limit, and vehicle length limit, respectively. If any estimate exceeds the corresponding limit, the vehicle is marked as an over-limit vehicle.

Images