Traffic flow parameter real-time sensing method based on air-ground cooperative sensing

Abstract

The invention discloses a traffic flow parameter real-time sensing method based on air-ground cooperative sensing, which can provide an accurate geometric alignment basis for fusion of air-based hovering video data and ground-based radar data by analyzing the tracking life cycle of a vehicle target to adaptively divide sensing areas and optimize calibration parameters. On the basis, through cross-modal feature matching and data association, the wide coverage advantage of an air-based video and the accurate distance measurement advantage of a ground-based radar are effectively converged, so that large-range vehicle accurate detection and tracking are realized, and the detection accuracy is improved. Furthermore, various key traffic flow parameters including queuing length, section traffic flow, interval vehicle number, vehicle existence and time headway are synchronously calculated and output.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation and computer vision technology, and in particular, it is a method for real-time perception of traffic flow parameters based on air-ground cooperative perception. Background Technology

[0002] With the continuous increase in urban traffic flow, real-time perception of traffic flow parameters at intersections has become an important requirement for intelligent traffic management. Accurate real-time traffic flow parameter information is of great significance for applications such as signal timing optimization, traffic congestion early warning, and route planning.

[0003] Currently, mainstream real-time traffic flow parameter perception solutions have the following limitations: pure video detection experiences a sharp decline in performance under adverse conditions such as nighttime, rain, snow, and fog, and has a limited detection range, making it difficult to cover a wide range of vehicles; while pure radar detection is accurate in ranging and unaffected by weather, traditional millimeter-wave radar has a limited detection range and cannot provide semantic information such as vehicle type.

[0004] The emergence of 4D millimeter-wave radar has improved detection range, but its standalone use still cannot meet the needs of large-scale intersection queue detection. Integrated air-ground perception offers a new technological approach to solving this problem: airborne UAV platforms, hovering at low altitudes, have the advantage of a wide field of view and a large detection range; ground-based roadside 4D millimeter-wave radar provides stable and reliable three-dimensional position and velocity information over relatively long distances. By working together, the distance limitations of a single sensor can be overcome, enabling accurate vehicle detection at large intersections. Furthermore, it can simultaneously calculate and output various key traffic flow parameters, including queue length, cross-sectional traffic flow, number of vehicles in a section, vehicle presence, and headway. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the prior art by providing a real-time traffic flow parameter perception method based on air-ground cooperative perception. This method adaptively divides the perception area and optimizes calibration parameters by analyzing the tracking lifecycle of vehicle targets, providing a precise geometric alignment basis for air-ground sensor data fusion. Furthermore, through cross-modal feature matching and data association, it effectively combines the wide coverage advantage of air-based video with the precise ranging advantage of ground-based radar, breaking through the detection distance limitations of a single sensor and achieving accurate vehicle detection at large intersections. Simultaneously, it calculates and outputs multiple key traffic flow parameters, including queue length, cross-sectional traffic flow, number of vehicles per section, vehicle presence, and headway.

[0006] The technical solution to achieve the purpose of this invention is: a real-time traffic flow parameter sensing method based on air-ground cooperative sensing, the method comprising the following steps:

[0007] Step 1, Multi-region division based on the fragmentation of radar-based data: Based on the continuity of tracking of vehicle targets in the same area by airborne video and ground-based radar, the reliability of sensor detection at different spatial locations is quantitatively evaluated, and the perception area is divided into multiple sub-regions of different quality levels accordingly.

[0008] Step 2, Multi-region adaptive calibration for vehicle detection: For each sub-region divided in Step 1, a unique coordinate mapping matrix is ​​initialized and iteratively optimized to accurately map the vehicle point cloud of the ground-based radar to the air-based image coordinate system.

[0009] Step 3, Fusion and detection of vehicle targets based on high-precision calibration of radar and video: Using the optimized coordinate mapping matrix of each sub-region obtained in Step 2, the collaborative perception region is divided, and the vehicle targets detected by ground-based radar and the vehicle targets detected by airborne video are associated at the feature level and position level, and the fused list of vehicle targets containing position, speed, size and category information is output.

[0010] Step 4, Synchronous perception and statistics of multi-dimensional traffic flow parameters: Based on the vehicle target list obtained in Step 3, traffic flow parameters are calculated in parallel, including vehicle queue length, cross-sectional traffic flow, number of vehicles in the section, vehicle presence, and headway.

[0011] Furthermore, the multi-region division based on the degree of fragmentation of radar-based data described in step 1 specifically includes:

[0012] Step 1-1, Ground-based 4D millimeter-wave radar preprocessing: The original point cloud output by the ground-based 4D millimeter-wave radar is clustered and denoised using the DBSCAN and RANSAC algorithms to obtain clustered target data; based on Kalman filtering and Hungarian matching, multi-target tracking is performed on the clustering results, and finally a radar-detected vehicle target set with target ID, target distance, target speed, and target height information is obtained.

[0013] Step 1-2, Preprocessing of space-based video data: Vehicle target detection is performed on the space-based video frames using a target detection algorithm. Based on Kalman filtering and Hungarian matching algorithm, multi-target tracking is performed on the target detection results. Finally, a set of video-detected vehicle targets with target ID, pixel position and target category information is obtained.

[0014] Steps 1-3, fragmented statistics of radar and video tracking data: Perform multi-frame time-domain statistics on the lifecycle of the tracked target for ground-based 4D millimeter-wave tracking results and airborne video detection and tracking results; classify the tracking quality based on the ID continuity of radar and video tracking results according to preset thresholds; For each spatial unit Statistical time domain window The mass distribution of the set of targets passing through this spatial unit, based on the degree of fragmentation at the current location; ; in, ; In the formula, For passing through spatial units The target set For the target set The total number of targets in the target range. Used to represent spatial units The degree of fragmentation in sensor data; a higher value indicates more severe fragmentation. For the tracking quality corresponding to target k, For indicator functions, Let k be the number of frames that target k survives. and The threshold for segmenting target tracking quality is used to classify the tracking quality of the ID continuity of radar and video tracking results into high, medium, and low, denoted as High, Medium, and Low, respectively.

[0015] Steps 1-4, Connected Component Partitioning: Based on the fragmented segmentation results of the target tracking data in Steps 1-3, spatial frequency statistics are performed on the targets in each segment, and connected components are solved using the seed method, ultimately dividing the sensing region into n connected components:

[0016]

[0017] In the formula, For the i-th seed The corresponding connected components, For the Laplace operator, This represents the threshold for the degree of fragmentation.

[0018] Furthermore, The following conditions must be met:

[0019] st

[0020] In the formula, Indicates a 24-adjacency relationship. From the seed point To space unit The last intermediate point on the path, i.e., the point before the end of the path; m is the total number of intermediate points on the path, representing the distance from the seed point to the target unit. The number of intermediate units passed through; The j-th intermediate point in the path is used to describe the spatial unit. With seed point The connected paths, j = 1, 2, ..., m; Seed point The fragmentation level value is used as a reference benchmark for the fragmentation level within the connected component. This is the consistency threshold within the region.

[0021] Furthermore, the multi-region adaptive calibration for vehicle detection described in step 2 specifically includes:

[0022] Step 2-1, Radar and image point pair initialization: Within the common field of view of air-ground cooperative sensing, select at least 4 corresponding point pairs of vehicle feature points in the video coordinate system and radar coordinate system, and calculate a global initial mapping matrix.

[0023] Step 2-2, Region Mapping Overlap Error Statistics: For each divided sub-region Using its corresponding current mapping matrix The vehicle point cloud detected by radar in this area is projected onto the airborne image, as shown in the following formula for 4D millimeter-wave radar coordinates. to camera coordinates Mapping;

[0024]

[0025]

[0026] In the formula, the mapping matrix All elements in the array are real numbers, with customizable values.

[0027] The matching error between the projection points and the vehicle detection results in the image is calculated using feature matching and the Hungarian algorithm. The sum of the errors of all matching pairs is the sub-region. The mapping overlap error;

[0028] Steps 2-3, Temporal Iterative Optimization: With the goal of minimizing the total mapping overlap error of all sub-regions, the mapping matrix parameters of each region are iteratively adjusted through optimization algorithms, and finally n optimized mapping matrices are obtained, where n is the total number of sub-regions.

[0029] Furthermore, the calculation method for the mapping overlap error of the region in step 2-2 is as follows:

[0030] The lower edge center points of the m video vehicle detection results in the sub-region are mapped to the radar coordinate system through the inverse matrix of the mapping matrix corresponding to the sub-region, and Hungarian matching is used to initially associate them with the points after radar clustering.

[0031] The matching points after radar clustering are then mapped back to the image coordinate system through a mapping matrix. The pixel error between the matching points and the corresponding video detection points is measured by Euclidean distance. The pixel errors of each matching point pair are summed and recorded as the mapping overlap error of the sub-region.

[0032] Furthermore, in steps 2-3, the Levenberg-Marquardt method is used for multiple rounds of iterative optimization to minimize the total mapping overlap error of the n sub-regions. The specific formula is as follows:

[0033]

[0034] In the formula, This represents the mapping overlap error of the i-th sub-region. This indicates the error per round.

[0035] Furthermore, step 3, the radar-based vehicle target fusion and detection based on high-precision calibration, specifically includes:

[0036] Step 3-1, Target Mapping and Association: The vehicle point cloud detected by the ground-based radar in real time is projected onto the airborne image according to its spatial location using the corresponding region coordinate mapping matrix optimized in Step 2, forming radar projection targets; at the same time, the vehicle targets detected by the airborne video are mapped to the radar coordinate system; then the Hungarian algorithm is used to perform target association matching in the selected coordinate system.

[0037] Step 3-2, Collaborative sensing region division:

[0038] Based on the fragmented statistics in step 1 and the multiple region mapping results in step 2, the multiple regions under the radar coordinate system are classified as follows: stable public view area, radar-dominated perception area, video-dominated perception area, and perception degradation area. Among them, the stable public view area is the region where both radar and video tracking quality are high; the radar-dominated perception area is the region where radar tracking quality is high and video tracking quality is medium / low; the video-dominated perception area is the region where radar tracking quality is medium / low and video tracking quality is high; and the perception degradation area is the region where both radar and video tracking quality are low.

[0039] Step 3-3, Data Fusion and Decision Making, specifically includes:

[0040] Step 3-3-1: For the vehicle targets successfully associated in Step 3-1, fuse their information: use the position and speed provided by radar data as a reference, and use the vehicle category and visual size provided by video data as semantic information.

[0041] Step 3-3-1: For unmatched radar targets, if they are within the radar's dominant sensing area, they are output as supplementary detection results.

[0042] Step 3-3-3: For unmatched video targets, if they are in the perceptual degradation zone, they are removed.

[0043] Steps 3-4, output the fusion result: output a fused list of vehicles, each target containing a globally unique ID, fused radar coordinates and image coordinates, speed, and vehicle type.

[0044] On the other hand, a real-time traffic flow parameter sensing system based on air-ground cooperative sensing is provided, the system comprising:

[0045] The first module is used to realize multi-region division based on the fragmentation degree of radar vision data: based on the tracking continuity of vehicle targets in the same area by air-based video and ground-based radar, the reliability of sensor detection at different spatial locations is quantitatively evaluated, and the perception area is divided into multiple sub-regions of quality levels accordingly.

[0046] The second module is used to implement multi-region adaptive calibration for vehicle detection: for each sub-region divided in step 1, a unique coordinate mapping matrix is ​​initialized and iteratively optimized to accurately map the vehicle point cloud of the ground-based radar to the air-based image coordinate system.

[0047] The third module is used to realize the fusion and detection of radar-visual vehicle targets based on high-precision calibration: using the optimized coordinate mapping matrix of each sub-region obtained by the second module, the collaborative perception area is divided, and the vehicle targets detected by ground-based radar and the vehicle targets detected by airborne video are associated at the feature level and position level, and the fused list of vehicle targets containing position, speed, size and category information is output.

[0048] The fourth module is used to realize the synchronous perception and statistics of multi-dimensional traffic flow parameters: based on the vehicle target list obtained from the third module, traffic flow parameters are calculated in parallel, including vehicle queue length, cross-sectional traffic flow, number of vehicles in the section, vehicle presence, and headway.

[0049] On the other hand, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the real-time perception method for traffic flow parameters based on air-ground cooperative perception.

[0050] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the real-time perception method for traffic flow parameters based on air-ground cooperative perception.

[0051] Compared with existing technologies, the significant advantages of this invention are as follows: This invention closely integrates the calibration process with the detection task. By analyzing the vehicle target tracking lifecycle, it adaptively divides the perception area and optimizes calibration parameters, providing a precise geometric alignment foundation for subsequent data fusion. Based on this, through cross-modal feature matching and data association, it effectively combines the semantic information advantages of airborne video with the ranging and velocity measurement advantages of ground-based radar, ultimately achieving more accurate and stable vehicle detection results than single-sensor or traditional fusion methods under complex lighting, weather, and occlusion scenarios. Through air-ground collaborative perception, the effective detection range is significantly expanded, enabling large-scale traffic flow parameter perception at intersections.

[0052] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0053] Figure 1 This is an overall flowchart of a real-time traffic flow parameter sensing method based on air-ground collaborative sensing in one embodiment.

[0054] Figure 2 This is a flowchart illustrating the multi-region division provided in one embodiment.

[0055] Figure 3 This is a flowchart of multi-region adaptive calibration provided in one embodiment.

[0056] Figure 4 This is a schematic diagram of the neighborhood used for region division provided in one embodiment.

[0057] Figure 5 This is a flowchart of radar-visual vehicle target fusion and detection provided in one embodiment. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0060] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0061] In one embodiment, combined Figure 1 This invention proposes a real-time traffic flow parameter sensing method based on air-ground cooperative sensing, the method comprising the following steps:

[0062] Step 1, Multi-region division based on the fragmentation of radar-based data: Based on the continuity of tracking of vehicle targets in the same area by airborne video and ground-based radar, the reliability of sensor detection at different spatial locations is quantitatively evaluated, and the perception area is divided into multiple sub-regions of different quality levels accordingly.

[0063] Step 2, Multi-region adaptive calibration for vehicle detection: For each sub-region divided in Step 1, a unique coordinate mapping matrix is ​​initialized and iteratively optimized to accurately map the vehicle point cloud of the ground-based radar to the air-based image coordinate system.

[0064] Step 3, Fusion and detection of vehicle targets based on high-precision calibration of radar and video: Using the optimized coordinate mapping matrix of each sub-region obtained in Step 2, the collaborative perception region is divided, and the vehicle targets detected by ground-based radar and the vehicle targets detected by airborne video are associated at the feature level and position level, and the fused list of vehicle targets containing position, speed, size and category information is output.

[0065] Step 4, Synchronous perception and statistics of multi-dimensional traffic flow parameters: Based on the vehicle target list obtained in Step 3, traffic flow parameters are calculated in parallel, including vehicle queue length, cross-sectional traffic flow, number of vehicles in the section, vehicle presence, and headway.

[0066] The method significantly expands the effective detection range through air-ground collaborative sensing, enabling large-scale vehicle detection and traffic flow parameter statistics at intersections.

[0067] Furthermore, in one embodiment, the multi-region division based on the degree of fragmentation of radar-based data in step 1 specifically includes:

[0068] Step 1-1, Ground-based 4D millimeter-wave radar preprocessing: The original point cloud output by the ground-based 4D millimeter-wave radar is clustered and denoised using the DBSCAN and RANSAC algorithms to obtain clustered target data; based on Kalman filtering and Hungarian matching, multi-target tracking is performed on the clustering results, and finally a radar-detected vehicle target set with target ID, target distance, target speed, and target height information is obtained.

[0069] Step 1-2, Preprocessing of space-based video data: Vehicle target detection is performed on the space-based video frames using a target detection algorithm. Based on Kalman filtering and Hungarian matching algorithm, multi-target tracking is performed on the target detection results. Finally, a set of video-detected vehicle targets with target ID, pixel position and target category information is obtained.

[0070] Steps 1-3, fragmented statistics of radar and video tracking data: Perform multi-frame time-domain statistics on the lifecycle of the tracked target for ground-based 4D millimeter-wave tracking results and airborne video detection and tracking results; classify the tracking quality based on the ID continuity of radar and video tracking results according to preset thresholds;

[0071] For each spatial unit Statistical time domain window The mass distribution of the set of targets passing through this spatial unit, based on the degree of fragmentation at the current location;

[0072]

[0073] in,

[0074]

[0075] In the formula, For passing through spatial units The target set For the target set The total number of targets in the target range. Used to represent spatial units The degree of fragmentation in sensor data; a higher value indicates more severe fragmentation. For the tracking quality corresponding to target k, For indicator functions, Let k be the number of frames that target k survives. and The threshold for segmenting target tracking quality is used to classify the tracking quality of the ID continuity of radar and video tracking results into high, medium, and low, denoted as High, Medium, and Low, respectively.

[0076] Steps 1-4, Connected Component Partitioning: Based on the fragmented segmentation results of the target tracking data in Steps 1-3, spatial frequency statistics are performed on the targets in each segment, and connected components are solved using the seed method, ultimately dividing the sensing region into n connected components:

[0077]

[0078] In the formula, For the i-th seed The corresponding connected components, For the Laplace operator, This represents the threshold for the degree of fragmentation. Indicates through calculation The second-order spatial reciprocal reflects the spatial rate of change of the degree of fragmentation; local minima refer to spatial units. The degree of fragmentation It is less than the fragmentation value of all its neighboring units.

[0079] here, The following conditions must be met:

[0080] st

[0081] In the formula, Indicates a 24-adjacency relationship. From the seed point To space unit The last intermediate point on the path, i.e., the point before the end of the path; m is the total number of intermediate points on the path, representing the distance from the seed point to the target unit. The number of intermediate units passed through; The j-th intermediate point in the path is used to describe the spatial unit. With seed point The connected paths, j = 1, 2, ..., m; Seed point The fragmentation level value is used as a reference benchmark for the fragmentation level within the connected component. This is the consistency threshold within the region.

[0082] Here, each seed is the center, and expansion is carried out according to the regional growth rules. Based on the degree of fragmentation of each point, n connected regions are eventually formed.

[0083] Furthermore, in one embodiment, step 2, the multi-region adaptive calibration for vehicle detection, specifically includes:

[0084] Step 2-1, Radar and image point pair initialization: Within the common field of view of air-ground cooperative sensing, select at least 4 corresponding point pairs of vehicle feature points in the video coordinate system and radar coordinate system, and calculate a global initial mapping matrix.

[0085] Step 2-2, Region Mapping Overlap Error Statistics: For each divided sub-region Using its corresponding current mapping matrix The vehicle point cloud detected by radar in this area is projected onto the airborne image, as shown in the following formula for 4D millimeter-wave radar coordinates. to camera coordinates Mapping;

[0086]

[0087]

[0088] In the formula, the mapping matrix All elements in the array are real numbers, with customizable values.

[0089] The matching error between the projection points and the vehicle detection results in the image is calculated using feature matching and the Hungarian algorithm. The sum of the errors of all matching pairs is the sub-region. The mapping overlap error;

[0090] Steps 2-3, Temporal Iterative Optimization: With the goal of minimizing the total mapping overlap error of all sub-regions, the mapping matrix parameters of each region are iteratively adjusted through optimization algorithms, and finally n optimized mapping matrices are obtained, where n is the total number of sub-regions.

[0091] Preferably, in some embodiments, the mapping overlap error of the region in step 2-2 is calculated as follows:

[0092] The lower edge center points of the m video vehicle detection results in the sub-region are mapped to the radar coordinate system through the inverse matrix of the mapping matrix corresponding to the sub-region, and Hungarian matching is used to initially associate them with the points after radar clustering.

[0093] The matching points after radar clustering are then mapped back to the image coordinate system through a mapping matrix. The pixel error between the matching points and the corresponding video detection points is measured by Euclidean distance. The pixel errors of each matching point pair are summed and recorded as the mapping overlap error of the sub-region.

[0094] Preferably, in some embodiments, in steps 2-3, multiple rounds of iterative optimization are performed using the Levenberg-Marquardt method to minimize the total mapping overlap error of the n sub-regions. The specific formula is as follows:

[0095]

[0096] In the formula, This represents the mapping overlap error of the i-th sub-region. This indicates the error per round.

[0097] Furthermore, in one embodiment, step 3, the radar-based vehicle target fusion and detection based on high-precision calibration, specifically includes:

[0098] Step 3-1, Target Mapping and Association: The vehicle point cloud detected by the ground-based radar in real time is projected onto the airborne image according to its spatial location using the corresponding region coordinate mapping matrix optimized in Step 2, forming radar projection targets; at the same time, the vehicle targets detected by the airborne video are mapped to the radar coordinate system; then the Hungarian algorithm is used to perform target association matching in the selected coordinate system.

[0099] Step 3-2, Collaborative sensing region division:

[0100] Based on the fragmented statistics in step 1 and the multiple region mapping results in step 2, the multiple regions under the radar coordinate system are classified as follows: stable public view area, radar-dominated perception area, video-dominated perception area, and perception degradation area. Among them, the stable public view area is the region where both radar and video tracking quality are high; the radar-dominated perception area is the region where radar tracking quality is high and video tracking quality is medium / low; the video-dominated perception area is the region where radar tracking quality is medium / low and video tracking quality is high; and the perception degradation area is the region where both radar and video tracking quality are low.

[0101] Step 3-3, Data Fusion and Decision Making, specifically includes:

[0102] Step 3-3-1: For the vehicle targets successfully associated in Step 3-1, fuse their information: use the position and speed provided by radar data as a reference, and use the vehicle category and visual size provided by video data as semantic information.

[0103] Step 3-3-1: For unmatched radar targets, if they are within the radar's dominant sensing area, they are output as supplementary detection results.

[0104] Step 3-3-3: For unmatched video targets, if they are in the perceptual degradation zone, they are removed.

[0105] Steps 3-4, output the fusion result: output a fused list of vehicles, each target containing a globally unique ID, fused radar coordinates and image coordinates, speed, and vehicle type.

[0106] In one embodiment, a real-time traffic flow parameter sensing system based on air-ground cooperative sensing is provided, the system comprising:

[0107] The first module is used to realize multi-region division based on the fragmentation degree of radar vision data: based on the tracking continuity of vehicle targets in the same area by air-based video and ground-based radar, the reliability of sensor detection at different spatial locations is quantitatively evaluated, and the perception area is divided into multiple sub-regions of quality levels accordingly.

[0108] The second module is used to implement multi-region adaptive calibration for vehicle detection: for each sub-region divided in step 1, a unique coordinate mapping matrix is ​​initialized and iteratively optimized to accurately map the vehicle point cloud of the ground-based radar to the air-based image coordinate system.

[0109] The third module is used to realize the fusion and detection of radar-visual vehicle targets based on high-precision calibration: using the optimized coordinate mapping matrix of each sub-region obtained by the second module, the collaborative perception area is divided, and the vehicle targets detected by ground-based radar and the vehicle targets detected by airborne video are associated at the feature level and position level, and the fused list of vehicle targets containing position, speed, size and category information is output.

[0110] The fourth module is used to realize the synchronous perception and statistics of multi-dimensional traffic flow parameters: based on the vehicle target list obtained from the third module, traffic flow parameters are calculated in parallel, including vehicle queue length, cross-sectional traffic flow, number of vehicles in the section, vehicle presence, and headway.

[0111] Specific limitations regarding the real-time traffic flow parameter sensing system based on air-ground cooperative perception can be found in the limitations of the real-time traffic flow parameter sensing method based on air-ground cooperative perception mentioned above, and will not be repeated here. Each module in the aforementioned real-time traffic flow parameter sensing system based on air-ground cooperative perception can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0112] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements:

[0113] Step 1, Multi-region division based on the fragmentation of radar-based data: Based on the continuity of tracking of vehicle targets in the same area by airborne video and ground-based radar, the reliability of sensor detection at different spatial locations is quantitatively evaluated, and the perception area is divided into multiple sub-regions of different quality levels accordingly.

[0114] Step 2, Multi-region adaptive calibration for vehicle detection: For each sub-region divided in Step 1, a unique coordinate mapping matrix is ​​initialized and iteratively optimized to accurately map the vehicle point cloud of the ground-based radar to the air-based image coordinate system.

[0115] Step 3, Fusion and detection of vehicle targets based on high-precision calibration of radar and video: Using the optimized coordinate mapping matrix of each sub-region obtained in Step 2, the collaborative perception region is divided, and the vehicle targets detected by ground-based radar and the vehicle targets detected by airborne video are associated at the feature level and position level, and the fused list of vehicle targets containing position, speed, size and category information is output.

[0116] Step 4, Synchronous perception and statistics of multi-dimensional traffic flow parameters: Based on the vehicle target list obtained in Step 3, traffic flow parameters are calculated in parallel, including vehicle queue length, cross-sectional traffic flow, number of vehicles in the section, vehicle presence, and headway.

[0117] For specific limitations on each step, please refer to the limitations on the real-time perception method of traffic flow parameters based on air-ground collaborative perception mentioned above, which will not be repeated here.

[0118] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being implemented when executed by a processor:

[0119] Step 1, Multi-region division based on the fragmentation of radar-based data: Based on the continuity of tracking of vehicle targets in the same area by airborne video and ground-based radar, the reliability of sensor detection at different spatial locations is quantitatively evaluated, and the perception area is divided into multiple sub-regions of different quality levels accordingly.

[0120] Step 2, Multi-region adaptive calibration for vehicle detection: For each sub-region divided in Step 1, a unique coordinate mapping matrix is ​​initialized and iteratively optimized to accurately map the vehicle point cloud of the ground-based radar to the air-based image coordinate system.

[0121] Step 3, Fusion and detection of vehicle targets based on high-precision calibration of radar and video: Using the optimized coordinate mapping matrix of each sub-region obtained in Step 2, the collaborative perception region is divided, and the vehicle targets detected by ground-based radar and the vehicle targets detected by airborne video are associated at the feature level and position level, and the fused list of vehicle targets containing position, speed, size and category information is output.

[0122] Step 4, Synchronous perception and statistics of multi-dimensional traffic flow parameters: Based on the vehicle target list obtained in Step 3, traffic flow parameters are calculated in parallel, including vehicle queue length, cross-sectional traffic flow, number of vehicles in the section, vehicle presence, and headway.

[0123] For specific limitations on each step, please refer to the limitations on the real-time perception method of traffic flow parameters based on air-ground collaborative perception mentioned above, which will not be repeated here.

[0124] As a specific example, the invention will be described in detail in one embodiment.

[0125] The ground-based platform of this invention employs an 80GHz 4D millimeter-wave radar mounted on the roadside, with a detection range of 1km. The airborne platform utilizes a DJI Matrice 350 RTK drone, hovering at a low altitude of 50 meters, equipped with an H20T gimbal camera, achieving a detection range of 2km. Through air-ground collaborative perception, the effective detection range is extended to 1.5km to 2km, specifically optimized for detecting large-scale vehicle queues at intersections. The computing unit uses an 8-core ARM Cortex-A55 edge computing processor with 2.5 TFLOPS of computing power, connected to the roadside radar via wired Ethernet, and establishing a low-latency communication link with the drone via a 5G CPE.

[0126] In this embodiment, the convolutional model used for object detection and recognition in video frames is Darknet, combined with the Yolov3 algorithm. The platform selected is an embedded device with a computing power of 2.5 TFLOPS and an operating system kernel version of Linux 5.10.0. The model training is completed under TensorFlow as the deep learning framework, and the NPU hardware acceleration is used to complete the model inference. The model is then deployed on the embedded device. The dataset used consists of traffic scene data from radar and visual sensors, including multiple categories such as pedestrians, non-motorized vehicles, and motorized vehicles. The training set has 8,000 images, the validation set has 2,000 images, and the test set has 3,000 images. All images have been resized to 416×416.

[0127] The video object detection algorithm used was YOLOv3. Feature extraction used a 3x3 convolution kernel with a stride of 1. During training, Adam was selected as the optimizer, and the learning rate was initialized to 0.001. The decay strategy used was cosine annealing scheduling, and the learning rate was adjusted every 5 epochs for a total of 20 epochs. After each epoch, a combination of cross-entropy loss and Focal loss was used to balance the loss of positive and negative samples. Finally, radar data preprocessing was used to convert the radar echoes into a bird's-eye view format and align them with the image data. The radar echo data resolution was 0.1 meters.

[0128] refer to Figure 1The diagram shown is an overall flowchart of a real-time traffic flow parameter perception method based on air-ground cooperative perception provided in this embodiment of the present disclosure. It can be seen that the main processes include: cooperative perception area division, multi-area adaptive calibration, and radar-visual vehicle target fusion and detection.

[0129] 1. Multi-regional division based on the degree of fragmentation of radar data

[0130] The purpose of this step is to identify the "high-quality detection areas" and "problem detection areas" in the air-ground collaborative system.

[0131] (1) Airborne video preprocessing: The camera on the UAV decodes the video stream and uses the YOLOv3 target detection algorithm to detect vehicles in real time. Then, it combines Kalman filtering and Hungarian algorithm to track the target and outputs a sequence of vehicle tracking boxes with IDs.

[0132] (2) Ground-based radar preprocessing: After the original point cloud of the roadside 4D millimeter-wave radar is clustered by DBSCAN and denoised by RANSAC, multi-target tracking is also performed, and vehicle point cloud clusters with IDs and their three-dimensional position and speed information are output.

[0133] (3) Data fragmentation statistics: Under a unified spatiotemporal reference system, the number of consecutive tracking frames for each vehicle ID in the temporal domain is counted. The tracking quality of each target is evaluated according to preset thresholds (e.g., high: >100 frames, medium: 50-100 frames, low: <50 frames). For each spatial unit (e.g., superpixel block) in the image, the quality distribution of vehicles passing through it within a certain time window is counted to quantify the perception reliability of that location.

[0134] (4) Connected region division: Based on the fragmented statistical map, the seed growth method is used to merge continuous regions with similar perception reliability, and finally the entire collaborative vision is divided into n connected regions.

[0135] 2. Multi-area adaptive calibration for vehicle detection

[0136] This step learns the optimal coordinate transformation for each region, ensuring precise alignment of the vehicle target in image and radar data. Combined with... Figure 2 Specifically:

[0137] (1) Radar and image point pair initialization: Select at least 4 clear vehicle feature point pairs (such as the center point of the bottom of the vehicle) manually or automatically in multiple areas, and calculate a global initial mapping matrix H.

[0138] (2) Statistics on the overlap of region mapping: For each divided sub-region Use its corresponding current mapping matrix The vehicle point cloud detected by radar in this area is projected onto the airborne image, as shown in the following formula for 4D millimeter-wave radar coordinates. to camera coordinates Mapping;

[0139]

[0140]

[0141] The matching error between the projected points and the vehicle detection results in the image is calculated using feature matching and the Hungarian algorithm. The sum of the errors of all matching pairs is the mapping overlap error of the region. Specifically, the lower edge center points of the m video vehicle detection results in the region are mapped to the radar coordinate system through the inverse matrix of the homography transformation matrix corresponding to the region. Hungarian matching is used to initially associate them with the points after radar clustering. The matching points after radar clustering are then mapped back to the image coordinate system through the homography transformation matrix. Euclidean distance is used to measure the pixel error between them and the corresponding video detection points. The sum of the errors of each matching point pair is recorded as the mapping overlap error of the region.

[0142] (3) Iterative optimization in the time domain: Collect a segment of real-time data (e.g., 1000 frames), and adjust the mapping matrix H of each region with the goal of minimizing the total mapping overlap error of all regions. i The parameters are adjusted until convergence. The Levenberg-Marquardt method is used for multi-round iterative optimization to minimize the sum of mapping overlap errors across the n regions.

[0143]

[0144] 2. Radar-based vehicle target fusion and detection based on high-precision calibration

[0145] Combination Figure 5 Specifically:

[0146] (1) Target mapping and association: The vehicle point cloud detected by the ground-based radar in real time is projected onto the airborne image according to its spatial location using the corresponding region mapping matrix optimized by S2 to form radar projection targets; at the same time, the vehicle targets detected by the airborne video are mapped to the radar coordinate system; then the Hungarian algorithm is used to perform target association matching in the selected coordinate system.

[0147] (2) Collaborative sensing region division:

[0148] Based on the fragmented statistics in 1 (3) above, and the multiple region mapping results in 2 above, the multiple regions under the radar coordinate system can be classified as: stable public view area, radar-dominated perception area, video-dominated perception area, and perception degradation area; among which, the stable public view area is the region where both radar and video sensor tracking quality are high; among which, the radar-dominated perception area is the region where radar tracking quality is high and video tracking quality is medium / low; among which, the video-dominated perception area is the region where radar tracking quality is medium / low and video tracking quality is high; among which, the perception degradation area is the region where both radar and video tracking quality are low.

[0149] (3) Data fusion and decision-making:

[0150] 1) For successfully associated vehicle targets, fuse their information: use the position and speed provided by radar data as a benchmark, and the vehicle category and visual size provided by video data as semantic information;

[0151] 2) For unmatched radar targets, if they are within the radar's dominant sensing area, they will be output as supplementary detection results.

[0152] 3) For unmatched video targets, if they are located in a video perception degradation region, they will be removed;

[0153] (4) Fusion result output: The system outputs a fused vehicle list. Each target contains a globally unique ID, fused radar coordinates and image coordinates, speed, and vehicle type.

[0154] 4. Synchronous perception and calculation of multi-dimensional traffic flow parameters: Based on the fusion of the vehicle target list and the pre-configured lane detection lines, the following traffic flow parameters are calculated.

[0155] (1) Vehicle queue length: The tail of the queue is dynamically identified by analyzing the spatial distribution and movement of vehicles;

[0156] (2) Cross-sectional traffic flow: The number of vehicles passing through per unit time is counted based on the virtual detection line;

[0157] (3) Number of vehicles in a section: Real-time statistics of the total number of vehicles in a specified road section;

[0158] (4) Vehicle Presence: Detect whether there are vehicles in key areas;

[0159] (5) Headway: Calculate the time interval between consecutive vehicles passing through the detection line.

[0160] The method of this invention utilizes the collaboration of air and ground sensors. When the UAV is hovering at low altitude, it integrates the detection capabilities of ground-based radar and air-based video, breaking through the distance limitations of a single sensor. This enables accurate detection of vehicle queue sequences at large intersections and provides traffic flow parameters such as traffic volume and headway across multiple areas and sections.

[0161] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A real-time traffic flow parameter sensing method based on air-ground cooperative sensing, characterized in that, The method includes the following steps: Step 1, Multi-region division based on the fragmentation of radar-based data: Based on the continuity of tracking of vehicle targets in the same area by airborne video and ground-based radar, the reliability of sensor detection at different spatial locations is quantitatively evaluated, and the perception area is divided into multiple sub-regions of different quality levels accordingly. Step 2, Multi-region adaptive calibration for vehicle detection: For each sub-region divided in Step 1, a unique coordinate mapping matrix is ​​initialized and iteratively optimized to accurately map the vehicle point cloud of the ground-based radar to the air-based image coordinate system. Step 3, Fusion and detection of vehicle targets based on high-precision calibration of radar and video: Using the optimized coordinate mapping matrix of each sub-region obtained in Step 2, the collaborative perception region is divided, and the vehicle targets detected by ground-based radar and the vehicle targets detected by airborne video are associated at the feature level and position level, and the fused list of vehicle targets containing position, speed, size and category information is output. Step 4, Synchronous perception and statistics of multi-dimensional traffic flow parameters: Based on the vehicle target list obtained in Step 3, traffic flow parameters are calculated in parallel, including vehicle queue length, cross-sectional traffic flow, number of vehicles in the section, vehicle presence, and headway.

2. The real-time traffic flow parameter sensing method based on air-ground cooperative sensing according to claim 1, characterized in that, Step 1, which describes the multi-regional division based on the degree of fragmentation of radar-based data, specifically includes: Step 1-1, Ground-based 4D millimeter-wave radar preprocessing: The original point cloud output by the ground-based 4D millimeter-wave radar is clustered and denoised using the DBSCAN and RANSAC algorithms to obtain clustered target data; based on Kalman filtering and Hungarian matching, multi-target tracking is performed on the clustering results, and finally a radar-detected vehicle target set with target ID, target distance, target speed, and target height information is obtained. Step 1-2, Preprocessing of space-based video data: Vehicle target detection is performed on the space-based video frames using a target detection algorithm. Based on Kalman filtering and Hungarian matching algorithm, multi-target tracking is performed on the target detection results. Finally, a set of video-detected vehicle targets with target ID, pixel position and target category information is obtained. Steps 1-3, fragmented statistics of radar and video tracking data: Perform multi-frame time-domain statistics on the lifecycle of the tracked target for ground-based 4D millimeter-wave tracking results and airborne video detection and tracking results; classify the tracking quality based on the ID continuity of radar and video tracking results according to preset thresholds; For each spatial unit Statistical time domain window The mass distribution of the set of targets passing through this spatial unit, based on the degree of fragmentation at the current location; ； in, ； In the formula, For passing through spatial units The target set For the target set The total number of targets in the target range. Used to represent spatial units The degree of fragmentation in sensor data; a higher value indicates more severe fragmentation. For the tracking quality corresponding to target k, For indicator functions, Let k be the number of frames that target k survives. and The threshold for segmenting target tracking quality is used to classify the tracking quality of the ID continuity of radar and video tracking results into high, medium, and low, denoted as High, Medium, and Low, respectively. Steps 1-4, Connected Component Partitioning: Based on the fragmented segmentation results of the target tracking data in Steps 1-3, spatial frequency statistics are performed on the targets in each segment, and connected components are solved using the seed method, ultimately dividing the sensing region into n connected components: ； In the formula, For the i-th seed The corresponding connected components, For the Laplace operator, This represents the threshold for the degree of fragmentation.

3. The real-time traffic flow parameter sensing method based on air-ground cooperative sensing according to claim 2, characterized in that, The following conditions must be met: ； s.t. ;; In the formula, Indicates a 24-adjacency relationship. From the seed point To space unit The last intermediate point on the path, i.e., the point before the end of the path; m is the total number of intermediate points on the path, representing the distance from the seed point to the target unit. The number of intermediate units passed through; The j-th intermediate point in the path is used to describe the spatial unit. With seed point The connected paths, j = 1, 2, ..., m; Seed point The fragmentation level value is used as a reference benchmark for the fragmentation level within the connected component. This is the consistency threshold within the region.

4. The real-time traffic flow parameter sensing method based on air-ground cooperative sensing according to claim 1, characterized in that, Step 2, the multi-region adaptive calibration for vehicle detection, specifically includes: Step 2-1, Radar and image point pair initialization: Within the common field of view of air-ground cooperative sensing, select at least 4 corresponding point pairs of vehicle feature points in the video coordinate system and radar coordinate system, and calculate a global initial mapping matrix. Step 2-2, Region Mapping Overlap Error Statistics: For each divided sub-region Using its corresponding current mapping matrix The vehicle point cloud detected by radar in this area is projected onto the airborne image, as shown in the following formula for 4D millimeter-wave radar coordinates. to camera coordinates Mapping; ； ； In the formula, the mapping matrix All elements in the array are real numbers, with customizable values. The matching error between the projection points and the vehicle detection results in the image is calculated using feature matching and the Hungarian algorithm. The sum of the errors of all matching pairs is the sub-region. The mapping overlap error; Steps 2-3, Temporal Iterative Optimization: With the goal of minimizing the total mapping overlap error of all sub-regions, the mapping matrix parameters of each region are iteratively adjusted through optimization algorithms, and finally n optimized mapping matrices are obtained, where n is the total number of sub-regions.

5. The real-time traffic flow parameter sensing method based on air-ground cooperative sensing according to claim 4, characterized in that, The method for calculating the mapping overlap error of the region in step 2-2 is as follows: The lower edge center points of the m video vehicle detection results in the sub-region are mapped to the radar coordinate system through the inverse matrix of the mapping matrix corresponding to the sub-region, and Hungarian matching is used to initially associate them with the points after radar clustering. The matching points after radar clustering are then mapped back to the image coordinate system through a mapping matrix. The pixel error between the matching points and the corresponding video detection points is measured by Euclidean distance. The pixel errors of each matching point pair are summed and recorded as the mapping overlap error of the sub-region.

6. The real-time traffic flow parameter sensing method based on air-ground cooperative sensing according to claim 4, characterized in that, In steps 2-3, the Levenberg-Marquardt method is used for multiple rounds of iterative optimization to minimize the total mapping overlap error of the n sub-regions. The specific formula is as follows: ； In the formula, This represents the mapping overlap error of the i-th sub-region. This indicates the error per round.

7. The real-time traffic flow parameter sensing method based on air-ground cooperative sensing according to claim 1, characterized in that, Step 3, the radar-based vehicle target fusion and detection based on high-precision calibration, specifically includes: Step 3-1, Target Mapping and Association: The vehicle point cloud detected by the ground-based radar in real time is projected onto the airborne image according to its spatial location using the corresponding region coordinate mapping matrix optimized in Step 2, forming radar projection targets; at the same time, the vehicle targets detected by the airborne video are mapped to the radar coordinate system; then the Hungarian algorithm is used to perform target association matching in the selected coordinate system. Step 3-2, Collaborative sensing region division: Based on the fragmented statistics in step 1 and the multiple region mapping results in step 2, the multiple regions under the radar coordinate system are classified as follows: stable public view area, radar-dominated perception area, video-dominated perception area, and perception degradation area. Among them, the stable public view area is the region where both radar and video tracking quality are high; the radar-dominated perception area is the region where radar tracking quality is high and video tracking quality is medium / low; the video-dominated perception area is the region where radar tracking quality is medium / low and video tracking quality is high; and the perception degradation area is the region where both radar and video tracking quality are low. Step 3-3, Data Fusion and Decision Making, specifically includes: Step 3-3-1: For the vehicle targets successfully associated in Step 3-1, fuse their information: use the position and speed provided by radar data as a reference, and use the vehicle category and visual size provided by video data as semantic information. Step 3-3-1: For unmatched radar targets, if they are within the radar's dominant sensing area, they are output as supplementary detection results. Step 3-3-3: For unmatched video targets, if they are in the perceptual degradation zone, they are removed. Steps 3-4, output the fusion result: output a fused list of vehicles, each target containing a globally unique ID, fused radar coordinates and image coordinates, speed, and vehicle type.

8. A real-time traffic flow parameter sensing system based on air-ground cooperative sensing according to the method of any one of claims 1 to 7, characterized in that, The system includes: The first module is used to realize multi-region division based on the fragmentation degree of radar vision data: based on the tracking continuity of vehicle targets in the same area by air-based video and ground-based radar, the reliability of sensor detection at different spatial locations is quantitatively evaluated, and the perception area is divided into multiple sub-regions of quality levels accordingly. The second module is used to implement multi-region adaptive calibration for vehicle detection: for each sub-region divided in step 1, a unique coordinate mapping matrix is ​​initialized and iteratively optimized to accurately map the vehicle point cloud of the ground-based radar to the air-based image coordinate system. The third module is used to realize the fusion and detection of radar-visual vehicle targets based on high-precision calibration: using the optimized coordinate mapping matrix of each sub-region obtained by the second module, the collaborative perception area is divided, and the vehicle targets detected by ground-based radar and the vehicle targets detected by airborne video are associated at the feature level and position level, and the fused list of vehicle targets containing position, speed, size and category information is output. The fourth module is used to realize the synchronous perception and statistics of multi-dimensional traffic flow parameters: based on the vehicle target list obtained from the third module, traffic flow parameters are calculated in parallel, including vehicle queue length, cross-sectional traffic flow, number of vehicles in the section, vehicle presence, and headway.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images