Traffic intersection vehicle trajectory fusion method and device, electronic equipment and medium

Abstract

The application provides a traffic intersection vehicle trajectory fusion method and device, electronic equipment and medium, and relates to the technical field of intelligent transportation, which comprises the following steps: standardizing and preprocessing the perception data of a millimeter wave radar and a camera to obtain effective data; matching single-camera and multi-camera fusion prediction targets to determine a visual measurement state vector; combining Mahalanobis distance and target binding ID to complete target association; performing adaptive Kalman filtering fusion tracking to obtain fusion target state data; performing new birth and disappearance judgment on unassociated targets after clustering to obtain an effective fusion tracking target set; extracting a preliminary trajectory, matching the preliminary trajectory with a new birth trajectory after interrupting trajectory prediction in a ROI (Region of Interest) of the intersection, and performing interpolation completion to obtain an optimized trajectory; and improving multi-target tracking accuracy, trajectory speed and position consistency, adapting to complex traffic scenarios at the intersection, and providing reliable trajectory data for traffic management and vehicle-road cooperation.

Description

Technical Field

[0001] This invention relates to the technical field of intelligent transportation, and in particular to a method, apparatus, electronic device, and medium for fusing vehicle trajectories at traffic intersections. Background Technology

[0002] Against the backdrop of the development of intelligent transportation and vehicle-road cooperative technologies, roadside perception sensors are the core foundation for traffic environment perception. Millimeter-wave radar and cameras are widely used for vehicle target perception at traffic intersections due to their respective technical characteristics. The data they collect, such as target position, speed, and trajectory, are key support for traffic management, scene modeling, and vehicle-road cooperative interaction. However, urban traffic intersections present complex scenarios with multi-directional vehicle interaction, mutual vehicle occlusion, and mixed traffic of motor vehicles and non-motor vehicles. Coupled with the physical limitations of millimeter-wave radar and cameras, accurately identifying and tracking vehicles and extracting high-quality driving trajectories still faces many technical challenges.

[0003] In existing technologies, millimeter-wave radar has inherent perception defects, making it difficult to accurately identify and track stationary and turning targets. It also has limited coverage, with blind spots easily formed by green belts and large vehicles. Multiple millimeter-wave radars need to be deployed at intersections to achieve full coverage, but this requires strict time synchronization of data from multiple radars and is prone to generating repeated trajectories of the same target, significantly increasing data transmission and storage costs. Cameras, on the other hand, have a limited effective identification range for vehicle targets, low accuracy in position and speed measurement, and the obstruction between vehicles can easily cause target tracking to be lost.

[0004] While the fusion of millimeter-wave radar and cameras can combine the advantages of both to improve perception, existing fusion solutions are mostly applied to in-vehicle autonomous driving or highway scenarios. In-vehicle fusion only focuses on the immediate, near-field traffic situation around the vehicle, while highway fusion relies on simplified road conditions with stable vehicle speeds and fixed paths. Neither can adapt to the complex interactive scenarios of urban traffic intersections. Furthermore, existing technologies lack effective optimization methods for the fused tracking trajectory, which easily leads to abnormal trajectory interruptions and omissions. This results in insufficient completeness and accuracy of the extracted vehicle trajectory, making it difficult to meet the high requirements of urban intelligent transportation for roadside vehicle perception. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, electronic device, and medium for fusing vehicle trajectories at traffic intersections, so as to alleviate the technical problem that the fusion results of vehicle driving trajectories at traffic intersections cannot meet reliability requirements.

[0006] In a first aspect, the present invention provides a method for fusing vehicle trajectories at a traffic intersection, comprising: Vehicle perception data acquired by millimeter-wave radar and cameras deployed at traffic intersections are standardized and preprocessed to obtain effective radar data and effective visual data. Based on the visually valid data, the vehicle targets detected by a single camera and the predicted targets fused by multiple cameras are matched, and the visual measurement state vector is determined by the association results used to characterize the matching relationship. Target association is performed based on the target Mahalanobis distance and target binding ID determined by the radar effective data and the visual measurement state vector, respectively, and fused target state data is obtained by fusion tracking after adaptive Kalman filtering; Based on the clustering results of the fused target state data, new and disappearance determinations are made for unrelated targets to obtain a set of effective fused tracking targets after removing invalid targets; Preliminary trajectory data is extracted from the effective fusion tracking target set based on the target ID. Abnormal interrupted trajectories within the intersection ROI are screened and predicted. The predicted trajectories are matched with the new trajectories and then interpolated to complete the data, resulting in the optimized vehicle driving trajectory.

[0007] In an optional implementation, the step of matching vehicle targets detected by a single camera with predicted targets fused from multiple cameras based on the visually valid data, and determining the visual measurement state vector through the association results used to characterize the matching relationship, includes: Vehicle targets are detected and continuously tracked using visually valid data. The vehicle position is estimated by calibrating the camera through homography transformation, and the motion parameters of the vehicle targets are calculated to obtain the visual recognition results of each single camera for the vehicle targets in the current frame. Based on the visual recognition results and visual fusion results, a Mahalanobis distance is calculated to characterize the similarity between the vehicle target detected by each single camera and the predicted target fused by the multi-camera system; wherein, the visual fusion result is the predicted value of the visual fusion result of the multi-camera system for the vehicle target in the current frame, obtained by using a uniform acceleration model and Kalman filter prediction step to perform state prediction on the visual fusion result of the multi-camera system for the vehicle target in the previous frame. A correlation cost matrix is ​​constructed using each Mahalanobis distance as a matrix element, and the correlation results are solved based on the Hungarian algorithm. Based on the association results, a unified visual measurement state vector for the target is determined.

[0008] In an optional implementation, the step of associating a target with the target based on the target Mahalanobis distance and target binding ID determined by the effective radar data and the visual measurement state vector respectively includes: The effective radar data and the visual measurement state vector are integrated into a global target set for intersection detection by the equipment. Calculate the Mahalanobis distance, which characterizes the similarity between each vehicle target in the target set detected by the global device at the intersection and the predicted target of the radar vision global fusion; wherein the predicted target of the radar vision global fusion is the predicted value of the global fusion result for the vehicle target in the current frame, obtained by using a uniform acceleration model and Kalman filter prediction step to perform state prediction on the global fusion result for the vehicle target in the previous frame of the global device. A radar vision global association cost matrix is ​​constructed using the Mahalanobis distance between each sensor as matrix elements, and the preliminary association results of the radar vision global association are obtained by using the Hungarian algorithm. Based on the preliminary association results, radar-detected targets, multi-camera fusion predicted targets, and radar vision global fusion predicted targets with matching relationships are identified. The original sensor IDs of the radar-detected targets and multi-camera fusion targets in the current frame are uniquely matched with the original sensor IDs of the radar vision global fusion predicted targets in the previous frame. The preliminary association results are corrected by eliminating unmatched association results and re-matching them.

[0009] In an optional implementation, the step of obtaining fused target state data through adaptive Kalman filtering fusion tracking includes: Based on the association type of the corrected association results, determine the measurement state vector of the Kalman filter corresponding to the current frame; The previous frame's fused target state data is processed using the state transition matrix of the uniform acceleration model and the Kalman filter prediction formula to obtain the prior predicted state of the current frame. Based on the measurement state vector and measurement covariance matrix corresponding to the association type, perform one or two updates to fuse the prior predicted state with the measurement state vector to obtain the fused target state data of the current frame. The fused target state data of the current frame is used as the new fused target state data of the previous frame. The state transition matrix of the uniform acceleration model and the Kalman filter prediction formula are used to process the fused target state data of the previous frame again to obtain the predicted target of the radar vision global fusion in the next frame.

[0010] In an optional implementation, the step of performing one or two updates based on the measurement state vector and measurement covariance matrix corresponding to the association type includes: If the corrected association result is only associated with a single radar-identified target or a visually identified target, then a single Kalman filter update is performed; If the corrected association result is associated with both radar-identified and visually-identified targets, then two Kalman filter updates are performed sequentially based on the measurement covariance matrices of radar and vision, respectively.

[0011] In an optional implementation, the steps of filtering and predicting abnormal interrupted trajectories within the intersection ROI, matching the predicted trajectory with the newly generated trajectory, and then interpolating to complete the optimized vehicle trajectory include: Based on the target ID, newly generated target trajectories and tracking interrupted trajectories within the intersection ROI are selected from the initial trajectory data. Based on the fused target state data of the last frame of the interrupted trajectory, a predicted trajectory is obtained through a uniformly accelerated motion model to characterize the real physical characteristics of the motion before the interruption. The predicted trajectory that meets the preset matching conditions is associated with the newly generated trajectory, and the missing frames of the successfully matched trajectory are filled in by interpolation to obtain the complete vehicle driving trajectory.

[0012] In an optional implementation, the step of standardizing and preprocessing the vehicle perception data acquired by millimeter-wave radar and cameras deployed at traffic intersections to obtain effective radar data and effective visual data includes: Using millimeter-wave radar deployed in the road area at traffic intersections, vehicle data of vehicles in the road area is acquired, wherein the vehicle data includes at least vehicle position, speed, acceleration and original tracking ID; Using cameras deployed in the intersection area of ​​a traffic intersection, image frame data of vehicles in the intersection area are acquired; The perception data, including the vehicle data and the image frame data, are processed by time synchronization, coordinate transformation, and ROI range filtering to obtain effective radar data and effective visual data.

[0013] Secondly, the present invention provides a device for fusing vehicle trajectories at a traffic intersection, comprising: The preprocessing module standardizes and preprocesses the vehicle perception data acquired by millimeter-wave radar and cameras deployed at traffic intersections to obtain effective radar data and effective visual data. The association module matches the vehicle targets detected by a single camera with the predicted targets fused by multiple cameras based on the visually valid data, and determines the visual measurement state vector through the association results used to characterize the matching relationship. The fusion module performs target association based on the target Mahalanobis distance and target binding ID determined by the radar effective data and the visual measurement state vector, respectively, and obtains fused target state data through adaptive Kalman filtering fusion tracking; The determination module performs new birth and disappearance determination on unrelated targets based on the clustering results of the fused target status data, and obtains a set of effective fused tracking targets after removing invalid targets; The optimization module extracts preliminary trajectory data from the effective fusion tracking target set based on the target ID, filters abnormal interrupted trajectories within the intersection ROI and makes predictions, matches the predicted trajectories with the newly generated trajectories and then interpolates to complete the data, thus obtaining the optimized vehicle driving trajectory.

[0014] Thirdly, the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the method as described in any of the foregoing embodiments.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed, implements the method described in any of the foregoing embodiments.

[0016] This invention provides a method, apparatus, electronic device, and medium for fusing vehicle trajectories at traffic intersections. First, standardized preprocessing is performed on vehicle perception data collected by millimeter-wave radar and cameras deployed at the traffic intersection to remove invalid data, resulting in uniformly formatted radar and visual valid data. Then, based on the visual valid data, matching is performed between single-camera detected vehicle targets and multi-camera fusion predicted targets. A unified visual measurement state vector is determined based on the matching association results. Subsequently, target association is completed by combining the target Mahalanobis distance and target binding ID. Adaptive Kalman filtering is used to fuse and track targets, obtaining fused target state data. Then, based on the clustering results of the fused target state data, new and disappearance determinations are performed on unassociated targets. After removing invalid targets, a set of valid fused tracking targets is obtained. Finally, preliminary trajectory data is extracted from this target set based on the target ID, and abnormal interrupted trajectories within the intersection's ROI are screened and further processed. The vehicle trajectory prediction process involves matching the predicted trajectory with the newly generated trajectory and then interpolating to fill in missing frames, ultimately obtaining an optimized vehicle trajectory. This invention effectively combines the sensing advantages of millimeter-wave radar and cameras, solving problems such as blind spots, low detection accuracy, and easy target loss in traffic intersections caused by single sensors. It also eliminates spatiotemporal biases and format differences in multi-sensor data, improving the accuracy of cross-sensor target association, avoiding false detections, missed detections, and invalid tracking, and compensating for trajectory interruptions caused by sensor occlusion and missed detections. This enables accurate and stable tracking of vehicle targets at traffic intersections. The extracted vehicle trajectory exhibits good spatiotemporal continuity and accuracy, while reducing the problem of duplicate trajectories caused by multi-radar deployment, lowering data transmission and storage costs. It is adaptable to complex traffic scenarios such as vehicle interaction, stopping and waiting, and mixed traffic at urban intersections, providing accurate and reliable intersection vehicle trajectory data for traffic management, vehicle-road cooperation, and traffic scenario modeling.

[0017] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained through the structures particularly pointed out in the description and the drawings.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 A flowchart of a method for fusing vehicle trajectories at a traffic intersection, provided in an embodiment of the present invention; Figure 2 A flowchart of another method for fusing vehicle trajectories at a traffic intersection provided in an embodiment of the present invention; Figure 3 A flowchart for target trajectory extraction and optimization provided in an embodiment of the present invention; Figure 4 A schematic diagram of the functional modules of a vehicle trajectory fusion device at a traffic intersection provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware architecture of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Currently, the accuracy of vehicle trajectory fusion cannot meet the requirements for complex interactive scenarios at urban traffic intersections. This invention provides a method, device, electronic device, and medium for vehicle trajectory fusion at traffic intersections. It can combine the respective sensing advantages of millimeter-wave radar and cameras to achieve accurate and stable tracking of vehicle targets at intersections, while reducing the problem of duplicate trajectories caused by the deployment of multiple radars. It can accurately and completely extract the driving trajectory of vehicles at urban traffic intersections and is suitable for complex traffic scenarios such as vehicle interaction, stopping and waiting, and mixed traffic of motor vehicles and non-motor vehicles at intersections.

[0023] To facilitate understanding of this embodiment, a method for fusing vehicle trajectories at traffic intersections disclosed in this embodiment of the invention will first be described in detail.

[0024] Figure 1 This is a flowchart of a method for fusing vehicle trajectories at a traffic intersection, provided as an embodiment of the present invention.

[0025] Reference Figure 1 The method mainly includes the following steps: S102 standardizes and preprocesses the vehicle perception data acquired by millimeter-wave radar and cameras deployed at traffic intersections to obtain effective radar data and effective visual data.

[0026] Here, standardized operations such as time synchronization, coordinate transformation, and ROI range filtering are performed on the raw vehicle perception data collected by millimeter-wave radar and cameras to remove invalid data outside the intersection ROI and unify the raw perception data of different formats into a standardized format, ultimately obtaining effective radar data and effective visual data. This preprocessing can eliminate the spatiotemporal deviation and format differences of multi-sensor data, remove invalid data to reduce the amount of subsequent calculations, and provide high-quality basic data with a unified format for subsequent multimodal fusion, which is a prerequisite for the accurate execution of the fusion algorithm.

[0027] In some embodiments, step S102 can be implemented through the following steps: Step 1.1: Use millimeter-wave radar deployed in the road area at traffic intersections to acquire vehicle data in the road area; The vehicle data includes at least vehicle position, speed, acceleration, and original tracking ID. Multiple millimeter-wave radars are deployed in the road area (within 30-100 meters of the stop line at traffic intersections) in each direction, with a radar acquisition frequency of 10Hz. These radars collect real-time vehicle data, including latitude and longitude positions, speeds, accelerations, and original radar tracking IDs. Specifically, one radar is placed every 100 meters, with approximately 200 meters between each intersection. This deployment of millimeter-wave radar in the road area fully leverages its long-range detection advantage, achieving comprehensive vehicle perception in the intersection area. The collected vehicle data contains complete motion parameters and unique IDs, providing foundational radar data for subsequent preprocessing and fusion.

[0028] For example, millimeter-wave radar and cameras are deployed according to the characteristics of the sensors. The millimeter-wave radar covers the road area. Millimeter-wave radar is suitable for detecting moving targets. Deploying it in the road area can give full play to its long-range detection advantage and make up for the insufficient recognition range of the camera in the road area.

[0029] Step 1.2: Use cameras deployed at the intersection area to acquire image frame data of vehicles in the intersection area; Multiple high-definition cameras are deployed on traffic light poles at the four corners of the intersection, with a frame rate of 25fps. These cameras collect real-time image frames of vehicles within the core area of ​​the intersection, and each frame is assigned a unique millisecond-level timestamp. This deployment in the core area of ​​the intersection leverages the cameras' high accuracy in target recognition, enabling precise visual perception of vehicles in the core interaction zone. The millisecond-level timestamped image frame data provides accurate timing data for subsequent time synchronization.

[0030] For example, millimeter-wave radar and cameras are deployed based on sensor characteristics, with the cameras covering the intersection area; the cameras are suitable for target recognition in the core interaction area of ​​the intersection, which can make up for the shortcomings of millimeter-wave radar in accurately recognizing stationary and turning targets at intersections.

[0031] Step 1.3 involves performing time synchronization, coordinate transformation, and ROI range filtering on the perception data, including vehicle data and image frame data, to obtain effective radar data and effective visual data.

[0032] Here, the NTP time of the intersection edge computing node is used as the benchmark to synchronize radar vehicle data and camera image frame data. A Gaussian projection transformation is used to convert the global geographic coordinates to the local coordinate system of the intersection, completing the coordinate transformation. The ROI range of the intersection is defined, and invalid data outside the ROI is removed, completing ROI range filtering, ultimately obtaining valid radar data and valid visual data. Time synchronization eliminates time discrepancies among multiple sensors, coordinate transformation unifies the spatial coordinates of multiple sensors, and ROI range filtering removes invalid data, reducing computational load. Finally, high-quality, uniformly formatted valid data is obtained, laying the foundation for the accurate execution of subsequent fusion algorithms.

[0033] For example, the recognition results of millimeter-wave radar and vision fusion are first synchronized in time, and one of the millimeter-wave radars is selected as the main sensor; the preprocessing module converts the global coordinates into local coordinates and filters out radar targets outside the ROI range; this preprocessing step is a necessary operation before the fusion of millimeter-wave radar and vision to ensure the spatiotemporal consistency of the fused data.

[0034] It should be noted that the recognition results of millimeter-wave radar and vision fusion are first synchronized in time, and one of the millimeter-wave radars is selected as the main sensor; the preprocessing module converts the global coordinates into local coordinates and filters out radar targets outside the ROI range. This preprocessing is a necessary preliminary step for millimeter-wave radar and vision fusion.

[0035] Specifically, this embodiment is based on the deployment of sensors at urban traffic intersections. First, eight millimeter-wave radars are deployed in the road area in each direction of the intersection, specifically within the lanes a certain distance from the stop line. These radars are set to a 10Hz acquisition frequency and collect real-time vehicle data within the road area. This data includes the vehicle's global latitude and longitude coordinates, speed, acceleration, and the radar's original tracking ID, such as R01_001. Simultaneously, four high-definition cameras are deployed on the traffic light poles at the four corners of the intersection. These cameras are set to a frame rate of 25fps and collect real-time image frames of vehicles within the core interaction area of ​​the intersection, adding a unique millisecond-level timestamp to each frame. Next, time synchronization processing is performed on the vehicle data collected by the radar and the image frame data collected by the cameras. Using the NTP time of the intersection edge computing node as a unified time reference, the timestamps of both the radar data and the camera image frames are calibrated to this reference, ensuring accurate matching between radar data and camera data at the same time point. Furthermore, coordinate transformation is performed. Using Gaussian projection transformation, the global geographic coordinates (latitude and longitude) from radar and camera data are converted into a local coordinate system for the intersection, with the intersection center as the origin, the x-axis representing the east-west direction, and the y-axis representing the north-south direction. This yields the vehicle's local coordinates, achieving coordinate unification of multi-sensor data. Finally, ROI (Region of Interest) filtering is performed. The intersection ROI is first defined as a rectangular area 50 meters horizontally and vertically from the intersection center. Then, based on the vehicle's local coordinates, invalid target data exceeding this ROI range is removed from the radar vehicle data and camera image frame data, retaining only the valid data within the ROI. This results in valid radar data and valid visual data. The valid radar data includes the vehicle's local coordinates, speed, acceleration, and original radar ID. The valid visual data includes timestamped image frames within the ROI and the vehicle's local coordinate calibration basis.

[0036] S104, based on visually valid data, matches the vehicle targets detected by a single camera with the predicted targets fused by multiple cameras, and determines the visual measurement state vector through the association results used to characterize the matching relationship.

[0037] Here, based on the visually valid data obtained from the steps of the aforementioned embodiments, the similarity between the target detected by a single camera and the target predicted by multi-camera fusion is calculated using Mahalanobis distance. The optimal matching between the two is then achieved using the Hungarian algorithm. Based on the association results obtained from the matching, the multi-camera detection data is integrated and calibrated to determine a globally unified visual measurement state vector. This solves the problem of inconsistent detection data caused by the viewing angle deviation of multiple cameras, realizes visual single-modal fusion, obtains a standardized visual measurement state vector, provides a unified visual input for radar visual cross-modal fusion, and improves the reliability of visual data.

[0038] In some embodiments, step S104 may be implemented specifically through the following steps, including: Step 2.1: Detect and continuously track vehicle targets using visually valid data. Estimate vehicle position by calibrating the camera through homography transformation and calculate the motion parameters of the vehicle targets to obtain the visual recognition results of each single camera for the vehicle targets in the current frame. For example, the YOLOv5 target detection algorithm and the ByteTrack multi-target tracking algorithm are used to detect and continuously track vehicle targets on visually valid data. The image pixel coordinates are converted into physical coordinates of the intersection through the homography transformation principle to achieve vehicle position estimation. At the same time, the vehicle's heading angle, velocity, acceleration and other motion parameters are calculated based on the position and time difference of adjacent frames to obtain the independent visual recognition result of each single camera in the current frame. This enables accurate detection and tracking of vehicle targets by a single camera, completes the conversion from pixel coordinates to physical coordinates, and obtains visual recognition results containing complete motion parameters. This provides basic detection data for multi-camera fusion and improves the practicality of single-camera visual detection.

[0039] Step 2.2: Calculate the Mahalanobis distance, which is used to characterize the similarity between the vehicle targets detected by each single camera and the predicted targets fused by the multi-camera system, based on the visual recognition results and the visual fusion results. Among them, the visual fusion result is the predicted value of the visual fusion result of multiple cameras for the vehicle target in the current frame, obtained by using the uniform acceleration model and Kalman filter prediction step to perform state prediction on the visual fusion result of multiple cameras for the vehicle target in the previous frame. Specifically, the state data from the previous frame after multi-camera visual fusion is retrieved. This data is then used to generate the current frame's multi-camera fusion prediction target through a uniformly accelerated model and Kalman filter prediction step. Using this prediction target as a benchmark, the Mahalanobis distance between each individual camera's visual recognition result and the predicted target is calculated, thus quantifying the similarity between targets. Compared to Euclidean distance, Mahalanobis distance eliminates the influence of data dimensionality and units, more accurately representing the similarity between individual camera targets and the fusion prediction target. The prediction target based on the previous frame's fusion result ensures the continuity of matching and improves the accuracy of multi-camera target matching.

[0040] For example, Mahalanobis distance is used to calculate target similarity. Mahalanobis distance is defined by combining the covariance matrix of the data, the visual recognition result of the current frame, and the predicted value of the fused tracking. The definition of Mahalanobis distance is as follows: ;in It is the covariance matrix of the data. This is the visual recognition result of the current frame. This is the predicted value for fusion tracking. The fusion of four cameras at the intersection is achieved by calculating the Mahalanobis distance between the visually recognized target and the visually fused target; the visually fused predicted target is obtained by performing state prediction on the previous frame's visual fusion result using a uniformly accelerated model and a Kalman filter prediction step.

[0041] Step 2.3: Construct an association cost matrix using each Mahalanobis distance as a matrix element, and solve for the association results based on the Hungarian algorithm; Here, the Mahalanobis distance between each single camera and the multi-camera fusion predicted target is used as matrix elements to construct a visual association cost matrix. The Hungarian algorithm is then used to solve this cost matrix to obtain the optimal matching visual association result, clarifying the affiliation of the same physical vehicle corresponding to targets detected by different single cameras. The Hungarian algorithm can efficiently solve the optimal matching problem in weighted bipartite graphs, ensuring optimal matching between targets detected by single cameras and targets predicted by multi-camera fusion, avoiding erroneous association problems in multi-target matching, and providing a reliable matching basis for multi-camera data integration.

[0042] For example, the association cost matrix is ​​constructed based on Mahalanobis distance, and the association result is solved using the Kuhn-Munkres (Hungarian) algorithm. The Kuhn-Munkres algorithm is mainly used to solve the optimal matching problem of weighted bipartite graphs and can achieve optimal matching of targets.

[0043] Step 2.4: Determine the unified visual measurement state vector of the target based on the association results.

[0044] Here, based on the visual association results, the visual recognition results of all single cameras matching the same physical vehicle are integrated and calibrated. Motion parameters are unified by averaging or removing bias values ​​to obtain a globally unified visual measurement state vector for the intersection. If a vehicle is detected by only a single camera, the recognition result of that single camera is directly used as its visual measurement state vector. This completes the single-modal fusion of multi-camera visual data, eliminates data differences caused by multi-camera viewpoint bias, further reduces detection noise from single cameras, and obtains a standardized visual measurement state vector, providing a unified input for subsequent radar vision cross-modal fusion.

[0045] For example, the fusion of four cameras at an intersection is achieved by calculating the Mahalanobis distance between the visually recognized target and the visually fused target, resulting in a unified visual recognition result. This unified result is the visual measurement state vector required for radar visual fusion.

[0046] S106: Target association is performed based on the target Mahalanobis distance and target binding ID determined by the effective radar data and the visual measurement state vector, respectively. The fused target state data is obtained by fusion tracking through adaptive Kalman filtering.

[0047] Here, the initial target association between radar effective data and visual measurement state vector is first completed using Mahalanobis distance and Hungarian algorithm. Then, the initial association result is corrected a second time using the original sensor ID. Based on the corrected final association result, adaptive Kalman filter fusion tracking is performed to output fused target state data. This can solve the error problem of cross-sensor target matching. Combining the dual advantages of accurate radar velocity measurement and accurate camera target recognition, it eliminates the measurement noise of a single sensor and obtains smooth and accurate fused target state data, providing core state support for trajectory extraction.

[0048] In some embodiments, the step of associating the target with the target Mahalanobis distance and the target binding ID in step S106 can be implemented through the following process, including: Step 3.1: Integrate the effective radar data and visual measurement state vectors into a global target set for intersection detection. Here, the preprocessed radar data and the visual measurement state vector obtained in the aforementioned embodiments are unified in data format. The vehicle position, speed, acceleration, and original radar ID in the radar data are standardized and aligned with the vehicle position, heading angle, speed, acceleration, and original camera ID in the visual measurement state vector according to the dimensions of position, speed, and timestamp. This is integrated into a global intersection device detection target set that includes radar-detected targets and visual fusion targets. This achieves cross-sensor data integration between radar and vision, solves the problem of inconsistent data formats among multiple sensors, and constructs a unified detection target set for radar-visual global fusion, ensuring the uniformity of subsequent correlation calculations.

[0049] For example, the visual fusion target needs to be fused with 8 millimeter-wave radars at the intersection. Before fusion, the detection data of the two need to be formatted to form a global set of detected targets, which serves as the basis for cross-sensor target association.

[0050] Step 3.2: Calculate the Mahalanobis distance, which characterizes the similarity between each vehicle target in the target set detected by the global equipment at the intersection and the predicted target from the global fusion of radar vision; Among them, the prediction target of radar vision global fusion is the prediction value of global fusion result for vehicle target in the current frame, obtained by using a uniform acceleration model and Kalman filter prediction step to perform state prediction on the global fusion result for vehicle target in the previous frame of the global device. Here, the state data after global fusion of radar vision from the previous frame is retrieved. Through uniform acceleration model and Kalman filter prediction step extrapolation, the predicted target for the current frame's global fusion of radar vision is generated. Using this predicted target as a benchmark, the cross-sensor Mahalanobis distance between each vehicle target in the intersection's global detection target set is calculated, quantifying the similarity between cross-sensor targets. The use of cross-sensor Mahalanobis distance adapts to the differences in data characteristics between radar and vision sensors, enabling more accurate measurement of the similarity between targets from different sensors. The predicted target based on the global fusion result of the previous frame ensures the temporal continuity of cross-sensor matching.

[0051] For example, target association involves matching the targets identified by all devices in the current frame with the results of fusion tracking prediction, and using Mahalanobis distance to calculate target similarity; the radar vision global fusion prediction target is obtained by performing state prediction on the global fusion result of the previous frame using a uniform acceleration model and a Kalman filter prediction step.

[0052] Step 3.3: Construct the radar vision global association cost matrix using the Mahalanobis distance between each sensor as matrix elements, and solve the preliminary association results of the radar vision global association using the Hungarian algorithm. Here, the Mahalanobis distances across each sensor are used as matrix elements to construct a global radar vision association cost matrix. This cost matrix is ​​then solved using the Hungarian algorithm to obtain the optimal matching preliminary global radar vision association result, clarifying the matching relationship between globally detected targets and globally fused predicted targets. The Hungarian algorithm achieves optimal preliminary matching of cross-sensor targets; it boasts high solution efficiency and accurate matching results, providing fundamental matching results for global radar vision target association and representing a core step in cross-sensor fusion.

[0053] For example, the association cost matrix is ​​constructed based on Mahalanobis distance and the association result is solved using the KuhnMunkres (Hungarian) algorithm. This method is also applicable to cross-sensor global target association between radar and vision.

[0054] Step 3.4: Based on the preliminary association results, determine the radar-detected targets, multi-camera fusion predicted targets, and radar vision global fusion predicted targets that have matching relationships. Perform a unique match between the original sensor IDs of the radar-detected targets and multi-camera fusion targets in the current frame and the original sensor IDs of the radar vision global fusion predicted targets bound to them in the previous frame. Correct the preliminary association results by eliminating the association results that fail to match and re-matching them.

[0055] Here, the original sensor IDs of matching radar / visual targets are extracted from the preliminary association results and uniquely matched with the original sensor IDs of the targets bound to the global fusion prediction targets in the previous frame. Association results with ID mismatches are removed, and the Hungarian algorithm is re-executed for the removed unmatched targets to obtain the corrected final association results. This effectively solves the association error problem caused by radar target splitting, misidentification, and sensor data frame deviation in Mahalanobis distance association, significantly improves the accuracy of cross-sensor target association, and provides a reliable basis for subsequent Kalman filter fusion tracking.

[0056] For example, the original sensor tracking ID of the previous frame is retained in each successfully fused target, and the new frame can be quickly matched based on the ID. Since the association based on Mahalanobis distance has association errors caused by radar target splitting, misidentification, etc., adding the ID association method can significantly reduce such errors.

[0057] Specifically, this embodiment can be implemented based on the fusion data of 8 millimeter-wave radars and 4 cameras deployed at a traffic intersection. First, the effective data from the 8 millimeter-wave radars and the visual measurement state vector obtained by fusing the data from the 4 cameras are standardized in data format. The vehicle position, speed, acceleration, and original radar ID in the effective radar data are standardized and aligned with the vehicle position, heading angle, speed, acceleration, and original camera ID in the visual measurement state vector according to the dimensions of position, speed, and timestamp, and integrated into a global target set for intersection detection. Second, the state data after global fusion of radar vision in the previous frame is retrieved and substituted into the state transition matrix of the uniform acceleration model and the Kalman filter prediction formula to deduce and generate the radar vision global fusion prediction target for the current frame. Then, the cross-sensor Mahalanobis distance between each vehicle target in the global target set and the predicted target is calculated to obtain the Mahalanobis distance matrix. Then, this Mahalanobis distance matrix is ​​used as the radar vision global association cost matrix and solved using the Hungarian algorithm to obtain the preliminary association result of the radar vision global set. This result clarifies the matching relationship between the targets in the global target set and the radar vision global fusion prediction targets. Finally, based on the preliminary association results, the original sensor IDs of the radar-detected targets and multi-camera fusion targets in the current frame are extracted, and the original sensor IDs of the radar vision global fusion predicted targets in the previous frame are also extracted. Uniqueness matching is performed on the two sets of IDs. If the original sensor ID of the target in the current frame is inconsistent with the ID bound in the previous frame, the association result is removed. For the radar / vision targets that are not matched after removal and the radar vision global fusion predicted targets that are not matched, the Hungarian algorithm is re-executed for matching, and finally the corrected radar vision global target association results are obtained.

[0058] In practical applications, step S106, which involves obtaining the fused target state data through adaptive Kalman filtering during fusion tracking, can be further refined to include: Step 4.1: Based on the association type of the corrected association results, determine the measurement state vector of the Kalman filter corresponding to the current frame; Here, based on the corrected final association result obtained from the steps of the aforementioned embodiments, the target association type is determined to be radar-only association, vision-only association, or radar and vision association. The radar measurement vector, vision measurement state vector, or radar and vision measurement vector are respectively determined as the measurement state vector for the current frame's Kalman filter, where the radar measurement vector is the average of all associated radar measurements. This achieves accurate matching between the measurement state vector and the association type, providing suitable sensor measurement input for the Kalman filter update step, ensuring the targeted nature of the filter fusion, and adapting to the full-scene requirements of radar-vision cross-modal fusion.

[0059] For example, there are three types of association results for radar and vision fusion: association only with targets identified by millimeter-wave radar, association only with targets identified by vision, and association with both millimeter-wave radar and targets identified by vision. When only radar is associated, the measurement vector is the average value of all millimeter-wave radar measurements. When only vision is associated, it is the output result of vision fusion. When both are associated, it includes the average value of radar measurements and the result of vision fusion.

[0060] Step 4.2: Process the fused target state data of the previous frame using the state transition matrix of the uniform acceleration model and the Kalman filter prediction formula to obtain the prior prediction state of the current frame. Here, the fused target state data and corresponding state covariance matrix from the previous frame are retrieved and substituted into the state transition matrix of the uniform acceleration model and the Kalman filter prediction formula. (3) Perform calculations to obtain the prior prediction state of the current frame. ,in is the prior state vector from the previous moment, and F is the state transition matrix of the uniformly accelerated model. It is the prior error covariance matrix. It is the Kalman filter state covariance matrix of the previous time step. This is the process noise covariance matrix, which requires manual initialization. The uniform acceleration model fits the short-term motion characteristics of vehicles at urban intersections. It uses a Kalman filter prediction step to predict the next frame based on the state of the previous frame, ensuring the temporal continuity of vehicle target tracking and providing prior reference data for the update step.

[0061] For example, the target tracking module uses a uniform acceleration model and a Kalman filter to track the fused target; the Kalman filter prediction step predicts the next frame based on the state of the previous frame. The state prediction process is formula (3), where F is the state transition matrix of the uniform acceleration model and Q is the process noise covariance matrix, and the initial value needs to be set manually.

[0062] Step 4.3: Based on the measurement state vector and measurement covariance matrix corresponding to the association type, perform one or two updates to fuse the prior predicted state with the measurement state vector to obtain the fused target state data of the current frame. Here, based on the measurement state vector corresponding to the association type and the specific measurement covariance matrix ( It is the measurement covariance matrix of millimeter-wave radar. (This refers to the camera's measurement covariance matrix). The Kalman gain is calculated, and one or two Kalman filter updates are performed. The prior predicted state is fused with the sensor's measured state vector to obtain fused target state data in the current frame, containing the fused state vector and covariance matrix. An adaptive one / two-update strategy achieves accurate fusion of radar visual data, eliminating measurement noise from a single sensor and obtaining smooth, accurate fused target state data, providing core data for trajectory extraction.

[0063] For example, the Kalman filter update step has different calculation processes depending on the three association results: a single update is performed when only radar / only visual association is performed, and two updates are performed when there is dual association; after Kalman filtering, relatively smooth target position, velocity and acceleration information can be obtained.

[0064] Step 4.4: Take the fused target state data of the current frame as the new fused target state data of the previous frame, and process the fused target state data of the previous frame again using the state transition matrix of the uniform acceleration model and the Kalman filter prediction formula to obtain the predicted target of the radar vision global fusion in the next frame.

[0065] Here, the fused target state data of the current frame is used as the new fused target state data of the previous frame. This data is then substituted into the state transition matrix of the uniform acceleration model and the Kalman filter prediction formula to perform calculations, obtaining the radar vision global fusion prediction target for the next frame. This prediction serves as the benchmark for cross-sensor target association in the next frame. This achieves iterative closed-loop Kalman filtering, ensuring the continuity of vehicle target tracking and providing a unified prediction benchmark for cross-sensor target association in the next frame, ensuring that the association always has a temporal reference.

[0066] For example, the core of the Kalman filter prediction step is to predict the state of the next frame based on the state of the previous frame. By using the target state data after fusion of the current frame as the new data of the previous frame, the radar vision global fusion prediction target of the next frame can be deduced, providing a matching benchmark for target association in the next frame.

[0067] Specifically, this embodiment is implemented based on three correlation results of radar visual global fusion, and the Kalman filter state vector (fused target state data) is set as follows. ,in , For the target location, For heading angle, , For speed, Angular velocity, and It is acceleration. The measurement vector is... Each tracked target may be associated with multiple identified targets; here, the measurement vector is the average of the measurements of all associated targets. The state transition matrix of the uniformly accelerated model is: (1) The measurement matrix is: (2) First, based on the corrected final association results, the association type of the target is determined. If it is radar-only association, the average of all associated millimeter-wave radar measurements is taken as the radar measurement vector, which is then used as the measurement state vector for the Kalman filter. If it is vision-only association, the vision measurement state vector is used as the measurement state vector for the Kalman filter. If it is both radar and vision association, both the radar measurement vector and the vision measurement state vector are used together as the measurement state vector for the Kalman filter. Second, the fused target state data and the corresponding state covariance matrix from the previous frame are retrieved and substituted into the state transition matrix of the uniform acceleration model and the Kalman filter prediction formula. (3) Process it to obtain the prior prediction state of the current frame, which includes the prior state vector. and prior state covariance matrix Here, Q is the process noise covariance matrix, with an initial value manually set based on engineering experience. Then, based on the measurement state vector and measurement covariance matrix corresponding to the association type, one or two Kalman filter updates are performed. The prior predicted state is fused with the sensor-measured state vector to calculate the fused target state data for the current frame, including the fused state vector and fused state covariance matrix. Finally, the fused target state data of the current frame is used as the new fused target state data of the previous frame. It is then substituted again into the state transition matrix of the uniform acceleration model and the Kalman filter prediction formula to perform the Kalman prediction step, obtaining the radar visual global fusion prediction target for the next frame. This prediction target is used as the benchmark for radar visual global target association in the next frame.

[0068] In some embodiments, the step of performing a single or two updates based on the measurement state vector and measurement covariance matrix corresponding to the association type includes: Step 5.1: If the corrected association result is only associated with a single radar-identified target or a visually identified target, then perform a single Kalman filter update. Here, if the corrected association result is a single association type (radar only / vision only), the Kalman gain is calculated once based on the measurement state vector and specific measurement covariance matrix of the corresponding sensor. Then, the prior predicted state and the single-sensor measured data are fused according to the Kalman filter update formula, and a single Kalman filter update is performed. This provides an adapted filtering update strategy for single-sensor association scenarios. The standardized single update process ensures the robustness of the filtering calculation and can effectively fuse single-sensor measured data and prior predicted state, improving the accuracy of fused data in single-sensor scenarios.

[0069] For example, when the association result is only associated with the target identified by millimeter-wave radar, the fused tracking target may be associated with the identification results of multiple millimeter-wave radars, and the measurement vector The average of all millimeter-wave radar measurements; the correlation result is the measurement vector when associated only with visually recognized targets. It is the output result of visual fusion; The Kalman filter update step is consistent with formula (4) (Kalman gain calculation) and formula (5) (state update process), that is, a single Kalman filter update is performed, as follows: During Kalman filter updates, the Kalman gain is first calculated, and then the updated state is calculated based on the predicted prior state and the current frame measurement. The Kalman gain is: (4) in, It is the vehicle motion fusion state vector of the previous frame, which is the accurate state after Kalman filtering update; It is the state covariance matrix of the previous frame, and it is the error matrix after Kalman filtering. It is the transpose of the state transition matrix F. Here, H is the measurement covariance matrix, and H is the measurement matrix. The Kalman filter update process is as follows: (5) in, It is the prior state vector at the current moment. This is the Kalman filter state covariance matrix at the current moment. After Kalman filtering, relatively smooth target position, velocity, and acceleration information can be obtained. The target heading angle is calculated using velocity. Since there is no velocity data in the first frame after the target is detected, the heading angle cannot be calculated. Therefore, starting from the second frame, the target heading angle is: (6) Step 5.2: If the corrected association result is associated with both radar-identified targets and visually identified targets, then perform two Kalman filter updates sequentially based on the measurement covariance matrices of radar and vision, respectively.

[0070] Here, if the corrected association result is a dual association type involving radar and vision, the Kalman gain is first calculated based on the radar measurement state vector and radar measurement covariance matrix, and a first Kalman update is performed. Then, based on the radar-updated state, the Kalman gain is calculated based on the vision measurement state vector and camera measurement covariance matrix, and a second Kalman update is performed. This completes the sequential fusion of the prior predicted state and the measured data from both sensors. By fully combining the dual advantages of accurate radar velocity measurement and accurate camera position and heading angle detection, the two sequential updates allow the advantageous data from both sensors to participate in the fusion, significantly improving the accuracy of the fused target state data compared to a single update.

[0071] For example, when the association result is associated with millimeter-wave radar and visual recognition targets, the update step updates twice based on radar and visual measurement vectors. First, the gain is calculated and updated using the radar measurement vector combined with the radar measurement covariance matrix (Equations 7 and 8). Then, the gain is calculated and updated using the visual measurement vector combined with the camera measurement covariance matrix (Equations 9 and 10), as detailed below: Update step based on and Updated twice, first use Update status, Kalman gain based on millimeter-wave radar measurements for: (7) in This is the measurement covariance matrix of the millimeter-wave radar, and the update process is as follows: (8) in This is the state vector after the first update following radar measurements. This is the prior predicted state vector for the current frame. This is the error covariance matrix after radar update.

[0072] Reuse Update state, Kalman gain based on visual measurement for: (9) in This is the camera's measurement covariance matrix, and the update process is as follows: (10) For example, this embodiment is based on three association results from radar vision global fusion and implemented using the Kalman filter update formula. First, the prior prediction state of the current frame is obtained, including the prior state vector. and prior state covariance matrix Simultaneously, the measurement matrix H is determined, which matches the state transition matrix of the uniform acceleration model. Next, the type of the corrected association result is determined; if it is a single association type involving only radar association, then the radar measurement vector is substituted. and radar measurement covariance matrix According to the formula (4) Calculate the Kalman gain, and then use the formula (5) Perform a single Kalman filter update to obtain the fused target state data; if it is a single association type with only visual association, then substitute the visual measurement state vector. and camera measurement covariance matrix Perform a single Kalman filter update using the same formula as described above.

[0073] Finally, if it is a dual association type involving both radar and vision, then it is first based on the radar measurement vector. and radar measurement covariance matrix According to the formula (7) Calculate the radar Kalman gain, and then use the formula (8) Perform the first Kalman filter update; then use the state vector X after the first update. kr (Intermediate state vector after the first Kalman update by millimeter-wave radar in a radar-visual dual-association scenario) and covariance matrix P kr Based on the intermediate covariance matrix, the visual measurement state vector is substituted. and camera measurement covariance matrix According to the formula (9) Calculate the visual Kalman gain, and finally apply the formula (10) Perform the second Kalman filter update to obtain the final fused target state data.

[0074] S108. Based on the clustering results of the fused target state data, new and disappearance determinations are made for unrelated targets to obtain an effective fused tracking target set after removing invalid targets.

[0075] Here, unassociated detected targets are clustered using the Euclidean distance global nearest neighbor algorithm. Based on the clustering results, new targets are identified and a first preset number of frames (e.g., 3 frames) is set for valid tracking verification. For tracked targets that fail to associate, a timing determination is made and a second preset number of frames (e.g., 5 frames) is set for disappearance if they are not associated. Invalid candidate new targets and disappeared targets are removed to obtain an effective fused tracking target set. This is done to avoid false detection, missed detection, and invalid tracking, to ensure the effectiveness and accuracy of the data in the fused tracking target set, and to reduce the error rate of subsequent trajectory extraction.

[0076] For example, all detected targets that are not correctly associated are clustered using a global nearest neighbor algorithm based on the Euclidean distance of the target location. Each cluster center is a new target and a new tracking target is created. If a new target is correctly tracked for 3 consecutive frames, it is considered to exist; otherwise, it is deleted. If a tracking target fails to be associated with a detected target for 5 consecutive frames, it is considered to have disappeared and is deleted.

[0077] It should be noted that detected targets that fail to be successfully associated with the tracked target may be newly generated targets, which may be identified by multiple sensors. The clustering method employs a global nearest neighbor algorithm based on Euclidean distance from the target's location. The center of each cluster is considered a newly generated target, and a new tracking target is created. The measurement value of the newly generated target is the average of all sensor measurements. If a newly generated target is correctly tracked for three consecutive frames, it is considered to exist and continues tracking; otherwise, the target is deleted. For tracked targets that fail to be associated with the detected target for five consecutive frames, the target is considered to have disappeared and is deleted.

[0078] S110: Based on the target ID, extract preliminary trajectory data from the effective fusion tracking target set, filter abnormal interrupted trajectories within the intersection ROI and predict them, match the predicted trajectory with the new trajectory and then interpolate to complete it, and obtain the optimized vehicle driving trajectory.

[0079] Here, preliminary trajectory data is extracted from the effective fusion tracking target set based on the unique ID of the fusion target. Abnormal interrupted trajectories within the intersection ROI are screened out and trajectory prediction is performed using a uniform acceleration motion model. The predicted trajectory is matched with the new trajectory according to preset conditions of position, heading angle, and timestamp. For the successfully matched trajectory, the missing frames are filled in using the interpolation method to obtain the optimized complete vehicle driving trajectory. This compensates for the trajectory interruption problem caused by sensor occlusion and missed detection, ensures the spatiotemporal continuity and accuracy of vehicle trajectory, improves the quality of trajectory extraction, and provides reliable intersection vehicle trajectory data for traffic management and vehicle-road cooperation.

[0080] In some embodiments, step S110 may also be implemented by the following steps: Step 6.1: Based on the target ID, filter out the new target trajectory and the tracking interruption trajectory within the intersection ROI from the preliminary trajectory data; Here, the unique ID of the fused target is used as the sole criterion for judgment. The trajectory of the new target corresponding to the valid new target retained by the new target judgment is selected from the preliminary trajectory data. At the same time, the trajectory of the tracking interruption within the intersection ROI caused by sensor occlusion, missed detection, and blind spot is also selected. The key frame status data and timestamp of the relevant trajectory are recorded. The use of the unique ID as the basis ensures the accuracy of trajectory selection, avoids confusion between trajectories of different vehicles, and filters invalid data by only selecting trajectories within the intersection ROI, thereby improving the calculation efficiency of subsequent trajectory optimization.

[0081] For example, the complete running trajectory of the target at the intersection can be obtained from each fused data frame based on the fused target ID; it is necessary to filter out all target trajectories that are interrupted in the ROI of the traffic intersection as well as all newly generated target trajectories; the interrupted trajectory is caused by the millimeter-wave radar missing detection and the obstruction of large vehicles, which is more obvious at the boundary between the radar and camera coverage areas.

[0082] Step 6.2: Based on the fused target state data of the last frame of the interrupted trajectory, a predicted trajectory is obtained through a uniform acceleration motion model to characterize the physical features of the real motion before the interruption. Here, for each intersection ROI with a tracking interruption trajectory, the fused target state data of the last frame is extracted. This data is then substituted into Formula 11 of the uniform acceleration motion model to generate a predicted trajectory. Each frame of the trajectory contains complete motion parameters and a timestamp, and the prediction frame length is controlled to not exceed 30 frames. The fused target state data is smoothed data after radar-visual fusion, which reduces prediction error. The uniform acceleration model fits the short-term motion characteristics of vehicles at intersections, and the 30-frame length limit avoids prediction deviations caused by over-extension. The generated predicted trajectory can accurately represent the true motion trend of the vehicle before the interruption.

[0083] For example, for all targets whose tracking is interrupted within the ROI of the intersection, the state of future frames is predicted using a uniformly accelerated motion model according to Formula 11 based on the motion state vector of the last frame; a maximum of 30 frames are predicted. It has been found through testing that predicting too many frames will reduce the accuracy of the prediction.

[0084] Step 6.3: Associate the predicted trajectory that meets the preset matching conditions with the newly generated trajectory, and use interpolation to fill in the missing frames of the successfully matched trajectory to obtain the complete vehicle driving trajectory.

[0085] Here, the predicted trajectory and the newly generated trajectory are correlated and matched according to preset conditions of position, heading angle, and timestamp. For successfully matched trajectories, based on the state data of the last frame of the interrupted trajectory and the first frame of the newly generated trajectory, interpolation is used to fill in the missing frames between the two trajectories, stitching them together to form a complete vehicle driving trajectory. The preset matching conditions ensure accurate matching between the predicted trajectory and the newly generated trajectory, avoiding incorrect stitching of different vehicle trajectories. The interpolation method effectively compensates for the trajectory interruption problem caused by sensor perception defects, significantly improving the integrity and continuity of the vehicle trajectory.

[0086] For example, this embodiment is implemented based on trajectory data within the ROI of a traffic intersection. First, using the unique ID of the fused target as the sole criterion, trajectory filtering is performed on the obtained preliminary trajectory data. The trajectory corresponding to the valid new target retained after new target determination is selected, and the trajectory of tracking interruption within the ROI of the intersection caused by sensor occlusion, missed detection, or blind spots is also selected. The last frame of fused target state data of each track interruption trajectory and the first frame of state data and timestamp of each new target trajectory are recorded. Second, for each track interruption trajectory within the ROI of the intersection, the fused target state data of its last frame is substituted into the uniform acceleration motion model formula 11 for deduction to generate a predicted trajectory. Each frame of the predicted trajectory contains motion parameters such as vehicle position, heading angle, velocity, and acceleration, and has a unique timestamp. At the same time, the prediction frame length is controlled to not exceed 30 frames to avoid prediction errors caused by over-deduction. Finally, the generated predicted trajectory and the newly generated target trajectory are matched according to preset matching conditions. The preset matching conditions are that the spatial distance between the Nth frame of the predicted trajectory and the first frame of the newly generated target trajectory is ≤2 meters, the absolute value of the heading angle difference is ≤10°, and the timestamps are exactly the same. For predicted trajectories and newly generated target trajectories that meet the preset matching conditions, they are determined to be trajectories of the same vehicle. Then, based on the last frame state data of the interrupted trajectory and the first frame state data of the newly generated trajectory, cubic spline interpolation is used to interpolate and complete the missing frames between the two trajectories, splicing the interrupted trajectory and the newly generated trajectory into a complete vehicle driving trajectory. If the predicted trajectory does not match any newly generated target trajectory, it is determined that the vehicle target is lost, and the trajectory is no longer completed.

[0087] Specifically, based on the fused target ID, the complete running trajectory of the target at the intersection can be obtained from each fused data frame; the target trajectory that is abnormally interrupted within the intersection ROI is predicted and stitched together for optimization, with a maximum of 30 frames predicted. Predicting too many frames will reduce the accuracy; the missing frames of the successfully matched trajectory are filled in using cubic spline interpolation; the multi-target tracking accuracy of this algorithm reaches about 90% on time, and the speed consistency of trajectory extraction is over 80%, and the position consistency is over 90%.

[0088] In some embodiments, the target trajectory extraction and optimization flowchart is as follows: Figure 3 As shown: Based on the fused target ID, the complete trajectory of the target at the intersection can be obtained from each fused data frame. However, due to the limitations of the coverage and viewing angle of millimeter-wave radar and cameras, millimeter-wave radar is prone to missed detections. Large vehicles can easily obscure small targets, causing abnormal interruptions in the tracking trajectory. These problems are more pronounced at the boundaries of the radar and camera coverage areas. To improve the accuracy and completeness of the extracted trajectory, the abnormally interrupted target trajectories within the intersection's ROI are predicted and stitched together for optimization. The main approach is as follows: Filter out all target trajectories that were interrupted within the ROI at the traffic intersection, as well as all newly generated target trajectories.

[0089] For all targets whose tracking is interrupted within the ROI of the intersection, based on their motion state vector in the last frame Using a uniformly accelerated motion model to predict the future The state of the frame is shown in Equation 11, where Generally, a maximum of 30 frames can be predicted. Tests have shown that predicting too many frames will decrease the accuracy of the prediction.

[0090] (11) For each predicted target trajectory, match it with all newly generated target trajectories. If the predicted trajectory's t-th... Distance between the first frame and the first frame of the newly generated target satisfy Meters, the absolute value of the heading angle difference satisfy If the timestamps of the two frames are the same, then the two trajectories are considered to be for the same target; if there is no new target trajectory to match, then the target is considered to have been lost.

[0091] For all trajectories that can be correctly matched, the missing frames are interpolated and filled in using cubic spline interpolation based on the last frame state of the interrupted trajectory and the first frame state of the new trajectory.

[0092] In a preferred embodiment for practical application, based on an urban intersection scenario, firstly, a number of millimeter-wave radars (e.g., 8) are deployed in the road area of ​​the intersection to cover the driving range of each lane. At the traffic light poles at the four corners of the intersection, a number of cameras (e.g., 4) are deployed to cover the core interaction area of ​​the intersection. One millimeter-wave radar is selected as the main sensor, and the core interaction area of ​​the intersection is designated as the Region of Interest (ROI), a rectangular area with a center of the intersection as the origin and dimensions of 50 meters horizontally and vertically. Secondly, the raw perception data collected by the millimeter-wave radar and cameras undergoes standardized preprocessing to obtain effective radar data and effective visual data. Then, based on the effective visual data, single-modal fusion of the four cameras is performed. Mahalanobis distance and the Hungarian algorithm are used to match the target detected by the single camera with the predicted target of the multi-camera fusion, determining a unified visual measurement state vector. Then, the effective radar data and visual measurement state vectors are integrated into a global target set for intersection detection. First, the preliminary global association results are obtained using Mahalanobis distance and the Hungarian algorithm. Then, the original sensor IDs are used to correct the preliminary association results. After obtaining the final association results, adaptive Kalman filtering fusion tracking is performed, outputting fused target state data. Furthermore, unassociated detected targets are clustered using the Euclidean distance global nearest neighbor algorithm, and new target identification is performed. Tracking targets that fail to associate are timed and disappearing targets are identified. After removing invalid targets, a valid fused tracking target set is obtained. Finally, preliminary trajectory data is extracted from the valid fused tracking target set based on the unique ID of the fused target. Abnormal interrupted trajectories within the intersection ROI are screened. Based on the fused target state data of the last frame of the interrupted trajectory, a uniform acceleration motion model is used to predict up to 30 frames to obtain the predicted trajectory. The predicted trajectory is matched with the new trajectory under preset conditions: position ≤ 2 meters, absolute value of heading angle difference ≤ 10°, and same timestamp. For successfully matched trajectories, cubic spline interpolation is used to fill in missing frames, ultimately obtaining the optimized complete vehicle driving trajectory for the city's intersection.

[0093] Figure 2 A flowchart illustrating another method for fusing vehicle trajectories at traffic intersections, provided in an embodiment of the present invention. (Refer to...) Figure 2 The specific steps of this method are as follows: First, raw vehicle perception data from 8 millimeter-wave radars and 4 cameras deployed at the traffic intersection are collected and preprocessed. Data time synchronization between the millimeter-wave radars and cameras is achieved based on data timestamps, and one millimeter-wave radar is selected as the master sensor to ensure time consistency of multi-sensor data. Then, the global geographic coordinates of the multi-sensor data are converted to the local coordinate system of the intersection, while radar targets outside the ROI of the traffic intersection are filtered out, and invalid data is removed to obtain valid radar and visual data with a unified format. During the fusion tracking process, the similarity between the target identified by the current frame sensor and the predicted target of the fusion tracking is calculated using Mahalanobis distance. An association cost matrix is ​​constructed based on Mahalanobis distance, and the optimal target matching result is solved using the Kuhn-Munkres algorithm. Simultaneously, secondary association is performed using the original sensor tracking IDs. The sensor IDs of the previous frame retained by the fused targets are used to quickly match the new frame target, solving the association errors caused by radar target splitting, misidentification, and data frame deviation that easily occur when relying solely on Mahalanobis distance association. Finally, state fusion tracking of the associated targets is performed based on a uniform acceleration motion model and a Kalman filter, divided into a prediction step and an update step: Prediction step: corresponding to... Figure 2 The Kalman filter prediction method, based on the vehicle fusion state and state covariance of the previous frame, uses the state transition matrix of the uniform acceleration model to deduce the prior predicted state of the current frame, providing a basis for state updates. The update step involves differentiating the target association results (association only with millimeter-wave radar, association only with vision, and association with both radar and vision). For single-sensor association (corresponding to label ③), a Kalman filter update is performed once. For dual-sensor association (corresponding to labels ① and ②), updates are first based on radar data and then a second update based on vision data, ultimately obtaining smooth vehicle fusion state data (including position, velocity, acceleration, heading angle, etc.). For detected targets that have not been successfully associated with the tracked target, a global nearest neighbor algorithm based on position Euclidean distance is used for clustering. The cluster center is the new target, and a new tracked target is created. If a new target is correctly tracked for 3 consecutive frames, it is considered a real target; otherwise, it is deleted. For tracked targets that fail to associate with the detected target, if they are not associated with the detected target for 5 consecutive frames, they are considered to have disappeared and the tracked target is deleted. Finally, a set of valid vehicle fusion tracked targets with unique IDs, after removing invalid targets, is output, providing core data for subsequent trajectory extraction.

[0094] In some embodiments, such as Figure 4 As shown, an embodiment of the present invention provides a vehicle trajectory fusion device for traffic intersections, comprising: The preprocessing module standardizes and preprocesses the vehicle perception data acquired by millimeter-wave radar and cameras deployed at traffic intersections to obtain effective radar data and effective visual data. The association module matches the vehicle targets detected by a single camera with the predicted targets fused by multiple cameras based on the visually valid data, and determines the visual measurement state vector through the association results used to characterize the matching relationship. The fusion module performs target association based on the target Mahalanobis distance and target binding ID determined by the radar effective data and the visual measurement state vector, respectively, and obtains fused target state data through adaptive Kalman filtering fusion tracking; The determination module performs new birth and disappearance determination on unrelated targets based on the clustering results of the fused target status data, and obtains a set of effective fused tracking targets after removing invalid targets; The optimization module extracts preliminary trajectory data from the effective fusion tracking target set based on the target ID, filters abnormal interrupted trajectories within the intersection ROI and makes predictions, matches the predicted trajectories with the newly generated trajectories and then interpolates to complete the data, thus obtaining the optimized vehicle driving trajectory.

[0095] The present invention provides an embodiment for implementing an electronic device. In this embodiment, the electronic device may be, but is not limited to, a personal computer (PC), a laptop computer, a monitoring device, a server, or other computer device with analysis and processing capabilities.

[0096] As an exemplary embodiment, see [link to example]. Figure 5 The electronic device 110 includes a communication interface 111, a processor 112, a memory 113, and a bus 114. The processor 112, the communication interface 111, and the memory 113 are connected via the bus 114. The memory 113 is used to store a computer program that supports the processor 112 in executing the above-described method. The processor 112 is configured to execute the program stored in the memory 113.

[0097] The machine-readable storage medium mentioned in this article can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0098] Non-volatile media can be non-volatile memory, flash memory, storage drives (such as hard disk drives), any type of storage disk (such as optical discs, DVDs, etc.), or similar non-volatile storage media, or combinations thereof.

[0099] It is understood that the specific operation methods of each functional module in this embodiment can be referred to the detailed description of the corresponding steps in the above method embodiment, and will not be repeated here.

[0100] The computer-readable storage medium provided in the embodiments of the present invention stores a computer program. When the computer program code is executed, it can implement the method described in any of the above embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0101] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0102] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention.

Claims

1. A method for fusing vehicle trajectories at a traffic intersection, characterized in that, include: Vehicle perception data acquired by millimeter-wave radar and cameras deployed at traffic intersections are standardized and preprocessed to obtain effective radar data and effective visual data. Based on the visually valid data, the vehicle targets detected by a single camera and the predicted targets fused by multiple cameras are matched, and the visual measurement state vector is determined by the association results used to characterize the matching relationship. Target association is performed based on the target Mahalanobis distance and target binding ID determined by the radar effective data and the visual measurement state vector, respectively, and fused target state data is obtained by fusion tracking after adaptive Kalman filtering; Based on the clustering results of the fused target state data, new and disappearance determinations are made for unrelated targets to obtain a set of effective fused tracking targets after removing invalid targets; Preliminary trajectory data is extracted from the effective fusion tracking target set based on the target ID. Abnormal interrupted trajectories within the intersection ROI are screened and predicted. The predicted trajectories are matched with the new trajectories and then interpolated to complete the data, resulting in the optimized vehicle driving trajectory.

2. The method according to claim 1, characterized in that, The steps of matching vehicle targets detected by a single camera with predicted targets fused from multiple cameras based on the visually valid data, and determining the visual measurement state vector through the association results used to characterize the matching relationship, include: Vehicle targets are detected and continuously tracked using visually valid data. The vehicle position is estimated by calibrating the camera through homography transformation, and the motion parameters of the vehicle targets are calculated to obtain the visual recognition results of each single camera for the vehicle targets in the current frame. Based on the visual recognition results and visual fusion results, a Mahalanobis distance is calculated to characterize the similarity between the vehicle target detected by each single camera and the predicted target fused by the multi-camera system; wherein, the visual fusion result is the predicted value of the visual fusion result of the multi-camera system for the vehicle target in the current frame, obtained by using a uniform acceleration model and Kalman filter prediction step to perform state prediction on the visual fusion result of the multi-camera system for the vehicle target in the previous frame. A correlation cost matrix is ​​constructed using each Mahalanobis distance as a matrix element, and the correlation results are solved based on the Hungarian algorithm. Based on the association results, a unified visual measurement state vector for the target is determined.

3. The method according to claim 1, characterized in that, The steps of associating targets based on the target Mahalanobis distance and target binding ID determined by the radar effective data and the visual measurement state vector respectively include: The effective radar data and the visual measurement state vector are integrated into a global target set for intersection detection by the equipment. Calculate the Mahalanobis distance, which characterizes the similarity between each vehicle target in the target set detected by the global device at the intersection and the predicted target of the radar vision global fusion; wherein the predicted target of the radar vision global fusion is the predicted value of the global fusion result for the vehicle target in the current frame, obtained by using a uniform acceleration model and Kalman filter prediction step to perform state prediction on the global fusion result for the vehicle target in the previous frame of the global device. A radar vision global association cost matrix is ​​constructed using the Mahalanobis distance between each sensor as matrix elements, and the preliminary association results of the radar vision global association are obtained by using the Hungarian algorithm. Based on the preliminary association results, radar-detected targets, multi-camera fusion predicted targets, and radar vision global fusion predicted targets with matching relationships are identified. The original sensor IDs of the radar-detected targets and multi-camera fusion targets in the current frame are uniquely matched with the original sensor IDs of the radar vision global fusion predicted targets in the previous frame. The preliminary association results are corrected by eliminating unmatched association results and re-matching them.

4. The method according to claim 1, characterized in that, The steps for obtaining fused target state data through adaptive Kalman filtering and fusion tracking include: Based on the association type of the corrected association results, determine the measurement state vector of the Kalman filter corresponding to the current frame; The previous frame's fused target state data is processed using the state transition matrix of the uniform acceleration model and the Kalman filter prediction formula to obtain the prior predicted state of the current frame. Based on the measurement state vector and measurement covariance matrix corresponding to the association type, perform one or two updates, fusing the prior predicted state with the measurement state vector to obtain the fused target state data for the current frame. The fused target state data of the current frame is used as the new fused target state data of the previous frame. The state transition matrix of the uniform acceleration model and the Kalman filter prediction formula are used to process the fused target state data of the previous frame again to obtain the predicted target of the radar vision global fusion in the next frame.

5. The method according to claim 4, characterized in that, Based on the measurement state vector and measurement covariance matrix corresponding to the association type, the steps of performing one or two updates include: If the corrected association result is only associated with a single radar-identified target or a visually identified target, then a single Kalman filter update is performed; If the corrected association result is associated with both radar-identified and visually-identified targets, then two Kalman filter updates are performed sequentially based on the measurement covariance matrices of radar and vision, respectively.

6. The method according to claim 1, characterized in that, The steps involved in filtering and predicting abnormally interrupted trajectories within the intersection's ROI, matching the predicted trajectories with the newly generated trajectories, and then interpolating to complete the optimized vehicle trajectory include: Based on the target ID, newly generated target trajectories and tracking interrupted trajectories within the intersection ROI are selected from the initial trajectory data. Based on the fused target state data of the last frame of the interrupted trajectory, a predicted trajectory is obtained through a uniformly accelerated motion model to characterize the real physical characteristics of the motion before the interruption. The predicted trajectory that meets the preset matching conditions is associated with the newly generated trajectory, and the missing frames of the successfully matched trajectory are filled in by interpolation to obtain the complete vehicle driving trajectory.

7. The method according to claim 1, characterized in that, The steps involved in standardizing and preprocessing vehicle perception data acquired by millimeter-wave radar and cameras deployed at traffic intersections to obtain effective radar and visual data include: Using millimeter-wave radar deployed in the road area at traffic intersections, vehicle data of vehicles in the road area is acquired, wherein the vehicle data includes at least vehicle position, speed, acceleration and original tracking ID; Using cameras deployed in the intersection area of ​​a traffic intersection, image frame data of vehicles in the intersection area are acquired; The perception data, including the vehicle data and the image frame data, are processed by time synchronization, coordinate transformation, and ROI range filtering to obtain effective radar data and effective visual data.

8. A device for fusing vehicle trajectories at a traffic intersection, characterized in that, include: The preprocessing module standardizes and preprocesses the vehicle perception data acquired by millimeter-wave radar and cameras deployed at traffic intersections to obtain effective radar data and effective visual data. The association module matches the vehicle targets detected by a single camera with the predicted targets fused by multiple cameras based on the visually valid data, and determines the visual measurement state vector through the association results used to characterize the matching relationship. The fusion module performs target association based on the target Mahalanobis distance and target binding ID determined by the radar effective data and the visual measurement state vector, respectively, and obtains fused target state data through adaptive Kalman filtering fusion tracking; The determination module performs new birth and disappearance determination on unrelated targets based on the clustering results of the fused target status data, and obtains a set of effective fused tracking targets after removing invalid targets; The optimization module extracts preliminary trajectory data from the effective fusion tracking target set based on the target ID, filters abnormal interrupted trajectories within the intersection ROI and makes predictions, matches the predicted trajectories with the newly generated trajectories and then interpolates to complete the data, thus obtaining the optimized vehicle driving trajectory.

9. An electronic device, characterized in that, It includes a memory, a processor, and a program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed, implements the method described in any one of claims 1-7.

Images