A highway vehicle trajectory correlation method based on radar and vision fusion

Abstract

This invention relates to a method for highway vehicle trajectory association based on radar-video fusion, comprising the following steps: Step 1, capturing core vehicle data from multiple sources; Step 2, completing missing radar trajectories in overlapping areas; Step 3, spatiotemporally aligning trajectory data; Step 4, fusing radar-video features and matching trajectories. The beneficial effects of this invention are: addressing the problem of insufficient robustness in vehicle trajectory association by proposing a trajectory association method based on radar-video fusion. This method first proposes a forward or reverse trajectory completion strategy to enhance trajectory continuity in cases of trajectory interruptions within overlapping areas. Subsequently, it performs temporal synchronization and spatial alignment processing on sensor data from two adjacent radar-video monitoring points to ensure consistency between data from different sources. Based on this, it extracts and fuses multidimensional features from radar and video, calculates the similarity between trajectories, constructs an association cost matrix, and achieves optimal association between trajectories through a cost-minimization matching method.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation, specifically relating to a method for associating vehicle trajectories on highways based on radar-visual fusion. Background Technology

[0002] Highways, as a crucial component of the national transportation infrastructure, bear a significant burden of passenger and freight transport. With the increasing number of vehicles and rising traffic safety requirements, intelligent transportation has become a key path to solving traffic congestion and improving safety. Intelligent transportation refers to the integration of modern sensing, communication, data processing, and artificial intelligence technologies to intelligently upgrade the transportation system, achieving a comprehensive transportation model that is efficient, safe, and controllable.

[0003] With increasing traffic volume and diversified travel demands, traditional traffic monitoring methods are insufficient to meet the needs for accurate vehicle perception and tracking. As an emerging intelligent traffic perception technology, radar-visual fusion technology combines the precise distance and speed measurement capabilities of millimeter-wave / LiDAR with the rich target recognition information of visual images, providing a more reliable and comprehensive solution for vehicle perception and tracking on highways.

[0004] The principle of laser-visual fusion technology: Millimeter-wave radar uses electromagnetic waves in the millimeter-wave band to detect targets, accurately measuring the distance, speed, and angle of target vehicles. Its advantage lies in its immunity to interference from lighting conditions and adverse weather, enabling stable acquisition of target dynamic parameters. LiDAR emits a laser beam and receives the reflected light to construct a three-dimensional point cloud model, accurately perceiving the vehicle's shape, position, and trajectory. In high-speed, complex road conditions, millimeter-wave radar monitors vehicle speed changes in real time, while lidar identifies the vehicle's outline and position, providing fundamental data for tracking.

[0005] The radar-visual fusion technology integrates millimeter-wave / LiDAR and visual perception information through a data layer, a feature layer, and a decision layer. The data layer directly fuses raw data, the feature layer fuses features from both, and the decision layer synthesizes the results of each layer. For example, the data layer fuses vehicle distance and speed measured by millimeter-wave radar with vehicle type and location identified by cameras, comprehensively and accurately perceiving vehicle status and providing reliable data for tracking.

[0006] Traditional methods for associating vehicle trajectories on highways typically use radar-detected target location data for trajectory correlation. This involves calculating the correlation between trajectories based on radar characteristics. However, because each radar operates relatively independently, even after spatiotemporal alignment, the trajectories of the same target on different radars may not overlap. Even with the introduction of speed information, mismatches can still easily occur in multi-target scenarios. Summary of the Invention

[0007] In summary, to address the challenge of target matching between radar and video data fusion during highway vehicle tracking, this invention aims to provide a highway vehicle trajectory association method based on radar-video fusion.

[0008] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A method for associating highway vehicle trajectories based on radar-visual fusion, comprising the following steps:

[0009] Step 1: Capture key data of the target vehicle through two radar monitoring points to obtain the vehicle's specific location, speed, and precise position coordinates on the image;

[0010] Step 2: Based on the precise position coordinate information obtained in Step 1, the missing trajectory is completed in the radar detection overlap area to obtain continuous vehicle trajectory data.

[0011] Step 3: Based on the continuous vehicle trajectory data obtained in Step 2, perform time synchronization and spatial alignment processing on the target trajectories of the two radar monitoring points to obtain spatiotemporally consistent vehicle trajectory data.

[0012] Step 4: Based on the spatiotemporally consistent vehicle trajectory data obtained in Step 3, the trajectory is correlated through radar and video fusion features to obtain continuous observation results of the vehicle between two radar and video monitoring points.

[0013] Based on the above technical solution, the present invention can be further improved as follows:

[0014] Furthermore, the completion strategy in step 2 adopts a two-way extrapolation method: if the vehicle moves from the monitoring range of the first radar to the monitoring range of the second radar, and the first radar has no target data in the overlapping area, then the trajectory points in the overlapping area are completed by forward extrapolation of the data of the first radar; if the second radar has no detection data in the overlapping area, then the trajectory points in the overlapping area are completed by backward extrapolation of the data of the second radar.

[0015] Furthermore, the time synchronization method described in step 3 is as follows: using the data time of one radar as a reference, the data of another radar is interpolated to eliminate the time difference between different radar data and achieve consistency in the time dimension.

[0016] The spatial alignment method is as follows: first, measure the geodetic latitude, longitude and altitude of each radar to determine the geocentric coordinates of the radar; then, transform the relative Cartesian coordinates of the vehicles detected by the radar to a rectangular coordinate system with the first radar as the origin, so as to achieve spatial unification of target position information of different radars.

[0017] Furthermore, step 4 specifically includes:

[0018] Step 4.1, Radar-Vision Feature Extraction and Heterogeneous Fusion: From the spatiotemporally consistent vehicle trajectory data obtained in Step 3, image features and radar data features are extracted respectively. Then, the two types of features are normalized and preprocessed. The normalized image features and radar features are fused by weighting to obtain the radar-visual fusion features.

[0019] Step 4.2, Data Denoising and Dimension Alignment: Based on the radar-visual fusion features obtained in Step 4.1, clean and dimensionally consistent standardized fusion features are obtained through dimensionality reduction and denoising processing and dimension unification.

[0020] Step 4.3: Based on the standardized fusion features obtained in Step 4.2, the trajectory association cost matrix that can be used for matching is obtained by calculating the similarity between features.

[0021] Furthermore, in step 4.1, image features are quickly extracted from vehicle images using image recognition algorithms; radar data feature extraction involves constructing a six-dimensional tensor from the information of each frame of data, and then integrating it into a feature matrix within a time window t.

[0022] According to claim 4, the method for highway vehicle trajectory association based on radar-visual fusion is characterized in that step 4.3 specifically involves: firstly, calculating the cosine similarity between the standardized fusion features of different radar trajectories to construct a trajectory association cost matrix; then, transforming the trajectory association problem into an optimal selection problem of different elements in the cost matrix: finding a set of elements without repetition in rows and columns in the cost matrix, and the total cost of this set being the minimum, and this optimal matching result corresponding to the trajectory pair of the same vehicle between the two radar-visual monitoring points; finally, connecting the successfully matched trajectory pairs in chronological order to complete the vehicle's motion trajectory segment between the two monitoring points, thereby achieving continuous observation of the vehicle between the two radar-visual monitoring points.

[0023] The beneficial effects of this invention are as follows: Addressing the problem of insufficient robustness in vehicle trajectory association, a trajectory association method based on radar-video fusion is proposed. This method first proposes a forward or reverse trajectory completion strategy to enhance trajectory continuity when trajectory interruptions occur within overlapping areas. Subsequently, sensor data from two adjacent radar-video monitoring points undergo time synchronization and spatial alignment processing to ensure consistency between data from different sources. Based on this, multi-dimensional features from radar and video are extracted and fused, the similarity between trajectories is calculated, an association cost matrix is ​​constructed, and optimal association between trajectories is achieved through a cost-minimization matching method. Attached Figure Description

[0024] Figure 1 This is a flowchart of the present invention;

[0025] Figure 2 A schematic diagram for trajectory completion;

[0026] Figure 3 A graph showing the data from the two radars on the time axis;

[0027] Figure 4 Here is a flowchart of the image feature extraction process;

[0028] Figure 5 This is a flowchart of trajectory association. Detailed Implementation

[0029] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0030] like Figure 1 As shown, a method for associating highway vehicle trajectories based on radar-visual fusion includes the following steps:

[0031] Step 1: Capture key data of the target vehicle through two radar monitoring points to obtain the vehicle's specific location, speed, and precise position coordinates on the image.

[0032] Leveraging the hardware foundation of radar-visual fusion technology, two adjacent radar-visual monitoring points utilize the advantages of radar and visual sensors respectively. Millimeter-wave / LiDAR is responsible for accurately collecting vehicle position (Cartesian coordinates relative to radar) and speed data, unaffected by lighting conditions or inclement weather; the camera captures visual images of the vehicle, then extracts precise position coordinates from the images, providing basic data support for subsequent feature fusion and enabling preliminary perception of multi-dimensional information about the vehicle target.

[0033] Step 2: Based on the precise position coordinate information obtained in Step 1, the missing trajectory is completed in the radar detection overlap area to obtain continuous vehicle trajectory data.

[0034] Unlike radar-visual fusion at a single monitoring point, overlapping radar trajectory points are few and located near radar detection limits or blind spots, resulting in weak signals, high noise, or even no trajectory points. Directly associating trajectories based on trajectory similarity is prone to failure and can lead to trajectory discontinuities and reduced tracking performance. Therefore, missing trajectories need to be filled in before trajectory association.

[0035] The completion strategy adopts a two-way deduction approach: such as Figure 2 As shown, if the measured object, i.e., the vehicle, moves from the monitoring range of the first radar to the monitoring range of the adjacent second radar, and the first radar does not generate target data in the overlapping area, then the data from the first radar needs to be forward-engineered to complete the trajectory data points within the overlapping area. If the second radar does not detect target data in the overlapping area, then its data needs to be backward-engineered. Through this forward or reverse trajectory completion strategy, the problem of trajectory interruption in the overlapping area is compensated, the continuity of the trajectory is enhanced, and a complete data foundation is laid for subsequent spatiotemporal alignment and trajectory association.

[0036] Step 3: Based on the continuous vehicle trajectory data obtained in Step 2, perform time synchronization and spatial alignment processing on the target trajectories of the two radar monitoring points to obtain spatiotemporally consistent vehicle trajectory data.

[0037] In the radar-visual fusion vehicle tracking process, each radar unit operates independently, resulting in differences in the timestamps of the target data they collect. Therefore, when associating vehicle trajectories detected by different radars, it is essential to synchronize the radar data in time. The time synchronization method involves using the data time of one radar as a reference and interpolating the data from another radar to eliminate the time difference between the different radar data and achieve consistency in the time dimension. Specifically:

[0038] Assume the data acquisition intervals of the two radars are as follows: and , The data from the two radars are then represented on the time axis as follows: Figure 3 As shown. Using the data time of the second radar as a reference, the data from the first radar is interpolated. Assume the data sampling time of the first radar is... The data sampling time of the second radar is To obtain the data from the first radar at time j, the data collected by the first radar before and after that time are... and Interpolation is performed to obtain the radar data at time j. The interpolation method is shown in equation (1).

[0039] (1)

[0040] Because different radars are installed in different locations, their output data use independent coordinate systems. To effectively correlate targets from different radars, the target position information output by each radar must be transformed and unified into the same global coordinate system, i.e., spatial alignment. The spatial alignment method is as follows: First, measure the geodetic latitude, longitude, latitude, and altitude of each radar to determine its geocentric coordinates; then, transform the relative Cartesian coordinates of the vehicles detected by the radars to a Cartesian coordinate system with the first radar as the origin, thus achieving spatial unification of the target position information from different radars. Specifically:

[0041] After the radar installation was completed, its geodetic latitude, longitude, and altitude were measured to obtain its coordinates. These represent the radar's longitude, latitude, and altitude, respectively. After detecting a vehicle, the radar outputs the Cartesian coordinates of the vehicle's relative position to the radar. If the geodetic coordinates of the first installed radar are... The geocentric coordinates are The vehicle target's coordinates in a Cartesian coordinate system with this radar as the origin are... The calculation method is shown in equation (2).

[0042]

[0043] in, The radius of the ellipsoid is the closed loop formed by the intersection of the plane passing through the normal and running east-west with the ellipsoid in the horizontal coordinate system. It is the first eccentricity of the Earth, with a value of 0.0167.

[0044] Step 4: Based on the spatiotemporally consistent vehicle trajectory data obtained in Step 3, the trajectory is correlated through radar and video fusion features to obtain continuous observation results of the vehicle between two radar and video monitoring points.

[0045] Step 4.1, Radar-Vision Feature Extraction and Heterogeneous Fusion: From the spatiotemporally consistent vehicle trajectory data obtained in Step 3, image features and radar data features are extracted respectively. Then, the two types of features are preprocessed by normalization. The normalized image features and radar features are then fused using a weighted method to obtain the radar-visual fusion feature. Specifically, as follows:

[0046] Image Feature Extraction: Vehicle images were obtained using image recognition algorithms. The following methods were then used to quickly extract vehicle image features (the process is as follows). Figure 4 (As shown).

[0047] Radar data feature extraction: From the target vehicle data detected by radar, the coordinates and velocity information can be obtained. Therefore, the radar target in each frame of data can be represented as a six-dimensional tensor. Assuming that the radar detects N frames of data within a time window t in the overlapping area, then the characteristics of the radar are... , can be represented as a matrix, as shown in equation (3).

[0048] (3)

[0049] Image feature and radar feature fusion: Radar data and image feature data come from different sensors. In order to fuse the information extracted from the image features with the object features extracted from the radar, the data needs to be preprocessed. Formula (3) is used to first normalize the features of the radar and the image, and then the normalized image feature information is weighted and the fused features are output. .

[0050] Step 4.2, Data Denoising and Dimension Alignment: Based on the radar-visual fusion features obtained in Step 4.1, clean and dimensionally consistent standardized fusion features are obtained through dimensionality reduction and denoising processing and dimension unification.

[0051] In the detection overlap region, the target is at the sensor's limit detection position, resulting in decreased sensor detection performance and thus, the data often contains significant noise. To reduce data noise and calculate the similarity between feature matrices of different sizes, it is necessary to reduce the dimensionality of the features, obtaining the dimensionality-reduced feature matrix, i.e., the standardized fused features. .

[0052] Step 4.3: Based on the standardized fusion features obtained in Step 4.2, the trajectory association cost matrix that can be used for matching is obtained by calculating the similarity between features. Through the preceding steps, the radar-visual fusion features of the trajectories within the detection overlap area are obtained. The cosine similarity of the trajectory features of different radars is calculated pairwise to construct a cost matrix. The trajectory association problem can be transformed into... In the matrix, find elements that are distinct in both rows and columns, maximizing the correlation of the overall trajectories. Using... Figure 5 The steps for calculating trajectory association are as follows:

[0053] First, the cosine similarity between the standardized fused features of different radar trajectories is calculated to construct the trajectory association cost matrix. The trajectory association problem is then transformed into an optimal selection problem for elements in a cost matrix that are distinct in each row and column: In the cost matrix... Find a set of elements without row or column repetitions, and find the combination with the minimum total cost. This optimal matching result corresponds to the trajectory pair of the same vehicle between the two radar monitoring points. Finally, connect the successfully matched trajectory pairs in chronological order to complete the vehicle's motion trajectory segment between the two monitoring points, thereby achieving continuous observation of the vehicle between the two radar monitoring points.

[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for associating highway vehicle trajectories based on radar-visual fusion, characterized in that, Includes the following steps: Step 1: Capture key data of the target vehicle through two radar monitoring points to obtain the vehicle's specific location, speed, and precise position coordinates on the image; Step 2: Based on the precise position coordinate information obtained in Step 1, the missing trajectory is completed in the radar detection overlap area to obtain continuous vehicle trajectory data. Step 3: Based on the continuous vehicle trajectory data obtained in Step 2, perform time synchronization and spatial alignment processing on the target trajectories of the two radar monitoring points to obtain spatiotemporally consistent vehicle trajectory data. Step 4: Based on the spatiotemporally consistent vehicle trajectory data obtained in Step 3, the trajectory is correlated through radar and video fusion features to obtain continuous observation results of the vehicle between two radar and video monitoring points.

2. The highway vehicle trajectory association method based on radar-visual fusion according to claim 1, characterized in that, In step 2, the completion strategy adopts a two-way extrapolation method: if the vehicle moves from the monitoring range of the first radar to the monitoring range of the second radar, and the first radar has no target data in the overlapping area, then the trajectory points in the overlapping area are completed by forward extrapolation of the data of the first radar; if the second radar has no detection data in the overlapping area, then the trajectory points in the overlapping area are completed by backward extrapolation of the data of the second radar.

3. The highway vehicle trajectory association method based on radar-visual fusion according to claim 1, characterized in that, The time synchronization method described in step 3 is as follows: using the data time of one radar as a reference, the data of another radar is interpolated to eliminate the time difference between different radar data and achieve consistency in the time dimension. The spatial alignment method is as follows: first, measure the geodetic latitude, longitude and altitude of each radar to determine the geocentric coordinates of the radar; then, transform the relative Cartesian coordinates of the vehicles detected by the radar to a rectangular coordinate system with the first radar as the origin, so as to achieve spatial unification of target position information of different radars.

4. The highway vehicle trajectory association method based on radar-visual fusion according to claim 1, characterized in that, Step 4 specifically includes: Step 4.1, Radar-Vision Feature Extraction and Heterogeneous Fusion: From the spatiotemporally consistent vehicle trajectory data obtained in Step 3, image features and radar data features are extracted respectively. Then, the two types of features are normalized and preprocessed. The normalized image features and radar features are fused by weighting to obtain the radar-visual fusion features. Step 4.2, Data Denoising and Dimension Alignment: Based on the radar-visual fusion features obtained in Step 4.1, clean and dimensionally consistent standardized fusion features are obtained through dimensionality reduction and denoising processing and dimension unification. Step 4.3: Based on the standardized fusion features obtained in Step 4.2, the trajectory association cost matrix that can be used for matching is obtained by calculating the similarity between features.

5. The highway vehicle trajectory association method based on radar-visual fusion according to claim 4, characterized in that, In step 4.1, image features are quickly extracted from vehicle images using image recognition algorithms; radar data feature extraction involves constructing a six-dimensional tensor from the information of each frame of data, and then integrating it into a feature matrix within a time window t.

6. The highway vehicle trajectory association method based on radar-visual fusion according to claim 4, characterized in that, Step 4.3 specifically involves: First, calculating the cosine similarity between the standardized fusion features of different radar trajectories to construct a trajectory association cost matrix; then, transforming the trajectory association problem into an optimal selection problem of elements with distinct rows and columns in the cost matrix: finding a set of elements without row or column repetitions in the cost matrix, and minimizing the total cost of this set. This optimal matching result corresponds to the trajectory pair belonging to the same vehicle between the two radar monitoring points; finally, connecting the successfully matched trajectory pairs in chronological order to complete the vehicle's motion trajectory segment between the two monitoring points, thereby achieving continuous observation of the vehicle between the two radar monitoring points.

Images