A method for multi-vehicle trajectory acquisition based on data fusion of roadside millimeter-wave radars and video cameras

Abstract

Multi-vehicle trajectory acquisition based on data fusion using at least two video cameras and at least two millimeter-wave radars. Once the calibration of the video cameras, target detection bounding

Description

Technical Field The invention belongs to the technical field of vehicle target detection, multi-sensor data fusion, vehicle tracking, and re-identification. In particular, the present invention relates to a method to track and re-identify multi-target vehicles based on fusion of the video camera data and millimeter-wave radar data. Technological Background In recent years, the rapid growth of vehicles has put increased pressure on traffic management. As surveillance video cameras take on an important role in public safety, traffic-related tasks based on roadside video cameras such as vehicle target detection, vehicle classification, and vehicle tracking are of increasing interest. How to judge whether the target detection results in different frames of a singlevideo camera or the images of multiple video cameras belonging to the same vehicle, has become an important problem for traffic management. The vehicle re-identification task refers to determining whether the vehicle data captured by distributed roadside sensors (such as video cameras, millimeter-wave radar, etc.) in different moments or non-overlapping areas belong to the same vehicle. This task is of great importance to regional traffic management, security, and vehicle-infrastructure cooperation. Currently, existing vehicle reidentification methods are divided into four main categories: (1) Methods based on the sensors: Using various sensors to detect and identify a vehicle is the most basic and earliest method of vehicle re-identification. These methods usually extract the vehicle’s features based on various sensors (such as infrared sensors, ultrasonic sensors, etc.) and re-identify the vehicles through feature matching. Induction loops are the most commonly used sensors for data acquisition in traffic scenarios, which can also obtain the vehicle attributes such as speed, volume, and vehicle footprint. Loop-based vehicle reidentification methods use the data perceived by induction loops to estimate travel times to complete the vehicle re-identification task. With the advent of emerging sensor techniques (such as Global Positioning Systems (GPS), Radio Frequency Identification (RFID), and mobile phones), some vehicle re-identification approaches use beacon-based vehicle tracking and monitoring systems. For example, RFID-based vehicle re-identification algorithms are usually used fortoil stations, and GPS-based travel time estimation methods can also be used for solving vehicle re-identification problems. Most of the sensor-based methods need to install a large number of hardware equipment, which limits their widespread use. Moreover, many methods are susceptible to environmental conditions (such as weather conditions, signal strength, traffic congestion, vehicle travel speed, etc.), which will impact the sensitivity of the sensor for vehicle re-identification tasks. (2) Methods based on plate information: With the development of computer vision technology, it has become possible to identify vehicles through the image. At the early stage, the vehicle re-identification methods based on computer vision mainly use vehicle license plate information as the vehicle features, which can be extracted through the license plate positioning, character segmentation, and recognition algorithm. The license plate positioning algorithm mainly uses the greyscale, color, and texture information of the license plate, and the character segmentation and recognition algorithm mainly uses template matching and artificial neural networks. The re-identification accuracy of these methods is high, but their disadvantages are also very obvious: Due to the shooting perspective changing, weather effects, lighting changes, and other factors in the actual traffic environment, the target vehicle license plate information may be unclear, or even difficult to obtain. Therefore, although license plate recognition is the simplest and most direct way to identify different vehicles, the tasks of vehicle re-identification cannot be completed solely relying on the license plate information in many scenarios. (3) Methods based on computer vision: There are other computer-vision-based methods using the non-plate information, which can enhance the stability and accuracy of vehicle re-identification. These methods mainly use the HSV features, LBP features, HOG, and other features of the vehicle image to carry out image feature matching, and then the vehicles can be re-identified through their color, type, vehicle windscreen, etc. These methods based on the non-plate information are highly interpretable, but the re-identification accuracy is still low due to the shooting perspective changing and occlusion. (4) Methods based on machine learning: With the development of artificial intelligence algorithms and deep neural networks in the field of computer vision recently, more and more vehicle re-identification methods based on machine learning have been proposed. These methods mainly use convolutional neural networks (CNN) to obtain multiple region segmentation of the target vehicles and extract regional feature vectors from the multiple region segmentation results. The regional feature vectors can then be fused with the global feature vectors to obtain the appearance feature vectors of the target vehicle, which are used for vehicle re-identification. These methods consider the influence of vehicle posture to make the re-identification result much more accurate, gradually becoming the mainstream method for vehicle re-identification tasks. However, the accuracy of the model is limited by the diversity of the training dataset, which must include images of the vehicles from various angles. In the actual traffic scenario, it is difficult to obtain such a massive dataset of all vehicle images at different angles, and key points of the different vehicle images have to be annotated separately in the collected dataset, resulting in a huge workload. In addition, images of the same vehicle under different viewing angles or lighting conditions may have large intraclass variation, and images of different vehicles with the same model may have a high inter-class similarity. These factors may also interfere with re-identification and then limit the accuracy of vehicle re-identification. All the above vehicle re-identification methods are based on images or video from the roadside video cameras and re-identify multiple vehicles according to their appearances. The flow diagram of the traditional vehicle re-identification method is shown in Figure 1. However, it is very common for different vehicles to be identical in appearance, except for their license plates. In such cases, the false positive rate of vehicle re-identification based on vehicle appearances will increase significantly. The motion features acquired by the millimeter-wave radars can expand the dimension of vehicle attributes based on appearance features. Millimeter-wave radar works in the frequency domain of SO-SOO GHz (with a wavelength of 1 to 10 mm). The wavelength of a millimeter-wave is between microwave and centimeter-wave, therefore it has the advantages of both microwave radar and optoelectronic radar. Compared to centimeter-wave seekers, millimeter-wave seekers are small, light, and have a high spatial resolution; Compared to optical seekers such as infrared seekers and laser seekers, millimeter-wave seekers have a strong ability to penetrate fog, smoke, and dust, thus have all-weather (except heavy rain) perception capability. For this reason, they are widely used in communications, guidance, remote sensing, radio astronomy, and wave spectroscopy. Their main advantages are the following: (1) Small antenna aperture and narrow beam: Millimeter-wave radar is resistant to ground multipath and clutter interference, and has high lateral resolution for near-air targets and high angular resolution for area imaging and target surveillance. Narrow beams bring high anti-jamming performance and high antenna gain, making it easy detection of small targets. (2) Large bandwidth: Millimeter-wave radar has a high information rate and wide spreading spectrum capability. It can use a narrow pulse or broadband FM signal to obtain the detailed structural characteristics of the target clutter and enhance its anti-jamming ability. The millimeter-wave radars with adjacent frequencies are easy to overcome mutual interference, making it easy to get accurate target tracking and identification capability. (3) High Doppler frequency: Millimeter-wave radar is easy to use target Doppler frequency characteristics for target detection. Their penetrating characteristics for dry atmospheric pollution also provide a good detection capability in dust, soot, and dry snow conditions. In summary, video cameras can extract the image features of multi-target vehicles and obtain their positioning data through camera calibration and target detection algorithms; millimeter-wave radar can also obtain the positioning data of multi-target vehicles, and data fusion algorithms can be used to obtain more accurate positioning data of multi-target vehicles. In addition, millimeter-wave radar can accurately perceive the motion information of the vehicles (such as speed and drive direction), which can expand the dimension of vehicle attributes. Combining the video camera data and millimeter-wave radar data can provide a solution to the above vehicle re-identification challenges. Summary of the Invention To clarify the content of the invention more clearly, the technical terms involved are explained as follows: Single-video camera: A single video camera that allows multiple vehicles to enter and exit its visual range. Cross-video camera: Two or more video cameras with different fields of view. Multiple vehicles can exit the visual range of one video camera and enter the visual range of another video camera. Vehicle image features: Vehicle image features include the HSV value, LBP, and HOG of the vehicle image, as well as the vehicle’s color. Vehicle motion features: Vehicle motion features include the longitude and latitude coordinates, speed, and heading angle of the vehicles. HSV: HSV (Hue, Saturation, Value) is a color space defined by A R. Smith in 1978, also known as the hexagonal cone model. The parameters of color in this model are hue (H), saturation (S), and lightness (V). LBP: LBP (Local Binary Pattern) is an effective texture descriptor, which has significant advantages such as rotation invariance and grayscale invariance. Its basic idea is to use the gray value of its central pixel as a threshold value and compare it with its neighborhood to obtain the binary code to express local texture features. HOG: HOG (Histogram of Oriented Gradient) is a feature descriptor used for object detection in computer vision and image processing. The features are extracted by calculating oriented gradient histograms of local regions of an image. Video camera data: The image data obtained by the video cameras. The pixel coordinates and corresponding world coordinates of the detection objects in the image can be obtained after camera calibration. Millimeter-wave radar data: The relative distance, azimuth, speed, and heading angle of the vehicle obtained by the millimeter-wave radar, which can be used to calculate the coordinates in the radar coordinate system. The world coordinates of the vehicle can be also obtained after millimeter-wave radar calibration. RTK: RTK device uses the differential positioning technology based on carrier phase observations, which can provide real-time positioning results and achieve centimeter-level accuracy. RTK devices can be divided into handheld and vehicle-mounted types according to their use. The handheld RTK can be used for single-point positioning, and the vehicle-mounted RTK can be installed on the vehicle for continuous positioning. RTK positioning data: Longitude and latitude coordinates measured by handheld and vehicle-mounted RTK devices. Dataset: A collection of data obtained by a data acquisition device. In this patent, it refers to the positioning data collected by video cameras, millimeter-wave radars, and vehicle-mounted RTK devices. Central computing server: The computer device used to receive, process, and store data obtained by various positioning data acquisition devices, including video cameras and millimeter-wave radars. Spatiotemporal matching: It refers to synchronizing the positioning data collected by different devices for the same target in time and space through the optimization method. World Coordinate System: The world coordinate system in this patent refers to the WGS-84 coordinate system. Pixel coordinate system: The pixel coordinate system represents the coordinates of the pixel in the image. Generally, the pixel in the upper left comer of the image is set as the origin, the right direction is set as the positive x-axis direction, and the lower direction is set as the positive y-axis direction, respectively. Radar coordinate system: The radar coordinate system represents the three-dimensional coordinate system with the installation position of the millimeter-wave radar as the origin. Homography matrix: The matrix which can be used for the coordinate transformation. It can be calculated by selecting corresponding key points (at least 4 pairs of non-collinear points) under different coordinate systems. Target vehicle: The vehicle that enters the field of view of the sensors (video camera and millimeterwave radar). Multi-target vehicle: The collection of the target vehicles containing at least one target vehicle. Bounding box: Bounding box (bb) is a rectangular box in an image for target detection, which is used to frame the detected target outline in the image based on the target detection algorithm. Multi-sensor data fusion algorithm: Multi-sensor data fusion algorithm is used to synthesize multisource data and obtain the best consistent estimate of the object. In this patent, the Kalman filtering algorithm is used for data fusion. Lost vehicle: The vehicle that has appeared within the field of view of the collective devices (video camera, millimeter-wave radar), but not captured by one of the devices currently. Lost vehicle database: Records the features of the last n frames before vehicle loss, including vehicle image and motion features. The value of n can be adjusted based on the effect. Vehicle re-identification system: A system framework used to re-identify lost vehicles. ID Reassignment: Reassign the historical ID to the retrieved vehicle. The purpose of the invention is to improve the accuracy of the vehicle tracking and re-identification method, using at least two video cameras and at least two millimeter-wave radars. The camera-radar fusion avoids the impact of high intra-class variation of the same vehicle and high inter-class similarity of the different vehicles due to the camera viewpoint, lighting changes, and other reasons. To achieve the above purpose, the implementation steps of the present invention include: 1) Video camera and millimeter-wave radar calibration. Because the accuracy of the vehiclemounted RTK positioning data Do is much higher than the accuracy of the video camera positioning data Di and millimeter-wave radar positioning data D2, Do is used as the truth value to establish the optimization models for the spatiotemporal matching of the positioning data Di and D2. Based on the optimization models, positioning data Di and D2 can achieve the spatiotemporal synchronization with Do, and then the video camera Si and millimeter-wave radar S2 can be calibrated. 2) Data fusion of the video camera and millimeter-wave radar. Based on the deep learning target detection model, multi-target vehicles in the video camera image can be detected and their image and motion features can be extracted. Using the vehicle-mounted RTK positioning data Do as the truth value, the video camera positioning data Di and millimeter-wave radar positioning data D2 can then be fused to obtain a better estimation of multi-target vehicles’ position. 3) Multi-target vehicle tracking and re-identification. Based on the above fusion data, multiple vehicle targets in different frames of a single-video camera and cross-video cameras can be tracked and re-identified. The present invention mainly involves the following technical issues: 1. RTK differential positioning technology; 2. Video camera calibration technology; 3. Millimeter-wave radar calibration technology; 4. Video-based vehicle detection technology; 5. Multi-sensor data spatiotemporal matching technology; 6. Multi-sensor data fusion technology; 7. Multi-target vehicles tracking technology; 8. Multi-target vehicles re-identification technology; The specific procedure to solve the above technical issues in the present invention is as follows: 1. Calibration: 1) Video camera calibration: Select at least 4 non-collinear points as the key points in the field of view of the video camera. Use a handheld RTK device to obtain the world coordinates of the selected key points, and calculate their pixel coordinates in the camera image. Based on the corresponding coordinates of the key points in the world coordinate system and pixel coordinate system, the homography matrix can be calculated and then the video camera can be calibrated. 2) Target detection bounding box calibration: Drive the experimental vehicle equipped with the RTK device into the video camera’s field of view. Collect the experimental vehicle’s positioning data through the vehicle-mounted RTK device and the video camera respectively over a period (the duration needs to be greater than 2 minutes, during which the experimental vehicle repeatedly appears within the visual field of the video camera). Obtain the bounding box of the experimental vehicle in the camera image using the vehicle detection algorithm (such as Y OLO v5). Take the midpoint at the bottom of the bounding box as the pixel coordinates of the experimental vehicle, and transform it to the world coordinate system based on the homography transformation matrix, which can be used as the video camera positioning coordinates of the experimental vehicle. Establish a spatiotemporal matching optimization model between the vehicle-mounted RTK positioning data Do and video camera positioning data Di, and solve the transformation parameters to complete the calibration of the target detection bounding box. 3) Millimeter-wave radar calibration: Collect the experimental vehicle’s positioning data through a vehicle-mounted RTK device and the millimeter-wave radar over some time (the duration needs to be greater than 2 minutes, during which the experimental vehicle repeatedly appears within the perception range of the millimeter-wave radar). Based on the raw millimeter-wave radar data (including the target’s distance and azimuth relative to the millimeter-wave radar), calculate the relative coordinates of the experimental vehicle in the radar coordinate system as the millimeter-wave radar positioning data of the experimental vehicle. Establish a spatiotemporal matching optimization model between the vehiclemounted RTK positioning data Do and millimeter-wave radar positioning data D2, and solve the transformation parameters to complete the calibration of the millimeter-wave radar. 2. Data acquisition: Use the video cameras to acquire the image data of target vehicles within its field of view. Use the millimeter-wave radars to acquire multi-target vehicle trajectory data of target vehicles within its field of view, including vehicles’ positioning coordinates (in the world coordinate system), speeds, heading angles, etc. Transmit the data obtained by the video cameras and millimeter-wave radars to the central computing server for subsequent calculations. 3. Vehicle detection and feature extraction: After receiving data uploaded from video cameras and millimeter-wave radars, the central computing server performs vehicle detection and feature extraction steps. For the image data from video cameras, the pre-trained deep learning target detection model can be used to detect the vehicles, and then the target detection bounding boxes with a high confidence level (confidence level greater than 0.6) are extracted. Based on the bounding box and vehicle images in the bounding box, the image features (including vehicle color, model, size, license plate information, etc.) and motion features (world coordinates) of each target vehicle can be extracted. For the multi-target vehicle trajectory data from millimeter-wave radars, the motion features (including world coordinates, speed, heading angle, etc.) of the target vehicle can be extracted. 4. Feature matching and data fusion: Use vehicle-mounted RTK positioning data of the experimental vehicle as the truth value, the multi-target vehicle trajectory data and image data can be matched based on the extracted vehicle motion features, and then the video camera positioning data and millimeter-wave radar positioning data can be fused to improve the precision of the positioning data through the multi-sensor data fusion algorithm. 5. Vehicle tracking and re-identification: 1) Vehicle tracking of a single-video camera: Use the pre-trained deep learning target detection model to detect the vehicles in the camera’s field of view, and obtain the target detection bounding boxes in every frame. Use the DeepSort multi-target tracking algorithm to match the target detection bounding boxes of the same target within different frames and assign the IDs to the vehicles. 2) Vehicle re-identification of the cross-video cameras: When a target vehicle Vx becomes lost in the field of view of camera 1, the vehicle re-identification system records its image and motion features before vehicle loss, and transmits the features to the lost vehicle database. When the target vehicle Vy enters the field of view of camera 2, the vehicle re-identification system captures its image and motion features and performs feature matching in the lost vehicle database according to the image and motion features. During the ID reassignment stage, if there are vehicles with high feature similarity (greater than 0.5) in the lost vehicle database, the ID of the vehicle with the highest feature similarity is reassigned to Vy. to achieve the vehicle re-identification; If all the vehicles’ feature similarity in lost vehicle database is less than 0.5, the vehicle Vy is assigned a new ID. Drawings: The specific content and advantages of the present invention will become apparent and readily understood in conjunction with the following drawings: Figure 1 shows the flow diagram of the traditional vehicle re-identification method; Figure 2 shows the flow diagram of the vehicle re-identification method of the present invention; Figure 3 shows the schematic diagram of sensor deployment; Figure 4 shows the schematic diagram of roadside video camera calibration; Figure 5 shows the coordinate transformation of the video target detection bounding box; Figure 6 shows the temporal synchronization of the video target detection bounding box and vehiclemounted RTK positioning data; Figure 7 shows the spatial error correction between the video target detection bounding box and vehiclemounted RTK positioning data; Figure 8 shows the schematic diagram of millimeter-wave radar rotational offset angle and target detection; Figure 9 shows the coordinate transformation from the millimeter-wave radar coordinates system to the world coordinates system; Figure 10 shows the flow diagram of the temporal synchronization optimization for millimeter-wave radar positioning data and vehicle-mounted RTK positioning data; Figure 11 shows the flow diagram of the spatial deviation correction for millimeter-wave radar positioning data and vehicle-mounted RTK positioning data; Figure 12 shows the schematic diagram of vehicle tracking of a single-video camera; Figure 13 shows the flow diagram of vehicle re-identification of the cross-video cameras; Figure 14 shows a schematic diagram of vehicle re-identification of the cross-video cameras. Specific Implementation To clarify the objects, technical solutions, and advantages of the present invention more clearly, the specific implementation of the present invention is described in detail below. Step 1: Sensor deployment and data acquisition The present invention relies on at least two video cameras and at least two millimeter-wave radars, to achieve multi-target vehicle detection, tracking, and vehicle re-identification. The video cameras use the directional video cameras, and the millimeter-wave radar uses a long-distance radar in the 79GHz frequency band. The basic sensor deployment plan is shown in Figure 3. The video cameras and millimeter-wave radars are installed at the same position on the roadside pole, ensuring that the world coordinates of the two sensors’ installation positions are consistent. The detection range of the sensor can be adjusted according to its installation height and angle. When its installation height is 6 meters and the downward tilt angle is 10°, the detection range of the sensor can reach 100-150m if there are no other obstacles in the scene. Both types of sensors collect data at a frequency of 25Hz and transmit the perception data to the central server for data storage and processing. For image data form video cameras, the pre-trained deep learning target detection model can be used to detect the target vehicles within the camera's field of view, and then the image features (including vehicle color, model, size, license plate information, etc.) and motion features (world coordinates) of each target vehicle can be calculated. For the multi-target vehicle trajectory data from millimeter-wave radars, the motion features (including world coordinates, speed, heading angle, etc.) of the target vehicle can be extracted. Variant deployment plan A: The millimeter-wave radars and video cameras are deployed on different roadside poles, and the overlap of the two sensors' field of view should be greater than 90%. The effective perception range is the intersection of the visual field range of the video cameras and millimeter-wave radars. Variant deployment plan B: The millimeter-wave radars and video cameras are deployed on the same gantry above the road, and the overlap of the two sensors' field of view should be greater than 90%. The effective perception range is the intersection of the visual field range of the video cameras and millimeter-wave radars. Variant deployment plan C: The millimeter-wave radars and video cameras are deployed at different gantry positions above the road, and the overlap of the two sensors' field of view should be greater than 90%. The effective perception range is the intersection of the visual field range of the video cameras and millimeter-wave radars. Step 2: Sensor calibration (1) Video camera calibration: The video camera calibration in the present invention refers to estimating a transformation matrix between the world coordinate system and the pixel coordinate system, i.e. the homography matrix H. as shown in equation (1). The world coordinate system refers to the WGS-84 coordinate system in the present invention. The world coordinates are acquired by vehicle-mounted RTK devices, which can provide three-dimensional positioning results through differential positioning technology and achieve centimeter-level accuracy in real time. The world coordinates obtained by RTK devices are used as truth values in the present invention because of their high positioning accuracy. The pixel coordinate system represents the coordinates of the image pixel in the image in the present invention. In the pixel coordinate system, the pixel in the upper left comer of the image is set as the origin generally, the right direction is set as the positive x-axis direction, and the lower direction is set as the positive y-axis direction, respectively. Equation (2) describes the transformation between the world coordinate system and the pixel coordinate system through the homography matrix, where (longitude, latitude) represents the world coordinate and (x, y) represents the pixel coordinate. Equation (3) and equation (4) are the formulae for the transformation. hit / 121 _ / 131 ^12 ^13 / 122 / ^23 / 132 / 133 _ (1) longitude latitude 1 longitude = latitude = / in / 121 _ / 131 / 112 / 122 / 132 / 123 y / 133 _ _ 1 _ / 1113? + / 112 Z / + / 113 / 131x + / 132 y + / 133 / 121“^ + / 122? / + / 123 h3iX + h32y + h33 (2) (3) (4) As shown in Figure 4, the handheld RTK device is placed within the field of view of the video camera. At least four non-colinear points are selected as the key points, and their world coordinates should be obtained through the handheld RTK device. The pixel coordinates of the key points in the camera image are also calculated. Through the key points, the homography matrix is calculated for the transformation between the world coordinate system and the pixel coordinate system, completing the calibration of the video camera. (2) Target detection bounding box calibration: The target detection bounding box calibration in the present invention refers to estimating a transformation between the world coordinate and the vehicle detection bounding box pixel coordinates in the camera image. Vehicle detection bounding box pixel coordinates refer to the coordinates of the pixel at the midpoint of the lower bottom of the bounding box when a video camera is located directly above the road. Variant A: Vehicle detection bounding box pixel coordinates refer to the coordinates of the pixel at the bottom right vertex of the bounding box when the video camera is located at the upper left of the road. Variant B: Vehicle detection bounding box pixel coordinates refer to the coordinates of the pixel at the bottom left vertex of the bounding box when the video camera is located at the upper right of the road. The target detection bounding box is obtained through the deep learning target detection model from the video camera data with a frequency of 25Hz, and the world coordinates are also obtained through the vehicle-mounted RTK device with a data frequency of 5Hz. The corresponding vehicle detection bounding box pixel coordinates should be acquired from the video camera data through the deep learning target detection model. The vehicle detection bounding box pixel coordinates can then be transformed into the world coordinates through the homography matrix. Considering the system clock deviation and the world coordinates deviation, a spatiotemporal matching optimization model can be established for two types of world coordinates obtained by the RTK device and vehicle detection bounding box in the image, including two stages of the temporal synchronization optimization and the spatial error correction. The temporal synchronization optimization procedure is shown in Figure 6. A linear interpolation method can be used to up-sample the world coordinates acquired from the vehicle-mounted RTK device to unify its sampling frequency with the video camera data. Then the optimization algorithm is used to find the optimal calibration parameters At, which can minimize the Euclidean distance between two types of world coordinates on the time series, as shown in equation 5: minf = ^(longiludehh t-longitudertk +(latitudebb t-latitude rtk (5) where (longitudebb tdatitudebb () represents the transformed world coordinates of the target vehicle acquired from the vehicle detection bounding box in the camera image at the time t, (longitudertk t+&tplatitudertk / +A / ) represents the world coordinates of the target vehicle acquired from the vehicle-mounted RTK device at the time t + At. The spatial error correction procedure is shown in Figure 7. The spatial error of the two types of world coordinates can be calculated according to equation (6), and then the world coordinates from the vehicle detection bounding box in the camera image can be corrected through the error curved surface fitting according to the world coordinates from the RTK, as shown in equation (7). error = (longitude bb tJatitudebb (longitude rtk t+M,latitude^ (6) cordinatecorrection=(longitudebb t,latitudebb J - Error Curved Surface (7) (3) Millimeter-wave radar calibration: The millimeter-wave radar can be calibrated using an experimental vehicle equipped with the vehicle-mounted RTK device. The millimeter-wave radar can obtain the target-level data containing the vehicle’s relative distance, azimuth, speed, etc., as shown in Figure 8. The relative coordinates of the experimental vehicle in the radar coordinate system can be calculated from the raw millimeter-wave radar data, as shown in equations (8) to (10): x = dis • cos 0i • cos 02 (8) y = dis • cos 0i • sin02 (9) z = - dis • cos0i (10) where dis denotes the relative distance between the target vehicle and the millimeter-wave, and 02 denote the pitch angle and the horizontal angle between the target vehicle and the millimeter-wave radar respectively, and (x, y, z) is the coordinates of the target vehicle in the radar coordinate system. As shown in Figure 9, there is the spatial position and posture angle deviation between the radar coordinate system and the world coordinate system. The posture angle deviation can be characterized as three angles in the orthogonal direction, denoted as Ct, / 3 and . The spatial position deviation is the deviation between the origin of the millimeter-wave radar coordinate system and the origin of the world coordinate system, denoted as longituderadar, latituderadar. and heightradar The transformation formula from the radar coordinate system to the world coordinate system is shown in equation (11)-(14). Rx(a) = 0 cosa 0 sin a 0 -sin a cos a (11) RM = cos / 3 0 -sin / 3 sin / 3 0 cos / 3 (12) COS7 -sin7 0 sin 7 cos 7 0 0 0 1 (13) Rx (a) Ry (^Rz (7) lOTlffitudOradar latitude radar height radar longtitude latitude height (14) where (x,y,z) is the coordinate of the target vehicle in the radar coordinate system, (longitude, latitude, height) is the world coordinate of the target vehicle detected by radar after transformation. The transformed world coordinates of the target vehicle acquired from the millimeter-wave radar can be denoted as (longitude, latitude, height), and the world coordinates of the target vehicle acquired from the vehicle-mounted RTK device can be denoted as (longitudertk ,latitude^ ,heightrtk). A spatiotemporal matching optimization model can be established for two types of world coordinates obtained by the RTK device and radar, including two stages of temporal synchronization optimization and spatial error correction. The temporal synchronization optimization procedure is shown in Figure 10. A linear interpolation method can be used to up-sample the world coordinates acquired from the vehicle-mounted RTK device to unify its sampling frequency with the radar data. Then the optimization algorithm is used to find the optimal calibration parameters (O' , (3 . y , At), which can minimize the Euclidean distance between two types of world coordinates on the time series, as shown in equation 15: min f=y / (longitudet — longituder tkt+At) 2 + (latitudet — latitudera- t+A^ (15) where (longitudet, latitudet) represents the transformation world coordinates of the target vehicle acquired from the millimeter-wave radar at the time t, (longitudertk t+At, latitude rtk t+At) represents the world coordinates of the target vehicle acquired from the vehicle-mounted RTK device at the time t + At. The spatial error correction procedure is shown in Figure 11. The spatial error of the two types of world coordinates can be calculated according to equation (14), and then the world coordinates from the millimeter-wave radar can be corrected through the error curved surface fitting according to the world coordinates from the RTK, as shown in equation (14). error = (longitude t, latitudet) - (longitudertk t+ / kt, latitudertk t+ / kt) (16) cordinatecorrectior - (longitudet, latitude t) - Error Curved Surface (17) Step 3: Vehicle detection and feature extraction For the image data from video cameras, the pre-trained deep learning target detection model can be used to detect the multi-target vehicles, and the image features (including vehicle color, model, size, license plate information, etc.) and motion features (world coordinates) of each target vehicle can be extracted. When license plate information cannot be obtained, this column can be fdled with a blank value. For the multi-target vehicle trajectory data from millimeter-wave radars, the motion features (including world coordinates, speed, heading angle, etc.) of the target vehicle can be extracted. Step 4: Feature matching and data fusion Using vehicle-mounted RTK positioning data of the experimental vehicle as the truth value, the multi-target vehicle trajectory data and image data can be matched based on the extracted vehicle motion features, and then the video camera positioning data and millimeter-wave radar positioning data can be fused to improve the precision of the positioning data through the multi-sensor data fusion algorithm. The data fusion algorithms include Kalman fdtering, fuzzy logic inference, deep neural networks, etc. Kalman fdter is one of the algorithms widely used in multi-sensor data fusion. Kalman fdtering is a recursive fdtering algorithm that doesn’t need to save past historical information. The estimated fdtering value is calculated based on the new data and the estimates from the previous frame. The principle of Kalman fdtering can be expressed in the following five equations: Predict: Xi = Fxt_1 +Bu^ (18) P; = FPt^FT + Q (19) Update: kt = Pt HT (HP;Ht + R) -1 (20) xt = xi + kt (zt — Hx^ ) (21) pt=(i-ktH)Pr (22) where F denotes the state transfer matrix; B denotes the control matrix; P denotes the state covariance matrices; Q denotes the state transfer covariance matrices; H denotes the observation matrices; R denotes the observation noise variance; Z denotes the actual observed value; k denotes the Kalman coefficient. Step 5: Vehicle tracking and re-identification 1) Vehicle tracking of a single-video camera: Use the pre-trained deep learning target detection model to detect the vehicles in the camera’s field of view, and obtain the target detection bounding boxes in every frame. Use the DeepSort multi-target tracking algorithm to match the target detection bounding boxes of the same target within different frames and assign the IDs to the vehicles, which can achieve vehicle tracking in the field of view of a single video camera. Figure 12 shows the schematic diagram of vehicle tracking of a single-video camera The DeepSort multi-target tracking algorithm, proposed by Nicolai Wojke, is an improvement on the Sort algorithm. Sort and DeepSort are the common algorithms used for Multiple Object Tracking (MOT) tasks. The Sort algorithm uses Kalman filtering to process the frame-by-frame data and the Hungarian algorithm to measure the correlation of the target, which can achieve good performance at high frame rates. However, as the Sort algorithm ignores the surface features of the object, it is only accurate when the uncertainty in the object state estimation is low. As for the DeepSort algorithm, a CNN neural network is trained on the large-scale dataset to extract image features to increase the robustness of the network. The DeepSort algorithm uses a detector and a tracker for target re-identification. The inputs of the DeepSort algorithm include target detection bounding box, target detection confidence, and target features (image features and motion features). The target detection confidence is used to filter the target detection bounding boxes, while the target features are used for the subsequent tracking. In the prediction stage, the Kalman filter with a uniform motion and linear observation model is used to predict the tracker. 2) Vehicle re-identification of the cross-video cameras: The flow diagram of vehicle reidentification of the cross-video cameras is shown in Figure 13. When a target vehicle become lost in the field of view of camera 1, the vehicle re-identification system records its image and motion features of the last n frames (n 10) before vehicle loss and transmits the features to the lost vehicle database. The predicted speed of the lost vehicle is calculated using linear fitting and Kalman filtering, and the predicted position of the vehicle is also calculated based on the driving direction before vehicle loss. The lost vehicle database can then be updated according to the predicted speed and position. When the target vehicle enters the field of view of camera 2, the vehicle re-identification system captures its image and motion features of the first n frames (n 10) after it enters, and performs feature matching in the lost vehicle database according to the image and motion features. During the ID reassignment stage, if there are vehicles with high feature similarity (greater than 0.5) in the lost vehicle database, the target vehicle is reassigned with the ID of the vehicle with the highest feature similarity. If there are no matching vehicles in the lost vehicle database, the target vehicle is assigned a new ID. In the present invention, the license plate information is not mandatory. If the camera can recognize part of the license plate information (e.g. key characters, license plate color, etc.) under good conditions, license plate information can be given a dynamic weight according to the confidence level of the recognition results, which can be used as a calibration basis for multi-target vehicle re-identification. The schematic diagram of vehicle re-identification of the cross-video cameras is shown in Figure 14. 19 08 25

Claims

1. A method for multi-vehicle trajectory acquisition based on data fusion of roadside millimeter-wave radars and video cameras, using at least two video cameras and at least two millimeter-wave radars, comprising the following steps:1) Calibration:1.1) Video camera calibration;1.2) Target detection bounding box calibration;1.3) Millimeter-wave radar calibration;2) Data acquisition:2.1) Use said video cameras to acquire the image data of target vehicles within its field of view; said image data from video cameras contain image features and motion features of said target vehicle;2.2) Use said millimeter-wave radars to acquire multi-target vehicle trajectory data of target vehicles within its field of view; said multi-target vehicle trajectory data from millimeter-wave radars contain the motion features of said target vehicle;3) Vehicle detection and feature extraction: After receiving data from video cameras and millimeter-wave radars, the central computing server performs vehicle detection and feature extraction;3.1) Feature extraction of said image data;3 .2) Feature extraction of said multi-target vehicle trajectory data;4) Feature matching and data fusion: Match said multi-target vehicle trajectory data and said image data based on the extracted vehicle motion features, and then fuse the video camera positioning data (Di) and millimeter-wave radar positioning data (D2);5) Vehicle tracking and re-identification:5.1) Vehicle tracking of a single-video camera5.1.1) Use a pre-trained deep learning target detection model to detect the vehicles in the camera’s field of view, and obtain target detection bounding boxes in every frame;5.1.2) Use a DeepSort multi-target tracking algorithm to match the target detection bounding boxes of the same target within different frames and assign IDs to the vehicles;5.2) Vehicle re-identification of cross-video cameras5.2.1) When a target vehicle becomes lost in the field of view of camera 1, the vehicle re-identification system records its image features and motion features before vehicle loss, and transmits both features to a lost vehicle database;5.2.2) When said target vehicle enters the field of view of camera 2, the vehicle reidentification system captures its image features and motion features and performs feature matching in said lost vehicle database according to both features; if there are vehicles with high feature similarity in said lost vehicle database, said target vehicle is reassigned with the ID of the vehicle with the highest feature similarity; if there are no matching vehicles in said lost vehicle database, said target vehicle is assigned with a new ID.

2. A method according to claim 1, wherein said video cameras uses the directional cameras, and said millimeter-wave radars use the long-distance radars in the 79GHz frequency band. The video cameras and millimeter-wave radars are installed at the same position on the roadside pole, ensuring that the world coordinates of the two sensors are consistent.

3. A method according to claim 1, wherein said video cameras and said millimeter-wave radars are deployed according to one of the following options:19 08 25Option A: The millimeter-wave radars and video cameras are deployed on different roadside poles, and the overlap of the two sensors' field of view should be greater than 90%;Option B: The millimeter-wave radars and video cameras are deployed on the same gantry above the road, and the overlap of the two sensors' field of view should be greater than 90%;Option C: The millimeter-wave radars and video cameras are deployed at different gantry positions above the road, and the overlap of the two sensors' field of view should be greater than 90%.

4. A method according to claims 2 or 3, wherein the detection range of said millimeter-wave radars and said video cameras was adjusted based on the installation height and angle: said installation height is 6 meters, the downward tilt angle is 10°; Both types of sensors collect data at a frequency of 25Hz and transmit the perception data to the central server for data storage and processing.

5. A method according to claim 1, wherein said video camera calibration refers to estimating transformation between the world coordinate and the vehicle detection bounding box pixel coordinates in the camera image; said vehicle detection bounding box pixel coordinates include the following three definitions:Definition A: Vehicle detection bounding box pixel coordinates refer to the coordinates of the pixel at the midpoint of the lower bottom of the bounding box when the video camera is located directly above the road;Definition B: Vehicle detection bounding box pixel coordinates refer to the coordinates of the pixel at the bottom right vertex of the bounding box when the video camera is located at the upper left of the road;Definition C: Vehicle detection bounding box pixel coordinates refer to the coordinates of the pixel at the bottom left vertex of the bounding box when the video camera is located at the upper right of the road.

6. A method according to claim 4, wherein said image data from video cameras contain the license plate information of said target vehicle.

7. A method according to claim 1, wherein said data fusion algorithm adopts the Kalman filtering algorithm.

Images