A multi-target vehicle detection and re-identification method based on radar and vision fusion

Abstract

A multi-target vehicle detection and re-identification method based on radar and vision fusion, the method mainly uses roadside perception equipment including roadside video camera and millimeter wave radar. In this method, two types of sensors, video camera and millimeter wave radar, are used to detect multiple target vehicles on the road, and a multi-sensor data fusion method is used to fuse video camera image data and millimeter wave radar detection target data to improve target detection data accuracy. This method considers two types of scenes: single video camera continuous tracking of multiple target vehicles and cross video camera continuous tracking of multiple target vehicles. Based on the multi-target vehicle image features extracted from video images, combined with the data detected by roadside millimeter wave radar, the geographical spatial features of multi-target vehicles are obtained, adding new dimensions to vehicle attributes. Applied to multi-target vehicle re-identification algorithm, it can effectively improve the accuracy of multi-target vehicle re-identification and improve the vehicle supervision level in the control area.

Description

Technical Field

[0001] This invention belongs to the field of mobile vehicle target detection, multi-sensor data fusion and vehicle re-identification technology, and relates to a method for detecting and re-identifying multiple target vehicles by fusing video camera image data and millimeter-wave radar data. Background Technology

[0002] In recent years, with the continuous development of my country's economy and the improvement of people's living standards, driving has become the preferred mode of transportation for many, and commuting by car has become a common phenomenon. The rapid growth of motor vehicles has brought greater pressure to traffic supervision departments. As video surveillance occupies an increasingly important position in the field of public safety, vehicle-related tasks are receiving more and more attention, such as vehicle target detection, vehicle classification, vehicle tracking, and driver behavior analysis. Determining whether vehicles in different image frames from a single video camera belong to the same vehicle, and determining whether vehicles in video images across different video cameras belong to the same vehicle, has become a crucial requirement for vehicle management. Only on this basis can subsequent work such as vehicle trajectory tracking and driver behavior analysis be better carried out.

[0003] Vehicle re-identification refers to the problem of determining whether vehicle data captured by roadside sensors (video cameras, millimeter-wave radar, etc.) at different times or in different locations in non-overlapping areas belong to the same vehicle within a specific traffic control scenario. Solving this problem is of great significance for precise vehicle control, regional security, vehicle-road cooperation, and many other aspects within a region.

[0004] Currently, existing vehicle re-identification methods are mainly divided into four categories:

[0005] Category 1: Sensor-based methods

[0006] Using various sensors to detect and identify vehicles is the most basic and earliest vehicle re-identification method. For each vehicle detected by the sensors, these methods typically extract vehicle features using specific techniques and determine the vehicle's identity through feature matching. The earliest vehicle re-identification methods were based on extracting vehicle feature information using various hardware detectors (such as infrared, ultrasonic, etc.). Subsequently, many vehicle identification methods utilizing other sensors or inductors were proposed, such as using three-dimensional magnetic sensors to detect multi-dimensional vehicle features and obtain time information from the sensors for training a Gaussian maximum likelihood classifier. Induction coils are the most commonly used tool for acquiring data in traffic scenarios, capable of monitoring various vehicle attributes (such as speed, volume, and vehicle footprint). Induction coils are generally deployed on main roads and highways in cities. Real-time vehicle re-identification methods based on induction coils extract vehicle features from the data provided by the induction coils and estimate vehicle travel time to complete the vehicle re-identification task. With the advent of some emerging sensors or technologies (such as Global Positioning System (GPS), Radio Frequency Identification (RFID), and mobile phones), some methods have explored beacon-based vehicle tracking and monitoring systems to complete vehicle re-identification tasks. For example, vehicle re-identification algorithms based on radio frequency identification (RFID) tags are applicable to various toll stations, while GPS-based vehicle travel time estimation methods are used to address the vehicle re-identification problem. Sensor-based methods, however, often require extensive hardware installation, resulting in demanding experimental environments and difficulties in reproduction. Furthermore, many methods are susceptible to environmental factors such as weather conditions, signal strength, traffic congestion, and vehicle speed, all of which reduce sensor sensitivity to varying degrees. Additionally, the lack of a unified performance evaluation standard means that these methods cannot be considered ideal vehicle re-identification approaches.

[0007] The second category: based on vehicle license plate information.

[0008] With the development of computer vision technology, it has become possible to identify vehicle identity information through image or video data, and the deployment of hardware facilities has been greatly reduced, saving significant hardware and labor costs. In the initial stage, vehicle re-identification methods based on computer vision technology mainly extracted vehicle license plate information through license plate localization, character segmentation, and recognition. Localization was primarily achieved using grayscale, color, and texture information, while segmentation and recognition mainly employed template matching and neural networks. License plate-based vehicle re-identification methods are highly accurate, but their drawbacks are also significant: traffic monitoring systems suffer from problems such as changing shooting angles, weather conditions, lighting variations, and low image resolution. Once the target vehicle's license plate information is lost, the method becomes ineffective. In real-world, complex traffic environments, it is common for license plate information to be difficult to obtain due to factors such as obscured license plates, low camera resolution, long shooting distances, and shooting angles, which significantly reduces the accuracy of license plate recognition. It is also ineffective against vehicles with fake license plates, cloned license plates, or even no license plates. Therefore, although license plate recognition is the simplest and most direct method for distinguishing different vehicles, in many cases, relying solely on license plate information is insufficient for re-identification tasks.

[0009] The third category: based on vehicle image features.

[0010] This method does not rely solely on license plates but integrates other non-license plate information to complete vehicle re-identification, thereby improving the stability and accuracy of vehicle re-identification. Traditional vehicle re-identification methods without license plate information mainly extract HSV, LBP, and HOG features from vehicle images for image feature matching, and then classify vehicles by color, model, and windshield to achieve vehicle re-identification. While vehicle re-identification methods without license plate information are highly interpretable, they are susceptible to changes in viewing angle and occlusion, resulting in relatively low re-identification accuracy.

[0011] The fourth category: based on machine learning and deep neural networks.

[0012] In recent years, with the development of artificial intelligence algorithms and deep neural networks in the field of computer vision, new technical approaches have emerged for vehicle re-identification. An increasing number of vehicle re-identification methods based on machine learning and deep learning have been proposed, significantly improving the accuracy of vehicle re-identification and gradually becoming the mainstream approach for vehicle re-identification.

[0013] Convolutional Neural Networks (CNNs) take images as input and extract features without pre-extracting complex, manually defined features. They do so through a continuous forward learning and backward feedback process. Each layer of a CNN primarily includes feature extraction and feature mapping operations. In feature extraction, the input to a neuron is the output of the previous layer. A convolutional kernel is used to convolve the input to obtain local features. Each layer can use multiple kernels, representing the extraction of multiple features from the input. Because the kernels share weights, the number of network parameters is significantly reduced. In feature mapping, sigmoid or tanh functions are used as activation functions, ensuring the extracted features are shift-invariant. CNNs automatically learn from training data, avoiding the fixed feature extraction methods of manually defined approaches. Instead, they implicitly learn from the training data, and due to the shared kernel weights, this parallel learning improves computational efficiency.

[0014] This approach is based on a convolutional neural network (CNN) and considers the impact of pose on recognition accuracy. The main method involves obtaining multiple region segmentation results for the target vehicle from the image to be recognized; using a CNN to extract region feature vectors from these segmentation results, and then fusing them with a global feature vector to obtain the appearance feature vector of the target vehicle. Finally, the fused feature vector is used for vehicle re-identification and retrieval. While this scheme considers the impact of pose on vehicle re-identification, the model's accuracy is limited by the diversity of the dataset. The dataset must include vehicle images from various angles and be sufficiently large. In real-world scenarios, collecting hundreds of thousands of vehicle images from different angles is quite difficult. Furthermore, keypoint annotation is required on the collected dataset for vehicle images from different angles. Since different images have different angles, the number and location of annotated keypoints vary, resulting in a huge workload. Therefore, from a feasibility and workload perspective, this method is quite complex.

[0015] Furthermore, the significant intra-class differences in the same vehicle due to variations in camera angle and lighting conditions, or the high inter-class similarity between different vehicles due to the same model, remain a major challenge limiting the accuracy of vehicle re-identification.

[0016] The aforementioned vehicle re-identification methods are all based on image or video data captured by roadside cameras, performing re-identification of multiple target vehicles at the vehicle appearance level. Their flowcharts are as follows: Figure 1 As shown, the process includes acquiring vehicle images or videos, vehicle detection, feature extraction and representation, similarity measurement calculation, and displaying detection results. However, it is quite common for different vehicles to be completely identical in appearance, except for their license plates. In such cases, re-identifying the target vehicle solely based on its appearance will significantly increase the false positive rate.

[0017] Millimeter-wave radar is a type of radar that operates in the millimeter-wave band. Millimeter waves typically refer to the 30–300 GHz frequency range (wavelength 1–10 mm). Since the wavelength of millimeter waves falls between microwaves and centimeter waves, millimeter-wave radar combines some advantages of both microwave and photoelectric radar. Compared to centimeter-wave seekers, millimeter-wave seekers are smaller, lighter, and have higher spatial resolution. Compared to infrared, laser, and television optical seekers, millimeter-wave seekers have a stronger ability to penetrate fog, smoke, and dust, and are suitable for all weather conditions (except heavy rain). Light waves experience significant attenuation during propagation in the atmosphere, requiring high precision in device manufacturing. Compared to light waves, millimeter waves experience less attenuation when propagating through atmospheric windows (frequencies where attenuation is minimal due to resonant absorption by gas molecules during propagation), and are less affected by natural light and thermal radiation sources. Therefore, they are of great significance in communications, radar, guidance, remote sensing, radio astronomy, and spectroscopy. Their advantages mainly include the following:

[0018] (1) Small antenna aperture and narrow beam: high tracking and guidance accuracy; easy to perform low elevation angle tracking, resist ground multipath and clutter interference; high lateral resolution for near-air targets; high angular resolution for area imaging and target surveillance; high anti-interference performance of narrow beam; high antenna gain; easy to detect small targets, etc.

[0019] (2) Large bandwidth: It has a high information rate and can easily obtain the detailed structural features of the target by using narrow pulse or wideband frequency modulation signals; it has a wide spread spectrum capability, which reduces multipath and clutter and enhances anti-interference capability; it can easily overcome mutual interference when adjacent frequency radar or millimeter wave identification devices work; it has high range resolution and can easily obtain accurate target tracking and identification capabilities.

[0020] (3) High Doppler frequency: good detection and identification capability for slow and vibrating targets; easy to use the target Doppler frequency characteristics for target feature identification; good detection capability under dust, smoke and dry snow conditions due to the penetration characteristics of dry air pollution.

[0021] Video cameras can extract image features from multiple vehicles, and camera calibration techniques and target detection algorithms can be used to obtain location data for these vehicles. Millimeter-wave radar can also detect the location data of multiple vehicles, and data fusion algorithms can be used to obtain more accurate location data. Furthermore, millimeter-wave radar can accurately capture geospatial information such as vehicle speed and direction of travel, adding a new dimension to vehicle attributes. Combining this with video image data can provide a solution to the aforementioned vehicle re-identification challenges.

[0022] Existing technology

[0023] Patent document CN108875754A

[0024] Patent document CN111582178A

[0025] Patent document CN111553205A

[0026] Patent document CN109508731A

[0027] Patent document CN111435421A

[0028] Explanation of related terms

[0029] 1. Single video camera: A single video camera that allows multiple vehicles to enter and exit the field of view of the camera.

[0030] 2. Cross-video camera: A set of two or more video cameras with different fields of view, in which a multi-target vehicle can move out of the field of view of one video camera and into the field of view of another video camera.

[0031] 3. Vehicle image features: HSV value, LBP and HOG of vehicle image are all vehicle image features. In addition, vehicle color, model, size and license plate information are also included.

[0032] 4. Vehicle geospatial characteristics: refers to the vehicle's latitude and longitude coordinates, speed, and heading angle information.

[0033] 5. HSV: HSV (Hue, Saturation, Value) is a color space created by A.S. Smith in 1978 based on the intuitive characteristics of color, also known as the hexagonal pyramid model. The parameters for color in this model are: Hue (H), Saturation (S), and Value (V).

[0034] 6. LBP: The LBP (Local Binary Pattern) operator is an efficient texture description operator with significant advantages such as rotation invariance and grayscale invariance. Its basic idea is to use the grayscale value of the center pixel as a threshold and compare it with its neighborhood to obtain a binary code that represents local texture features.

[0035] 7. HOG: Histogram of Oriented Gradients is a feature descriptor used in computer vision and image processing for object detection. It constructs features by calculating and statistically analyzing the gradient orientation histograms of local image regions. HOG features combined with SVM classifiers have been widely applied in image recognition.

[0036] 8. Significant intra-class variability: Targets within the same category exhibit substantial differences in their features. In vehicle re-identification, this manifests as follows: for the same vehicle, factors such as different camera angles and lighting intensities lead to significant variations in its image features.

[0037] 9. High inter-class similarity: Targets of different categories exhibit high similarity in their features. In vehicle re-identification, this is reflected in the high similarity of different vehicles in image features due to similarities in model, color, size, or belonging to the same brand and model.

[0038] 10. Video camera data: Video image data. After calibrating the camera and performing target detection on the image, the pixel coordinates and world geographic coordinates of the detected target can also be obtained.

[0039] 11. Video camera positioning data: Target positioning data output by the calibrated video camera, i.e., coordinates in the world geographic coordinate system obtained by converting the pixel coordinate system.

[0040] 12. Millimeter-wave radar data: The vehicle's distance, azimuth, speed, and heading angle relative to the millimeter-wave radar can be used to calculate the vehicle's coordinates in the radar coordinate system. After calibrating the millimeter-wave radar, the vehicle's coordinates in the world coordinate system can be calculated.

[0041] 13. Millimeter-wave radar positioning data: Target positioning coordinate data in millimeter-wave radar data.

[0042] 14. RTK: High-precision GPS measurements require carrier phase observations. RTK equipment uses differential positioning technology, which is a real-time dynamic positioning technology based on carrier phase observations. It can provide real-time three-dimensional positioning results of the measurement station in a specified coordinate system, achieving centimeter-level accuracy. RTKs can be divided into handheld and vehicle-mounted types according to their application. Handheld RTKs can be used for single-point coordinate measurements by hand. Vehicle-mounted RTKs can be installed on test vehicles for continuous coordinate measurements.

[0043] 15. RTK positioning data: latitude and longitude coordinates obtained from handheld and vehicle-mounted RTK measurements.

[0044] 16. Data set: The collection of data acquired by the data acquisition device. In this patent, it refers to the positioning data of the experimental test vehicle collected by the video camera, millimeter-wave radar and vehicle-mounted RTK equipment.

[0045] 17. Central computing server: A computer device used to receive, process and store data acquired by various data acquisition devices (including video camera image data and millimeter-wave radar positioning data).

[0046] 18. Spatiotemporal matching: refers to using an optimal matching method to ensure that the datasets acquired by different devices are synchronized in time and consistent in space for the positioning data collected from the same target, i.e., the error is within an acceptable range (0.5 meters).

[0047] 19. World geographic coordinate system: The world geographic coordinate system mentioned in this patent is the WGS-84 coordinate system among geographic coordinate systems.

[0048] 20. Pixel coordinate system: The pixel coordinate system represents the position of an image pixel in the image. Usually, the top left pixel of the image is taken as the origin, the right direction is defined as the positive x-axis, and the down direction is defined as the positive y-axis. The horizontal and vertical coordinates of the pixel represent the number of pixels that the pixel is away from the y-axis and the x-axis, respectively.

[0049] 21. Radar coordinate system: namely, millimeter-wave radar coordinate system, a three-dimensional spatial coordinate system with the millimeter-wave radar itself as the origin.

[0050] 22. Homography transformation matrix: The coordinate transformation matrix under different coordinate systems, which can be calculated by selecting the corresponding key points (at least 4 pairs of non-collinear points) under different coordinate systems.

[0051] 23. Multiple times: at least twice.

[0052] 24. Target vehicle: A vehicle that enters the field of view of the sensors (video cameras and millimeter-wave radar).

[0053] 25. Multi-target vehicle: A set of vehicles containing at least one target vehicle.

[0054] 26. Multi-target vehicle data: Location data of at least one vehicle, obtained by video cameras and millimeter-wave radar.

[0055] 27. Deep learning object detection algorithm: The deep learning object detection algorithm used in this patent is the YOLO v5 object detection algorithm to detect vehicles on the road.

[0056] 28.bb: Bounding box, the target detection rectangle, returned by the target detection algorithm, used to select the detected target contour in the video image.

[0057] 29. Multi-sensor data fusion algorithm: This algorithm synthesizes multi-source data after integrated processing to form the best consistent estimate of the measured object and its properties. This patent uses the Kalman filter algorithm for data fusion.

[0058] 30. Mahalanobis Distance: Proposed by Indian statistician PC Mahalanobis, Mahalanobis distance represents the distance between a point and a distribution. It is an efficient method for calculating the similarity between two unknown sample sets. Unlike Euclidean distance, it takes into account the relationships between various characteristics (e.g., a piece of information about height will lead to a piece of information about weight, because the two are related), and it is scale-invariant, meaning it is independent of the measurement scale.

[0059] 31. Cosine distance: Cosine distance, also known as cosine similarity, is used to evaluate the similarity between two vectors by calculating the cosine of the angle between them.

[0060] 32. Roadside: Located in other locations that are not part of the road surface itself, such as the side of the road or above the road (supported by poles or gantry frames).

[0061] 33. Lost vehicle: A vehicle that was present within the field of view of the aforementioned sensor devices (video cameras, millimeter-wave radar) but has not been captured by any of the devices.

[0062] 34. Sensor devices: In this patent, "sensor" refers to instruments that can acquire data, including video cameras, millimeter-wave radar, and RTK positioning devices (handheld and vehicle-mounted).

[0063] 35. Vehicle Re-identification System: A system framework for re-identifying lost vehicles.

[0064] 36. Vehicle Loss Database: Records the features of the last n frames before a vehicle is lost (the value of n can be adjusted according to the desired effect), including vehicle image features and geospatial features.

[0065] 37. ID Reassignment: When the vehicle re-identification system recovers a lost vehicle, it assigns the vehicle's historical ID to the recovered vehicle. [Summary of the Invention]

[0066] To address the problem of low recognition accuracy in current video image-based vehicle re-identification technologies due to variations in camera angle and lighting conditions, which result in significant intra-class differences for the same vehicle under different viewing angles or lighting conditions, or high inter-class similarity between different vehicles due to the same model, the technical solution adopted in this invention is as follows:

[0067] A multi-target vehicle detection and re-identification method based on radar-visual fusion adds geospatial information dimension to vehicle features by incorporating millimeter-wave radar detection data.

[0068] The specific plan is as follows:

[0069] 1) Calibrate the video camera S1 and the millimeter-wave radar S2, using the vehicle-mounted RTK positioning data D0 as the relative truth (because the error level of the vehicle-mounted RTK positioning data is at the centimeter level, while the error level of the video camera positioning data D1 and the millimeter-wave radar positioning data D2 is at the meter level, the accuracy of the vehicle-mounted RTK positioning data is much higher than that of the video camera and the millimeter-wave radar positioning data). Establish spatiotemporal matching optimization models between positioning data D1 and D2 and positioning data D0, respectively. Use the optimization algorithm to perform spatiotemporal matching of positioning data D1 and D2 to complete the sensor calibration work.

[0070] 2) Utilizing multi-target vehicle data acquired by video cameras and millimeter-wave radar, a deep learning target detection algorithm is employed to detect multiple target vehicles within the video images from the video cameras and extract image features and geospatial features of each target vehicle. Using the vehicle-mounted RTK positioning data D0 as the relative ground truth, a multi-sensor data fusion algorithm is used to fuse the multi-target vehicle data acquired by the video cameras and millimeter-wave radar, obtaining the optimal estimate of the multi-target vehicle positioning accuracy. The fused positioning data D3 exhibits higher positioning accuracy compared to D1 and D2.

[0071] By utilizing the fused multi-target vehicle data, continuous tracking and re-identification of multi-target vehicles between different image frames of a single video camera and multi-target vehicles across video camera images are performed.

[0072] The specific technical problems involved in this invention mainly include the following aspects:

[0073] 1. RTK differential positioning technology;

[0074] 2. Camera calibration technology;

[0075] 3. Multi-target vehicle detection technology based on image data;

[0076] 4. Millimeter-wave radar calibration technology;

[0077] 5. Multi-sensor data spatiotemporal matching optimization technology;

[0078] 6. Multi-sensor data fusion technology;

[0079] 7. Continuous tracking and re-identification technology for multiple targets in single-video camera footage;

[0080] 8. Multi-target vehicle continuous tracking and re-identification technology across video camera images;

[0081] The specific process of the technical solution adopted by the present invention to solve its technical problem is as follows:

[0082] 1. Calibration:

[0083] 1) Camera Calibration: Using a handheld RTK device, accurately locate key points within the camera image frame using world geographic coordinates. Select at least four non-collinear key points for location. Determine the pixel coordinates of the key points in the camera image pixel coordinate system. Calculate the homography transformation matrix between the pixel coordinate system and the world geographic coordinate system to complete the camera calibration.

[0084] 2) Target Detection Box Coordinate Calibration: An onboard RTK device is installed on the experimental test vehicle to acquire real-time vehicle positioning data. The experimental test vehicle is driven into the camera's field of view, and a deep learning target detection algorithm is used to annotate the detection box for the experimental vehicle in the video image. The pixel coordinates of the midpoint of the bottom edge of the detection box are used as the pixel coordinates of the experimental test vehicle. The world geographic coordinates of the experimental vehicle are calculated based on the homography transformation matrix corresponding to the video camera image pixel coordinate system and the world geographic coordinate system, which is the positioning data of the experimental test vehicle detected by the video camera. The positioning data of the experimental vehicle output by the video camera and the synchronous onboard RTK positioning data D0 are collected for a period of time (the duration must be greater than 2 minutes, and the experimental vehicle must appear repeatedly in the field of view of the video camera). A spatiotemporal matching optimization model between the two datasets D0 and D1 is established, and the parameters are solved to complete the coordinate calibration of the target detection box of the video camera. 3) Millimeter-Wave Radar Calibration: The relative coordinates of the experimental vehicle with respect to the millimeter-wave radar are calculated based on the raw millimeter-wave radar data (including the distance and azimuth data of the target relative to the millimeter-wave radar), which is the coordinates of the experimental vehicle in the radar coordinate system. The millimeter-wave radar acquires positioning data D2 of the experimental vehicle (the duration must be greater than 2 minutes, and the experimental vehicle must repeatedly appear within the millimeter-wave radar's field of view) over a period of time, along with synchronous onboard RTK positioning data D0. The homography transformation matrix of the two datasets (D0 and D2) is calculated, and a spatiotemporal matching optimization model between the two datasets is established. The parameters are then solved to complete the millimeter-wave radar calibration. Millimeter-wave radar calibration data acquisition and target detection box coordinate calibration data acquisition can be performed simultaneously or sequentially (the order is not important).

[0085] 2. Data Acquisition: Video image data within the camera's field of view is acquired using roadside video cameras, and multi-target vehicle data within the radar's field of view is acquired using roadside millimeter-wave radar, including each vehicle's target ID and millimeter-wave radar data (positioning coordinates, speed, heading angle, etc. in a world geographic coordinate system). The data acquired by the video cameras and millimeter-wave radar (including video image data and millimeter-wave radar positioning data) is transmitted to the central computing server for subsequent calculations.

[0086] 3. Vehicle Target Recognition and Feature Extraction: After receiving data (including video image data and millimeter-wave radar positioning data) uploaded by video cameras and millimeter-wave radar, the central computing server performs vehicle target recognition and feature extraction steps. For video image data: A pre-trained deep learning target detection algorithm is used to identify multiple target vehicles within the camera's field of view. Target vehicle detection bounding boxes (the portion of the video image within the bounding box, a subset of the complete video image) with high confidence (confidence greater than 0.6) are extracted. The image features of each target vehicle (including vehicle color, model, size, and license plate information (optional)) and geospatial features, i.e., positioning data (world geographic coordinates), are calculated. For millimeter-wave radar data: Geospatial features of the target vehicles are extracted, including vehicle world geographic coordinates, speed, heading angle, etc.

[0087] 4. Feature matching and data fusion: The geospatial features of multiple targets vehicles acquired by video cameras and millimeter-wave radar are matched, and the positioning data acquired by video cameras and millimeter-wave radar are fused using a multi-sensor data fusion method to improve the accuracy of positioning data.

[0088] 5. Vehicle re-tracking:

[0089] 1) Vehicle re-tracking within the field of view of a single video camera: A pre-trained deep learning object detection algorithm is used to identify vehicles within the field of view of a single video camera and mark the target boxes. The Deepsort multi-target tracking algorithm is used to assign IDs to multiple target vehicles and to re-identify and match the target boxes between different frames to achieve vehicle re-tracking within the field of view of a single video camera.

[0090] 2) Cross-video camera vehicle re-tracking: When a target vehicle V x When the target vehicle is lost within the field of view of camera 1, the vehicle re-identification system records the target vehicle V. x The image features and geospatial features of the vehicle before it was lost are collected and transmitted to the lost vehicle database. The lost target vehicle V x When a vehicle enters the field of view of camera 2, the vehicle re-identification system captures vehicle features (including image features and geospatial features) and performs feature matching in the vehicle loss database. Vehicles with high similarity (similarity greater than 0.5) are reassigned IDs. During ID reassignment, the ID with the highest similarity to the lost vehicle in the database is used for ID reassignment, thus achieving vehicle re-tracking. If no matching vehicle is found in the vehicle loss database (i.e., the feature similarity with all lost vehicles in the database is less than 0.5), then the target vehicle V is not tracked. x Assign a new ID.

[0091] Brief description of the attached figures

[0092] Figure 1Flowchart of traditional vehicle re-identification technology;

[0093] Figure 2 Flowchart of the vehicle re-identification technology roadmap of this invention;

[0094] Figure 3 This is a schematic diagram of the sensor deployment;

[0095] Figure 4 A diagram illustrating camera calibration;

[0096] Figure 5 A schematic diagram of camera calibration images;

[0097] Figure 6 A schematic diagram illustrating the coordinate transformation of the target bounding box in camera image inspection.

[0098] Figure 7 Flowchart for optimizing the time synchronization model between target detection bounding box data and vehicle-mounted RTK positioning data;

[0099] Figure 8 Flowchart for spatial error calibration between target detection bounding box data and vehicle-mounted RTK positioning data;

[0100] Figure 9 This is a schematic diagram illustrating the relationship between the rotation offset angle of a millimeter-wave radar and target detection.

[0101] Figure 10 A schematic diagram illustrating the calculation of world geographic coordinates for targets detected by millimeter-wave radar.

[0102] Figure 11 Flowchart of the optimization model for time synchronization between millimeter-wave radar detection data and vehicle-mounted RTK positioning data;

[0103] Figure 12 is a flowchart of spatial calibration between millimeter-wave radar detection data and vehicle-mounted RTK positioning data;

[0104] Figure 13 is a schematic diagram of vehicle re-identification using a single video camera;

[0105] Figure 14 is a flowchart of vehicle recognition across video cameras;

[0106] Figure 15 is a schematic diagram of vehicle re-identification across video cameras. Detailed Implementation

[0107] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0108] This invention relates to a multi-target vehicle detection and re-identification method based on radar-visual fusion, the overall technical approach of which is shown in the appendix. Figure 2As shown, the specific process consists of five steps, including: sensor deployment and data acquisition, sensor calibration (video cameras and millimeter-wave radar), multi-sensor data fusion, and vehicle re-identification methods (single video camera and cross-video camera).

[0109] Step 1: Sensor Deployment and Data Acquisition

[0110] This invention utilizes data from two types of sensors—video cameras and millimeter-wave radar—to achieve multi-vehicle target detection, tracking, data fusion, and vehicle re-identification. The video cameras are directional, and the millimeter-wave radar is a long-range radar operating in the 79GHz band. The basic deployment plan is attached. Figure 3 As shown, the video camera and millimeter-wave radar are deployed at the same location on the roadside pole (ensuring the latitude and longitude distance between the two sensors is less than 0.5 meters), thus maintaining consistency in their world geographic coordinates. The sensor detection range can be adjusted according to the installation height and angle. When the sensor is installed at a height of 6 meters, with a downward tilt angle of 10° and no other obstructions in the scene, its field of view can reach 100-150 meters. Both types of sensors collect data at a frequency of 25Hz and connect to the central server for data storage and processing. For video image data: a pre-trained deep learning object detection algorithm is used to identify multiple target vehicles within the camera's field of view, and the image features of each target vehicle (including: vehicle color, model, size, license plate information (optional)) and geospatial features (world geographic coordinates) are extracted. For millimeter-wave radar data: the geospatial features of the target vehicles are extracted (including: vehicle world geographic coordinates, speed, heading angle, etc.).

[0111] Variant A: The millimeter-wave radar and video camera are deployed on different roadside poles to ensure that the overlap rate of the two sensors' fields of view is greater than 90%. The sensing range of this road section is the intersection of the fields of view of the video camera and the millimeter-wave radar.

[0112] Variant B: The millimeter-wave radar and video camera are deployed on the same gantry above the road, ensuring that the overlap of the two sensors' fields of view is greater than 90%. The sensing range of this road section is the intersection of the fields of view of the video camera and the millimeter-wave radar.

[0113] Variant C: Millimeter-wave radar and video cameras are deployed at different gantry locations above the road, ensuring that the overlap rate of the two sensors' fields of view is greater than 90%. The sensing range of this road section is the intersection of the fields of view of the video cameras and the millimeter-wave radar.

[0114] Step 2: Calibration

[0115] (1) Video camera calibration

[0116] In this invention, the calibration of the video camera refers to establishing the mapping relationship between the pixel coordinate system and the world geographic coordinate system in the camera image, i.e., the homography transformation matrix H, as shown in equation (1). The world geographic coordinate system is a reference coordinate system selected in the environment to describe the position of the camera and objects. This coordinate system is called the world geographic coordinate system. In this invention, the world geographic coordinate system is defined as the WGS-84 coordinate system in the geographic coordinate system. In this invention, the world geographic coordinates are obtained by the RTK device. High-precision GPS measurement must use carrier phase observations. The RTK device uses differential positioning technology, i.e., real-time dynamic positioning technology based on carrier phase observations. It can provide the three-dimensional positioning results of the measurement station in the specified coordinate system in real time and achieve centimeter-level accuracy. Because of its high positioning accuracy, the world geographic coordinates obtained by RTK are regarded as relative true values ​​in this invention. The pixel coordinate system represents the position of the image pixel in the image. Usually, the upper left pixel of the image is taken as the origin, the right direction is defined as the positive x-axis direction, and the lower direction is defined as the positive y-axis direction. The horizontal and vertical coordinates of the pixel represent the number of pixels that the pixel is far from the y-axis and the x-axis, respectively. Equation (2) describes the transformation between the world geographic coordinate system and the pixel coordinate system through a homography matrix, where longitude represents longitude, latitude represents latitude, x represents the horizontal coordinate of the pixel coordinate, and y represents the horizontal coordinate of the pixel coordinate. Equations (3) and (4) are the conversion calculation formulas.

[0117]

[0118]

[0119]

[0120]

[0121] As attached Figure 4 With appendix Figure 5 As shown, the handheld RTK rover is placed within the field of view of the video camera, and the world geographic coordinates of at least four non-collinear key points are collected. The pixel coordinates of the bottom of the handheld RTK rover's handheld stick are selected as the pixel coordinates corresponding to the positioning points in the camera image. Finally, the homography transformation matrix between the two coordinate systems is calculated to establish the mapping relationship between the video image pixel coordinates and the world geographic coordinates, thus completing the video camera parameter calibration.

[0122] (3) Target detection box coordinate calibration

[0123] Object detection box coordinate calibration establishes a mapping between the pixel coordinates of the vehicle object detection box in the camera image and the vehicle's world geographic coordinates. Specifically, the pixel coordinates of the vehicle object detection box refer to the pixel coordinates of the midpoint of the bottom edge of the detection box. (The camera is located directly above the road in the direction of travel.)

[0124] Variant A: The camera is located at the upper left of the road, and the pixel coordinates of the vehicle target detection box refer to the pixel coordinates of the lower right vertex of the detection box.

[0125] Variant B: The camera is located at the upper right of the road, and the pixel coordinates of the vehicle target detection box refer to the pixel coordinates of the lower left vertex of the detection box.

[0126] The target detection bounding boxes were acquired using a deep learning target detection algorithm at a data frequency of 25Hz, while the vehicle's world geographic coordinates were acquired using an onboard RTK at a data frequency of 5Hz. An onboard RTK was installed on the experimental test vehicle to acquire real-time world geographic coordinate data. The vehicle was driven into the camera's field of view, and the deep learning target detection algorithm was used to annotate and detect the vehicle in the video image, obtaining its pixel coordinates. The world geographic coordinates of the detected vehicle were calculated based on the homography transformation matrix corresponding to the camera-detected target pixel coordinates and world geographic coordinates. Clock asynchrony between different sensors is a common phenomenon. Furthermore, the world geographic coordinates after the target detection box pixel coordinates are transformed differ from the vehicle's real-world geographic coordinates. Therefore, a spatiotemporal matching optimization model for these two types of world geographic coordinates needs to be established, including a time synchronization optimization model and a spatial error calibration model.

[0127] As attached Figure 7 As shown, in the time synchronization optimization model, it is first necessary to unify the acquisition frequencies of the two types of data. The world geographic coordinates acquired by the vehicle-mounted RTK are upsampled using a linear interpolation method to make their sampling frequency consistent with the target detection box data frequency. The objective function of the time synchronization optimization model is established as shown in Equation (5), and an optimization algorithm is used to find a suitable set of parameters Δt to minimize the Euclidean distance between the two world geographic coordinates in the time series.

[0128]

[0129] Among them, longitude bb_t and latitude bb_t Longitude represents the world geographic coordinates corresponding to the pixel coordinates of the target detection bounding box at time t. rtk_t+Δt and latitude rtk_t+Δt The world geographic coordinates represent the measurements taken by the vehicle-mounted RTK at time t+Δt.

[0130] As attached Figure 8 As shown, in spatial error calibration, two types of world geographic coordinates are established in the pixel coordinate system. The error calculation formula is shown in Equation (6). The error spatial distribution surface is calculated by the surface fitting method, and the world geographic coordinates calculated by the target detection box are corrected according to Equation (7).

[0131] Error = (longitude bb_t latitude bb_t )-(longitude rtk_t+Δt latitude rtk_t+Δt (6)

[0132] coordinate correction = (longitude) bb_t latitude bb_t Error Curced Surface(7)

[0133] (3) Millimeter-wave radar calibration

[0134] Based on the suitability of millimeter-wave radar for detecting moving targets, an experimental vehicle equipped with a vehicle-mounted RTK system was used to calibrate the millimeter-wave radar. The radar returned target-level data, including the detected target's range, azimuth, and velocity. (See attached image.) Figure 9 A schematic diagram of millimeter-wave radar target detection is shown. The relative coordinates of the experimental vehicle with respect to the radar are calculated based on the original data of the millimeter-wave radar. The calculation formulas are shown in Equations (8) to (10).

[0135] x=dis·cosθ1·cosθ2(8)

[0136] y = dis·cosθ1·cosθ2(9)

[0137] z = -dis·cosθ1 (10)

[0138] Where distance represents the distance between the target vehicle and the radar, θ1 and θ2 represent the elevation angle and horizontal angle between the target vehicle and the radar, respectively, and x, y and z represent the coordinates of the target detected by the radar in the radar coordinate system.

[0139] As attached Figure 10 As shown, there are pose angle deviations and spatial position deviations between the radar coordinate system and the world geographic coordinate system. The pose angle deviation includes deviation angles in three directions, denoted as α, β, and γ. The spatial position deviation is the deviation between the origin of the millimeter-wave radar coordinate system and the origin of the world geographic coordinate system, which is the coordinate of the millimeter-wave radar in the world geographic coordinate system, denoted as longitude. radar latitude radar and height radar .

[0140] The pose deviation angle correction matrices in the three directions are shown in Equations (11), (12), and (13), respectively.

[0141]

[0142]

[0143]

[0144] The conversion formula for converting the radar coordinate system to the world geographic coordinate system is shown in equation (14).

[0145]

[0146] Where x, y, and z represent the coordinates of the radar-detected target in the radar coordinate system, and longitude, latitude, and height represent the world geographic coordinates of the radar-detected target converted from the radar coordinate system to the world geographic coordinate system.

[0147] Let longitude, latitude, and height be the world geographic coordinates of the detected target calculated from radar coordinates, and let longitude be the world geographic coordinates of the vehicle acquired by the onboard RTK. rtk latitude rtk and height rtk Two types of spatiotemporal matching optimization models for world geographic coordinates are established, including a time synchronization optimization model and a spatial error calibration model.

[0148] As attached Figure 11 As shown, in the time synchronization optimization model, it is first necessary to unify the acquisition frequencies of the two types of data. The world geographic coordinates acquired by the vehicle-mounted RTK are upsampled using a linear interpolation method to make their sampling frequency consistent with the target detection box data frequency. The objective function is established as shown in Equation (5), and an optimization algorithm is used to find a suitable set of parameters α, β, γ and Δt to minimize the Euclidean distance between the two world geographic coordinates in the time series.

[0149]

[0150] Among them, longitude t and latitude t This indicates the transformation of the radar-detected target's coordinates in the radar coordinate system to world geographic coordinates in the world geographic coordinate system at time t. rtk_t+Δt and lattitude rtk_t+Δt This represents the world geographic coordinates of the vehicle acquired by the onboard RTK at time t+Δt.

[0151] As shown in Figure 12, in spatial error calibration, the spatial error distribution of two types of world geographic coordinates in the pixel coordinate system is established. The error calculation formula is shown in Equation (14). The error spatial distribution surface is calculated by the surface fitting method, and the world geographic coordinates calculated by the target detection box are corrected according to Equation (15).

[0152] Error = (longitude t latitude t )-(longitude rtk_t+Δt latitude rtk_t+Δt (16)

[0153] coordinate correction = (longitude) t latitude t Error Curved Surface(17)

[0154] Step 3: Video Image Target Recognition and Feature Extraction

[0155] A pre-trained deep learning object detection algorithm is used to identify multiple vehicles within the camera's field of view. Image features (including vehicle color, model, size, and license plate information (optional)) and geospatial features (world geographic coordinates) are extracted for each vehicle. If license plate information is unavailable, this field is left blank. For millimeter-wave radar data: geospatial features of the target vehicles are extracted (including vehicle world geographic coordinates, speed, heading angle, etc.).

[0156] Step 4: Feature Matching and Data Fusion

[0157] Target vehicle data acquired from video and millimeter-wave radar are matched based on geospatial features. A multi-sensor data fusion method is used to fuse vehicle speed and geographic location data acquired from video and millimeter-wave radar to improve data accuracy. Specifically, the fusion algorithm uses the high-precision positioning coordinates obtained from the experimental vehicle's onboard RTK as the reference ground value, and employs sensor data fusion methods such as Kalman filtering, multi-Bayes estimation, fuzzy logic inference, and deep neural networks to fuse and calculate the geographic location data of multiple target vehicles.

[0158] Kalman filtering is one of the most widely used algorithms in multi-source data fusion. Kalman filtering is a recursive filtering algorithm characterized by its lack of need to store historical information. New data is combined with the estimated value obtained in the previous frame (or previous moment) and the system's state equations to obtain a new estimated value in a specific way. The principle of Kalman filtering can be expressed by the following five formulas:

[0159] predict:

[0160]

[0161]

[0162] renew:

[0163]

[0164]

[0165]

[0166] The meanings of each parameter are as follows:

[0167] F: State transition matrix;

[0168] B: Control matrix;

[0169] P: State covariance matrix;

[0170] Q: State transition covariance matrix;

[0171] H: Observation matrix;

[0172] R: Observation noise variance;

[0173] The state variables at time t-1 are inferred, but have not yet been corrected for the observations at this time.

[0174] The state variables at this moment are inferred from the previous moment and have been corrected based on the observed values ​​at this moment;

[0175] z: Actual observed value;

[0176] k: Kalman coefficient;

[0177] t: Time t.

[0178] Step 5: Vehicle Re-tracking

[0179] 1) Vehicle re-tracking within the field of view of a single video camera: A pre-trained deep learning object detection algorithm is used to identify vehicles within the field of view of a single video camera and mark the target boxes. The Deepsort multi-object tracking algorithm is then used to assign IDs to multiple target vehicles and to re-identify and match the target boxes between different frames, thereby achieving vehicle re-tracking within the field of view of a single video camera. A schematic diagram of the re-identification results is shown in Figure 13.

[0180] The Deepsort algorithm, proposed by Nicolai Wojke et al., is an improvement upon the Sort algorithm. Both Sort and Deepsort are commonly used algorithms in Multiple Object Tracking (MOT). The Sort algorithm uses a simple Kalman filter to process the correlation of frame-by-frame data and a Hungarian algorithm for correlation measurement. This simple algorithm achieves good performance at high frame rates. However, because the Sort algorithm ignores the surface features of the detected objects, it is only accurate when the uncertainty in object state estimation is low. In the Deepsort algorithm, a more reliable metric is used instead of the correlation metric, and a CNN neural network is trained on a large-scale dataset to extract features, increasing the network's robustness to loss and obstacles.

[0181] The Deepsort algorithm constructs a detector and a tracker for target re-identification. The detector is based on the target detection bounding boxes, while the tracker is used for target tracking and re-identification. The Deepsort algorithm inputs include: target detection bounding boxes, target detection confidence scores, and target features (image features and motion features). The target detection confidence scores are primarily used to filter a portion of the detection boxes, while the target detection bounding boxes and target features are used for tracker construction and subsequent tracking calculations. In the algorithm's prediction module, a Kalman filter is used to predict the tracker's behavior, constructing a model based on uniform motion and linear observation. The algorithm's update module includes matching, tracker updates, and target feature set updates.

[0182] Cascaded matching algorithm: Each detector is assigned a tracker, and each tracker has a timer parameter. If a tracker completes a match and updates, the timer parameter is reset to 0; otherwise, it is incremented by one unit. In cascaded matching, the trackers are prioritized based on their timer parameters, with smaller parameters matching first and larger parameters matching later. In other words, the tracker that matched first in the previous frame has higher priority, while trackers that haven't matched for several frames have lower priority.

[0183] Feature Comparison: Mahalanobis distance and cosine distance are introduced for comparing motion information and image information. Mahalanobis distance avoids the risk of different data feature variances in Euclidean distance by adding a covariance matrix to the calculation. The purpose of this is to normalize the variance, making the distance value more consistent with the data characteristics and practical meaning. Mahalanobis distance is a distance metric for measuring dissimilarity, while cosine distance is a similarity metric. The former distinguishes based on location, while the latter distinguishes based on direction. Cosine distance can be used to measure differences between different individuals across dimensions. Combining these two types of feature distances allows for a more comprehensive measurement of the differences in features between different targets.

[0184] Feature matching: Based on Mahalanobis distance and cosine distance features, target feature matching is performed between different image frames to complete the target re-identification (ID transfer) process.

[0185] 2) Cross-Camera Vehicle Re-tracking: The overall logic flowchart for cross-camera vehicle re-tracking is shown in Figure 14. When a target vehicle is lost within the field of view of camera 1, the system records the vehicle image features and geospatial features of the n frames (n≥10) prior to its loss and imports them into the lost vehicle database. The geospatial features include world geographic coordinates, vehicle speed, and driving direction features acquired by millimeter-wave radar. After a vehicle is lost, the system calculates the predicted vehicle speed using methods such as linear fitting and Kalman filtering, and calculates the predicted vehicle position based on its driving direction before loss, dynamically updating the lost vehicle database. When the target vehicle enters the field of view of camera 2, the system captures the features of the n frames (n≥10) prior to the vehicle's appearance and performs feature matching within the lost vehicle database. Vehicles with high similarity are reassigned IDs to achieve vehicle re-tracking. If no matching vehicle is found in the lost vehicle database, a new ID is assigned to the target vehicle. In this invention, license plate information is not mandatory. If environmental factors are favorable (lighting, shooting angle, etc.), the camera can recognize some license plate information (such as key characters, license plate color, etc.). Dynamic weights are assigned based on the confidence level of the recognition results of consecutive frames within the field of view of a single camera, which can serve as the verification basis for multi-target vehicle re-tracking. Ultimately, cross-video camera re-tracking of multiple target vehicles is achieved, and the schematic result is shown in Figure 15.

Claims

1. A multi-target vehicle detection and re-identification method based on radar-visual fusion, employing at least two pre-deployed video cameras and at least two millimeter-wave radars, comprising the following steps: 1) Calibration: 1.1) Video camera calibration; 1.2) Target detection box coordinate calibration; 1.3) Millimeter-wave radar calibration; The calibration process is as follows: for video cameras With millimeter-wave radar Calibration was performed using vehicle-mounted RTK positioning data with higher accuracy than that of video cameras and millimeter-wave radar. As relative truth values, video camera positioning data are established separately. With millimeter-wave radar positioning data The spatiotemporal matching optimization model uses an optimization algorithm to analyze video camera positioning data. and millimeter-wave radar positioning data Spatiotemporal matching is performed to complete sensor calibration. The spatiotemporal matching optimization model includes a time synchronization optimization model and a spatial error calibration. The time synchronization optimization model uses linear interpolation to upsample the world geographic coordinates acquired by the vehicle-mounted RTK, making its sampling frequency consistent with the target detection box data frequency. The spatial error calibration establishes the spatial error distribution of two types of world geographic coordinates in the pixel coordinate system, calculates the error spatial distribution surface through surface fitting, and corrects the world geographic coordinates calculated for the target detection box. 2) Data Acquisition: 2.1) Acquire video image data within the field of view of roadside video cameras; 2.2) Acquire multi-target vehicle data within the radar's field of view using millimeter-wave radar deployed along the roadside; 3) Vehicle target recognition and feature extraction: After receiving the video image data and the multi-target vehicle data, the central computing server performs vehicle target recognition and feature extraction; 3.1) Feature extraction from video image data; 3.2) Feature extraction from millimeter-wave radar data; 4) Feature matching and data fusion: Matching the geospatial features of multiple targets vehicles acquired by video cameras and millimeter-wave radar, using the experimental vehicle's onboard RTK positioning data ( As a relative truth, a multi-sensor data fusion method is used to analyze the positioning data of video cameras. ) and millimeter-wave radar positioning data ( To integrate; 5) Vehicle Re-tracking: Utilizing fused multi-target vehicle data, continuous tracking and re-identification are performed on multi-target vehicles across different image frames from a single video camera and across multiple video camera images. This includes the following two scenarios. 5.1) Re-tracking of vehicles within the field of view of a single video camera; 5.2) Cross-video camera vehicle re-tracking; the cross-video camera vehicle re-tracking includes the following steps: 5.2.1) When a target vehicle is lost within the field of view of camera 1, the system records the vehicle's image features and geospatial features before it is lost, and transmits them to the lost vehicle database; 5.2.2) When the target vehicle enters the field of view of camera 2, the system captures the vehicle features and performs feature matching in the vehicle loss database. Vehicles with high similarity are reassigned IDs to achieve vehicle retracking. 5.2.3) If no matching vehicle is found in the vehicle loss database, a new ID is assigned to the target vehicle.

2. The method as described in claim 1, characterized in that, The video camera is a directional camera, and the millimeter-wave radar is a long-range radar in the 79GHz band; the video camera and the millimeter-wave radar are deployed at the same position on the roadside pole so that their latitude and longitude coordinates are consistent.

3. The method as described in claim 1, characterized in that, The video camera and the millimeter-wave radar are deployed using one of the following schemes: Option A: The millimeter-wave radar and video camera are deployed on different roadside poles, with a field of view overlap of more than 90%. Option B: The millimeter-wave radar and video camera are deployed on the same gantry above the road, with a field of view overlap of more than 90%. Option C: Millimeter-wave radar and video cameras are deployed at different gantry locations above the road, with a field of view overlap of more than 90%.

4. The method as described in claim 1, characterized in that, The detection range of the video camera and the millimeter-wave radar is adjusted according to the installation height and angle: the installation height is 6 meters and the downward tilt angle is 10°; both the video camera and the millimeter-wave radar collect data at a frequency of 25Hz and connect to the central server for data storage and processing.

5. The method as described in claim 1, characterized in that, Target detection box coordinate calibration establishes a mapping relationship between the pixel coordinates of the vehicle target detection box in the camera image and the vehicle's world geographic coordinates; the target detection box coordinate calibration is divided into the following three cases: 1.2.1) When the video camera is located directly above the road in the direction of travel, the pixel coordinates of the vehicle target detection box refer to the pixel coordinates of the midpoint of the bottom edge of the detection box; 1.2.2) When the video camera is located at the upper left of the road, the pixel coordinates of the vehicle target detection box refer to the pixel coordinates of the lower right vertex of the detection box; 1.2.3) When the video camera is located on the upper right of the road, the pixel coordinates of the vehicle target detection box refer to the pixel coordinates of the lower left vertex of the detection box.

6. The method as described in claim 1, characterized in that, The video image data includes the following target vehicle image features: vehicle color, model, size, and geographic spatial features.

7. The method as described in claim 4, characterized in that, The video image data also includes license plate information.

8. The method as described in claim 1, characterized in that, The millimeter-wave radar data includes the following geospatial characteristics of the target vehicle: world geographical coordinates, speed, and heading angle.

9. The method as described in claim 1, characterized in that, The data fusion employs the Kalman filter method.

10. The method as described in claim 1, characterized in that, The vehicle re-tracking within the field of view of a single video camera includes the following steps: 5.1.1) Use a pre-trained deep learning object detection algorithm to identify vehicles within the field of view of a single video camera and mark the target boxes; 5.1.2) The Deepsort multi-target tracking algorithm is used to assign IDs to multiple target vehicles and to re-identify and match target boxes between different image frames to achieve vehicle re-tracking within the field of view of a single video camera.

Images