First-level road non-ETC vehicle toll collection method and system based on multi-source data fusion

Abstract

The invention discloses a first-level road non-ETC vehicle toll collection method and system based on multi-source data fusion, and the method comprises the steps: scanning a vehicle in a radar sensing recognition region in real time through a radar, so as to obtain radar scanning information; generating a vehicle tracking record based on the radar scanning information, wherein the vehicle tracking record comprises the target vehicle and first vehicle information corresponding to the target vehicle; in response to the target vehicle entering the visual identification area, an image acquisition instruction is sent to image identification equipment by using a radar, and the image identification equipment comprises license plate identification equipment and vehicle body identification equipment; acquiring second vehicle information acquired by the image recognition equipment; fusing the first vehicle information and the second vehicle information to obtain vehicle fusion information; and determining identification information of the target vehicle based on the vehicle fusion information, and determining a charging amount of the target vehicle according to the identification information. According to the invention, through fusion processing of the multi-source data, the accuracy of vehicle identification is effectively improved, and the conditions of misidentification and missed identification are reduced.

Description

Technical Field

[0001] This invention relates to the field of information fusion technology, specifically to a toll collection method and system for non-ETC vehicles on primary highways based on multi-source data fusion. Background Technology

[0002] Highway toll collection systems are a key technological means to ensure the recovery of highway construction funds and achieve sustainable operation of highway facilities. In my country's highway network, Class I highways bear the important responsibility of connecting economic and political centers with transportation hubs. They experience high traffic volumes, but compared to fully enclosed expressways, Class I highways differ significantly in terms of closure methods, entrance / exit spacing, and traffic speed. They often adopt a "partially controlled access" management model, allowing the retention of at-grade intersections and the flexible placement of closely spaced entrances / exits. This flexibility makes it difficult for Class I highways to universally establish toll booths for intermittent toll collection like expressways. Therefore, "free-flow toll collection" for non-stop traffic has become an inevitable requirement for the development of toll collection technology on Class I highways.

[0003] Currently, free-flow toll collection technology is mainly divided into two categories: for vehicles equipped with ETC on-board units, toll deduction is completed via microwave communication through roadside units (RSUs); for vehicles without ETC, vehicle identification and subsequent toll collection mainly rely on vision-based license plate image recognition technology. Visual recognition technology is widely used due to its low cost and rich information acquisition capabilities; however, this technology has significant limitations in practical deployment.

[0004] First, visual recognition technology suffers from inaccurate trigger timing, leading to high rates of missed and false captures. When the camera operates alone, both continuous video stream analysis and fixed-interval capture have limitations: video stream analysis suffers from judgment lag, easily causing license plates to deviate from the optimal recognition angle or become blurry in captured images when vehicles are traveling at high speeds; fixed-interval capture generates numerous empty frames, wasting resources in sparse traffic, and in dense traffic, improper interval settings can lead to multiple vehicles appearing in the same frame or missed captures. Second, pure visual solutions can only acquire static information such as license plate characters, failing to correlate with dynamic data like vehicle speed and precise location, making it difficult to construct a complete chain of vehicle passage evidence. This limits its capabilities in applications such as vehicle tracing and vehicle type confirmation (e.g., addressing issues like "large vehicles with small license plates" or "cloned license plates"). Therefore, the accuracy of vehicle recognition is low, making it difficult to accurately charge vehicles without ETC (Electronic Toll Collection) devices. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a toll collection method and system for non-ETC vehicles on primary highways based on multi-source data fusion. The aim is to improve the accuracy of vehicle identification and the precision of toll collection through the collaborative work of radar and visual recognition, and to achieve efficient and reliable free-flow toll collection for vehicles without ETC.

[0006] This invention discloses a toll collection method for non-ETC vehicles on primary highways based on multi-source data fusion, comprising: The radar is used to scan vehicles in the radar sensing and identification area in real time in order to obtain radar scanning information. Vehicle tracking records are generated based on radar scan information. The vehicle tracking records include the target vehicle and its corresponding first vehicle information, which includes the target vehicle's position, speed, and shape. In response to a target vehicle entering the visual recognition area, the radar sends an image acquisition command to the image recognition equipment, which includes license plate recognition equipment and vehicle body recognition equipment. Acquire the second vehicle information collected by the image recognition device. The second vehicle information includes the license plate image information collected by the license plate recognition device and the vehicle body image information collected by the vehicle body recognition device. The first and second vehicle information are merged to obtain vehicle fusion information; The identification information of the target vehicle is determined based on the vehicle fusion information, and the toll amount for the target vehicle is determined based on the identification information.

[0007] Preferably, generating vehicle tracking records based on radar scan information includes: The system uses a visual sensor to collect images of vehicles in the radar-sensing and identification area in real time, thereby obtaining multiple frames of images. The radar scanning information is preprocessed to generate radar target data; Visual target data is generated from multiple frames of acquired images using a target tracking algorithm. By performing temporal and spatial alignment on radar target data and visual target data, radar-visual fusion information is obtained. Vehicle tracking records for the target vehicle are generated based on the fusion of radar and visual information.

[0008] Preferably, the radar scanning information is preprocessed to generate radar target data, including: Data analysis is performed on the radar scan information to obtain the raw channel information, which includes the detection time of the detected target, radar image, relative speed, relative distance and relative angle. The detected target is the vehicle scanned by the radar. The original channel information is filtered using preset filtering rules to select radar target vehicles from the detected targets. The preset filtering rules include target filtering and lane range filtering. Target filtering is used to filter out empty target signals, which are based on preset relative speed, preset relative distance, and preset relative angle markers. Lane range filtering is used to select vehicles within the effective lane range, which is determined based on preset distance thresholds, including lateral distance thresholds and longitudinal distance thresholds. The motion state of radar target vehicles is predicted based on the Kalman filter algorithm, and radar vehicle information of radar target vehicles is obtained, including distance information, lane information and contour information. Establish the correlation between detection time, radar target vehicle, radar image, and radar vehicle information to generate radar target data.

[0009] Preferably, the visual target data is generated based on multiple frames of acquired images using a target tracking algorithm, including: Multiple frames of acquired images are input into the target detection model to obtain initial visual target information, which includes acquisition time, target category, visual detection box, and visual vehicle information. The target detection model is a deep learning model built based on the YOLO-V3 algorithm. The target tracking algorithm is used to correlate the initial visual target information to obtain visual target vehicles in multiple frames of acquired images, and a visual identifier ID is set for the visual target vehicles. The preset indicators of the target tracking algorithm include one or more of the following: intersection-union ratio, mean accuracy, region of interest, non-maximum suppression, and frame rate per second. Establish the correlation between acquisition time, visual target vehicle, and visual vehicle information to generate visual target data.

[0010] Preferably, radar target data and visual target data are temporally and spatially aligned to obtain radar-visual fusion information, including: Obtain the radar sampling frequency and the visual sampling frequency, and determine the reference sampling frequency for time alignment based on the least common multiple of the radar sampling frequency and the visual sampling frequency; By combining the reference sampling frequency, sampling time, and detection time, the visual target data and radar target data are time-aligned; Obtain the radar coordinate system and the visual coordinate system; Establish a spatial transformation matrix between the radar coordinate system and the visual coordinate system, and spatially align the radar image and the acquired image according to the spatial transformation matrix; Feature-level fusion is performed on spatiotemporally aligned radar target data and visual target data to generate radar-visual fusion information containing multi-dimensional attributes.

[0011] Preferably, generating vehicle tracking records for the target vehicle based on radar-visual fusion information includes: The radar detection bounding boxes in the radar image and the visual detection bounding boxes in the acquired image are obtained based on the radar-visual fusion information. The radar detection bounding boxes are used to represent the position of the radar target vehicle, and the visual detection bounding boxes are used to represent the position of the visual target vehicle. Calculate the intersection-union ratio (CIU) between the radar detection boxes and the visual detection boxes; the CIU is the ratio of the intersection of the radar detection boxes and the visual detection boxes to the union of the radar detection boxes and the visual detection boxes. The target vehicle is marked based on the crossover ratio, and a vehicle tracking record is generated by combining the radar-visual fusion information.

[0012] Preferably, marking the target vehicle based on the crossover-union ratio includes: When the crossover ratio is 0, obtain the radar detection box area and the visual detection box area. If the radar detection box area is 0 and the visual detection box area is not 0, then mark the visual target vehicle as the target vehicle; if the visual detection box area is 0 and the radar detection box area is not 0, then mark the radar target vehicle as the target vehicle. If the cross-union ratio (CUI) is greater than 0 and less than the preset CUI threshold, the radar target vehicle corresponding to the current detection time is associated with the radar target vehicle corresponding to the previous detection time to obtain a radar association result. The visual target vehicle corresponding to the current acquisition time is also associated with the visual target vehicle corresponding to the previous acquisition time to obtain a visual association result. If both the radar association result and the visual association result are successful, the radar target vehicle and the visual target vehicle are merged into the same target vehicle. If only one association is successful, the target vehicle is marked according to the successful association result. If both associations fail, the target vehicle recognition is deemed unsuccessful. If the cross-connection ratio (CBR) is greater than or equal to a preset CBR threshold, the radar target vehicle and the visual target vehicle will be associated as the same target vehicle.

[0013] Preferably, the image acquisition command is sent to the image recognition device using radar, including: Based on the target vehicle's position and speed, predict the time when the target vehicle enters the visual recognition area; Based on the vehicle shape and speed of the target vehicle, a region of interest is dynamically generated. The region of interest is used to represent the coordinate information of the target vehicle in a preset coordinate system. Based on the entry time and region of interest, an image acquisition command is generated and sent to the image recognition device.

[0014] Preferably, the identification information of the target vehicle is determined based on vehicle fusion information, and the toll amount for the target vehicle is determined based on the identification information, including: The identification information of the target vehicle is obtained based on vehicle fusion information. The identification information includes: license plate information, body information and vehicle model information. Based on the identification information, query the historical passage records of the target vehicle from the registration information database; The toll amount for the target vehicle is generated based on historical traffic records.

[0015] This invention discloses a toll collection system for non-ETC vehicles on primary highways based on multi-source data fusion, used to execute a toll collection method for non-ETC vehicles on primary highways based on multi-source data fusion. The toll collection system for non-ETC vehicles on primary highways based on multi-source data fusion includes: The radar scanning module is configured to use radar to scan vehicles in the radar sensing and identification area in real time in order to obtain radar scanning information; The vehicle tracking module is configured to generate vehicle tracking records based on radar scan information. The vehicle tracking records include the target vehicle and its corresponding initial vehicle information, which includes the target vehicle's position, speed, and shape. The image command module is configured to: in response to a target vehicle entering the visual recognition area, send an image acquisition command to the image recognition equipment using radar, the image recognition equipment including license plate recognition equipment and vehicle body recognition equipment; The image acquisition module is configured to acquire secondary vehicle information acquired by the image recognition device, which includes license plate image information acquired by the license plate recognition device and vehicle body image information acquired by the vehicle body recognition device. The data fusion module is configured to fuse the first vehicle information and the second vehicle information to obtain vehicle fusion information. The toll generation module is configured to: determine the identification information of the target vehicle based on the vehicle fusion information, and determine the toll amount for the target vehicle based on the identification information.

[0016] Compared with existing technologies, the beneficial effects of this invention are as follows: by using multi-source data fusion technology, the accuracy and real-time performance of vehicle identification are effectively improved, and the false detection rate of a single sensor in complex environments is reduced; by using radar to predict the trajectory of the target vehicle and dynamically generating regions of interest, the image recognition equipment is precisely guided to collect key information at the optimal time, further improving data matching efficiency and toll accuracy; by constructing a complete vehicle passage evidence chain, including multi-dimensional information such as license plate, vehicle body, and vehicle type, the traceability capability and toll reliability of vehicles without ETC are improved. Attached Figure Description

[0017] Figure 1 A flowchart illustrating the toll collection method for non-ETC vehicles on primary highways based on multi-source data fusion provided by this invention; Figure 2 This invention provides a millimeter-wave radar target detection map; Figure 3 This is a network structure diagram of the YOLO model provided by the present invention; Figure 4 The synchronization processing flowchart provided by this invention; Figure 5 A top view of the radar coordinate system and world coordinate system provided by the present invention; Figure 6 This invention provides a mapping diagram showing the relationship between the world coordinate system and the visual sensor coordinate system. Figure 7 This invention provides a mapping diagram showing the relationship between the visual sensor coordinate system and the image coordinate system. Figure 8 This invention provides a target vehicle matching and confirmation strategy diagram; Figure 9 This is a schematic diagram of the structure of the non-ETC vehicle toll collection system for primary highways based on multi-source data fusion provided by the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The present invention will now be described in further detail with reference to the accompanying drawings.

[0020] like Figure 1 As shown in the figure, this embodiment of the invention provides a toll collection method for non-ETC vehicles on primary highways based on multi-source data fusion, including the following steps.

[0021] S1. Use radar to scan vehicles in the radar sensing and identification area in real time to obtain radar scanning information.

[0022] In this embodiment of the invention, the technical grade of a Class I highway is between that of a highway and a Class II highway, characterized by separate directions and lanes for traffic. Class I highways differ from highways in terms of closure methods, entrance / exit spacing restrictions, and traffic speed. The core of Class I highway toll collection technology is accurate vehicle identification, including license plate and vehicle model information. Vehicles equipped with ETC-OBU can be identified through ETC-RSU, but vehicles without ETC are primarily identified through license plate image recognition equipment. Relying on cameras for identification is highly susceptible to factors such as lighting, weather, vehicle speed, vehicle model, and algorithms. Especially during vehicle travel, speed, vehicle type, obstruction by large vehicles, overtaking by other vehicles, and nighttime lighting all affect the instantaneous position of the captured image and its clarity, thus impacting the identification effect.

[0023] Therefore, to improve the accuracy of identifying vehicles without ETC (Electronic Toll Collection) devices, radar is used to scan vehicles within the radar's detection and identification area in real time. The radar can be millimeter-wave radar, a ranging sensor that uses millimeter waves with wavelengths of 1-10mm (frequency range of 30-300GHz) as the propagation medium. Millimeter-wave radar is mainly divided into two frequency bands: 24GHz and 80GHz. 24GHz radar is a short-to-medium range radar with a maximum detection range of approximately 50 meters, primarily used for detecting short-to-medium range targets such as lane change assist and blind spot detection. 80GHz and above radars are long-range radars with a maximum detection range of approximately 160 meters, commonly used in intelligent vehicles for adaptive cruise control, emergency collision avoidance, and other functions. The millimeter-wave radar used in this embodiment of the invention operates at 80GHz, capable of simultaneously detecting wide-angle short-to-medium range targets and high-resolution long-range targets—essentially an active millimeter-wave radar integrating short-to-medium range and long-range detection.

[0024] In this embodiment of the invention, the radar is installed in a suitable location, such as on a dedicated bracket above or to the sides of a road, so that its scanning range can completely cover the radar's sensing and recognition area, avoiding blind spots. The radar continuously emits electromagnetic waves and receives reflected waves. By analyzing and processing the reflected waves, radar scanning information such as the vehicle's distance, speed, angle, and vehicle body point cloud data is obtained. This information will serve as the basis for subsequent vehicle tracking and identification.

[0025] S2. Generate vehicle tracking records based on radar scan information.

[0026] In this embodiment of the invention, a vehicle tracking record is generated based on radar scanning information. The vehicle tracking record includes the target vehicle and its corresponding first vehicle information, which includes the target vehicle's position, speed, and shape.

[0027] In this embodiment of the invention, a radar and video fusion scheme can be adopted to fuse radar-acquired information with information acquired by a high-definition camera, thereby improving the accuracy of vehicle tracking. A visual sensor is used to acquire vehicles within the radar's sensing and recognition area in real time, obtaining multiple frames of images. Then, the radar scanning information is preprocessed to generate radar target data. A target tracking algorithm is used to generate visual target data based on the multiple frames of images. The radar target data and visual target data are time-aligned and spatially aligned to obtain radar-visual fusion information. Finally, a vehicle tracking record for the target vehicle is generated based on the radar-visual fusion information.

[0028] Specifically, in the process of data preprocessing of radar scan information, the radar scan information is first parsed to obtain raw channel information. The raw channel information includes the detection time of the detected target, radar image, relative speed, relative distance, and relative angle. The detected target is a vehicle scanned by the radar.

[0029] Furthermore, the original channel information is filtered using preset filtering rules to select radar target vehicles from the detected targets. The preset filtering rules include target filtering and lane range filtering. Target filtering is used to filter out empty target signals, which are based on preset relative speed, preset relative distance, and preset relative angle markers. Lane range filtering is used to select vehicles within the effective lane range, which is determined based on preset distance thresholds, including lateral distance thresholds and longitudinal distance thresholds.

[0030] It should be noted that millimeter-wave radar returns multiple frames of scan data per second, with each frame containing 64 channels of information. Each channel includes information such as the relative speed, relative distance, and relative angle of the detected target ahead. Due to the radar's operating principle and vehicle swaying caused by road bumps, the channel information returned by the radar contains a large number of invalid targets, empty targets, and false targets, interfering with the identification of targets ahead by intelligent vehicles. Therefore, it is necessary to filter the raw target data returned by the millimeter-wave radar based on road conditions and vehicle motion information to provide reliable motion information for subsequent stable radar tracking.

[0031] For example, Table 1 shows the original channel information acquired by the radar. According to the radar resolution protocol, the data information with a relative velocity of 81.91 m / s, a relative distance of 0 m, and a relative angle of 0° in the radar return signal is marked as an empty target signal.

[0032] Table 1 Original Channel Information As shown in Table 1, the signals returned by the radar in each frame contain a large number of empty targets, and these empty target signals need to be filtered out first.

[0033] Furthermore, radar possesses both mid-range and long-range detection capabilities, resulting in a wide detection range. While retaining the original detection range would allow for the detection of more targets in front of the vehicle, it would also increase data processing for non-dangerous targets, raising computational costs and processing time. Therefore, it is necessary to constrain the detection range of millimeter-wave radar at both longitudinal and lateral distances.

[0034] For example, a Cartesian coordinate system is established with the center of the radar plane on the gantry as the origin and the vehicle's direction of travel as the y-axis. A millimeter-wave radar target detection diagram is shown below. Figure 2 As shown, where, Let d be the relative angle between the radar and the target in front, and d be the relative distance between the radar and the target in front. Let X be the lateral distance between this vehicle and the vehicle in front. Assuming X is the effective lane range threshold, and according to road technical specifications, the standard width between lane lines is 3.75 meters, the effective lane range threshold X = 4.5. Therefore, the lateral distance between this vehicle and the vehicle in front should meet the following conditions: .

[0035] The maximum detection range of the millimeter-wave radar is 180 meters. Based on the actual road traffic environment and the detection range of the visual sensor, a longitudinal distance threshold of 80 meters is selected, meaning that target information beyond 80 meters in front of the vehicle is not considered. Thus, after target filtering and lane range filtering, the initial selection results of the radar's effective target data are shown in Table 2.

[0036] Table 2 Radar Target Vehicles Furthermore, the motion state of the radar target vehicle is predicted based on the Kalman filter algorithm, and the radar vehicle information of the radar target vehicle is obtained, including distance information, lane information and contour information. Finally, the correlation between detection time, radar target vehicle, radar image and radar vehicle information is established to generate radar target data.

[0037] Specifically, Kalman filtering is an algorithm that estimates the state at the current moment using the state estimate from the previous moment and the observed state at the current moment. State estimation algorithms based on linear system models and Gaussian noise models estimate the value of the unobserved state by performing a weighted average between the observed values ​​and model predictions of the time-series data. Due to its small memory footprint, good real-time performance, and ease of implementation, it has wide applications in target tracking.

[0038] For a target moving at a constant velocity, a constant velocity motion model can be used for modeling and estimation. The observation equation for Kalman filtering is: X kLet Z be the system state at time k. k Let v be the target observation value at time k, H be the observation matrix, and v be the target observation value. k The Kalman filter algorithm consists of two steps: prediction and update. In the prediction step, the current state is estimated based on the posterior estimate from the previous time step, yielding the prior estimate for the current time step. The update equation is as follows: In the above formula, This represents the prior state estimate at time k. Let A be the prior estimate of the covariance at time k, and let A be the state transition matrix. For motion noise, Q is The covariance. The target measurement value at the current time is used to correct the predicted estimate, resulting in the posterior estimate at the current time. The update equation is as follows: Among them, K k Let be the Kalman gain at time k, and R be the covariance of the system's observation noise v. Let P be the posterior state estimate at time k, i.e., the optimal estimate; k Let be the posterior covariance estimate at time k, representing the uncertainty of the state. A uniform motion model is used to model the target state, where the target state X is an eight-dimensional vector, represented as follows: Where (u,v) is the center of the target border, a is the aspect ratio, and h is the height. These represent the speeds of the first four items, respectively.

[0039] In this way, the Kalman filter algorithm can accurately predict the motion state of radar target vehicles, obtaining more precise radar vehicle information. After obtaining the radar target data, it is time-aligned and spatially aligned with the visual target data acquired by the high-definition camera.

[0040] In this embodiment of the invention, during the process of generating visual target data based on multiple frames of acquired images using the target tracking algorithm, the multiple frames of acquired images are first input into the target detection model to obtain initial visual target information. The initial visual target information includes acquisition time, target category, visual detection box, and visual vehicle information. The target detection model is a deep learning model built based on the YOLO-V3 algorithm.

[0041] The initial visual target information is then correlated using a target tracking algorithm to obtain visual target vehicles from multiple frames of acquired images, and visual identifiers (IDs) are set for the visual target vehicles. The preset indicators of the target tracking algorithm include one or more of the following: intersection-union ratio, mean accuracy, region of interest, non-maximum suppression, and frame rate per second. Finally, the correlation between acquisition time, visual target vehicles, and visual vehicle information is established to generate visual target data.

[0042] In some embodiments, the YOLO network structure references the GoogLeNet model, a novel deep learning model that employs 24 convolutional layers, with the last two being fully connected layers, and uses 1×1 convolutional kernels for dimensionality reduction. The YOLO model network structure diagram is shown below. Figure 3 As shown. The evaluation metrics for object detection in the YOLO model can be one or more of the following: intersection-union ratio, mean precision, region of interest, non-maximum suppression, and frames per second. The mean precision is the average of multiple precision metrics, and the PR curve reflects the performance of each category. The calculation formula is: Where Q represents the set of object detection categories, AveP(q) is the average accuracy of objects within each category, and AP is the average precision. A higher AP value indicates a better classifier, and mAP ranges from [0,1]. The Region of Interest (ROI) is extracted to more accurately obtain object category and location information, reducing data processing load for subsequent work. Non-maximum suppression reflects optimal object detection. This process involves sorting the predicted bounding boxes, selecting the highest-scoring box, and removing others with large overlapping areas. Frames per second (fps) is a metric for evaluating the speed of object detection algorithms, representing the number of images that can be processed per second. fps is used to evaluate real-time detection performance. Faster detection speed indicates better real-time performance; this metric is generally evaluated using the time required to process one image. The above object detection evaluation metrics are among the most commonly used currently available. Evaluating model performance using these metrics can fully reflect the performance of object detection.

[0043] Object detection using object tracking algorithms is trained by finding the largest object among two objects based on their intersection and union ratios. It then predicts which bounding box is closest to the actual labeled object and selects the largest object for detection. Therefore, object tracking algorithms perform well in detecting large objects. By detecting the entire input image and then selecting bounding boxes before outputting the results, the time rate is much faster than other object detection methods, making it suitable for detecting moving vehicles.

[0044] In this embodiment of the invention, it is necessary to perform spatiotemporal alignment of radar target data and visual target data. First, the radar sampling frequency and visual sampling frequency are obtained, and a reference sampling frequency for time alignment is determined based on the least common multiple of the radar sampling frequency and visual sampling frequency. Then, by combining the reference sampling frequency, sampling time, and detection time, the visual target data and radar target data are time-aligned.

[0045] It should be understood that because the sampling frequency of millimeter-wave radar differs from that of high-definition cameras, the asynchrony in data sampling periods will cause temporal differences in the data during acquisition. Ensuring that millimeter-wave radar data and visual sensor data are aligned in time is a key factor in fusion processing. Meaningful fusion results can only be obtained when the data originates from the same point in time.

[0046] Therefore, the visual target data and radar target data first need to be unified and synchronized in time. In order to synchronize the two, the least common multiple period of the two is selected as the minimum sampling period, based on the time of the millimeter-wave radar. The sampling time of the camera and millimeter-wave radar data coincides with the sampling time of the two, and the sampled image is the same frame.

[0047] In some embodiments, the time-synchronized values ​​of radar and video can also be time-synchronized and aligned with the RSU antenna data. Specifically, firstly, a unified start time is used to retrieve the radar-video integrated machine data and RSU antenna data; then, the time of each RSU data entry is formatted to a second-level system timestamp; then, the time of each radar-video integrated machine data entry is formatted to a millisecond-level system timestamp; finally, a matching time threshold is set, and the RSU data is paired one-to-many with the corresponding radar-video integrated machine data within the positive and negative time thresholds. This ensures that the radar-video integrated machine data and RSU antenna data are highly synchronized in time, providing a data foundation for subsequent data fusion processing.

[0048] After completing time synchronization alignment, the next step is to spatially align the radar target data and visual target data. Due to differences in the installation positions and viewing angles of the radar and cameras, the target data they acquire are biased in the spatial coordinate system. Therefore, it is necessary to obtain the radar coordinate system and the visual coordinate system, establish a spatial transformation matrix between the radar coordinate system and the visual coordinate system, and spatially align the radar image and the acquired image according to the spatial transformation matrix. Finally, the spatiotemporally aligned radar target data and visual target data are fused at the feature level to generate radar-visual fusion information containing multi-dimensional attributes.

[0049] For example, Figure 4 For the synchronous processing flowchart, Figure 5 A top view of the radar coordinate system and the world coordinate system, such as... Figure 4 and Figure 5As shown, during spatial synchronization, a coordinate system is first established; the radar coordinate system is... World coordinate system is The corresponding planes in both coordinate systems are parallel. In the Z-axis direction, plane and The distance between the planes is In the Y-axis direction, plane is lower than surface The distance is .

[0050] Projecting a point in the world coordinate system onto the pixel coordinate system requires a rigid body transformation to convert it to the vision sensor coordinate system, at which point the camera extrinsic parameter matrix is ​​obtained. Then, the transformation relationship between the vision sensor coordinate system and the image coordinate system is determined using the principle of similar triangles. Finally, the transformation relationship between the image coordinate system and the pixel coordinate system is obtained using the camera intrinsic parameter matrix. The world coordinate system is... The visual sensor coordinate system is The relationship between the two is as follows: Figure 6 As shown. Let the coordinate system of the vision sensor be... Image coordinate system is Let the coordinates of point P in the visual sensor coordinate system be ( After being projected onto the image coordinate system, it becomes p(x,y) The focal length of the vision sensor is f, and the overall relationship is as follows: Figure 7 As shown.

[0051] The transformation relationship between the visual sensor coordinate system and the image coordinate system is as follows: After the transformation between the pixel coordinate system and the world coordinate system is completed, the data collected by the millimeter-wave radar can be gradually transformed into the image through the world coordinate system, thus achieving spatial alignment between the radar and the vision sensor.

[0052] Furthermore, feature-level fusion is performed on the spatiotemporally aligned radar target data and visual target data to generate radar-visual fusion information containing multi-dimensional attributes. Specifically, feature-level fusion mainly involves extracting and integrating feature information from the spatiotemporally aligned radar target data and visual target data. Features extracted from radar target data can include motion features such as distance, speed, and angle of the target vehicle, as well as shape features such as contours; features extracted from visual target data cover visual features such as color, texture, and shape details of the target, as well as semantic features such as category and location obtained through target detection and tracking algorithms. During feature fusion, feature concatenation methods can be used to directly concatenate the feature vectors extracted from the radar target data and visual target data to form a longer feature vector.

[0053] In this embodiment of the invention, a vehicle tracking record for a target vehicle is generated based on radar-visual fusion information. Specifically, radar detection boxes in the radar image and visual detection boxes in the acquired image are obtained according to the radar-visual fusion information. The radar detection boxes are used to represent the position of the radar target vehicle, and the visual detection boxes are used to represent the position of the visual target vehicle. The intersection-union ratio (IUGR) between the radar detection boxes and the visual detection boxes is calculated; the IUGR is the ratio of the intersection of the radar detection boxes and the visual detection boxes to the union of the radar detection boxes and the visual detection boxes. The target vehicle is marked according to the IUGR value, and combined with the radar-visual fusion information, a vehicle tracking record for the target vehicle is generated.

[0054] In some embodiments, by receiving, parsing, preprocessing, and managing the target period of radar data, a valid output target is obtained. This target is then projected onto an image and matched with the detected and tracked visual target for Region of Interest (ROI). The matched target is then matched with the vehicle target output at the previous time step for ID confirmation, ultimately yielding the vehicle target at the current time step. Based on the two sets of coordinate transformation relationships mentioned above, the rectangular ROI region of the vehicle in the radar coordinate system can be derived into the rectangular ROI region in the pixel coordinate system. The transformed ROI region is used as the detection box for the radar target.

[0055] After obtaining the bounding boxes of the targets output by the visual sensor and millimeter-wave radar in the image, it is necessary to perform similarity correlation between the two bounding boxes. The targets output by the millimeter-wave radar have undergone data preprocessing and periodic management, and each output target has a corresponding ID and ROI region. If the target exists for a long time, the ID will continue. The targets output by the visual sensor also have unique IDs after detection and tracking. After each target matching, the output vehicle target saves the IDs of both sensors. The correlation between the two bounding boxes is checked using the IOU (Interval of Value) method, and the calculation formula is as follows: Where Sr is the area of ​​the radar detection box, and Sc is the area of ​​the visual detection box. The intersection of the two detection boxes. This is the union of two detection boxes. IoU represents the degree of intersection between the two detection boxes, ranging from 0 to 1. A larger IoU indicates a stronger correlation. After sensor target matching, the target ID and attribute information of each sensor are stored, and the target vehicle is output.

[0056] The process of marking the target vehicle according to the value of the intersection over union (IoU) is as follows: When the value of the IoU is 0, obtain the area of the radar detection box and the area of the visual detection box. If the area of the radar detection box is 0 and the area of the visual detection box is not 0, mark the visual target vehicle as the target vehicle; if the area of the visual detection box is 0 and the area of the radar detection box is not 0, mark the radar target vehicle as the target vehicle. When the value of the IoU is greater than 0 and less than the preset IoU threshold, associate the radar target vehicle corresponding to the current detection time with the radar target vehicle corresponding to the previous detection time to obtain a radar association result, and associate the visual target vehicle corresponding to the current acquisition time with the visual target vehicle corresponding to the previous acquisition time to obtain a visual association result. If both the radar association result and the visual association result are successfully associated, fuse the radar target vehicle and the visual target vehicle into the same target vehicle; if only one is successfully associated, mark the target vehicle according to the successfully associated result; if both are associated unsuccessfully, determine that the target vehicle recognition fails. When the value of the IoU is greater than or equal to the preset IoU threshold, associate the radar target vehicle and the visual target vehicle as the same target vehicle.

[0057] Exemplarily, Sr is the area of the radar detection box, and Sc is the area of the visual detection box. When the detection boxes do not overlap, that is, when the IoU is 0, it is necessary to further judge the situations of Sr and Sc. If and , at this time, the radar does not detect a target in this area, while the visual sensor detects a target. Since the picture is detected by a neural network model, it has a good detection effect on vehicle targets with similar features. Ignoring the influence of the misdetection of the visual sensor, it is considered that the radar has missed detection, and finally the target detected by the visual sensor is output; If and , at this time, the visual sensor does not detect a target in this area, while the radar detects a target. When the light is insufficient, there is rainy or foggy weather, or for long-distance detection, the probability of the visual sensor missing detection increases. The radar detection can maintain a good detection effect in bad weather and is relatively accurate for long-distance detection. It can be considered that the visual sensor has missed detection, and finally the target detected by the radar is output.

[0058] As Figure 8 shown, when 0 < IoU < a, the two target detection boxes have an overlapping area, but the correlation is insufficient. Match the target ID at the current moment with the ID saved at the previous moment. If both can be matched, it indicates that the target fusion is successful. If only one is matched, it indicates that one party has missed detection, and the comprehensive target is output. When a < IoU < 1, the correlation between the target detection boxes of the two sensors is relatively large, and it can be considered that the detection targets of the two are the same, and the comprehensive vehicle target information is output.

[0059] S3. In response to a target vehicle entering the visual recognition area, use radar to send an image acquisition command to the image recognition equipment.

[0060] In this embodiment of the invention, the image recognition device includes a license plate recognition device and a vehicle body recognition device. Based on the target vehicle's position and speed, the entry time of the target vehicle into the visual recognition area is predicted; based on the target vehicle's shape and speed, a region of interest (ROI) is dynamically generated, which represents the target vehicle's coordinate information in a preset coordinate system; based on the entry time and the ROI, an image acquisition command is generated and sent to the image recognition device.

[0061] For example, the radar utilizes its millisecond-level response speed and long-range detection capabilities to detect and track target vehicles, outputting target information including vehicle position (x, y, z), velocity (v), contour point set, and preliminary vehicle type probability distribution. Furthermore, through strict spatiotemporal alignment, the target position in the radar coordinate system is precisely mapped to the image pixel coordinate system. A mapping relationship is established between the pre-calibrated camera intrinsic parameter matrix and the radar-camera extrinsic parameter matrix, projecting the core coordinates of the radar target onto the image. Then, centered on the projection point, and combining the target contour, velocity, and type probability provided by the radar, an optimized region of interest (ROI) is dynamically generated.

[0062] It's important to note that for longitudinal ROI expansion of high-speed vehicles (speed > 80 km / h), vehicles moving at high speeds experience significant displacement within a short period. If only the target position in the current frame is captured, the target may already be blurred or partially out of the frame. The system significantly extends the ROI length along the target's direction of motion (based on the heading angle) to ensure that the entire vehicle (especially the front) falls completely within this "predicted area" at the moment the camera actually triggers. For lateral ROI expansion of large vehicles (trucks), if the ROI is generated only based on the basic outline, its side features (such as door logos and cargo box details) may be at the edge of the frame or even cropped when the vehicle approaches the camera. Large vehicles have wide bodies, and lateral expansion ensures their complete side features, including the cargo box, wheels, and other key components, ensuring that the entire vehicle's features, especially the front / side features, are completely within the capture area, thus providing complete image data for accurate vehicle type identification. This expansion is not fixed but dynamically calculated based on the target's real-time speed and type probability, forming an ROI area with optimized size and position. Finally, the radar encapsulates the generated ROI region and its coordinate information into a precise trigger command, which is then sent to the image recognition device in real time via a high-speed communication link.

[0063] S4. Obtain the second vehicle information collected by the image recognition device.

[0064] In this embodiment of the invention, the second vehicle information includes license plate image information collected by the license plate recognition device and vehicle body image information collected by the vehicle body recognition device. When no trigger command is received, the image recognition device can be in a low-power "monitoring" mode. It may simply be continuously imaging, but it does not need to call complex AI recognition models to analyze the entire image frame by frame, thus greatly saving computing resources and power consumption. Once the image recognition device receives a ROI command from the radar, it is immediately "awakened." The image recognition device no longer processes the entire image, but instead concentrates all computing resources (such as AI accelerators) on the area specified by the ROI. Within this area, it performs rapid focusing, exposure optimization, and captures a high-resolution image.

[0065] S5. Merge the first vehicle information with the second vehicle information to obtain vehicle fusion information.

[0066] In this embodiment of the invention, the first vehicle information collected by radar, such as dynamic data like vehicle location, speed, and model, is organically fused with the second vehicle information collected by image recognition equipment, including static data such as license plate images and vehicle body images. Through algorithmic processing, these data from different sources and of different types can be integrated into a comprehensive and accurate vehicle fusion information. This fused information not only includes the vehicle's physical characteristics, such as license plate number and body color, but also records the vehicle's dynamic behavior, such as driving trajectory and speed changes, providing strong support for subsequent toll collection and traffic management of non-ETC vehicles on primary highways.

[0067] S6. Determine the identification information of the target vehicle based on the vehicle fusion information, and determine the toll amount for the target vehicle based on the identification information.

[0068] In this embodiment of the invention, the identification information of the target vehicle is obtained based on vehicle fusion information. The identification information includes license plate information, vehicle body information, and vehicle model information. The historical passage records of the target vehicle are retrieved from the registration information database based on the identification information. The toll amount for the target vehicle is generated based on the historical passage records.

[0069] like Figure 9 As shown, the present invention also provides a toll collection system for non-ETC vehicles on primary highways based on multi-source data fusion, used to execute a toll collection method for non-ETC vehicles on primary highways based on multi-source data fusion. The toll collection system for non-ETC vehicles on primary highways based on multi-source data fusion includes: a radar scanning module 201, a vehicle tracking module 202, an image instruction module 203, an image acquisition module 204, a data fusion module 205, and a toll generation module 204.

[0070] The radar scanning module 201 is configured to: scan vehicles within the radar sensing and recognition area in real time using radar to obtain radar scanning information. The vehicle tracking module 202 is configured to: generate vehicle tracking records based on the radar scanning information. These records include the target vehicle and its corresponding initial vehicle information, which includes the target vehicle's position, speed, and shape. The image command module 203 is configured to: send an image acquisition command to image recognition devices (including license plate recognition and vehicle body recognition devices) using radar in response to a target vehicle entering the visual recognition area. The image acquisition module 204 is configured to: acquire secondary vehicle information acquired by the image recognition devices. This secondary information includes license plate image information acquired by the license plate recognition device and vehicle body image information acquired by the vehicle body recognition device. The data fusion module 205 is configured to: fuse the initial and secondary vehicle information to obtain vehicle fusion information. The toll generation module 206 is configured to: determine the target vehicle's identification information based on the vehicle fusion information and determine the toll amount based on this identification information.

[0071] As can be seen from the above technical solution, this invention discloses a toll collection method and system for non-ETC vehicles on primary highways based on multi-source data fusion. The toll collection method includes: using radar to scan vehicles in the radar sensing and recognition area in real time to obtain radar scanning information; generating vehicle tracking records based on the radar scanning information, the vehicle tracking records including the target vehicle and its corresponding vehicle position, vehicle speed, and vehicle shape; responding to the target vehicle entering the visual recognition area, sending an image acquisition command to an image recognition device using radar, the image recognition device including a license plate recognition device and a vehicle body recognition device; acquiring second vehicle information collected by the image recognition device; fusing the first vehicle information and the second vehicle information to obtain vehicle fusion information; determining the identification information of the target vehicle based on the vehicle fusion information, and determining the toll amount of the target vehicle based on the identification information.

[0072] This invention effectively improves the accuracy and real-time performance of vehicle identification through multi-source data fusion technology, and reduces the false detection rate of a single sensor in complex environments. By predicting the trajectory of the target vehicle with radar and dynamically generating regions of interest, it accurately guides the image recognition equipment to collect key information at the optimal time, further improving data matching efficiency and toll accuracy. By constructing a complete vehicle passage evidence chain, including multi-dimensional information such as license plate, vehicle body, and vehicle type, it improves the traceability of vehicles without ETC and the reliability of toll collection.

[0073] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A first-class highway non-ETC vehicle toll collection method based on multi-source data fusion, characterized in that, The method comprises the steps of: scanning vehicles in a radar sensing area in real time by using a radar to obtain radar scanning information; generating a vehicle tracking record based on the radar scanning information, the vehicle tracking record comprising a target vehicle and corresponding first vehicle information of the target vehicle, the first vehicle information comprising vehicle position, vehicle speed and vehicle shape of the target vehicle; in response to the target vehicle entering a visual recognition area, sending an image acquisition instruction to an image recognition device by using the radar, the image recognition device comprising a license plate recognition device and a vehicle body recognition device; obtaining second vehicle information acquired by the image recognition device, the second vehicle information comprising license plate image information acquired by the license plate recognition device and vehicle body image information acquired by the vehicle body recognition device; fusing the first vehicle information and the second vehicle information to obtain vehicle fusion information; determining identification information of the target vehicle based on the vehicle fusion information, and determining a toll amount of the target vehicle according to the identification information.

2. The primary road non-ETC vehicle toll collection method according to claim 1, wherein The method comprises the steps of: acquiring vehicles in the radar sensing area in real time by using a visual sensor to obtain multiple frames of acquisition images; performing data preprocessing on the radar scanning information to generate radar target data; generating visual target data from the multiple frames of acquisition images by using a target tracking algorithm; aligning the radar target data and the visual target data in time and space to obtain radar-visual fusion information; generating the vehicle tracking record of the target vehicle based on the radar-visual fusion information.

3. The primary road non-ETC vehicle toll collection method according to claim 2, wherein The method comprises the steps of: performing data analysis on the radar scanning information to obtain original channel information, the original channel information comprising detection time of a detection target, a radar image, relative speed, relative distance and relative angle, the detection target being a vehicle scanned by the radar; screening the original channel information by using a preset screening rule to screen out radar target vehicles from the detection target, the preset screening rule comprising target screening and lane range screening, the target screening being used to filter out empty target signals, the empty target signals being marked based on a preset relative speed, a preset relative distance and a preset relative angle; the lane range screening being used to select vehicles in an effective lane range, the effective lane range being determined based on a preset distance threshold, the preset distance threshold comprising a lateral distance threshold and a longitudinal distance threshold; predicting a motion state of the radar target vehicle based on a Kalman filtering algorithm, and obtaining radar vehicle information of the radar target vehicle, the radar vehicle information comprising distance information, lane information and contour information; establishing an association relationship among the detection time, the radar target vehicle, the radar image and the radar vehicle information to generate the radar target data.

4. The primary road non-ETC vehicle toll collection method according to claim 3, wherein The method comprises the steps of: Inputting multiple frames of the collected images into a target detection model to obtain initial visual target information, the initial visual target information including a collection time, a target category, a visual detection frame, and visual vehicle information, the target detection model being a deep learning model constructed based on a YOLO-V3 algorithm; Performing association processing on the initial visual target information by using a target tracking algorithm to obtain a visual target vehicle in the multiple frames of the collected images, and setting a visual identification ID for the visual target vehicle, preset indicators of the target tracking algorithm including one or more of an intersection union ratio, an average precision mean, a region of interest, non-maximum suppression, and a frame rate per second; Establishing an association relationship among the collection time, the visual target vehicle, and the visual vehicle information to generate the visual target data.

5. The primary road non-ETC vehicle toll collection method according to claim 4, wherein The time alignment and space alignment of the radar target data and the visual target data to obtain radar-visual fusion information includes: Obtaining a radar sampling frequency and a visual sampling frequency, and determining a reference sampling frequency for time alignment according to a least common multiple of the radar sampling frequency and the visual sampling frequency; Performing time alignment of the visual target data and the radar target data in combination with the reference sampling frequency, the sampling time, and the detection time; Obtaining a radar coordinate system and a visual coordinate system; Establishing a space conversion matrix between the radar coordinate system and the visual coordinate system, and performing space alignment of the radar image and the collected image according to the space conversion matrix; Performing feature-level fusion of the radar target data and the visual target data that are time and space aligned to generate radar-visual fusion information containing multi-dimensional attributes.

6. The primary road non-ETC vehicle toll collection method according to claim 5, wherein The generation of the vehicle tracking record of the target vehicle based on the radar-visual fusion information includes: Obtaining a radar detection frame in the radar image and a visual detection frame in the collected image according to the radar-visual fusion information, the radar detection frame being used to represent the position of the radar target vehicle, and the visual detection frame being used to represent the position of the visual target vehicle; Calculating an intersection union ratio between the radar detection frame and the visual detection frame, the intersection union ratio being the ratio of the intersection between the radar detection frame and the visual detection frame to the union of the radar detection frame and the visual detection frame; Marking the target vehicle according to the numerical value of the intersection union ratio, and generating a vehicle tracking record of the target vehicle in combination with the radar-visual fusion information.

7. The primary road non-ETC vehicle toll collection method according to claim 6, wherein The marking of the target vehicle according to the numerical value of the intersection union ratio includes: In a case where the numerical value of the intersection union ratio is 0, obtaining a radar detection frame area and a visual detection frame area, if the radar detection frame area is 0 and the visual detection frame area is not 0, marking the visual target vehicle as the target vehicle, and if the visual detection frame area is 0 and the radar detection frame area is not 0, marking the radar target vehicle as the target vehicle. In a case where the value of the intersection over union is greater than 0 and less than a preset intersection over union threshold, a radar target vehicle corresponding to a current detection time is associated with a radar target vehicle corresponding to a previous detection time to obtain a radar association result, and a visual target vehicle corresponding to a current acquisition time is associated with a visual target vehicle corresponding to a previous acquisition time to obtain a visual association result; if both the radar association result and the visual association result are successful, the radar target vehicle and the visual target vehicle are fused into the same target vehicle; if only one of them is associated successfully, the target vehicle is marked according to the result of the successful association; if both of them fail to be associated, it is determined that the target vehicle identification fails; In a case where the value of the intersection over union is greater than or equal to the preset intersection over union threshold, the radar target vehicle and the visual target vehicle are associated as the same target vehicle.

8. The primary road non-ETC vehicle toll collection method according to claim 1, wherein The sending of the image acquisition instruction to the image recognition device by the radar comprises: predicting an entering time of the target vehicle into the visual recognition area according to the vehicle position and the vehicle speed of the target vehicle; dynamically generating a region of interest according to the vehicle shape and the vehicle speed of the target vehicle, the region of interest being used to represent coordinate information of the target vehicle in a preset coordinate system; generating the image acquisition instruction based on the entering time and the region of interest, and sending the image acquisition instruction to the image recognition device.

9. The primary road non-ETC vehicle toll collection method according to claim 8, wherein, The determination of the identification information of the target vehicle based on the vehicle fusion information, and the determination of the toll amount of the target vehicle according to the identification information, comprise: obtaining the identification information of the target vehicle based on the vehicle fusion information, the identification information comprising license plate information, vehicle body information and vehicle model information; querying a historical passing record of the target vehicle from a record database according to the identification information; generating the toll amount of the target vehicle according to the historical passing record. 10.A first-class highway non-ETC vehicle charging system based on multi-source data fusion, configured to perform the first-class highway non-ETC vehicle charging method based on multi-source data fusion according to any one of claims 1-9. It comprises: a radar scanning module configured to scan vehicles in a radar sensing recognition area in real time by using a radar to obtain radar scanning information; a vehicle tracking module configured to generate a vehicle tracking record based on the radar scanning information, the vehicle tracking record comprising a target vehicle and corresponding first vehicle information of the target vehicle, the first vehicle information comprising a vehicle position, a vehicle speed and a vehicle shape of the target vehicle; an image instruction module configured to send an image acquisition instruction to an image recognition device by using the radar in response to the target vehicle entering a visual recognition area, the image recognition device comprising a license plate recognition device and a vehicle body recognition device; an image acquisition module configured to obtain second vehicle information collected by the image recognition device, the second vehicle information comprising license plate image information collected by the license plate recognition device and vehicle body image information collected by the vehicle body recognition device; a data fusion module configured to fuse the first vehicle information and the second vehicle information to obtain vehicle fusion information; The toll generation module is configured to determine the identification information of the target vehicle based on the vehicle fusion information, and determine the toll amount of the target vehicle according to the identification information.

Images