Method and system for monitoring a bridge vehicle

Abstract

This application proposes a method and system for monitoring vehicles on bridges, relating to the field of bridge safety monitoring. The method includes: acquiring environmental point cloud data and radar echo data; calculating the point cloud density of the environmental point cloud data and performing planar segmentation based on the calculated point cloud density to obtain bridge surface point cloud data and vehicle candidate point cloud data; dividing the vehicle candidate point cloud data into regions based on the point cloud density and radar echo data to obtain core candidate regions for vehicles; constructing a directed bounding box based on the core candidate regions for vehicles and adjusting the directed bounding box using radar echo data to obtain vehicle bounding box data; and calculating the vehicle's dynamic parameters based on the bridge surface point cloud data, vehicle bounding box data, and radar echo data. These dynamic parameters include spatial position, driving speed, driving direction, and real-time distance to a preset safety boundary. This technical solution enables accurate and blind-spot-free dynamic identification of vehicle operating status in bridge scenarios.

Description

Technical Field

[0001] This application relates to the field of bridge safety monitoring, and more specifically, to a method and system for monitoring bridge vehicles. Background Technology

[0002] As a key node in the transportation network, the real-time monitoring of vehicle operation status on bridges is crucial for ensuring traffic safety. Traditional methods for real-time monitoring of vehicle operation status mainly include manual inspection, fixed laser ranging, millimeter-wave radar, and optical vision systems.

[0003] However, each of these traditional methods has its limitations. Manual inspections have a low monitoring frequency and data lag, making it difficult to meet the needs of real-time dynamic monitoring. Fixed laser ranging can only cover local areas, and its identification efficiency is low for non-fixed-point parked or high-speed moving vehicles. Although millimeter-wave radar can cover long distances, its low vertical resolution makes it difficult to distinguish the echoes from vehicles on the bridge surface and the water or ground below. It is also easily obstructed by bridge railings, beam bottoms, and other structures, resulting in monitoring blind spots at the middle and far ends of the bridge. Optical vision systems are greatly affected by rain, fog, and changes in day and night lighting. Their recognition accuracy drops significantly in bad weather or insufficient light, and they have difficulty accurately distinguishing between lanes going onto and off the bridge.

[0004] Therefore, how to accurately and dynamically identify the operating status of vehicles in bridge scenarios has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for monitoring bridge vehicles, which can accurately and dynamically identify the operating status of vehicles in bridge scenarios without blind spots.

[0006] This application is implemented as follows: In a first aspect, this application provides a method for monitoring bridge vehicles, comprising the following steps: acquiring environmental point cloud data and radar echo data, wherein the environmental point cloud data is data collected by lidar on the target bridge area, and the radar echo data is data collected by millimeter-wave radar on the target bridge area; calculating the point cloud density of the environmental point cloud data, and performing planar segmentation based on the calculated point cloud density to obtain bridge surface point cloud data and vehicle candidate point cloud data; dividing the vehicle candidate point cloud data into regions based on the point cloud density and radar echo data to obtain vehicle core candidate regions; constructing a directed bounding box based on the vehicle core candidate regions, and adjusting the directed bounding box using radar echo data to obtain vehicle bounding box data; and calculating the vehicle's dynamic parameters based on the bridge surface point cloud data, vehicle bounding box data, and radar echo data, wherein the dynamic parameters include spatial position, driving speed, driving direction, and real-time distance from a preset safety boundary.

[0007] In some implementations, the lidar is deployed at a first height at the bridgehead of the target bridge to collect environmental point cloud data of the near end of the bridge. The millimeter-wave radar includes a first millimeter-wave radar and a second millimeter-wave radar; wherein, the first millimeter-wave radar is deployed at a first height at the bridgehead of the target bridge to collect radar echo data of the far end of the bridge, and the second millimeter-wave radar is deployed at a second height at the bridgehead of the target bridge to collect radar echo data of the middle and far ends of the bridge; the second height is lower than the first height.

[0008] In some implementations, the steps of calculating the point cloud density of the environmental point cloud data and performing planar segmentation based on the calculated point cloud density to obtain bridge surface point cloud data and vehicle candidate point cloud data include: projecting a bird's-eye view onto the environmental point cloud data and calculating the point cloud density of each region in the projected two-dimensional image using a sliding window; marking regions with point cloud density below a preset density threshold as background regions and removing them to obtain high-density regions; performing planar fitting using a random sampling consensus algorithm within the high-density regions to obtain the main bridge surface plane and the main vehicle plane; determining the bridge surface point cloud data based on the main bridge surface plane and determining the vehicle candidate point cloud data based on the main vehicle plane.

[0009] In some implementations, the step of using a random sampling consensus algorithm for plane fitting in a high-density region includes: iterating through a randomly sampled point cloud plane in the high-density region, and using the calculated distance from each point to the randomly sampled point cloud plane as the residual; monitoring the fluctuation amplitude of the residual during the iteration process, and dynamically reducing the residual threshold when the fluctuation amplitude exceeds a preset fluctuation threshold; and selecting target point clouds with residuals smaller than the final threshold after dynamic adjustment to form the vehicle main plane.

[0010] In some implementations, the process of fitting the main plane of the bridge deck includes verifying the main plane of the bridge deck using radar echo data.

[0011] In some implementations, radar echo data includes echo intensity information; the step of dividing vehicle candidate point cloud data into regions based on point cloud density and radar echo data to obtain vehicle core candidate regions includes: in vehicle candidate point cloud data, determining overlapping regions where point cloud density is greater than or equal to a first preset threshold and echo intensity information is greater than or equal to a second preset threshold, and using the overlapping regions as vehicle core candidate regions.

[0012] In some implementations, the steps of constructing a directed bounding box based on the vehicle's core candidate region and adjusting the directed bounding box using radar echo data to obtain the vehicle bounding box data include: calculating the covariance matrix of the point cloud within the vehicle's core candidate region and extracting the largest eigenvector of the covariance matrix as the vehicle's principal direction; constructing an initial directed bounding box for the vehicle based on the vehicle's principal direction; parsing the vehicle's velocity vector from the radar echo data and calculating the angle between the velocity vector and the vehicle's principal direction to obtain the steering angle; determining that the vehicle is in a steering state in response to the steering angle exceeding a preset steering threshold, performing curve fitting on the front and rear wheel regions of the subsequent point cloud of the vehicle's core region, and correcting the shape of the initial directed bounding box based on the fitting result to generate the vehicle bounding box data; or, in response to the steering angle not exceeding a preset steering threshold, using the initial directed bounding box as the vehicle bounding box data.

[0013] In some implementations, the vehicle bounding box data carries the vehicle's center coordinates and main direction information. The steps for calculating the vehicle's dynamic parameters based on bridge surface point cloud data, vehicle bounding box data, and radar echo data include: determining the bridge coordinate system and the bridge principal axis vector based on the bridge surface point cloud data; mapping the vehicle's center coordinates onto the bridge coordinate system to obtain the vehicle's spatial position; determining the vehicle's speed based on the temporal changes in the environmental point cloud data corresponding to the vehicle bounding box data and the Doppler shift data of the radar echo data corresponding to the vehicle bounding box data; calculating the angle between the vehicle's main direction information and the bridge principal axis vector, and determining the vehicle's direction of travel based on the calculated angle; and calculating the distance between the vehicle and a preset bridge edge based on the vehicle bounding box data and bridge surface point cloud data, using the calculated distance as the real-time distance between the vehicle and a preset safety boundary.

[0014] In some implementations, the monitoring method further includes: comparing the spatial position with a preset lane boundary; if the comparison result shows that the spatial position exceeds the preset lane boundary, it is determined to be a lane departure; comparing the driving speed with a preset speed threshold; if the driving speed exceeds the preset speed threshold, it is determined to be speeding; comparing the real-time distance with a preset distance threshold; if the real-time distance is less than the preset distance threshold, it is determined to be too close to the bridge; and triggering the warning module when at least one of the following conditions is determined: lane departure, speeding, or being too close to the bridge.

[0015] In a second aspect, this application provides a bridge vehicle monitoring system, comprising: a lidar for acquiring environmental point cloud data of a target bridge area; a millimeter-wave radar for acquiring radar echo data of the target bridge area; a memory for storing one or more programs; and a processor connected to the lidar and the millimeter-wave radar respectively, for receiving the environmental point cloud data and the radar echo data; when the one or more programs are executed by the processor, the method as described in any one of the first aspects above is implemented.

[0016] Compared with the prior art, this application has at least the following advantages or beneficial effects: This application proposes a method for monitoring bridge vehicles. By utilizing the combined operation of lidar and millimeter-wave radar, it overcomes the blind spots of single sensors under long-distance or obstructed conditions, achieving comprehensive, dead-angle-free coverage of the entire bridge area. Simultaneously, through point cloud density filtering and dual-verification of region division, invalid or low-quality data such as bridge structure, water surface clutter, and vehicle window edges can be effectively eliminated, significantly improving the accuracy and anti-interference capability of vehicle identification. Furthermore, by dynamically adjusting the directed bounding box and fusing multi-source data, accurate modeling of vehicle geometry and motion state can be achieved, providing high-precision vehicle bounding box data for subsequent processing. Finally, through coordinate system mapping and multi-source data fusion, synchronous high-precision calculation of four dynamic parameters—spatial position, driving speed, driving direction, and safe distance—can be achieved, providing comprehensive and real-time data support for bridge vehicle safety monitoring. This effectively supports safety applications such as speeding warnings, lane departure warnings, and collision risk warnings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of an embodiment of a bridge vehicle monitoring method according to this application; Figure 2 This is a flowchart illustrating the specific steps of planar segmentation of point cloud density in one embodiment of this application; Figure 3 This is a flowchart illustrating the specific steps of plane fitting using a random sampling consensus algorithm in a high-density region in one embodiment of this application. Figure 4 This is a flowchart illustrating the steps for calculating the dynamic parameters of a vehicle in one embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0020] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0021] Bridge vehicle monitoring is a technology that uses sensors to collect dynamic data on vehicles on bridges and calculate their dynamic parameters, aiming to detect dangerous behaviors such as speeding and lane departure in a timely manner. Traditional monitoring methods include manual inspection, fixed laser ranging, millimeter-wave radar single-point monitoring, and camera-based visual recognition systems.

[0022] While traditional monitoring methods are diverse, they all have their limitations. Manual inspections are inefficient and cannot provide real-time continuous monitoring; fixed laser ranging has limited coverage; millimeter-wave radar single-point monitoring can detect vehicles at long distances, but its low vertical resolution makes it difficult to distinguish between vehicles on the bridge surface and clutter under the bridge, and it is easily blocked by the bridge structure, creating blind spots; camera-based visual recognition systems are severely affected by weather and lighting conditions.

[0023] In other words, while traditional monitoring methods are diverse, none can effectively address the technical challenge of accurately and dynamically identifying vehicle operating status in bridge scenarios without blind spots. To solve this problem, this application employs a series of techniques, including the coordinated deployment of lidar and millimeter-wave radar, point cloud density-guided plane segmentation, dual-verification region division, dynamic adjustment of bounding boxes for motion state perception, and dynamic parameter calculation through multi-source data fusion. These methods achieve full-bridge coverage without blind spots, accurate extraction of vehicle targets in complex environments, and synchronous high-precision calculation of dynamic parameters.

[0024] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings. Unless otherwise specified, the various embodiments and features described below can be combined with each other.

[0025] Please see Figure 1 The method for monitoring vehicles on this bridge includes the following steps: Step S101: Acquire environmental point cloud data and radar echo data. The environmental point cloud data is data collected by the lidar on the target bridge area, and the radar echo data is data collected by the millimeter-wave radar on the target bridge area.

[0026] Step S101 above, by simultaneously acquiring environmental point cloud data from the lidar and radar echo data from the millimeter-wave radar, provides a multi-source, complementary information foundation for subsequent steps. The environmental point cloud data records the spatial location information of the bridge structure, vehicles, and surrounding environment in the form of discrete three-dimensional coordinate points; the radar echo data records the distance data, velocity vectors, and echo intensity of each target within the detection range. The two sensors operate synchronously, acquiring complementary raw data from different physical dimensions.

[0027] It should be noted that, to ensure the accuracy of subsequent calculations, spatiotemporal alignment of the data from both the LiDAR and millimeter-wave radar is necessary when acquiring environmental point cloud data and radar echo data. This improves the temporal and spatial consistency between the LiDAR and millimeter-wave radar data. For temporal alignment, hardware synchronization signals such as the Precision Time Protocol (PTP) can be used to ensure that the timestamp error between the environmental point cloud data acquired by the LiDAR and the radar echo data acquired by the millimeter-wave radar does not exceed 10 milliseconds. For spatial alignment, coordinate system mapping can be used to transform the Cartesian coordinate system of the millimeter-wave radar to the bridge coordinate system of the LiDAR. The transformation relationship is x... fusion =R·x mm +T, where x mm The coordinates are in the original coordinate system of the millimeter-wave radar, where R is the rotation matrix, T is the translation vector, and x... fusion These are the coordinates after conversion, in a coordinate system consistent with that of the lidar.

[0028] Step S102: Calculate the point cloud density of the environmental point cloud data, and perform planar segmentation based on the calculated point cloud density to obtain bridge surface point cloud data and vehicle candidate point cloud data.

[0029] Step S102 preprocesses and initially separates the environmental point cloud data. Specifically, by calculating the point cloud density and segmenting the plane, candidate targets for the bridge surface and vehicles are quickly separated from the original environmental point cloud data, which greatly reduces the amount of data that needs to be processed in subsequent steps and improves processing efficiency.

[0030] Specifically, step S102 first calculates the point cloud density of each local region in the environmental point cloud data, i.e., the number of points per unit space. Point cloud density reflects the extent to which the area has been scanned by the LiDAR; vehicle surfaces typically have higher point cloud density, while open areas or bridge structures have relatively lower density. Next, planar segmentation is performed based on the calculated point cloud density, dividing the environmental point cloud data into different planar regions according to density distribution characteristics. From these regions, bridge surface point cloud data (belonging to the bridge surface) and vehicle candidate point cloud data (potentially belonging to vehicles) are separated. The bridge surface point cloud data is used for subsequent spatial reference, while the vehicle candidate point cloud data serves as the preliminary result for vehicle identification and proceeds to the next step of processing.

[0031] Step S103: Based on the point cloud density and radar echo data, divide the vehicle candidate point cloud data into regions to obtain the vehicle core candidate region.

[0032] Step S103 further filters the vehicle candidate point cloud data obtained in step S102, that is, by verifying the point cloud density and radar echo intensity, it removes unstable regions such as vehicle edges and windows to obtain highly reliable core candidate regions of the vehicle, providing a high-quality data foundation for subsequent accurate modeling.

[0033] Specifically, step S103 involves dividing the vehicle candidate point cloud data into regions based on two dimensions of information: the point cloud density calculated in step S102 and the radar echo data (primarily echo intensity). Regions with high point cloud density represent areas with strong lidar reflection, while regions with high echo intensity represent areas with strong millimeter-wave radar reflection. Both are typical characteristics of the vehicle entity (especially the metal body). Thus, by identifying overlapping regions where both point cloud density and echo intensity meet the requirements, the most reliable and representative regions can be further selected from the vehicle candidate point cloud data and designated as the core candidate regions for the vehicle.

[0034] Step S104: Construct a directed bounding box based on the core candidate region of the vehicle, and adjust the directed bounding box using radar echo data to obtain the vehicle bounding box data.

[0035] Step S104 generates accurate vehicle bounding box data by constructing a directed bounding box for the core candidate region of the vehicle and dynamically adjusting it in conjunction with radar echo data, providing an accurate geometric reference for subsequent dynamic parameter calculations. Specifically, step S104 first constructs a directed bounding box that tightly encloses the region based on the spatial distribution characteristics of the point cloud within the core candidate region of the vehicle. The directed bounding box is a minimum rectangle with directional attributes, whose length, width, and height are aligned with the vehicle's main direction. Then, using distance and velocity data from the radar echo data, the size, orientation, or position of the directed bounding box is fine-tuned to better fit the vehicle's true outline, ultimately obtaining accurate vehicle bounding box data. The vehicle bounding box data includes geometric parameters such as the vehicle's length, width, height, and center coordinates.

[0036] Step S105: Calculate the dynamic parameters of the vehicle based on the bridge surface point cloud data, vehicle bounding box data, and radar echo data. The dynamic parameters include spatial position, driving speed, driving direction, and real-time distance from the preset safety boundary.

[0037] Step S105 primarily uses the spatial coordinate system constructed from bridge surface point cloud data as a reference benchmark. Based on the center coordinates and main direction information of the vehicle bounding box data, and combined with the velocity information from the radar echo data, it synchronously calculates the vehicle's dynamic parameters: spatial position (the vehicle's real-time coordinates on the bridge surface), driving speed (the vehicle's instantaneous speed value), driving direction (the vehicle's direction of travel relative to the bridge's main axis), and real-time distance from the preset safety boundary (the shortest distance from the vehicle's edge to the bridge surface edge or guardrail, etc.). In other words, step S105, under a unified spatial and temporal benchmark, synchronously outputs four dynamic parameters: spatial position, driving speed, driving direction, and real-time distance from the preset safety boundary, providing comprehensive and real-time data support for bridge vehicle safety monitoring.

[0038] In simple terms, the overall interaction logic of steps S101-S105 is as follows: the lidar and millimeter-wave radar respectively collect environmental point cloud data and radar echo data as raw inputs; subsequent steps sequentially filter, refine, and model the raw data, ultimately outputting the vehicle's dynamic parameters. The environmental point cloud data provides high-precision three-dimensional spatial information, while the radar echo data provides supplementary information such as speed, distance, and echo intensity. The two types of data mutually verify and complement each other in each step.

[0039] In conclusion, this application addresses the issue of inaccurate vehicle identification or blind spots in bridge scenarios caused by single sensors by fusing data from LiDAR and millimeter-wave radar. This is primarily achieved through a multi-source data fusion and progressive refinement process, which effectively improves the accuracy of vehicle identification and the comprehensiveness of monitoring, eliminating the blind spot problem inherent in single sensors.

[0040] Specifically, this application first acquires environmental point cloud data collected by LiDAR and radar echo data collected by millimeter-wave radar. These two types of data are complementary in their physical principles, providing high-precision spatial location information and wide-area velocity and distance information for subsequent processing. Based on this, this application calculates the point cloud density of the environmental point cloud data and performs planar segmentation according to the point cloud density. This operation utilizes the difference in point cloud density distribution on the vehicle surface and background area, enabling rapid separation of bridge surface point cloud data from vehicle candidate point cloud data. This not only eliminates a large amount of irrelevant bridge surface structure interference but also significantly reduces the amount of data that needs to be processed in subsequent steps. Next, this application combines the point cloud density and radar echo data to regionalize the vehicle candidate point cloud data. Through dual verification by LiDAR and millimeter-wave radar, core candidate regions of the vehicle that exhibit high response under both sensors are selected, while edge regions with unstable features such as sunroofs and headlights are eliminated, ensuring the data quality for subsequent modeling. Then, this application constructs a directed bounding box based on the core candidate region of the vehicle and adjusts it using radar echo data. By combining the spatial distribution characteristics of the point cloud and the dynamic parameter information of the millimeter-wave radar, the generated vehicle bounding box data can accurately fit the real contour of the vehicle, especially overcoming the problem of inaccurate fitting in complex motion states such as vehicle turning by traditional methods. Finally, this application integrates bridge surface point cloud data, vehicle bounding box data, and radar echo data, and simultaneously calculates four dynamic parameters—spatial position, driving speed, driving direction, and real-time distance from the preset safety boundary—in a unified bridge coordinate system, realizing the transformation from raw data to complete vehicle state information.

[0041] Based on the aforementioned scheme, in some implementations of this application, the lidar is deployed at a first height at the bridgehead of the target bridge to collect environmental point cloud data of the near end of the bridge. The millimeter-wave radar includes a first millimeter-wave radar and a second millimeter-wave radar; wherein, the first millimeter-wave radar is deployed at a first height at the bridgehead of the target bridge to collect radar echo data of the far end of the bridge, and the second millimeter-wave radar is deployed at a second height at the bridgehead of the target bridge to collect radar echo data of the middle and far ends of the bridge; the second height is lower than the first height.

[0042] Understandably, this implementation deploys a lidar at a first height (30-50 meters) at the bridgehead of the target bridge, scanning obliquely downwards (e.g., 120° horizontal field of view, 25° vertical field of view) towards the near end of the bridge (e.g., the lane entrance area), to collect environmental point cloud data of the near end of the bridge. A first millimeter-wave radar is deployed at the same height as the lidar, i.e., the first height at the bridgehead, scanning obliquely downwards to cover the far end of the bridge (e.g., the middle and rear sections of the lanes), specifically for collecting radar echo data from the far end of the bridge. A second millimeter-wave radar is deployed at a second height below the first height (e.g., 20-30 meters), scanning obliquely downwards towards the middle and far ends of the bridge, specifically for collecting radar echo data from the middle and far ends of the bridge. The lidar covers the near end, the first millimeter-wave radar covers the far end, and the second millimeter-wave radar covers the middle and far ends at a lower position, forming a three-dimensional coverage of the entire bridge area.

[0043] In other words, this implementation method uses a deployment strategy of high-low combination and long-range division of labor to enable the first millimeter-wave radar to make up for the insufficient long-range coverage of the lidar, and the second millimeter-wave radar to bypass the obstruction of the bridge structure from a lower angle, thereby eliminating the inherent monitoring blind spots of a single sensor and improving the monitoring continuity under complex bridge structures.

[0044] Based on the aforementioned solution, please refer to Figure 2 In some implementations of this application, the steps of calculating the point cloud density of environmental point cloud data and performing planar segmentation based on the calculated point cloud density to obtain bridge surface point cloud data and vehicle candidate point cloud data include: Step S201: Projecting a bird's-eye view onto the environmental point cloud data and calculating the point cloud density of each region in the projected two-dimensional image using a sliding window; Step S202: Marking regions with point cloud density lower than a preset density threshold as background regions and removing them to obtain high-density regions; Step S203: Performing planar fitting using a random sampling consensus algorithm within the high-density regions to obtain the main bridge surface plane and the main vehicle plane; Step S204: Determining the bridge surface point cloud data based on the main bridge surface plane and determining the vehicle candidate point cloud data based on the main vehicle plane.

[0045] Understandably, this implementation, through a two-step strategy of first removing the background and then fitting a plane, can accurately and effectively identify vehicles from cluttered point clouds. Specifically, by first removing low-density background areas, this implementation significantly reduces the amount of data that needs to be processed for subsequent plane fitting, thus improving computational efficiency. Simultaneously, targeted plane fitting within high-density areas avoids interference from non-target regions such as bridge structures, thereby improving the accuracy of bridge surface and vehicle recognition.

[0046] Specifically, this implementation first projects a bird's-eye view onto the environmental point cloud data, converting the 3D spatial point cloud into a 2D top-down image. Then, a sliding window is used to calculate the point cloud density (points per unit area) for each region in the projected 2D image. Regions with point cloud density below a preset density threshold (e.g., 10 points / m²) are marked as background regions (non-vehicle target regions) and removed, retaining only high-density regions corresponding to vehicles and other physical targets. Next, within the selected high-density regions, a Random Sample Consensus (RANSAC) algorithm is used for plane fitting, respectively fitting the main bridge plane belonging to the bridge surface and the main vehicle plane belonging to the vehicles. Finally, the bridge surface point cloud data is determined based on the main bridge plane, and candidate vehicle point cloud data is determined based on the main vehicle plane.

[0047] Based on the aforementioned solution, please refer to Figure 3 In some implementations of this application, the step of using a random sampling consensus algorithm for plane fitting in a high-density region includes: Step S301: Iterating through a randomly sampled point cloud plane in the high-density region, and using the calculated distance from each point to the randomly sampled point cloud plane as the residual; Step S302: Monitoring the fluctuation amplitude of the residual during the iteration process, and dynamically reducing the residual threshold when the fluctuation amplitude exceeds a preset fluctuation threshold; Step S303: Filtering target point clouds with residuals less than the final threshold after dynamic adjustment to form the vehicle main plane.

[0048] Understandably, this implementation dynamically adjusts the residual threshold to enable the random sampling consensus algorithm to adapt to the differences in flatness in different areas of the bridge scenario. That is, by dynamically adjusting the residual threshold, the random sampling consensus algorithm maintains a more lenient selection criterion in flat areas to preserve the complete vehicle roof, while automatically tightening the criterion in complex areas to eliminate noise points, thereby improving the accuracy of the vehicle's principal plane fitting.

[0049] Specifically, within high-density areas, the random sampling consensus algorithm repeatedly performs the following operations: randomly selecting several point cloud points to form a candidate plane, and then calculating the distance from all other points in the area to this candidate plane, using this distance as the residual. During multiple iterations, the random sampling consensus algorithm continuously monitors the fluctuation range of the residual. When the residual fluctuation range exceeds a preset fluctuation threshold, it indicates that the terrain of the current area is relatively complex, such as the presence of uneven structures like vehicle roof racks or sunroofs. In this case, the random sampling consensus algorithm dynamically lowers the residual threshold (e.g., if the initial threshold is 0.2m, and the plane fitting error fluctuates more than 0.05m during iterations, the threshold can be lowered to 0.1m), only including points closer to the candidate plane in the statistics. After multiple iterations, the random sampling consensus algorithm selects all point cloud points whose residuals are less than the dynamically adjusted final threshold, and the plane formed by these points is taken as the vehicle's main plane.

[0050] It should be noted that traditional random sample consensus algorithms do not dynamically adjust the residual threshold, making it difficult to distinguish between bridge deck structures and vehicle targets, which can easily lead to misclassification, such as misidentifying bridge guardrails as vehicles. In this implementation, however, by dynamically adjusting the residual threshold within the random sample consensus algorithm, the accuracy of vehicle principal plane fitting can be effectively improved.

[0051] Based on the aforementioned scheme, some implementations of this application include using radar echo data to verify the main plane of the bridge deck when fitting the main plane of the bridge deck.

[0052] Understandably, this implementation method cross-validates the fitted bridge deck main plane by introducing radar echo data from millimeter-wave radar, thus solving the problem that a single lidar might mistakenly identify non-bridge deck structures as bridge decks in complex environments.

[0053] Specifically, after fitting a candidate bridge deck main plane from the environmental point cloud data using the random sample consensus algorithm, this implementation does not directly identify it as the real bridge deck. Instead, it further verifies the plane using radar echo data synchronously acquired by millimeter-wave radar. The basic logic of the verification is that the real bridge deck, as a continuous physical surface, should be detectable by millimeter-wave radar at all spatial locations and generate continuous and stable echo responses. If the radar echo data shows that the spatial location corresponding to the plane has discontinuous echoes, abnormal intensity, or no echoes at all, it indicates that the candidate plane may be a misidentification caused by bridge railings, water reflections under the bridge, or other noise.

[0054] In other words, this implementation method effectively eliminates the misidentification results that may be generated by a single sensor through the cross-verification mechanism of lidar and millimeter-wave radar, improves the reliability of the extraction of the main plane of the bridge deck, and lays an accurate spatial reference benchmark for the subsequent calculation of vehicle dynamic parameters based on the bridge deck coordinate system.

[0055] Based on the aforementioned scheme, in some implementations of this application, the radar echo data includes echo intensity information; the step of dividing the vehicle candidate point cloud data into regions according to the point cloud density and radar echo data to obtain the vehicle core candidate region includes: in the vehicle candidate point cloud data, determining the overlapping region where the point cloud density is greater than or equal to a first preset threshold and the echo intensity information is greater than or equal to a second preset threshold, and taking the overlapping region as the vehicle core candidate region.

[0056] Understandably, this implementation method, by simultaneously requiring point cloud density and echo intensity to meet thresholds, can eliminate non-metallic areas with dense point clouds but weak echoes, such as sunroofs, and can also eliminate edge areas with strong echoes but sparse point clouds, such as rearview mirrors, thereby selecting high-reliability areas representing the main structure of the vehicle from the vehicle candidate point cloud data as the core candidate areas of the vehicle.

[0057] It should be noted that high point cloud density means strong laser reflection, which is usually seen on solid parts such as the vehicle's roof or hood; high echo intensity means strong millimeter-wave radar reflection, which is also a typical characteristic of metal car bodies.

[0058] Based on the aforementioned scheme, in some implementations of this application, the steps of constructing a directed bounding box based on the vehicle core candidate region and adjusting the directed bounding box using radar echo data to obtain vehicle bounding box data include: calculating the covariance matrix of the point cloud within the vehicle core candidate region and extracting the largest eigenvector of the covariance matrix as the vehicle's principal direction; constructing an initial directed bounding box for the vehicle based on the vehicle's principal direction; parsing the vehicle's velocity vector from the radar echo data and calculating the angle between the velocity vector and the vehicle's principal direction to obtain the steering angle; determining that the vehicle is in a steering state in response to the steering angle exceeding a preset steering threshold, performing curve fitting on the front wheel region and rear wheel region of the subsequent region point cloud of the vehicle core, and correcting the shape of the initial directed bounding box based on the fitting result to generate vehicle bounding box data; or, in response to the steering angle not exceeding a preset steering threshold, using the initial directed bounding box as vehicle bounding box data.

[0059] Understandably, this implementation uses the velocity vector of millimeter-wave radar to sense the vehicle's motion state and performs special curve fitting correction for the steering state, which can improve the fitting accuracy of the vehicle bounding box in turning scenarios and provide a more accurate geometric benchmark for subsequent dynamic parameter calculations.

[0060] Specifically, after obtaining the core candidate region of the vehicle, this implementation first calculates the covariance matrix of the point cloud within that region and extracts the largest eigenvector of the covariance matrix as the vehicle's principal direction, which represents the approximate orientation of the vehicle body. Then, based on the vehicle's principal direction, an initial directed bounding box is constructed that can encompass the core candidate region of the vehicle. Simultaneously, this implementation parses the vehicle's velocity vector from the radar echo data, which represents the vehicle's true direction of motion. The steering angle is obtained by calculating the angle between the velocity vector and the vehicle's principal direction; the magnitude of this angle reflects whether the vehicle is turning. When the steering angle exceeds a preset steering threshold (e.g., 15°), the vehicle is determined to be in a turning state. In a turning state, the front and rear wheel regions of the vehicle will form non-parallel geometric structures, and a simple rectangular bounding box cannot accurately fit them. Therefore, this implementation performs curve fitting (e.g., quadratic curve fitting) on ​​the front and rear wheel regions and corrects the shape of the initial directed bounding box based on the fitting results, generating vehicle bounding box data that more closely resembles the actual vehicle body. When the steering angle does not exceed the steering threshold, the vehicle is in a straight-line driving state, and the initial oriented bounding box is directly used as the vehicle bounding box data. When obtaining the vehicle bounding box data, the accuracy of the center coordinates of the vehicle bounding box data can be corrected using the distance data carried in the radar echo data. For example, if the error exceeds 0.3m, weighted correction can be performed using the radar echo data.

[0061] Based on the aforementioned scheme, in some implementations of this application, the vehicle bounding box data carries the vehicle center coordinates and the vehicle's main direction information. Please refer to... Figure 4 The steps for calculating the dynamic parameters of a vehicle based on bridge surface point cloud data, vehicle bounding box data, and radar echo data include: Step S401: Determine the bridge coordinate system and the bridge principal axis vector based on the bridge surface point cloud data; Step S402: Map the vehicle center coordinates onto the bridge coordinate system to obtain the vehicle's spatial position; Step S403: Determine the vehicle's speed based on the temporal changes of the environmental point cloud data corresponding to the vehicle bounding box data and the Doppler frequency shift data of the radar echo data corresponding to the vehicle bounding box data; Step S404: Calculate the angle between the vehicle's principal direction information and the bridge principal axis vector, and determine the vehicle's driving direction based on the calculated angle; Step S405: Calculate the distance between the vehicle and a preset bridge edge based on the vehicle bounding box data and the bridge surface point cloud data, and use the calculated distance as the real-time distance between the vehicle and a preset safety boundary.

[0062] Understandably, this implementation method, by integrating bridge surface point cloud data, vehicle bounding box data, and radar echo data, synchronously calculates the vehicle's spatial position, speed, direction of travel, and real-time distance from the preset safety boundary under a unified bridge coordinate system, thereby providing comprehensive and real-time data support for bridge safety monitoring.

[0063] Specifically, this implementation first determines a bridge coordinate system with the bridge surface as a reference based on the bridge surface point cloud data, and extracts the longitudinal centerline of the bridge as the principal axis vector of the bridge (the vector direction is predefined, for example, the positive direction can be defined as from the end of the bridge to the beginning of the bridge). Since the vehicle center coordinates carried in the vehicle bounding box data are initially described based on the coordinate system of the LiDAR itself, it is necessary to map and transform these coordinates in the bridge coordinate system to obtain the true spatial position of the vehicle on the bridge surface.

[0064] In terms of speed calculation, this implementation adopts a dual-source fusion strategy. On the one hand, it uses the position changes of the environmental point cloud data corresponding to the vehicle bounding box data in the preceding and following time frames to calculate the speed. On the other hand, it uses the Doppler frequency shift data unique to millimeter-wave radar to directly obtain the speed value. The two are combined to determine the final driving speed.

[0065] In terms of direction calculation, this implementation calculates the angle between the vehicle's main direction information carried by the vehicle bounding box data and the bridge's main axis vector, and determines whether the vehicle is going up or down the bridge and its specific driving direction based on the sign and magnitude of this angle. It should be noted that this implementation pre-sets the direction of the bridge's main axis vector pointing towards the "going up bridge direction" as the positive direction. Therefore, when calculating the angle between the vehicle's main direction information and this main axis, the calculation includes: if the angle is acute (i.e., the vehicle is pointing in the same direction as the positive direction, the angle is 0°-90°), it is determined to be a positive angle, and the vehicle is in the process of going up the bridge; if the angle is obtuse (i.e., the vehicle is pointing in the opposite direction to the positive direction, the angle is 90°-180°), it is determined to be a negative angle, and the vehicle is in the process of going down the bridge. The specific value of the angle can be used to determine the vehicle's yaw angle relative to the bridge axis, i.e., the specific driving direction (such as whether it deviates from the lane line). Furthermore, when the angle error calculated by the lidar exceeds a preset threshold (e.g., 10°) due to vehicle obstruction or excessive distance, the velocity vector measured by millimeter-wave radar can be introduced for correction. For example, the correction formula can be: ;in, This is the corrected angle value. To determine the vehicle's yaw angle relative to the bridge axis using lidar alone. The velocity vector is measured by millimeter-wave radar. Let the principal axis vector of the bridge body be . To correct the weights, 0 ≤ The value is ≤1, determined by the reliability of the millimeter-wave radar velocity vector. Millimeter-wave radar directly senses the vehicle's true direction of motion (approaching or moving away from the radar) through the Doppler effect or displacement changes between consecutive frames. This velocity vector is independent of visual imaging and unaffected by obstructions, thereby correcting the angle calculation deviation of the lidar and ensuring the accuracy of lane identification for entering / exiting the bridge.

[0066] In terms of safe distance monitoring, this implementation method calculates the spatial distance based on the boundary position of the vehicle bounding box data and the preset bridge edge position in the bridge surface point cloud data, and uses the shortest distance obtained as the real-time distance between the vehicle and the preset safe boundary.

[0067] It should be noted that when both LiDAR and millimeter-wave radar are needed for comprehensive judgment, the features of the two sensors can be weighted and fused. Specifically, a self-attention mechanism can be used to calculate the feature similarity between the geometric features of the LiDAR point cloud and the dynamic features of the millimeter-wave radar. The geometric features of the LiDAR point cloud can include the length, width, and height of the vehicle bounding box, while the dynamic features of the millimeter-wave radar can include speed and distance. The feature similarity is calculated by the dot product of the feature vectors. Then, a gated fusion mechanism is used to evaluate the importance of the features from the two sensors. The gated fusion formula is g=σ(W·f+b), where σ() is the Sigmoid function, f is the input feature vector, W and b are learnable parameters, and g is the output gate weight. The input feature vector f is a multi-dimensional vector formed by concatenating the vehicle geometric features (such as the length, width, and height of the vehicle bounding box) extracted from the LiDAR environmental point cloud data and the dynamic features (such as speed and distance) extracted from the millimeter-wave radar radar echo data after spatiotemporal alignment. This gate weight is used to weight and fuse features from different sources. After weighted fusion, the vehicle dynamic parameters output by this application have high accuracy. Under normal circumstances, the error in spatial position does not exceed 0.5 meters, the error in driving speed does not exceed 2%, the real-time distance error from the preset safety boundary does not exceed 0.3 meters, and the error in driving direction does not exceed 5 degrees.

[0068] Based on the aforementioned scheme, in some implementations of this application, the monitoring method further includes: comparing the spatial position with a preset lane boundary; if the comparison result shows that the spatial position exceeds the preset lane boundary, it is determined to be a lane departure; comparing the driving speed with a preset speed threshold; if the driving speed exceeds the preset speed threshold, it is determined to be speeding; comparing the real-time distance with a preset distance threshold; if the real-time distance is less than the preset distance threshold, it is determined to be too close to the bridge; when it is determined to be at least one of lane departure, speeding, or being too close to the bridge, a warning module is triggered.

[0069] Understandably, this implementation method achieves automatic identification and early warning of abnormal driving behavior by comparing the calculated dynamic parameters with preset safety thresholds one by one. Specifically, after calculating the vehicle's dynamic parameters, this implementation method performs three independent comparison judgments. When at least one of these three comparison judgments is determined to be abnormal, the early warning module will be immediately triggered to issue an alert to management personnel or the driver. The first type compares the vehicle's spatial position with the preset lane boundary. If the spatial position exceeds the allowable range of the preset lane boundary, the vehicle is determined to be in a dangerous state of deviating from the lane. The second type compares the vehicle's speed with a preset speed threshold. If the speed exceeds the threshold, the vehicle is determined to be speeding. The third type compares the real-time distance between the vehicle and the preset safety boundary with a preset distance threshold. If the real-time distance is less than the threshold, the vehicle is determined to be too close to the edge of the bridge, posing a risk of colliding with the guardrail or falling off the bridge.

[0070] This application provides a bridge vehicle monitoring system, which includes at least one lidar, at least one millimeter-wave radar, at least one processor, and at least one memory. The lidar is used to collect environmental point cloud data of the target bridge area, and the millimeter-wave radar is used to collect radar echo data of the target bridge area. The processor, connected to both the lidar and the millimeter-wave radar, is used to receive the environmental point cloud data and the radar echo data. The processor and the memory are directly connected to each other, or communicate with each other through a communication interface, or are electrically connected through one or more communication buses or signal lines to achieve data transmission or interaction. The memory stores program instructions that can be executed by the processor, which can call and execute the program instructions to implement any of the bridge vehicle monitoring methods provided by the various implementations described above. For example, the following implementation is provided: The process involves acquiring environmental point cloud data and radar echo data. The environmental point cloud data is collected by lidar from the target bridge area, while the radar echo data is collected by millimeter-wave radar from the same area. Point cloud density is calculated from the environmental point cloud data, and planar segmentation is performed based on the calculated density to obtain bridge surface point cloud data and vehicle candidate point cloud data. Based on the point cloud density and radar echo data, the vehicle candidate point cloud data is further divided into regions to obtain core candidate vehicle regions. A directed bounding box is constructed based on the core candidate vehicle regions, and this bounding box is adjusted using radar echo data to obtain the vehicle bounding box data. Finally, based on the bridge surface point cloud data, vehicle bounding box data, and radar echo data, the dynamic parameters of the vehicle are calculated. These dynamic parameters include spatial position, speed, direction of travel, and real-time distance from a preset safety boundary.

[0071] For the specific implementation process of the above system, please refer to the bridge vehicle monitoring method provided in the above embodiment, which will not be repeated here.

[0072] The memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0073] A processor can be an integrated circuit chip with signal processing capabilities. This processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0074] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for monitoring bridge vehicles, characterized in that, Includes the following steps: The system acquires environmental point cloud data and radar echo data. The environmental point cloud data is collected by lidar on the target bridge area, and the radar echo data is collected by millimeter-wave radar on the target bridge area. The point cloud density is calculated from the environmental point cloud data, and the plane is segmented according to the calculated point cloud density to obtain bridge surface point cloud data and vehicle candidate point cloud data. Based on the point cloud density and the radar echo data, the vehicle candidate point cloud data is divided into regions to obtain the vehicle core candidate region. A directed bounding box is constructed based on the core candidate region of the vehicle, and the directed bounding box is adjusted using the radar echo data to obtain the vehicle bounding box data. Based on the bridge surface point cloud data, vehicle bounding box data, and radar echo data, the dynamic parameters of the vehicle are calculated. These dynamic parameters include spatial position, driving speed, driving direction, and real-time distance from the preset safety boundary.

2. The monitoring method according to claim 1, characterized in that, The lidar is deployed at a first height at the bridgehead of the target bridge to collect environmental point cloud data near the bridge. The millimeter-wave radar includes a first millimeter-wave radar and a second millimeter-wave radar; wherein, the first millimeter-wave radar is deployed at a first height at the bridgehead of the target bridge to collect radar echo data at the far end of the bridge body, and the second millimeter-wave radar is deployed at a second height at the bridgehead of the target bridge to collect radar echo data at the middle and far ends of the bridge body; the second height is lower than the first height.

3. The monitoring method according to claim 1, characterized in that, The steps of calculating the point cloud density of the environmental point cloud data and performing planar segmentation based on the calculated point cloud density to obtain bridge surface point cloud data and vehicle candidate point cloud data include: The environmental point cloud data is projected into a bird's-eye view, and the point cloud density of each region is calculated in the projected two-dimensional image using a sliding window. Regions with point cloud density below a preset density threshold are marked as background regions and removed to obtain high-density regions. Within the high-density region, a random sampling consensus algorithm is used for plane fitting to obtain the main plane of the bridge deck and the main plane of the vehicles. The bridge deck point cloud data is determined based on the main plane of the bridge deck, and the vehicle candidate point cloud data is determined based on the main plane of the vehicle.

4. The monitoring method according to claim 3, characterized in that, The step of performing plane fitting using a random sampling consensus algorithm within the high-density region includes: Iterate through the random sampling point cloud plane within the high-density region, and use the calculated distance from each point to the random sampling point cloud plane as the residual; During the iteration process, the fluctuation range of the residual is monitored, and when the fluctuation range exceeds the preset fluctuation threshold, the residual threshold is dynamically reduced. Target point clouds with residuals less than the final threshold after dynamic adjustment are selected to form the vehicle main plane.

5. The monitoring method according to claim 3, characterized in that, The process of fitting the main plane of the bridge deck includes verifying the main plane of the bridge deck using the radar echo data.

6. The monitoring method according to claim 1, characterized in that, The radar echo data includes echo intensity information; the step of dividing the vehicle candidate point cloud data into regions based on the point cloud density and the radar echo data to obtain the vehicle core candidate region includes: In the vehicle candidate point cloud data, an overlapping region is determined where the point cloud density is greater than or equal to a first preset threshold and the echo intensity information is greater than or equal to a second preset threshold, and the overlapping region is used as the vehicle core candidate region.

7. The monitoring method according to claim 1, characterized in that, The step of constructing a directed bounding box based on the vehicle's core candidate region and adjusting the directed bounding box using the radar echo data to obtain the vehicle bounding box data includes: Calculate the covariance matrix of the point cloud in the core candidate region of the vehicle, and extract the largest eigenvector of the covariance matrix as the main direction of the vehicle. Construct the initial oriented bounding box of the vehicle based on the vehicle's main direction; The vehicle's velocity vector is parsed from the radar echo data, and the angle between the velocity vector and the vehicle's main direction is calculated to obtain the steering angle. In response to the steering angle exceeding a preset steering threshold, the vehicle is determined to be in a steering state, and curve fitting is performed on the front wheel region and rear wheel region of the subsequent point cloud of the vehicle core. The shape of the initial oriented bounding box is corrected according to the fitting result to generate the vehicle bounding box data; or, in response to the steering angle not exceeding the preset steering threshold, the initial oriented bounding box is used as the vehicle bounding box data.

8. The monitoring method according to claim 1, characterized in that, The vehicle bounding box data carries the vehicle center coordinates and the vehicle's main direction information; the step of calculating the vehicle's dynamic parameters based on the bridge surface point cloud data, vehicle bounding box data, and radar echo data includes: The bridge coordinate system and the bridge principal axis vector are determined based on the bridge surface point cloud data. The vehicle's center coordinates are mapped into the bridge coordinate system to obtain the vehicle's spatial position. The vehicle's speed is determined based on the temporal changes of the environmental point cloud data corresponding to the vehicle bounding box data and the Doppler frequency shift data of the radar echo data corresponding to the vehicle bounding box data. Calculate the angle between the vehicle's main direction information and the bridge's main axis vector, and determine the vehicle's driving direction based on the calculated angle; Based on the vehicle bounding box data and the bridge surface point cloud data, the distance between the vehicle and the preset bridge edge is calculated, and the calculated distance is used as the real-time distance between the vehicle and the preset safety boundary.

9. The monitoring method according to claim 1, characterized in that, The monitoring method also includes: The spatial position is compared with the preset lane boundary. If the comparison result shows that the spatial position exceeds the preset lane boundary, it is determined to be a lane deviation. The driving speed is compared with a preset speed threshold. If the driving speed exceeds the preset speed threshold, it is determined to be speeding. The real-time distance is compared with a preset distance threshold. If the real-time distance is less than the preset distance threshold, it is determined that the distance to the bridge is too close. The warning module is triggered when the driver is determined to be in at least one of the following situations: lane departure, speeding, or being too close to the bridge.

10. A monitoring system for bridge vehicles, characterized in that, include: LiDAR is used to collect environmental point cloud data of the target bridge area; Millimeter-wave radar is used to collect radar echo data of the target bridge area; Memory, used to store one or more programs; The processor is connected to the lidar and the millimeter-wave radar respectively, and is used to receive the environmental point cloud data and radar echo data. When the one or more programs are executed by the processor, the method as described in any one of claims 1-9 is implemented.

Images