Integrated method and system for cooperative perception and intelligent management and control of vehicle and road

Abstract

The present application relates to the field of vehicle engineering, and more particularly to a vehicle-road cooperative perception and intelligent management and control integrated method and system, which extracts and fuses the multi-source data collected by roadside perspective on each vehicle in the road to obtain the roadside observation state of each vehicle; obtains the vehicle-end observation state of each vehicle from the vehicle itself, and performs state consistency verification on the obtained vehicle-end observation state and roadside observation state, so that the roadside observation state and the vehicle-end observation state correspond to the same vehicle; fuses the roadside observation state and the roadside observation state of the same vehicle to obtain the fusion observation state of the corresponding vehicle; and judges the risk of an accident of a certain vehicle in the road in combination with the fusion observation state of each vehicle. The present application generates a comprehensive perception target by feature-level fusion and vehicle-road information coupling, in combination with threshold matching and mutual verification correction, so as to determine the collision risk and give real-time early warning, thereby improving the perception accuracy and early warning timeliness.

Description

Technical Field

[0001] This invention belongs to the field of vehicle engineering technology, and in particular relates to a method and system for integrating vehicle-road cooperative perception and intelligent control. Background Technology

[0002] With the rapid development of the intelligent connected vehicle industry and the in-depth implementation of the "Transportation Powerhouse" strategy, vehicle-road cooperative technology has become a key path to improve traffic safety, traffic efficiency, and support advanced autonomous driving. Roadside units (RSUs) and onboard units (OBUs), as the core infrastructure of vehicle-road cooperation, are seeing their deployment and application become a focus of industry attention. Currently, several vehicle-road cooperative terminal solutions exist both domestically and internationally, such as Baidu's ACE Intelligent Transportation Engine, Qianfang Omni-T, and Hitachi ODD. However, practical engineering verification has revealed the following significant shortcomings in existing technologies regarding perception capabilities, information interaction, intelligent control, and deployment costs.

[0003] Some patented technologies have already made attempts to address the above problems. For example: Chinese patent publication number CN112116031A, published on December 22, 2020, entitled "Target Fusion Method, System, Vehicle and Storage Medium Based on Roadside Equipment", uses roadside cameras and radar as front-end basic equipment. Through a video microwave detection fusion computing unit, it fuses and deeply analyzes the video from the camera and some data from the radar to complete the output of information such as the position, speed, acceleration, and heading angle of the detected target vehicles and pedestrians, so as to provide perception fusion for autonomous vehicles and realize decision control of vehicle-road cooperative perception fusion. Chinese patent publication number CN115578709A, published on January 6, 2023, entitled "A Feature-Level Cooperative Perception Fusion Method and System for Vehicle-Road Cooperation," establishes a feature-level cooperative perception fusion framework for vehicle-road cooperation. Both the vehicle-side and roadside ends use PointPillars networks for point cloud feature extraction. By compressing the point cloud feature data of the roadside ends and transmitting it to the vehicle-side end, feature space correction and fusion are performed on the vehicle-side end to achieve vehicle-road cooperative perception.

[0004] The above solutions address some of the challenges of multi-sensor fusion perception on the roadside and feature-level collaborative fusion between vehicles and roads. However, the former is limited to the fusion output of the roadside and does not involve a two-way mutual verification and complementary correction mechanism between the information reported by the vehicle and the roadside perception results. Although the latter achieves feature-level fusion between the vehicle and the roadside, the fusion process is completed entirely on the vehicle. It does not treat the roadside edge computing module as an independent fusion decision unit, nor does it involve state consistency verification between the vehicle and the roadside or differentiated processing strategies when deviations exceed limits. Furthermore, it fails to directly apply the fusion results to real-time collision risk assessment and active safety warnings at the roadside.

[0005] Therefore, there is an urgent need to provide a vehicle-road cooperative perception and intelligent management terminal and method with high integration, excellent perception accuracy, strong real-time performance, and controllable cost. Summary of the Invention

[0006] In view of this, the present invention aims to provide a method and system for integrating vehicle-road cooperative perception and intelligent control, which performs feature-level fusion and vehicle-road information coupling, generates a comprehensive perception target through threshold matching and cross-verification correction, calculates collision risk, and provides real-time warnings. The present invention improves perception accuracy and warning timeliness while reducing deployment costs.

[0007] To achieve the above objectives, the technical solution created by this invention is implemented as follows: A method for integrating vehicle-road cooperative perception and intelligent control includes: S1: Feature extraction and fusion of multi-source data collected from each vehicle on the road from a roadside perspective will be performed to obtain the roadside observation status of each vehicle. S2: Obtain the vehicle-end observation status of each vehicle in the road, and verify the consistency between the obtained vehicle-end observation status and the roadside observation status obtained in step S1, so that the roadside observation status and the vehicle-end observation status correspond to the same vehicle. S3: Fuse the roadside observation status and vehicle-side observation status of the same vehicle determined in step S2 to obtain the fused observation status of the corresponding vehicle. S4: Combine the fused observation status of each vehicle on the road obtained from step S3, and determine the risk of an accident involving a certain vehicle on the road.

[0008] Furthermore, step S1 employs a cross-modal fusion model based on the YOLOv8 network to extract and fuse features from the captured road images and the radar point clouds of each vehicle on the road, obtaining the roadside observation status of each vehicle. The cross-modal fusion model includes a feature extraction unit, comprising a road feature extraction branch and a radar feature extraction branch. In the road feature extraction branch, the road image undergoes feature extraction processing by the backbone network of the YOLOv8 network, and then the extracted features are further processed by the neck network of the YOLOv8 network to perform cross-scale feature fusion and enhancement processing, obtaining... The system extracts road features; the radar feature extraction branch extracts features from the radar point cloud of each vehicle to obtain radar features; the feature fusion unit uses a cross-attention mechanism to perform bidirectional fusion of radar features and road features to obtain fused features; the detection unit includes a vehicle detection head and a lane detection head: in the vehicle detection head, the fused features are processed by the detection head of the original YOLOv8 network to output the driving information of the target vehicle in the roadside observation state; in the lane detection head, the fused features are processed by multiple convolution and activation operations to output the lane information in the roadside observation state.

[0009] Furthermore, the state consistency verification in step S2 includes: calculating the Euclidean distance between the vehicle's position in the roadside observation state and the vehicle's position in the vehicle-end observation state; calculating the minimum angle difference between the vehicle's heading angle in the roadside observation state and the vehicle's heading angle in the vehicle-end observation state; if the obtained Euclidean distance and minimum angle difference do not exceed their respective preset thresholds, then it is determined that the current roadside observation state and the vehicle-end observation state correspond to the same vehicle.

[0010] Furthermore, the preset thresholds for Euclidean distance and minimum included angle difference are as follows: D thr =D0+k D ×v V2X ; ; Among them, D thr and D0 and D1 represent the preset thresholds for Euclidean distance and minimum angle difference, respectively. k represents the preset static deviation base corresponding to the Euclidean distance and the minimum included angle difference, respectively. D and k θ These represent the threshold coefficients corresponding to the Euclidean distance and the minimum included angle difference, respectively. ， v V2X This indicates the vehicle's speed during the observation process at the vehicle end.

[0011] Furthermore, step S3 includes: S31: Use Kalman filtering to predict the fused observation state of the previous time step to obtain the estimated observation state of the current time step, as well as the corresponding prior estimated covariance matrix. S32: Combine the vehicle position coordinates in the current roadside observation state and the vehicle-end observation state into a position observation vector; S33: Based on the prior estimated covariance matrix obtained in step S31, calculate the Kalman gain at the current time using the following formula: ; Among them, K k Let K represent the Kalman gain at time k, H represent the observation matrix, and R represent the observation noise covariance matrix. Based on the Kalman gain at the current moment, the fused observation state at the current moment is obtained using the position observation vector obtained in step S32 through the following formula: ; in, Z represents the fused observation state at time k. k This represents the position observation vector at time k; S34: Using the fused observation state obtained in step S33 as the fused observation state of the previous moment, repeat the operations of steps S31 to S34 until the fusion of the roadside observation state and the vehicle-side observation state is completed.

[0012] Furthermore, the process of determining vehicle accident risk in step S4 includes determining the risk of a collision between the current vehicle and the vehicle in front, as well as the risk of the current vehicle colliding with other vehicles when passing through an intersection. Specifically, for determining the risk of a collision between the current vehicle and the vehicle in front: based on the fused observation states of the current vehicle and the vehicle in front, the relative distance and relative speed between the two vehicles are calculated; when the relative speed is less than or equal to 0, it indicates that there is no risk of a transient collision between the two vehicles; when the relative speed is greater than 0, the time of collision between the two vehicles is calculated using the following formula: ; Where TTC represents the collision time. Indicates relative distance. It indicates relative speed; if the collision time is greater than the preset time, it is determined that the two vehicles will collide; otherwise, it is determined that the two vehicles will not collide. To determine the risk of a collision between a vehicle and another vehicle at an intersection: At a given intersection, based on the historical trajectories of the current vehicle and the vehicle closest to it, an LSTM network is used to predict the future trajectories of the two vehicles. Based on the future trajectories of the two vehicles, the time difference between the two vehicles reaching the intersection point is calculated. If the time difference is less than a preset safe time, it is determined that the two vehicles will collide; otherwise, it is determined that the two vehicles will not collide.

[0013] A vehicle-road cooperative perception and intelligent control integrated system includes an integrated roadside perception and control unit deployed on the road and an in-vehicle intelligent terminal deployed on the vehicle. The in-vehicle intelligent terminal acquires the vehicle's own observation status. The integrated roadside perception and control unit includes a roadside multi-source perception module and an edge computing module. The roadside multi-source perception module collects multi-source data from each vehicle on the road from a roadside perspective. The edge computing module, based on the multi-source data and the vehicle's observation status, executes the vehicle-road cooperative perception and intelligent control integrated method provided by this invention to determine the risk of accidents involving vehicles on the road and transmits the risk information to the corresponding vehicle's in-vehicle intelligent terminal. The in-vehicle intelligent terminal issues an alarm notification based on the risk information.

[0014] Furthermore, the integrated roadside perception and control unit also includes: a communication module, which transmits the output information of the edge computing module to the vehicle-mounted intelligent terminal via V2X direct communication; and a power supply and environmental management module, which provides regulated power to all internal modules of the integrated roadside perception and control unit.

[0015] Furthermore, the in-vehicle intelligent terminal includes: a CAN bus interface module, which accesses the vehicle's internal network through dual CAN channels to collect vehicle driving information and operating status; a high-precision positioning module, which acquires the vehicle's location in real time; a V2X communication module, which packages the vehicle's driving information, location, and operating status into a vehicle-side observation status and sends it to the edge computing unit, as well as receiving risk information from the edge computing unit; an intelligent processing unit, which performs safety warning logic judgment on risk information and determines the warning level; a safety warning module, which issues corresponding prompts based on the warning level; and for high-level warnings, the safety warning module sends a linkage braking request to the vehicle to automatically complete emergency braking.

[0016] Compared with the prior art, the present invention can achieve the following beneficial effects: The vehicle-road cooperative perception and intelligent control integration method and system described in this invention achieves complementary and mutual verification between precise vehicle status and wide-area roadside perception through multi-sensor feature-level fusion and vehicle-road information coupling, improving the accuracy of perception and recognition and detection precision. Simultaneously, perception, fusion, and decision-making are all completed locally, significantly reducing end-to-end warning latency and meeting the requirements of active safety applications, which is significantly superior to cloud-dependent solutions. This invention achieves beyond-line-of-sight perception and millisecond-level warnings, effectively reducing the risk of typical accidents such as intersection collisions and ramp merging conflicts, and can effectively improve overall traffic efficiency. Attached Figure Description

[0017] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 A schematic flowchart illustrating the vehicle-road cooperative perception and intelligent control integration method described in an embodiment of the present invention; Figure 2 A flowchart illustrating the vehicle-road cooperative perception and intelligent control integration method described in the embodiments of the present invention; Figure 3 This invention provides an architectural diagram of the vehicle-road cooperative perception and intelligent control integrated system described in an embodiment of the invention. Figure 4 Architecture diagram of the integrated roadside perception and control unit described in the embodiment of the present invention; Figure 5 An architecture diagram of the in-vehicle intelligent terminal described in an embodiment of the present invention; Figure 6 A schematic diagram illustrating the process by which the intelligent processing unit and the security early warning module collaborate to complete an alarm, as described in an embodiment of the present invention. Figure 7Comparison results of the simulation fusion recognition rate comparison experiment described in the embodiments of the present invention; Figure 8 The comparison experiment results diagram of the early warning comparison experiment described in the embodiment of the present invention; Figure 9 The collision time-layered early warning statistical chart described in the embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0019] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0020] The invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] like Figure 1 and Figure 2 As shown in the embodiments of the present invention, the vehicle-road cooperative perception and intelligent control integration method includes: S1: Feature extraction and fusion of multi-source data collected from each vehicle on the road from a roadside perspective will be performed to obtain the roadside observation status of each vehicle.

[0022] In some embodiments, a cross-modal fusion model based on the YOLOv8 network is used to extract and fuse features from captured road images and radar point clouds of each vehicle on the road to obtain the roadside observation status of each vehicle. The cross-modal fusion model includes a feature extraction unit, a feature fusion unit, and a detection unit.

[0023] The feature extraction unit includes a road feature extraction branch and a radar feature extraction branch. In the road feature extraction branch, the road image is processed by the backbone network of the YOLOv8 network, and then the extracted features are fused and enhanced across scales by the neck network of the YOLOv8 network to obtain road features. The radar feature extraction branch extracts features from the radar point cloud of each vehicle to obtain radar features. In this embodiment of the invention, a channel attention mechanism is integrated into the backbone network of the YOLOv8 network. Specifically, the channel attention mechanism is inserted after the third and fourth C2f modules in the backbone network to perform channel-level recalibration on the output features of the previous C2f module, thereby enhancing the features of small targets. In addition, the radar feature extraction branch in this embodiment of the invention uses the PointPillars algorithm or the CenterPoint algorithm to extract features from the radar point cloud to generate radar features.

[0024] In the feature fusion unit, a cross-attention mechanism is used to perform bidirectional fusion of radar features and road features to obtain fused features. In this embodiment of the invention, the multi-head cross-attention mechanism specifically includes: Using road features as the first query matrix Q cam Radar features are used as the first key matrix K rad and the first value matrix V rad The first attention feature A1 is calculated using the following formula: ; Where d represents the number of dimensions, and softmax represents the softmax function; Using radar features as the second query matrix Q rad Road features are used as the second key matrix K cam Second general value matrix V cam The second attention feature A2 is calculated using the following formula: ; The first attention feature A1 and the second attention feature A2 are concatenated and then convolved to obtain the fused feature F. fused The process is as follows: F fused =Conv 1×1 (Concat(A1,A2)); Among them, Conv 1×1 This indicates a convolution operation with a kernel size of 1×1, and Concat indicates a channel concatenation operation.

[0025] The detection unit includes a vehicle detection head and a lane detection head. In the vehicle detection head, the fused features are processed by the detection head of the original YOLOv8 network to output the driving information of the target vehicle in the roadside observation state. In the lane detection head, the fused features are processed by multiple convolution and activation operations to output the lane information in the roadside observation state.

[0026] In this embodiment of the invention, the driving information of the target vehicle includes the target vehicle's two-dimensional detection box, category confidence score, three-dimensional position (latitude / longitude / elevation), and velocity vector (understandably, for a stationary target vehicle, its velocity vector is 0). The lane line information includes the lane line's two-dimensional detection box in the road image, category confidence score, and three-dimensional position (latitude / longitude / elevation). Furthermore, the specific feature processing procedure of the lane line detection head provided in this embodiment of the invention includes: using consecutive convolution operations, batch normalization, and SiLU activation operations as one convolution operation; fusing features through two consecutive convolution operations; and then performing a convolution operation with a kernel size of 1×1 to output the lane line information.

[0027] Furthermore, the cross-modal fusion model provided in this embodiment of the invention is trained using the following multi-task loss function: L total =λ det L det +λ line L line ; Among them, L total L represents the multi-task loss function. det L represents the loss function for the driving information of the target vehicle. line This represents the loss function for lane line information. Both loss functions use the cross-entropy loss function, λ. det and λ line This represents the corresponding loss weight, specifically set to λ. det =1.0, λ line =0.5.

[0028] In this embodiment of the invention, the road images and radar point clouds are also aligned in practice and in space. Specifically, based on GPS / PTP, each frame of the captured road image and radar point cloud is stamped with a nanosecond-level timestamp to ensure the temporal alignment of the data. The radar point cloud is transformed from the radar coordinate system to the coordinate system of the road image, and then the two are unified to the WGS-84 geographic coordinate system. The coordinate transformation is completed using an offline calibrated extrinsic parameter matrix (such as a rotation matrix and a translation vector).

[0029] S2: Obtain the vehicle-side observation state of each vehicle in the road, and verify the consistency between the obtained vehicle-side observation state and the roadside observation state obtained in step S1, so that the roadside observation state and the vehicle-side observation state correspond to the same vehicle.

[0030] In this embodiment of the invention, the vehicle-side observation status is obtained from the vehicle's built-in BSM (Blind Spot Monitoring System), and its content is consistent with the roadside observation status. Furthermore, this embodiment of the invention also performs spatiotemporal synchronization and alignment between the vehicle-side observation status and the roadside observation status.

[0031] In some embodiments, state consistency verification includes: calculating the Euclidean distance between the vehicle's position in the roadside observation state and the vehicle's position in the vehicle-end observation state, expressed as: ; Where D represents the Euclidean distance between the positions of the two vehicles. This indicates the vehicle's position in the vehicle-side observation state. Indicates the position of the vehicle in the roadside observation state; The minimum angle difference between the vehicle's heading angle in the roadside observation state and the vehicle's heading angle in the vehicle-end observation state is expressed as: ; in, θ represents the minimum angle difference. V2X and θ RS These represent the heading angles of the vehicle in the vehicle-side observation state and the roadside observation state, respectively. If the obtained Euclidean distance and the minimum included angle difference do not exceed their respective preset thresholds, then the current roadside observation state and the vehicle-side observation state are considered to correspond to the same vehicle.

[0032] The preset thresholds for Euclidean distance and minimum angle difference are as follows: D thr =D0+k D ×v V2X ; ; Among them, D thr and D0 and D1 represent the preset thresholds for Euclidean distance and minimum angle difference, respectively. k represents the preset static deviation base corresponding to the Euclidean distance and the minimum included angle difference, respectively. D and k θ These represent the threshold coefficients corresponding to the Euclidean distance and the minimum included angle difference, respectively. ， v V2X This indicates the vehicle's speed during the observation process at the vehicle end.

[0033] In this embodiment of the invention, D0 = 1m. k D =0.03s (multiplied by vehicle speed to obtain dynamic margin, unit: m), k θ =0.2° / (m / s). As the vehicle speed increases, the preset threshold increases linearly to avoid misjudgment due to sensor delay or positioning noise. In low-confidence scenarios (such as rainy days or when the vehicle is partially obscured), the two preset thresholds are multiplied by an expansion coefficient of 1.5.

[0034] S3: Fuse the roadside observation status and vehicle-side observation status of the same vehicle determined in step S2 to obtain the fused observation status of the corresponding vehicle.

[0035] In some embodiments, step S3 includes: S31: Use Kalman filtering to predict the fused observation state of the previous time step, and obtain the estimated observation state of the current time step, as well as the corresponding prior estimated covariance matrix.

[0036] In this embodiment of the invention, the fused observation state vector is defined as X=[x,y,v] x ,v y ] T Where (x,y) represents the position of the vehicle, (v... x ,v y () represents the vehicle's speed. The process of predicting the fused observation state using Kalman filtering includes: Based on the estimated observation state at time k-1 The covariance matrix P k-1 The following Kalman filtering process is performed to obtain the estimated observation state at time k. and prior estimation of the covariance matrix : ; ; Where F represents the state transition matrix and Q represents the noise covariance matrix in the prediction process. In this embodiment of the invention, the state transition matrix F uses a constant velocity model for trajectory prediction, that is: ; in, This represents the trajectory prediction interval. Furthermore, the noise covariance matrix Q is a diagonal matrix, i.e., Q = diag(q). x ,q y ,q vx ,q vy ), q x and q y Let q represent the process noise variances of the vehicle's position in the x and y directions, respectively. vx and qvy These represent the process noise variances of the vehicle speed in the x and y directions, respectively. Specifically, in this embodiment of the invention, q is taken as... x =q y =0.1m 2 q vx =q vy =0.5m 2 / s 2 .

[0037] S32: Combine the vehicle position coordinates in the current roadside observation state and the vehicle-end observation state into a position observation vector.

[0038] Understandably, the position observation vector Z can be represented as Z=[x V2X ,y V2Y ,x RS ,y RS ], where (x V2X ,y V2Y (x) represents the vehicle position in the vehicle-side observation state. RS ,y RS () indicates the vehicle position in the roadside observation state.

[0039] S33: Based on the prior estimated covariance matrix obtained in step S31, calculate the Kalman gain at the current time using the following formula: ; Among them, K k Let K represent the Kalman gain at time k, H represent the observation matrix, and R represent the observation noise covariance matrix. Based on the Kalman gain at the current moment, the fused observation state at the current moment is obtained using the position observation vector obtained in step S32 through the following formula: ; in, Z represents the fused observation state at time k. k This represents the position observation vector at time k.

[0040] In this embodiment of the invention, the observation matrix H is specifically: ; The observation noise covariance matrix R is dynamically assigned values ​​based on the confidence level of each observation state: R=diag(σ V2X 2 ,σ V2X 2 ,σ RS 2 ,σ RS 2 ); Where, σV2X This represents the standard deviation of the vehicle position in the vehicle-end observation state. In this embodiment of the invention, σ is taken as... V2X =0.1m, σ RS The standard deviation of the vehicle position in the roadside observation state is represented by the detection confidence score for the vehicle output by the cross-modal fusion model, which is obtained by mapping using the following formula: σ RS =0.5×(1-Conf); Where Conf represents the detection confidence level for the vehicle output by the cross-modal fusion model. It should be noted that, as shown in the above equation, only the detection confidence level Conf is the independent variable on the right side of the equation. However, the detection confidence level Conf has no dimension. Therefore, in this embodiment of the invention, the result obtained through the above equation will be forcibly appended with units (meters) to obtain the standard deviation σ. RS .

[0041] S34: Using the fused observation state obtained in step S33 as the fused observation state of the previous moment, repeat the operations of steps S31 to S34 until the fusion of the roadside observation state and the vehicle-side observation state is completed.

[0042] In this embodiment of the invention, the fusion observation status is also determined based on the data loss situation, specifically: If the vehicle-side observation status is lost or cannot be obtained, then the roadside perception will be relied upon entirely, that is, the fused observation status is the roadside observation status. If the roadside observation status is lost, the fused observation status becomes the vehicle-side observation status, and tracking is maintained. If both the vehicle-side observation state and the roadside observation state are valid, but the Euclidean distance D and the minimum angle difference calculated in step S2 are not valid, then the vehicle-side observation state and the roadside observation state are not valid. If the preset matching conditions are not met, but the preset reasonable range is not exceeded, it is determined that there is a conflict between the vehicle-side observation state and the roadside observation state, but both are within a reasonable range. The operation of step S3 is adopted to fuse the roadside observation state and the vehicle-side observation state of the same vehicle through Kalman filtering. If both the vehicle-side observation and the roadside observation are valid, but the Euclidean distance D and the minimum angle difference are different... If both exceed a reasonable range, it is determined that one of the vehicle-side observation status and the roadside observation status has failed, and a cross-verification correction is initiated: if the vehicle-side signal is strong, the vehicle-side signal is trusted; if the roadside signal has high confidence, the roadside signal is trusted.

[0043] In this invention, the effective determination of the vehicle-side observation status and the roadside observation status is as follows: the vehicle-side BSM is receiving normally and the confidence level in the roadside observation status is not less than 0.5; the matching condition is: the Euclidean distance D is greater than the preset Euclidean distance, or the minimum included angle difference. The difference is greater than the preset minimum angle difference; a reasonable range is: the Euclidean distance D is less than three times the preset Euclidean distance, and the minimum angle difference... Less than three times the preset minimum included angle difference; This invention uses Kalman filtering to fuse the roadside observation status and vehicle-side observation status of the same vehicle, realizing "software and hardware complementarity" of vehicle-road perception, effectively solving the blind spots and false detection problems of single perception sources, and achieving a comprehensive perception and recognition accuracy of over 98%.

[0044] S4: Combine the fused observation status of each vehicle on the road obtained from step S3, and determine the risk of an accident involving a certain vehicle on the road.

[0045] In some embodiments, the process of determining vehicle accident risk includes determining the risk of a collision between the current vehicle and the vehicle in front, and the risk of the current vehicle colliding with other vehicles when passing through an intersection.

[0046] To determine the risk of a collision between the current vehicle and the vehicle ahead: Based on the fused observations of the current and ahead vehicles, calculate the relative distance and relative speed between the two vehicles; when the relative speed is less than or equal to 0, it indicates that there is no risk of a transient collision between the two vehicles; when the relative speed is greater than 0, calculate the time of collision using the following formula: ; Wherein, TTC represents the collision time, relative distance represents the relative speed; when the collision time is greater than the preset time, it is determined that the two vehicles will collide, otherwise it is determined that the two vehicles will not collide.

[0047] In this embodiment of the invention, the safety margin is adjusted based on the safe following distance recommended by AASHTO (National Highway Traffic and Transportation Association) (2 seconds) and the typical braking reaction time of urban drivers (0.8~1.2 seconds), combined with the calculated actual processing delay (≤80ms). Specifically, the preset time is set to 3s in this embodiment of the invention.

[0048] To determine the risk of a collision between a vehicle and another vehicle at an intersection: At a given intersection, based on the historical trajectories of the current vehicle and the vehicle closest to it, an LSTM network is used to predict the future trajectories of the two vehicles. Based on the future trajectories of the two vehicles, the time difference between the two vehicles reaching the intersection point is calculated. If the time difference is less than a preset safe time, it is determined that the two vehicles will collide; otherwise, it is determined that the two vehicles will not collide.

[0049] In this embodiment of the invention, a safety time is set to 1.5 seconds. That is, when the time difference is less than the preset 1.5 seconds, it is determined to be a high-risk conflict, meaning a conflict between the two vehicles is expected; otherwise, it is determined that no conflict will occur. Furthermore, this embodiment of the invention combines the collision time with the conflict risk at the intersection for further judgment. That is, when the time difference is within the range of [1.5, 3), it is determined to be a moderate risk, and a collision warning is generated. In this embodiment of the invention, a public dataset (such as nuScenes) is used to pre-train the LSTM network, and then fine-tuned using local traffic flow data, so that the root mean square error (RMSE) of the resulting LSTM network is less than 0.25m (2-second prediction).

[0050] This invention also provides an integrated system for vehicle-road cooperative perception and intelligent control, combining... Figures 1-3 This includes an integrated roadside perception and control unit deployed on the road and an in-vehicle intelligent terminal deployed on the vehicle. The in-vehicle intelligent terminal acquires the vehicle's own observation status. The integrated roadside perception and control unit includes a roadside multi-source perception module and an edge computing module, wherein: the roadside multi-source perception module collects multi-source data from each vehicle on the road from a roadside perspective; the edge computing module, based on the multi-source data and the vehicle's observation status, executes the vehicle-road cooperative perception and intelligent control integration method provided by this invention to determine the risk of accidents involving vehicles on the road and transmits the risk information to the corresponding vehicle's in-vehicle intelligent terminal. The in-vehicle intelligent terminal issues an alarm notification based on the risk information.

[0051] In this embodiment of the invention, the edge computing module is embedded within the RSU and is implemented using the "Horizon Journey 5" automotive-grade heterogeneous computing chip. This chip has passed AEC-Q100 Grade 2 automotive-grade certification and operates at temperatures ranging from -40℃ to 85℃. The edge computing module includes an ARM architecture, CPU, GPU, and neural network processing unit (NPU), with a computing power of ≥20 TOPS (INT8). Furthermore, the edge computing module is equipped with LPDDR4X / 5 memory (≥8GB) and eMMC / UFS storage (≥64GB), running an embedded Linux or real-time operating system. In addition, the risk information output by the edge computing module in this embodiment includes: hazard type (forward collision, intersection collision, pedestrian in blind spot, etc.), location of the hazardous target, and recommended measures (deceleration, braking, lane change and avoidance, etc.).

[0052] In this embodiment of the invention, the roadside multi-source perception module adopts a multi-sensor coaxial integrated structure, including a camera device, a main radar, and a secondary lidar. The optical axis of the camera device is parallel to the line-of-sight axis of the main radar, pointing in the direction of vehicle movement (or longitudinal direction of the road). It uses an 8-megapixel or higher global shutter CMOS sensor, equipped with an electric zoom lens, and supports HDR and low-light enhancement. The main radar uses a 77GHz or 79GHz high-resolution imaging radar, with a detection beam horizontal angle within the range of ±45° to ±60°, outputting point cloud and target list of the road scene. The detection range of the main radar is not less than 250 meters. The secondary lidar is used in complex intersections or scenarios requiring high-precision curb detection to jointly collect point cloud information of the scene with the main radar. The secondary lidar can be a 16-line or 32-line lidar, with a scanning angle of 120° and a point cloud frame rate of not less than 10Hz. In this embodiment of the invention, the camera, main radar, and secondary lidar are jointly calibrated at the factory: the intrinsic parameters of the camera are solved using a checkerboard calibration board and a corner detection algorithm, resulting in a reprojection error of less than 0.5 pixels after intrinsic parameter calibration; the extrinsic parameter matrices between each sensor are solved based on a common observation target, resulting in positional deviations between the camera and the main radar and secondary lidar of no more than 0.1m and 0.05m, respectively, after extrinsic parameter calibration. This calibration unifies the coordinate systems of the three components to the base coordinate system, eliminating the need for manual on-site calibration.

[0053] In this embodiment of the invention, the integrated roadside sensing and control unit is integrated into an integrated protective housing. Specifically, the integrated protective housing is made of cast aluminum or sheet metal with an IP65 or higher protection rating, and has a layered support structure inside. The back of the integrated protective housing has a mounting bracket interface for fixing to streetlight poles or dedicated columns; the bottom has an aviation plug interface for external power supply and network connection; the sides of the housing have louvered heat dissipation fins, an internal airflow duct, and a condensate drain hole at the bottom, ensuring waterproof, dustproof, and heat dissipation performance for all-weather outdoor operation. Furthermore, the camera equipment, main radar, and secondary lidar are rigidly fixed to the internal substrate of the integrated protective housing using shock-resistant brackets. The distance between the center of the camera equipment and the center of the main radar is 15cm, and the distance between the center of the main radar and the center of the secondary lidar is 12cm, with an installation error of no more than 2mm. All three components have a shared transparent window in front, with an ITO heat-coated surface and an airflow nozzle interface for automatic defrosting, defogging, and dust removal.

[0054] In some embodiments, the integrated roadside sensing and control unit, such as Figure 4 As shown, the integrated roadside perception and control unit also includes a communication module and a power supply and environmental management module. The communication module transmits the output information of the edge computing module to the onboard intelligent terminal via V2X direct communication. The power supply and environmental management module provides regulated power to all internal modules in the integrated roadside perception and control unit.

[0055] In this embodiment of the invention, the communication module is based on the LTE-V2X PC5 interface or 5G-V2X direct communication, with a built-in C-V2X protocol stack and RF front-end supporting 3GPP Rel.14 / 16. It has a maximum transmit power of 23dBm, a receive sensitivity better than -95dBm, a communication distance >300 meters (line-of-sight), end-to-end latency ≤80ms, and a packet loss rate ≤4%. This module is used to exchange vehicle-side observation status information with the vehicle-mounted intelligent terminal in real time. Furthermore, the communication module provided in this embodiment of the invention also utilizes cellular communication supporting 4G / 5G NR access to transmit statistical data (such as cross-sectional traffic, average vehicle speed, and event summaries) and event summaries processed by the edge computing module to the cloud control platform, and receives regional scheduling instructions issued by the platform.

[0056] In some embodiments, the in-vehicle intelligent terminal, such as Figure 5 As shown, the system includes a CAN bus interface module, a high-precision positioning module, a V2X communication module, an intelligent processing unit, and a safety warning module. The CAN bus interface module accesses the vehicle's internal network via dual CAN channels to collect vehicle driving information and operating status. The high-precision positioning module acquires the vehicle's location in real time. The V2X communication module packages the vehicle's driving information, location, and operating status into a vehicle-side observation status and sends it to the edge computing unit, while also receiving risk information from the edge computing unit. The intelligent processing unit performs safety warning logic judgment on the risk information to determine the warning level. The safety warning module issues corresponding prompts based on the warning level; for high-level warnings, the safety warning module sends a linkage braking request to the vehicle to automatically complete emergency braking. In this embodiment, the in-vehicle intelligent terminal is integrated inside an automotive-grade compact metal housing (IP54 protection).

[0057] In this embodiment of the invention, the CAN bus interface module connects to the vehicle's internal network via dual CAN channels (CAN2.0B or CAN FD) to collect data such as vehicle speed, RPM, accelerator pedal opening, brake pressure, steering angle, gear position, and ESC / ABS status. The high-precision positioning module integrates a BeiDou-3 / GPS dual-frequency RTK positioning module (horizontal positioning accuracy ≤2.5cm+1ppm, orientation accuracy ≤0.2°) and a built-in 6-axis IMU (accelerometer ±8g, gyroscope ±500° / s). It supports a tightly coupled GNSS / IMU navigation algorithm with an output frequency ≥10Hz. The V2X communication module uses the same LTE-V2X or 5G-V2X module as the communication module in the integrated roadside perception and control unit, supporting PC5 direct communication and Uu cellular communication to achieve V2V and V2I information exchange. The intelligent processing unit is an automotive-grade MCU or SoC (such as Infineon AURIX or NXP S32K) with a main frequency ≥200MHz. It is used for vehicle data organization, safety warning logic judgment, and message parsing with the integrated roadside perception and control unit. The safety warning module includes an audible and visual alarm (buzzer + LED indicator) and a CAN bus active safety interface. The buzzer emits different frequency alerts according to the warning level, and the LEDs display different colors synchronously. High-level warnings send a linkage braking request to the vehicle system via the CAN bus (compliant with ISO 15623 standard). In this embodiment, the in-vehicle intelligent terminal also includes a storage unit with built-in 8GB eMMC secure storage, which cyclically records driving data and communication logs and supports event latching. The alarm process completed by the intelligent processing unit and the safety warning module in this embodiment is as follows: Figure 6 As shown.

[0058] To verify the effectiveness of the vehicle-road cooperative perception and intelligent control integration method and system provided by the present invention, two comparative experiments were conducted in the embodiments of the present invention.

[0059] The first comparative experiment was a comparison of the simulated fusion recognition rate. Specifically, in this embodiment of the invention, the publicly available dataset nuScenes and a self-built dataset of challenging scenarios such as nighttime and rainy days on urban expressways were used to compare the fusion method provided by this invention (i.e., the improved Yolov8 network and cross-modal fusion model) with the conventional post-fusion method (the classic CenterPoint + image 2D bounding box IoU matching strategy). The evaluation metrics were average precision, recall, and location error. The comparison results are as follows: Figure 7 As shown. From Figure 7 As can be seen, the feature-level fusion method of this invention has a significantly better recognition rate than post-fusion in all scenarios, especially at night (improving by about 8.8 percentage points), in rainy weather (improving by about 13.2 percentage points), and in occluded scenarios (improving by about 16.5 percentage points), demonstrating the robustness of multimodal feature-level fusion to environmental interference.

[0060] The first comparative experiment was a warning comparison experiment. Specifically, this embodiment of the invention was tested at the merging area of ​​an expressway ramp in a certain region, from November 2025 to February 2026 (covering winter icy and snowy weather). Specifically, an integrated roadside perception and control unit was installed on a light pole at the intersection of the ramp and the main road. The camera and radar of the integrated roadside perception and control unit covered approximately 150 meters of the ramp and 200 meters of the main road. Five vehicles (five in total) were equipped with onboard units (OBUs) and were compared with five control vehicles that relied solely on onboard FCW (Forward Collision Warning) systems and did not have the system provided by this invention. During the test, if the collision time TTC ≤ 3.0s, the system issued a warning, and the driver's subjective evaluation was "timely warning, avoiding danger" or the data record showed that the driver implemented effective braking / avoidance, then the warning was considered successful. The test results are as follows: Figure 8 and Figure 9 As shown, where, Figure 8 The results of the comparative experiment are shown. Figure 9 The collision time-based warning statistics are shown. From... Figure 8 As can be seen, the false alarm rate of the terminal of this invention is 4.2%, while the false alarm rate of the control group relying solely on vehicle-mounted FCW is as high as 25.4%, a reduction of approximately 83.5%. The principle is that vehicle-mounted FCW is limited by the sensor installation position and field of view. In scenarios with obstructed views, such as ramp merging areas, vehicles approaching from the front or sides / rear are easily blocked and cannot be detected in time. This invention uses an integrated roadside perception and control unit to perform beyond-line-of-sight perception from the roadside perspective (covering approximately 150 meters of ramps and 200 meters of main road). Combined with the vehicle-road information coupling processing in the edge computing module, the vehicle-mounted BSM and roadside perception results are cross-checked bidirectionally, effectively compensating for the blind spots of single vehicle-mounted perception, thereby significantly reducing the false alarm rate.

[0061] At the same time, from Figure 8 As can be seen, the false alarm rate of the system provided by this invention is 6.2%, while that of the control group is 13.5%, a reduction of approximately 54.1%. The principle is as follows: In the feature-level fusion stage, this invention uses a cross-attention mechanism to fuse camera and millimeter-wave radar data, and suppresses false alarms from a single sensor through multimodal complementary verification; in the decision-making stage, it further performs spatiotemporal correlation and consistency verification of the target through vehicle-road information coupling. When the deviation between the two exceeds a reasonable range, mutual verification correction is triggered to avoid false warnings triggered by false detections from a single sensing source, thereby improving the credibility of the warning and the driver's acceptance of the system.

[0062] In addition, from Figure 8It can be seen that the average early warning advance time of the system provided by the present invention is 3.8 seconds, while that of the control group is 2.1 seconds, representing an improvement of approximately 81%. The principle lies in that the present invention sinks all perception, fusion and decision-making to the local roadside edge computing module for completion, and the end-to-end early warning delay does not exceed 100ms, which is far superior to the solution relying on cloud decision-making (usually exceeding 200ms). Meanwhile, the roadside perception coverage is much larger than the perception distance of a single vehicle, so potential conflict targets can be identified at a farther distance. Taking Figure 9 the secondary early warning (1.5s<TTC≤3s) as an example, the driver's braking response rate reaches 66.7%, indicating that the average early warning advance time of 3.8 seconds has reserved sufficient reaction and operation time for the driver.

[0063] Figure 9 shows that the tertiary early warning (TTC≤1.5s) is triggered 2 times. Although the driver has no time to react due to the too short time (the response rate is 0%), the AEB linked braking is successfully triggered both times, with a success rate of 100%. The principle lies in that during the tertiary early warning, in addition to the acousto-optic alarm, the present invention also sends a linked braking request to the vehicle AEB system through the CAN bus, and the vehicle active safety system takes over the emergency braking, which makes up for the physiological limitation that the driver cannot react within an extremely short time window. For the primary early warning (37 times) and the secondary early warning (6 times), braking is suggested through HMI prompt and acousto-optic alarm respectively, and the driver takes the lead in the hazard avoidance operation, which reflects the rationality of hierarchical response: no intervention in driving in low-risk scenarios, and active intervention in high-risk scenarios.

[0064] In summary, through vehicle-road information coupling and edge computing collaboration, the present invention is significantly superior to the solution relying only on the vehicle-mounted FCW in four dimensions: missing alarm rate, false alarm rate, early warning advance time and AEB linkage success rate, which verifies the effectiveness and engineering value of the integrated solution of collaborative vehicle-road perception and intelligent management and control in real road scenarios.

[0065] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in the present invention can be achieved, which is not limited herein.

[0066] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for integrating vehicle-road cooperative perception and intelligent control, characterized in that, include: S1: Feature extraction and fusion of multi-source data collected from each vehicle on the road from a roadside perspective will be performed to obtain the roadside observation status of each vehicle. S2: Obtain the vehicle-end observation status of each vehicle in the road, and verify the consistency between the obtained vehicle-end observation status and the roadside observation status obtained in step S1, so that the roadside observation status and the vehicle-end observation status correspond to the same vehicle. S3: Fuse the roadside observation status and vehicle-side observation status of the same vehicle determined in step S2 to obtain the fused observation status of the corresponding vehicle. S4: Combine the fused observation status of each vehicle on the road obtained from step S3, and determine the risk of an accident involving a certain vehicle on the road.

2. The vehicle-road cooperative perception and intelligent control integration method according to claim 1, characterized in that, Step S1 uses a cross-modal fusion model based on the YOLOv8 network to extract and fuse features from the captured road images and the radar point clouds of each vehicle in the road to obtain the roadside observation status of each vehicle. Cross-modal fusion models include: The feature extraction unit includes a road feature extraction branch and a radar feature extraction branch. In the road feature extraction branch, the road image is processed by the feature extraction of the backbone network of the YOLOv8 network, and then the extracted features are fused and enhanced by the neck network of the YOLOv8 network to obtain the road features. The radar feature extraction branch extracts features from the radar point cloud of each vehicle to obtain the radar features. The feature fusion unit uses a cross-attention mechanism to fuse radar features and road features bidirectionally to obtain fused features; The detection unit includes a vehicle detection head and a lane detection head: In the vehicle detection head, the fused features are processed by the detection head of the YOLOv8 network to output the driving information of the target vehicle in the roadside observation state; In the lane detection head, the fused features are processed by multiple convolution and activation operations to output the lane information in the roadside observation state.

3. The vehicle-road cooperative perception and intelligent control integration method according to claim 1, characterized in that, The state consistency check in step S2 includes: Calculate the Euclidean distance between the vehicle's position in the roadside observation state and the vehicle's position in the vehicle-end observation state; Calculate the minimum angle difference between the heading angle of the vehicle in the roadside observation state and the heading angle of the vehicle in the vehicle-end observation state; If the obtained Euclidean distance and minimum included angle difference do not exceed their respective preset thresholds, then the current roadside observation state and the vehicle-side observation state are considered to correspond to the same vehicle.

4. The vehicle-road cooperative perception and intelligent control integration method according to claim 3, characterized in that, The preset thresholds for Euclidean distance and minimum angle difference are as follows: D thr =D0+k D ×v V2X ; ； Among them, D thr and D0 and D1 represent the preset thresholds for Euclidean distance and minimum angle difference, respectively. k represents the preset static deviation base corresponding to the Euclidean distance and the minimum included angle difference, respectively. D and k θ These represent the threshold coefficients corresponding to the Euclidean distance and the minimum included angle difference, respectively. ， v V2X This indicates the vehicle's speed during the observation process at the vehicle end.

5. The vehicle-road cooperative perception and intelligent control integration method according to claim 1, characterized in that, Step S3 includes: S31: Use Kalman filtering to predict the fused observation state of the previous time step to obtain the estimated observation state of the current time step, as well as the corresponding prior estimated covariance matrix. S32: Combine the vehicle position coordinates in the current roadside observation state and the vehicle-end observation state into a position observation vector; S33: Based on the prior estimated covariance matrix obtained in step S31, calculate the Kalman gain at the current time using the following formula: ； Among them, K k Let K represent the Kalman gain at time k, H represent the observation matrix, and R represent the observation noise covariance matrix. Based on the Kalman gain at the current moment, the fused observation state at the current moment is obtained using the position observation vector obtained in step S32 through the following formula: ； in, Z represents the fused observation state at time k. k This represents the position observation vector at time k; S34: Using the fused observation state obtained in step S33 as the fused observation state of the previous moment, repeat the operations of steps S31 to S34 until the fusion of the roadside observation state and the vehicle-side observation state is completed.

6. The vehicle-road cooperative perception and intelligent control integration method according to claim 1, characterized in that, The process of determining vehicle accident risk in step S4 includes determining the risk of a collision between the current vehicle and the vehicle in front, and the risk of the current vehicle colliding with other vehicles when passing through an intersection, wherein: Regarding the assessment of the risk of a collision between the current vehicle and the vehicle in front: Based on the fused observations of the current vehicle and the vehicle ahead, the relative distance and relative speed between the two vehicles are calculated. When the relative speed is less than or equal to 0, it indicates that there is no risk of transient collision between the two vehicles. When the relative speed is greater than 0, the time of collision between the two vehicles is calculated using the following formula: ； Where TTC represents the collision time. Indicates relative distance. It indicates relative speed; if the collision time is greater than the preset time, it is determined that the two vehicles will collide; otherwise, it is determined that the two vehicles will not collide. To determine the risk of a collision with other vehicles when the current vehicle passes through an intersection: At a certain intersection, based on the historical driving trajectories of the current vehicle and the vehicle closest to the current vehicle, an LSTM network is used to predict the future driving trajectories of the two vehicles. Based on the future driving trajectories of the two vehicles, the time difference between the two vehicles reaching the intersection point is calculated. If the time difference is less than the preset safety time, it is determined that the two vehicles will collide; otherwise, it is determined that the two vehicles will not collide.

7. A vehicle-road cooperative perception and intelligent control integrated system, characterized in that, This includes integrated roadside sensing and control units deployed on roads and in-vehicle intelligent terminals deployed on vehicles, wherein: The in-vehicle intelligent terminal acquires the vehicle's own on-board observation status; The integrated roadside perception and control unit includes a roadside multi-source perception module and an edge computing module, wherein: the roadside multi-source perception module collects multi-source data from each vehicle on the road from a roadside perspective; the edge computing module, based on the multi-source data and the vehicle-end observation status, executes the vehicle-road cooperative perception and intelligent control integration method as described in any one of claims 1 to 6, determines the risk of accidents involving vehicles on the road, and transmits the risk information to the on-board intelligent terminal of the corresponding vehicle; The in-vehicle intelligent terminal sends alarm notifications based on risk information.

8. The vehicle-road cooperative perception and intelligent control integrated system according to claim 7, characterized in that, The integrated roadside sensing and control unit also includes: The communication module transmits the output information of the edge computing module to the vehicle-mounted intelligent terminal via V2X direct communication; The power supply and environmental management module provides regulated power to all internal modules in the integrated roadside sensing and control unit.

9. The vehicle-road cooperative perception and intelligent control integrated system according to claim 7, characterized in that, In-vehicle intelligent terminals include: The CAN bus interface module connects to the vehicle's internal network through dual CAN channels to collect vehicle driving information and vehicle operating status. A high-precision positioning module can acquire the vehicle's location in real time; The V2X communication module packages the vehicle's driving information, location, and operating status into a vehicle-side observation status and sends it to the edge computing unit, as well as receiving risk information from the edge computing unit; The intelligent processing unit performs security warning logic judgments on risk information and determines the warning level. The safety warning module issues corresponding prompts based on the warning level; for high-level warnings, the safety warning module sends a linkage braking request to the vehicle and automatically completes emergency braking.

Images