Roadside sensing method based on Leiyu fusion

Through the method based on lightning vision fusion, the main sensor is selected, the timestamp and space are synchronized, and the fusion radar and camera data are matched, which solves the stability and accuracy of the existing roadside perception system, and realizes the full-process event detection and Traffic management application.

CN118915069APending Publication Date: 2024-11-08NEWLAND DIGITAL TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411129415.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-16
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing roadside perception system has defects in the shortage of a single sensor, poor system stability, difficulty in time synchronization, low matching accuracy, and failure to achieve full-process event detection and traffic management applications, and cannot meet the needs of intelligent traffic systems.

Method used

The method based on lightning vision fusion is adopted, and the main sensor is selected, the time stamp synchronization is used to use the NTP server, the coordinate system conversion is used to perform spatial synchronization, the fusion radar and camera data are matched, the road target trajectory is updated, the traffic flow is counted and abnormal events are detected.

Benefits of technology

Real-time trajectory updates and traffic statistics of road targets within 400m, enhances system stability, ensures accurate synchronization and high-precision matching and integration between sensors, and can report abnormal events such as high-speed, low-speed, reverse rotation, and illegal parking in real time. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915069A_ABST
    Figure CN118915069A_ABST
Patent Text Reader

Abstract

The invention discloses a roadside sensing method based on Leiyu fusion, and the method comprises the following steps: selecting a main sensor according to the state of a current sensor; the NTP server is used to synchronize the timestamps of the thundersight devices; performing space synchronization on the thunder vision equipment by using coordinate system conversion; matching and fusing the radar and the camera; updating the track of the road target vehicle; counting the traffic flow according to the lane and the vehicle type classification; and detecting and reporting the abnormal event. According to the invention, real-time updating of track information of road target speed, distance, longitude and latitude, type, ID, lane number and license plate number in the whole range of 400m can be realized, real-time statistics of traffic flow in different lanes and types can be realized, and real-time reporting of abnormal events such as high speed, low speed, converse driving, illegal parking, congestion, pedestrian intrusion, non-motor vehicle intrusion and the like can be realized; when the main sensor breaks down, the system still operates stably, and the stability of the system is enhanced; accurate correction within a 400ms error range between sensors can be realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention is applied to the fields of intelligent perception and intelligent traffic technology, and specifically is a roadside perception method based on radar and vision fusion. Background Art

[0002] In intelligent transportation systems, sensor data fusion technology is the key to improving the accuracy and robustness of environmental perception. Millimeter-wave radar and camera are two main sensors, each with its own unique advantages, but also with certain limitations. Therefore, fusing millimeter-wave radar and camera data to make up for the shortcomings of a single sensor has become a hot topic in current research. The purpose of existing technologies is mainly to achieve more accurate and comprehensive environmental perception through the data fusion of millimeter-wave radar and camera. Millimeter-wave radar can provide accurate distance, speed and angle information, and is less affected by weather; while cameras can capture rich color and texture information, which is helpful for target recognition and classification. Through data fusion, the aim is to combine the advantages of both and improve the overall performance of the roadside perception system.

[0003] The existing roadside perception systems mainly include three types: single camera system, single millimeter wave radar system and fusion system based on millimeter wave radar and camera fusion. Although the single camera system has a strong perception ability of the texture features of road targets, it cannot accurately measure the distance, speed and angle information of road targets, and cannot maintain high perception accuracy at night and in bad weather conditions; although the single millimeter wave radar system can have a strong measurement accuracy of the distance, speed and angle information of moving targets, it is insensitive to stationary targets and cannot accurately detect pedestrians and non-motor vehicles, which are abnormal events that highways often pay attention to.

[0004] Existing systems based on the fusion of millimeter-wave radar and camera often only consider the fusion of a single millimeter-wave radar and a single monitoring camera, which cannot meet the needs of actual scenarios. The working range of millimeter-wave radar can reach 500m, while the working range of a monitoring camera cannot fully cover 500m, and thus cannot realize the detection of events throughout the entire process. At the same time, on highways, the installation position of monitoring cameras is relatively high, and license plate recognition cannot be effectively realized. Further fusion of bayonet cameras is required, and in current research, there is no roadside perception system that integrates millimeter-wave radar, multiple monitoring cameras and bayonet cameras. In addition, existing fusion systems often do not consider the situation where the system fails when a sensor fails, and the system stability is poor; and due to delays and failures in video streaming, the time synchronization of different sensors cannot be accurately achieved; due to errors in the calibration matrix, the front fusion and 3D target detection-based

[0005] The matching accuracy is not high; at the same time, the existing fusion system has not formed roadside applications directly related to traffic management and road safety, such as trajectory, traffic flow statistics and abnormal event detection.

[0006] In the prior art:

[0007] A single camera perception system has the following drawbacks:

[0008] First, it is impossible to accurately measure the distance, speed, and angle information of road targets;

[0009] Second, it is impossible to maintain high perception accuracy at night and in bad weather conditions.

[0010] A single millimeter-wave radar perception system also has the following defects:

[0011] First, they are insensitive to stationary targets;

[0012] Second, pedestrians and non-motor vehicles cannot be accurately detected.

[0013] The perception system based on the fusion of millimeter-wave radar and camera also has the following defects:

[0014] First, it is impossible to effectively detect all events within a 400m range;

[0015] Second, there is no roadside perception system that integrates millimeter-wave radar, multiple surveillance cameras, and bayonet cameras;

[0016] Third, the failure of the system when the sensor fails is not considered, and the system stability is poor;

[0017] Fourth, it is impossible to accurately synchronize the time of different sensors;

[0018] Fifth, the matching accuracy is not high;

[0019] Sixth, there are no roadside applications directly related to traffic management and road safety, such as road target trajectory, traffic flow statistics and abnormal event detection. Summary of the invention

[0020] The technical problem to be solved by the present invention is to provide a roadside perception method based on radar-vision fusion in view of the deficiencies in the prior art.

[0021] In order to solve the above technical problems, a roadside perception method based on radar-visual fusion of the present invention comprises the following steps:

[0022] Select the main sensor according to the current sensor status;

[0023] Use NTP server to synchronize the time stamp of Leishi equipment;

[0024] Use coordinate system transformation to spatially synchronize radar vision equipment;

[0025] Match and fuse radar and camera;

[0026] Update the trajectory of the target vehicle on the road;

[0027] Count the traffic flow by lane and vehicle type;

[0028] Detect and report abnormal events.

[0029] As a possible implementation manner, further, the step of selecting the main sensor according to the current sensor state specifically includes:

[0030] Real-time monitoring of the working status of each radar sensor;

[0031] Determine whether the sensor fails according to the preset failure criteria;

[0032] Automatically switch to backup sensor when main sensor failure is detected;

[0033] By default, the millimeter-wave radar is set as the main sensor, followed by the long- and short-focus surveillance cameras. The bayonet camera is not set as the main sensor, and only the main sensor frame can trigger the fusion and reporting functions.

[0034] As a possible implementation mode, further, the step of synchronizing the timestamp of the radar vision device by using the NTP server specifically includes:

[0035] Use NTP server to synchronize the clock of each sensor;

[0036] Generate a timestamp for each sensor data frame;

[0037] If there are multiple sensor data frames involved in the fusion within the fusion time interval, the frame closest to the fusion time is selected as the fusion data frame;

[0038] Motion information prediction is used to eliminate timestamp errors caused by jitter and streaming, and to compensate for spatial errors; the compensation distance S c For: S c =v(t f -t s )

[0039] Where v is the predicted speed, t f is the main sensor fusion timestamp, t s is the sensor frame timestamp.

[0040] As a possible implementation, further, the step of spatially synchronizing the radar vision device by using coordinate system transformation specifically includes:

[0041] The point cloud output by the millimeter-wave radar is clustered and processed by Kalman filtering to obtain the attribute information R (id r ,v x ,v y ,r x ,r y ,lon,lat,lane_id,type r ), where id r ,v x ,v y ,r x ,r y ,lon,lat,lane_id r ,type r Respectively represent the ID, lateral speed, longitudinal speed, lateral distance, longitudinal distance, longitude, latitude, lane number, and target type of the road moving target;

[0042] The image target detection method based on YOLOX is used to accurately determine the type, confidence and detection frame of the image target. At the same time, the ByteTrack algorithm is used to achieve high-precision tracking of the image target and obtain the attribute information C (id c, type c ,fcore,left,top,right,bottom), where id c ,type c ,fcore,left,top,right,bottom respectively represent the ID,type,confidence,left,top,right,bottom coordinates of the detection box of the image target;

[0043] The output information of the bayonet camera is B(id b ,type b ,land_id b ,car_num);

[0044] Joint calibration matrix M based on millimeter wave radar and camera RC The distance coordinate (r x ,r y ) is converted to pixel coordinates in the image coordinate system (i u ,i v ),

[0045]

[0046] The attribute information of the road moving target output by the millimeter wave radar becomes R(id r ,v x ,v y ,r x ,ry ,lon,lat,lane_id,typer,i u ,i v );

[0047] Based on the camera intrinsic parameter matrix M C The pixel coordinates of the image coordinate system (i p ,i q ) is converted to the distance coordinate (c x ,c y ),

[0048]

[0049] Among them, i p =(left+right)2,i q =(top+bottom) / 2, the road target attribute information output by the camera becomes C(id c ,type c ,fcore,left,top,right,bottom,c x ,c y ).

[0050] As a possible implementation, further, the step of matching and fusing the radar and the camera specifically includes:

[0051] Divide the action areas of the monitoring long-focus camera and short-focus camera, and perform matching and fusion within their respective action areas;

[0052] Preprocess the image detection frame, filter the incomplete detection of the image edge, and expand the detection frame by 1.4 times;

[0053] The similarity calculation S of the mapping coordinates of the millimeter wave radar on the image and the image detection coordinates is performed along the u and v axes of the image respectively. u , S v :

[0054]

[0055] Among them, set L thresh =10,L scale =3.

[0056] For the joint action area of ​​the millimeter wave radar and the camera, in order to prevent the mismatch when the vehicle density is high, the position similarity calculation S is added loc :

[0057]

[0058] Calculate the similarity S of the matching fusion match:

[0059]

[0060] Among them, R and C represent the action areas of radar and camera respectively.

[0061] The Hungarian matching algorithm is used to match the fusion similarity matrix S match Processing is performed to obtain the matching fusion result of the millimeter wave radar and the camera;

[0062] The fusion of millimeter-wave radar and bayonet camera is determined by lane number and matching distance, and the constraints are as follows:

[0063]

[0064] Among them, Tri_distance and Bay_Err represent the trigger distance and error distance of the bayonet trip wire respectively. Generally, Tri_distance=20 and Bay_Err=5 are set.

[0065] As a possible implementation manner, further, the step of updating the trajectory of the road target vehicle specifically includes:

[0066] Update the target fused by millimeter-wave radar, surveillance camera and bayonet camera to obtain trajectory data RCB (id, v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num);

[0067] Set a lifetime for unmatched targets and delete them if the lifetime expires.

[0068] As a possible implementation, further, the step of counting the traffic flow by lane and vehicle type classification specifically includes:

[0069] Two traffic flow tripwires are set at intervals, and the number of vehicles crossing the two tripwires during the statistical time is counted, and traffic flow statistics are conducted by lane and vehicle type.

[0070] As a possible implementation mode, further, the step of detecting and reporting abnormal events specifically includes:

[0071] High-speed event judgment: |RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)vx |>v max , where v max The maximum allowed speed is set;

[0072] Low speed event judgment: |RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)v x |<v min , where v min The minimum allowed speed is set;

[0073] Retrograde event judgment: RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|v x <0, assuming that the actual normal driving speed value is positive;

[0074] Illegal parking event judgment: |RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|v x |<2, and ensure that the target type is vehicle, and last for 5s;

[0075] Congestion event judgment: According to RCB (id, v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num) calculate the queue length and determine whether it is congested based on the queue length;

[0076] Pedestrian intrusion event judgment: RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|type=person;

[0077] Non-motor vehicle intrusion event judgment: RCB (id, v x ,v y ,r x ,ry ,lon,lat,lane_id,type,fscore,car_num)|type=nonmotor.

[0078] A roadside perception system based on radar and vision fusion, comprising:

[0079] The radar sensor module includes a millimeter-wave radar, a telephoto camera, a short-focus camera, and a bayonet camera;

[0080] Time synchronization module, using NTP server to synchronize the timestamp of Leishi equipment;

[0081] The spatial synchronization module uses coordinate system conversion to synchronize the radar vision equipment in space;

[0082] Matching fusion module, which matches and fuses radar and camera;

[0083] A trajectory update module updates the trajectory of the target vehicle on the road;

[0084] Traffic statistics module, which counts traffic flow by lane and vehicle type;

[0085] The event detection module detects and reports abnormal events.

[0086] The present invention adopts the above technical solution and has the following beneficial effects:

[0087] By adopting the present invention, real-time update of the target speed, distance, longitude and latitude, type, ID, lane number and license plate number trajectory information of the road within the entire range of 400m, real-time statistics of traffic flow by lane and type, and real-time reporting of abnormal events such as high speed, low speed, wrong-way driving, illegal parking, congestion, pedestrian intrusion, non-motor vehicle intrusion, etc. can be achieved; when the main sensor fails, the system still runs stably, thereby enhancing the stability of the system; accurate correction within the 400ms error range between sensors can be achieved; and high-precision matching and fusion can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0089] Figure 1 Schematic diagram of the system structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0091] Example 1

[0092] A roadside perception method based on radar and visual fusion includes the following steps: according to the current sensor state, the main sensor is selected, and the step specifically includes: real-time monitoring of the working state of each radar and visual sensor; judging whether the sensor fails according to a preset fault standard; automatically switching to a backup sensor when a main sensor failure is detected; setting the millimeter-wave radar as the main sensor by default, followed by long- and short-focus monitoring cameras, and not setting the bayonet camera as the main sensor, and only the main sensor frame can trigger the fusion and reporting functions.

[0093] The steps of synchronizing the timestamp of the radar vision equipment using the NTP server include: using the NTP server to synchronize the clocks of each sensor; generating a timestamp for each sensor data frame; if the sensor data frames involved in the fusion appear multiple frames within the fusion time interval, selecting the frame closest to the fusion time as the fusion data frame; using motion information prediction to eliminate the timestamp error caused by jitter and streaming, and to compensate for the spatial error; the compensation distance S c For: S c =v(t f -t s ), where v is the predicted speed, t f is the main sensor fusion timestamp, t s is the sensor frame timestamp.

[0094] The steps of spatial synchronization of radar vision equipment using coordinate system conversion specifically include: clustering and Kalman filtering the point cloud output by the millimeter wave radar to obtain the attribute information R (id r ,v x ,v y ,r x ,r y ,lon,lat,lane_id,type r ), where id r ,v x ,v y ,r x ,r y ,lon,lat,lane_id r ,type r They represent the ID, lateral speed, longitudinal speed, lateral distance, longitudinal distance, longitude, latitude, lane number, and target type of the moving target on the road respectively; the image target detection method based on YOLOX is used to accurately determine the type, confidence, and detection frame of the image target, and the ByteTrack algorithm is used to achieve high-precision tracking of the image target to obtain the attribute information C (id c ,type c ,fcore,left,top,right,bottom), where idc ,type c ,fcore,left,top,right,bottom respectively represent the ID, type, confidence, left, top, right, bottom coordinates of the detection box of the image target; the output information of the camera is B(id b ,type b ,land_id b ,car_num); Joint calibration matrix M based on millimeter wave radar and camera RC The distance coordinate (r x ,r y ) is converted to pixel coordinates in the image coordinate system (i u ,i v ), The attribute information of the road moving target output by the millimeter wave radar becomes R(id r ,v x ,v y ,r x ,r y ,lon,lat,lane_id,type r ,i u ,i v );Based on the camera intrinsic parameter matrix M C The pixel coordinates of the image coordinate system (i p ,i q ) is converted to the distance coordinate (c x ,c y ), Among them, i p =(left+right) / 2, i q =(top+bottom) / 2, the road target attribute information output by the camera becomes C(id c ,type c ,fcore,left,top,right,bottom,c x ,c y ).

[0095] The steps of matching and fusing the radar and the camera specifically include: dividing the action areas of the monitoring telephoto camera and the short-focus camera, and matching and fusing them in their respective action areas; preprocessing the image detection frame, filtering the incomplete detection of the image edge, and expanding the detection frame by 1.4 times; calculating the similarity S between the mapping coordinates of the millimeter-wave radar on the image and the image detection coordinates along the u and v axes of the image respectively. u , S v : Among them, set L thresh =10,L scale= 3. For the joint action area of ​​the millimeter wave radar and the camera, in order to prevent the mismatch when the vehicle density is high, the position similarity calculation S is increased. loc : Calculate the similarity S of the matching fusion match : Among them, R and C represent the action areas of radar and camera respectively. The Hungarian matching algorithm is used to match the fusion similarity matrix S match After processing, the matching fusion result of the millimeter-wave radar and the camera is obtained; the fusion of the millimeter-wave radar and the bayonet camera is determined by the lane number and the matching distance, and the constraints are as follows: Among them, Tri_distance and Bay_Err represent the trigger distance and error distance of the bayonet trip wire respectively. Generally, Tri_distance=20 and Bay_Err=5 are set.

[0096] The specific steps of updating the trajectory of the road target vehicle include: updating the target fused by the millimeter wave radar, the monitoring camera and the bayonet camera, and obtaining the trajectory data RCB (id, v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num); set a lifetime for unmatched targets and delete them if the lifetime exceeds.

[0097] The specific steps for counting traffic flow by lane and vehicle type include: setting two traffic tripwires at intervals, counting the number of vehicles crossing the two tripwires during the statistical time, and counting traffic flow by lane and vehicle type.

[0098] The steps for detecting and reporting abnormal events include: High-speed event judgment: |RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|v x |>v max , where v max The maximum allowed speed is set; low speed event judgment: |RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|v x |<v min , where v minis the minimum allowed speed; reverse event judgment: RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|v x <0, assuming that the actual normal driving speed value is positive; illegal parking event judgment: |RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|v x |<2, and ensure that the target type is vehicle and lasts for 5s; Congestion event judgment: according to RCB (id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num) calculates the queue length and determines whether it is congested based on the queue length; pedestrian intrusion event judgment: RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|type=person; Non-motor vehicle intrusion event judgment: RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|type=nonmotor.

[0099] A roadside perception system based on radar and vision fusion, comprising:

[0100] The radar sensor module specifically includes millimeter-wave radar, telephoto camera, short-focus camera and bayonet camera; the time synchronization module uses the NTP server to synchronize the timestamp of the radar device; the space synchronization module uses the coordinate system conversion to synchronize the space of the radar device; the matching fusion module matches and fuses the radar and the camera; the trajectory update module updates the trajectory of the target vehicle on the road; the traffic statistics module counts the traffic flow by lane and vehicle type; the event detection module detects and reports abnormal events.

[0101] Example 2

[0102] A roadside perception method and system based on millimeter-wave radar and camera fusion with 400m full coverage, strong stability, high-precision matching processing and strong application relevance, specifically including:

[0103] Main sensor screening module;

[0104] Time synchronization module;

[0105] Space synchronization module;

[0106] Matching processing module;

[0107] Trajectory update module;

[0108] Traffic flow statistics module;

[0109] Abnormal event detection module.

[0110] The specific implementation method is described in detail below:

[0111] 1. Main sensor screening module: used to determine the current operating status of the sensor, intelligently select the main sensor, and solve the system failure problem caused by a single sensor failure in the existing technology.

[0112] Real-time monitoring of the working status of millimeter-wave radar, bayonet camera, telephoto surveillance camera, and short-focus surveillance camera sensors. The working range of the millimeter-wave radar is 15-450m, the working range of the short-focus camera is 10-120m, and the working range of the telephoto camera is 120-400m.

[0113] Determine whether the sensor is faulty based on the preset fault criteria.

[0114] When a primary sensor failure is detected, it automatically switches to the backup sensor to ensure continuous operation of the system.

[0115] By default, the millimeter-wave radar is set as the main sensor, followed by the long- and short-focus surveillance cameras. The bayonet camera is not set as the main sensor. Only the main sensor frame can trigger the fusion and reporting functions.

[0116] 2. Time synchronization module: mainly used to synchronize the timestamps of millimeter-wave radar, bayonet camera, long-focus surveillance camera, and short-focus surveillance camera.

[0117] Use NTP server to synchronize the clock of each sensor;

[0118] Generate accurate timestamps for each sensor data frame;

[0119] If there are multiple sensor data frames involved in the fusion within the fusion time interval, the frame closest to the fusion time is selected as the fusion data frame;

[0120] In order to eliminate the timestamp error caused by jitter and streaming, motion information prediction can be often used to compensate for the spatial error caused by it. Since in a short period of time (400ms), the moving target can be equivalent to uniform motion, the compensated distance S c Can be simplified to:

[0121] S c =v(t f -t s )

[0122] Where v is the predicted speed, t f The main sensor fusion timestamp, usually millimeter wave radar, t s It is the sensor frame timestamp, usually a surveillance camera or a bayonet camera.

[0123] 3. Spatial synchronization module: mainly includes the conversion from image coordinate system to camera coordinate system and the conversion from millimeter wave radar coordinate system to image coordinate system.

[0124] The point cloud output by the millimeter-wave radar is clustered and processed by Kalman filtering to obtain the attribute information R (id r ,v x ,v y ,r x ,r y ,lon,lat,lane_id,type r ), where id r ,v x ,v y ,r x ,r y ,lon,lat,lane_id r ,type r Respectively represent the ID, lateral speed, longitudinal speed, lateral distance, longitudinal distance, longitude, latitude, lane number, and target type of the road moving target;

[0125] The image target detection method based on YOLOX is used to accurately determine the type, confidence and detection frame of the image target. At the same time, the ByteTrack algorithm is used to achieve high-precision tracking of the image target and obtain the attribute information C (id c ,type c ,fcore,left,top,right,bottom), where id c ,type c ,fcore,left,top,right,bottom respectively represent the ID,type,confidence,left,top,right,bottom coordinates of the detection box of the image target;

[0126] The output information of the bayonet camera is B(id b ,type b ,land_id b ,car_num);

[0127] Joint calibration matrix M based on millimeter wave radar and camera RC The distance coordinate (r x ,r y ) is converted to pixel coordinates in the image coordinate system (i u ,i v ),

[0128]

[0129] The attribute information of the road moving target output by the millimeter wave radar becomes R(id r ,v x ,v y ,r x ,r y ,lon,lat,lane_id,type r ,i u ,i v ).

[0130] Based on the camera intrinsic parameter matrix M C The pixel coordinates of the image coordinate system (i p ,i q ) is converted to the distance coordinate (c x ,c y ),

[0131]

[0132] Among them, i p =(left+right) / 2, i q =(top+bottom) / 2, the road target attribute information output by the camera becomes C(id c ,type c ,fcore,left,top,right,bottom,c x ,c y ).

[0133] 4. Matching processing module: including the matching fusion of millimeter-wave radar and surveillance camera, and the matching fusion of millimeter-wave radar and bayonet camera.

[0134] Divide the action areas of the monitoring long-focus camera and short-focus camera, and perform matching and fusion within their respective action areas;

[0135] The image detection frame is preprocessed to filter the incomplete detection of the image edge. At the same time, in order to improve the matching accuracy, the detection frame is enlarged by 1.4 times;

[0136] The similarity calculation S of the mapping coordinates of the millimeter wave radar on the image and the image detection coordinates is performed along the u and v axes of the image respectively. u , S v :

[0137]

[0138] Among them, L is generally set thresh =10,L scale =3.

[0139] For the joint action area of ​​the millimeter wave radar and the camera, in order to prevent the mismatch when the vehicle density is high, the position similarity calculation S is added loc :

[0140]

[0141] Calculate the similarity S of the matching fusion match :

[0142]

[0143] Among them, R and C represent the action areas of radar and camera respectively.

[0144] The Hungarian matching algorithm is used to match the fusion similarity matrix S match Processing is performed to obtain the matching fusion result of the millimeter wave radar and the camera;

[0145] The fusion of millimeter-wave radar and bayonet camera can be determined by lane number and matching distance, and the constraints are as follows:

[0146]

[0147] Among them, Tri_distance and Bay_Err represent the trigger distance and error distance of the bayonet trip wire respectively. Generally, Tri_distance=20 and Bay_Err=5 are set.

[0148] 5. Trajectory update module: updates the trajectory of the target vehicle on the road.

[0149] Update the target fused by millimeter-wave radar, surveillance camera and bayonet camera to obtain trajectory data RCB (id, v x ,v y ,r x ,r y,lon,lat,lane_id,type,fscore,car_num);

[0150] Set a life cycle for unmatched targets and delete them if the life cycle expires;

[0151] 6. Traffic flow statistics module: statistics are classified by lane and vehicle type.

[0152] A. Set up two traffic trip wires and count the vehicles that cross the two trip wires at the same time during the statistical period;

[0153] B. Carry out traffic statistics by lane and by vehicle type (sedans, buses, trucks, hazardous chemicals vehicles).

[0154] 7. Abnormal event detection module: report abnormal events such as high speed, low speed, wrong driving, illegal parking, congestion, pedestrian intrusion, non-motor vehicle intrusion, etc.

[0155] High-speed event judgment: |RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|v x |>v max ,

[0156] Among them, v max The maximum allowed speed is set.

[0157] Low speed event judgment: |RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|v x <v min , where v min The minimum allowed speed is set.

[0158] Retrograde event judgment: RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|v x <0, assuming that the actual normal driving speed value is positive.

[0159] Illegal parking event judgment: |RCB(id,v x ,v y ,rx ,r y ,lon,lat,lane_id,type,fscore,car_num)|v x |<2, and ensure that the target type is vehicle and lasts for 5s.

[0160] Congestion event judgment: According to RCB (id, v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num) calculates the queue length and determines whether it is congested based on the queue length.

[0161] Pedestrian intrusion event judgment: RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|type=person.

[0162] Non-motor vehicle intrusion event judgment: RCB (id, v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|type=nonmotor.

[0163] The above are embodiments of the present invention. For ordinary technicians in this field, according to the teachings of the present invention, all equivalent changes, modifications, substitutions and variations made within the scope of the patent application of the present invention without departing from the principles and spirit of the present invention should fall within the scope of the present invention.

Claims

1. A roadside perception method based on radar and visual fusion, characterized in that: The steps include: Select the main sensor according to the current sensor status; Use NTP server to synchronize the time stamp of Leishi equipment; Use coordinate system transformation to spatially synchronize radar vision equipment; Match and fuse radar and camera; Update the trajectory of the target vehicle on the road; Count the traffic flow by lane and vehicle type; Detect and report abnormal events.

2. The roadside perception method based on radar and visual fusion according to claim 1 is characterized by: The step of selecting the main sensor according to the current sensor state specifically includes: Real-time monitoring of the working status of each radar sensor; Determine whether the sensor fails according to the preset failure criteria; Automatically switch to backup sensor when main sensor failure is detected; By default, the millimeter-wave radar is set as the main sensor, followed by the long- and short-focus surveillance cameras. The bayonet camera is not set as the main sensor, and only the main sensor frame can trigger the fusion and reporting functions.

3. The roadside perception method based on radar and visual fusion according to claim 1 is characterized in that: The step of synchronizing the timestamp of the Leishi device using the NTP server specifically includes: Use NTP server to synchronize the clock of each sensor; Generate a timestamp for each sensor data frame; If there are multiple sensor data frames involved in the fusion within the fusion time interval, the frame closest to the fusion time is selected as the fusion data frame; Motion information prediction is used to eliminate timestamp errors caused by jitter and streaming, and to compensate for spatial errors; the compensation distance S c For: S c =v(t f -t s ) Where v is the predicted speed, t f is the main sensor fusion timestamp, t s is the sensor frame timestamp.

4. The roadside perception method based on radar and visual fusion according to claim 1 is characterized by: The step of spatially synchronizing the radar vision equipment by using coordinate system conversion specifically includes: The point cloud output by the millimeter-wave radar is clustered and processed by Kalman filtering to obtain the attribute information R (id r ,v x ,v y ,r x ,r y ,lon,lat,lane_id,type r ), where idr,v x ,v y ,rx,r y ,lon,lat,lane_id r ,type r Respectively represent the ID, lateral speed, longitudinal speed, lateral distance, longitudinal distance, longitude, latitude, lane number, and target type of the road moving target; The image target detection method based on YOLOX is used to accurately determine the type, confidence and detection frame of the image target. At the same time, the ByteTrack algorithm is used to achieve high-precision tracking of the image target and obtain the attribute information C (id c ,type c ,fcore,left,top,right,bottom), where id c ,type c ,fcore,left,top,right,bottom respectively represent the ID,type,confidence,left,top,right,bottom coordinates of the detection box of the image target; The output information of the bayonet camera is B(id b ,type b ,land_id b ,car_num); Joint calibration matrix M based on millimeter wave radar and camera RC The distance coordinate (r x ,r y ) is converted to pixel coordinates in the image coordinate system (i u ,i v ), The attribute information of the road moving target output by the millimeter wave radar becomes R(id r ,v x ,v y ,r x ,r y ,lon,lat,lane_id,type r ,i u ,i v ); Based on the camera intrinsic parameter matrix M C The pixel coordinates of the image coordinate system (i p ,i q ) is converted to the distance coordinate (c x ,c y ), Among them, i p =(left+right) / 2, i q =(top+bottom) / 2, the road target attribute information output by the camera becomes C(id c ,type c ,fcore,left,top,right,bottom,c x ,c y ).

5. The roadside perception method based on radar and visual fusion according to claim 1 is characterized by: The step of matching and fusing the radar and the camera specifically includes: Divide the action areas of the monitoring long-focus camera and short-focus camera, and perform matching and fusion within their respective action areas; Preprocess the image detection frame, filter the incomplete detection of the image edge, and expand the detection frame by 1.4 times; The similarity calculation S of the mapping coordinates of the millimeter wave radar on the image and the image detection coordinates is performed along the u and v axes of the image respectively. u , S v : Among them, set L thresh =10,L scale =3. For the joint action area of ​​the millimeter wave radar and the camera, in order to prevent the mismatch when the vehicle density is high, the position similarity calculation S is added loc : Calculate the similarity S of the matching fusion match : Among them, R and C represent the action areas of radar and camera respectively. The Hungarian matching algorithm is used to match the fusion similarity matrix S match Processing is performed to obtain the matching fusion result of the millimeter wave radar and the camera; The fusion of millimeter-wave radar and bayonet camera is determined by lane number and matching distance, and the constraints are as follows: Among them, Tri_distance and Bay_Err represent the trigger distance and error distance of the bayonet trip wire respectively. Generally, Tri_distance=20 and Bay_Err=5 are set.

6. The roadside perception method based on radar and visual fusion according to claim 1 is characterized by: The step of updating the trajectory of the road target vehicle specifically includes: Update the target fused by millimeter-wave radar, surveillance camera and bayonet camera to obtain trajectory data RCB (id, v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num); Set a lifetime for unmatched targets and delete them if the lifetime expires.

7. The roadside perception method based on radar and visual fusion according to claim 1 is characterized by: The step of counting the traffic volume by lane and vehicle type classification specifically includes: Two traffic flow tripwires are set at intervals, and the number of vehicles crossing the two tripwires during the statistical time is counted, and traffic flow statistics are conducted by lane and vehicle type.

8. The roadside perception method based on radar and visual fusion according to claim 1 is characterized by: The steps of detecting and reporting abnormal events specifically include: High-speed event judgment: |RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)v x |>v max , where v max is the maximum allowed speed set; Low speed event judgment: |RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)v x |<v min , where v min The minimum allowed speed is set; Retrograde event judgment: RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)v x <0, assuming that the actual normal driving speed value is positive; Illegal parking event judgment: |RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)v x |<2, and ensure that the target type is vehicle, and last for 5s; Congestion event judgment: According to RCB (id, v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num) calculate the queue length and determine whether it is congested based on the queue length; Pedestrian intrusion event judgment: RCB(id,v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|type=person; Non-motor vehicle intrusion event judgment: RCB (id, v x ,v y ,r x ,r y ,lon,lat,lane_id,type,fscore,car_num)|type=nonmotor.

9. A roadside perception system based on radar and visual fusion, characterized in that: It includes: The radar sensor module includes a millimeter-wave radar, a telephoto camera, a short-focus camera, and a bayonet camera; Time synchronization module, using NTP server to synchronize the timestamp of Leishi equipment; The spatial synchronization module uses coordinate system conversion to synchronize the radar vision equipment in space; Matching fusion module, which matches and fuses radar and camera; A trajectory update module updates the trajectory of the target vehicle on the road; Traffic statistics module, which counts traffic flow by lane and vehicle type; The event detection module detects and reports abnormal events.

Citation Information

Cited By

  • Traffic abnormal event detection method based on multi-source data fusion

    CN121415576A