Roadside perception control system and method based on radar and vision fusion

Abstract

The application relates to the technical field of roadside perception, in particular to a roadside perception control system and method based on radar and vision fusion; the system comprises a sensor unit, a data processing unit and a pre-warning control unit; the application realizes more comprehensive and accurate perception of traffic participants by comprehensively processing laser radar data and video data; meanwhile, based on the space matching of calibration objects, the conversion relationship between the laser radar coordinate system and the vision coordinate system is determined, so that the same target detected by different sensors can be accurately corresponded in space, the accuracy of the judgment of the traffic condition of the whole roadside perception system is greatly improved, and when compared with the pre-stored control threshold, the system can more scientifically and reasonably judge whether to issue a pre-warning.

Description

Technical Field

[0001] This invention relates to the field of roadside perception technology, specifically to a roadside perception control system and method based on radar-visual fusion. Background Technology

[0002] The purpose of roadside perception is to achieve instantaneous intelligent perception of traffic participants and road conditions on the road section. Roadside perception can expand the perception range of driving vehicles and drivers.

[0003] When vehicles achieve autonomous or assisted autonomous driving functions, they not only need their own sensors to acquire surrounding information, but also rely on roadside perception systems to provide more macroscopic and comprehensive traffic scene data as supplements and references. While radar can accurately measure the motion parameters of a target, it struggles to precisely determine the target's specific type because radar primarily analyzes the physical characteristics of reflected signals, lacking visual features such as the target's appearance and shape. Accurately obtaining target motion parameters from video images is also difficult and relatively inaccurate, with significant errors. Traditional single-sensor perception methods, due to these limitations, struggle to comprehensively and accurately capture these abnormal behaviors and provide timely warnings, failing to meet the ever-increasing demands for traffic safety.

[0004] Based on this, the present invention provides a roadside perception control system and method based on radar-visual fusion to solve the aforementioned technical problems. Summary of the Invention

[0005] The purpose of this invention is to provide a roadside perception control system and method based on laser radar and video fusion. By integrating laser radar data and video data, it achieves a more comprehensive and accurate perception of traffic participants. At the same time, based on the spatial matching of calibration objects, it determines the transformation relationship between the laser radar coordinate system and the visual coordinate system, so that the same target detected by different sensors can be accurately matched in space, which greatly improves the accuracy of the entire roadside perception system in judging traffic conditions. In addition, when compared with pre-stored control thresholds, it can more scientifically and reasonably determine whether to issue a warning.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A first aspect of the present invention provides a roadside perception and control system based on radar-visual fusion, comprising a sensor unit, a data processing unit, and a warning control unit, wherein:

[0008] The sensor unit is used to acquire target data information, which includes lidar data, video data, and positioning information data.

[0009] The data processing unit is used to spatially match lidar data and video data, and extract first feature data and second feature data based on lidar data and video data, as well as feature data extracted by fusing lidar and vision. The data processing unit is connected to the sensor unit.

[0010] The early warning control unit is used to upload preset control thresholds, receive early warning instructions, and issue early warning information according to the received early warning instructions. The early warning control unit is connected to the data processing unit.

[0011] The present invention is further configured such that: the sensor unit includes a lidar module, a location positioning module, a video acquisition module, and a first communication module, wherein:

[0012] The data processing unit is used for real-time lidar data of the target;

[0013] The location positioning module is used to obtain the real-time location information of the target;

[0014] The video acquisition module is used to acquire real-time video information of the target;

[0015] The first communication module is used to realize information interaction between the sensor unit and the data processing unit.

[0016] The present invention is further configured such that: the data processing unit includes a second communication module, a coordinate transformation module, and a feature extraction module, wherein:

[0017] The second communication module is used to realize information interaction between the data processing unit, the sensor unit, and the early warning control unit;

[0018] The coordinate transformation module, based on the calibration object acquired by the lidar and the calibration object acquired by vision, maps the coordinates of the calibration object acquired in the lidar coordinate system to the coordinates in the vision coordinate system, determines the transformation relationship between the lidar coordinate system and the vision coordinate system, and realizes the transformation between lidar coordinates and vision coordinates. The coordinate transformation module is connected to the second communication module.

[0019] The feature extraction module extracts first feature data based on LiDAR data and second feature data based on video data. The feature extraction module is connected to the coordinate transformation module.

[0020] The present invention is further configured such that: the data processing unit further includes a feature fusion module, a threshold comparison module, and a database module, wherein:

[0021] The feature fusion module is used to fuse the feature data extracted by LiDAR and vision to obtain a matching target with first feature data and second feature data. The feature fusion module is connected to the feature extraction module.

[0022] The threshold comparison module compares the fused target feature data with pre-stored control thresholds to determine whether to issue a corresponding warning command. The threshold comparison module is connected to the feature fusion module.

[0023] The database module is used to store the acquired sensor data and preset control threshold information. The database module is connected to both the second communication module and the threshold comparison module.

[0024] The present invention is further configured such that: the early warning control unit includes a third communication module, an early warning issuing module, and a threshold setting module, wherein:

[0025] The third communication module is used to realize information interaction between the early warning control unit and the data processing unit;

[0026] The early warning release module is used to receive early warning instructions and release early warning information according to the received early warning instructions. The early warning release module is connected to the third communication module.

[0027] The threshold setting module is used to preset the upload of control thresholds, and the threshold setting module is connected to the third communication module.

[0028] A second aspect of the present invention also provides a roadside perception control method based on radar-visual fusion, which is applied to the aforementioned roadside perception control system based on radar-visual fusion, comprising the following steps:

[0029] Acquire sensor data, wherein the sensor data includes lidar data and video data;

[0030] Spatial matching of LiDAR data and video data;

[0031] Based on LiDAR data, extract the first feature data;

[0032] Extract the second feature data based on the video data;

[0033] Integrate feature data extracted from LiDAR and vision;

[0034] Based on the characteristic data of the fused target, a warning command is issued by comparing it with the pre-stored control threshold.

[0035] The present invention is further configured such that the matching process is as follows:

[0036] Based on the calibration objects acquired by LiDAR and vision, the coordinates of the calibration objects acquired in the LiDAR coordinate system are mapped one-to-one with the coordinates in the vision coordinate system.

[0037] Based on the coordinates of the calibrated object, determine the transformation relationship between the lidar coordinate system and the visual coordinate system, (x v ,y v ,z v )=T×(x r ,y r ,z r ), where (x v ,y v ,z v (x) represents the target's coordinates in the lidar coordinate system. r ,y r ,z r ) represents the coordinates of the target in the visual coordinate system, and T is the transformation matrix.

[0038] The present invention is further configured such that the process of extracting the first feature data is as follows:

[0039] Velocity feature extraction: Obtain the transmitted signal frequency f1, received signal frequency f2, and lidar propagation speed c of the target from the lidar, and calculate the target's velocity.

[0040] Acceleration feature extraction: Obtain velocity data of the target at several consecutive time points, and calculate the acceleration of the target at time i. In the formula, v i+1 Let v be the velocity at time i+1. i-1 Let Δt be the velocity at time i-1, and Δt be the lidar update cycle. Then, calculate the average value of several consecutive accelerations within the i+n cycle, which is the acceleration within that cycle.

[0041] The present invention is further configured such that the process of extracting the second feature data is as follows:

[0042] Extract the target contour shape from video data; identify the extracted target contour shape to obtain the target's classification information.

[0043] The present invention is further configured such that the process of fusing the feature data extracted by lidar and vision is as follows:

[0044] Match the target coordinates with the first feature data to the visual coordinate system to obtain the target coordinates with the first feature data in the visual coordinate system, i.e., transform the target coordinates;

[0045] Calculate the distance between the transformed target coordinates and the target coordinates with the second feature data;

[0046] The target with the smallest gap is selected for matching to obtain the matched target, which has first feature data and second feature data.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] This invention acquires sensor data, spatially matches LiDAR data and video data, extracts first feature data based on the LiDAR data, and extracts second feature data based on the video data. Then, it fuses the feature data extracted from LiDAR and vision, compares the fused target feature data with pre-stored control thresholds, and issues a warning command. By integrating LiDAR and video data, this invention not only acquires motion-related information such as target distance, speed, and acceleration, but also reveals the target's appearance and specific classification. The fusion of these two data sets enables a more comprehensive and accurate perception of traffic participants. Furthermore, based on spatial matching of calibration objects, the transformation relationship between the LiDAR coordinate system and the vision coordinate system is determined, ensuring that the same target detected by different sensors can be accurately matched spatially. This avoids target information mismatch during data fusion, greatly improving the accuracy of the entire roadside perception system's judgment of traffic conditions. In addition, the fused feature data covers multiple key features of the target, allowing for a more scientific and reasonable judgment on whether to issue a warning when compared with pre-stored control thresholds. Attached Figure Description

[0049] Figure 1 This is a system diagram of the roadside perception control system based on radar-visual fusion of the present invention;

[0050] Figure 2 This is a system diagram of the sensor unit in the roadside perception control system based on radar-visual fusion of the present invention;

[0051] Figure 3 This is a system diagram of the data processing unit in the roadside perception control system based on radar-visual fusion of the present invention;

[0052] Figure 4 This is a system diagram of the early warning control unit in the roadside perception control system based on radar-visual fusion of the present invention.

[0053] Explanation of icon numbers:

[0054] 10. Sensor Unit; 11. LiDAR Module; 12. Positioning Module; 13. Video Acquisition Module; 14. First Communication Module; 20. Data Processing Unit; 21. Second Communication Module; 22. Coordinate Transformation Module; 23. Feature Extraction Module; 24. Feature Fusion Module; 25. Threshold Comparison Module; 26. Database Module; 30. Early Warning Control Unit; 31. Third Communication Module; 32. Early Warning Issuance Module; 33. Threshold Setting Module. Detailed Implementation

[0055] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0056] Example:

[0057] like Figures 1-4 As shown, this embodiment provides a roadside perception control system based on laser-visual fusion, including a sensor unit 10, a data processing unit 20, and a warning control unit 30. The sensor unit 10 is used to acquire target data, including laser radar data, video data, and positioning information. The data processing unit 20 is used to spatially match the laser radar data and video data, extract first feature data and second feature data based on the laser radar data and video data, and fuse the feature data extracted from the laser radar and vision. The data processing unit 20 is connected to the sensor unit 10. The warning control unit 30 is used to upload preset control thresholds, receive warning commands, and issue warning information according to the received warning commands. The warning control unit 30 is connected to the data processing unit 20.

[0058] In this embodiment, it should be noted that the sensor unit 10 acquires LiDAR data, video data, and positioning information data of different targets in real time, and transmits the acquired information to the data processing unit 20. The data processing unit 20 first determines the transformation relationship between the LiDAR coordinate system and the visual coordinate system based on the calibration objects acquired by the LiDAR and the visual coordinate system, so as to perform target matching in the future. Then, it uses the LiDAR data and video data respectively to obtain the first feature data and the second feature data. Finally, it fuses the obtained first feature data and the second feature data to obtain the matching target with the first feature data and the second feature data. The data processing unit 20 retrieves the pre-stored threshold control information and determines whether to issue a warning command by comparison. When issuing a warning command, it is uploaded to the warning control unit 30, which issues the warning.

[0059] In this invention, the sensor unit 10 includes a lidar module 11, a location positioning module 12, a video acquisition module 13, and a first communication module 14, wherein: the data processing unit 20 is used for real-time lidar data of the target; the location positioning module 12 is used to acquire real-time location information of the target; the video acquisition module 13 is used to acquire real-time video information of the target; and the first communication module 14 is used to realize information interaction between the sensor unit 10 and the data processing unit 20.

[0060] In this embodiment, it should be noted that the lidar module 11, the location module 12, and the video acquisition module 13 respectively acquire lidar data, real-time video information, and real-time location information, and upload the above information to the data processing unit 20 through the first communication module 14. The lidar continuously emits laser pulses into the monitoring area and receives the laser signals reflected back from the target. A high-definition camera installed at a suitable location on the roadside acquires road scene images, and then transmits the image data to the data processing unit 20 for storage and subsequent processing via wired (Ethernet, etc.) or wireless (5G network, etc.) communication methods.

[0061] In this invention, the data processing unit 20 includes a second communication module 21, a coordinate transformation module 22, and a feature extraction module 23. The second communication module 21 enables information interaction between the data processing unit 20, the sensor unit 10, and the early warning control unit 30. The coordinate transformation module 22, based on the calibration objects acquired by the lidar and the visual coordinate system, establishes a one-to-one correspondence between the coordinates of the calibration objects acquired in the lidar coordinate system and the coordinates in the visual coordinate system, determining the transformation relationship between the lidar coordinate system and the visual coordinate system, and realizing the transformation between lidar coordinates and visual coordinates. The coordinate transformation module 22 is connected to the second communication module 21. The feature extraction module 23 extracts first feature data based on lidar data and extracts second feature data based on video data. The feature extraction module 23 is connected to the coordinate transformation module 22.

[0062] In addition, the data processing unit 20 also includes a feature fusion module 24, a threshold comparison module 25, and a database module 26. The feature fusion module 24 is used to fuse the feature data extracted by the lidar and vision to obtain a matching target with first feature data and second feature data. The feature fusion module 24 is connected to the feature extraction module 23. The threshold comparison module 25 is used to compare the feature data of the fused target with the pre-stored control threshold to determine whether to issue a corresponding warning command. The threshold comparison module 25 is connected to the feature fusion module 24. The database module 26 is used to store the acquired sensor data and the preset control threshold information. The database module 26 is connected to both the second communication module 21 and the threshold comparison module 25.

[0063] In this embodiment, it should be noted that the second communication module 21 receives the information uploaded by the sensor unit 10, the coordinate transformation module 22 determines the transformation relationship between the lidar coordinates and the visual coordinates, and then the feature extraction module 23 extracts the first feature data based on the lidar data and the second feature data based on the video data. The first and second feature data are then uploaded to the feature fusion module 24, which fuses the first and second feature data. The specific process is as follows: the target coordinates with the first feature data are matched to the visual coordinate system to obtain the target coordinates with the first feature data in the visual coordinate system, i.e., the transformed target coordinates; the distance between the transformed target coordinates and the target coordinates with the second feature data is calculated; the target with the smallest distance is selected for matching to obtain the matched target. The matched target has both the first and second feature data. The matched target is compared with the preset control threshold information through the threshold comparison module 25 to determine whether to issue a warning. After fusion, each target contains multi-dimensional features, avoiding the one-sidedness of judging the target solely based on lidar or video data. This allows for a more accurate grasp of the overall situation of the target and improves the accuracy of traffic condition analysis and decision-making.

[0064] In this invention, the early warning control unit 30 includes a third communication module 31, an early warning release module 32, and a threshold setting module 33, wherein: the third communication module 31 is used to realize information interaction between the early warning control unit 30 and the data processing unit 20; the early warning release module 32 is used to receive early warning instructions and release early warning information according to the received early warning instructions, and the early warning release module 32 is connected to the third communication module 31; the threshold setting module 33 is used to upload preset control thresholds, and the threshold setting module 33 is connected to the third communication module 31.

[0065] In this embodiment, it should be noted that the warning release module 32 receives the warning release instruction through the third communication module 31. After receiving the instruction, the warning release module 32 will release the corresponding warning information. At the same time, the threshold setting module 33 can preset the control threshold and upload it to the data processing unit 20 for storage through the third communication module 31.

[0066] Furthermore, this embodiment also provides a roadside perception control method based on radar-visual fusion, which is applied to the aforementioned roadside perception control system based on radar-visual fusion, and includes the following steps:

[0067] Acquire sensor data, including LiDAR data and video data;

[0068] Spatial matching of LiDAR data and video data; the matching process is as follows:

[0069] Based on the calibration objects acquired by LiDAR and vision, the coordinates of the calibration objects acquired in the LiDAR coordinate system are mapped one-to-one with the coordinates in the vision coordinate system.

[0070] Based on the coordinates of the calibrated object, determine the transformation relationship between the lidar coordinate system and the visual coordinate system, (x v ,y v ,z v )=T×(x r ,y r ,z r ), where (x v ,y v ,z v (x) represents the target's coordinates in the lidar coordinate system. r ,y r ,z r ) represents the coordinates of the target in the visual coordinate system, and T is the transformation matrix.

[0071] Based on LiDAR data, first feature data is extracted; the process of extracting the first feature data is as follows:

[0072] Velocity feature extraction: Obtain the transmitted signal frequency f1, received signal frequency f2, and lidar propagation speed c of the target from the lidar, and calculate the target's velocity.

[0073] Acceleration feature extraction: Obtain velocity data of the target at several consecutive time points, and calculate the acceleration of the target at time i. In the formula, v i+1 Let v be the velocity at time i+1. i-1 Let Δt be the velocity at time i-1, and Δt be the lidar update cycle. Then, calculate the average value of several consecutive accelerations within the i+n cycle, which is the acceleration within that cycle.

[0074] Based on the video data, second feature data is extracted; the process of extracting the second feature data is as follows:

[0075] Extract the target contour shape from video data; identify the extracted target contour shape to obtain the target's classification information.

[0076] The feature data extracted from LiDAR and vision are fused; the process of fusing the feature data from LiDAR and vision is as follows:

[0077] Match the target coordinates with the first feature data to the visual coordinate system to obtain the target coordinates with the first feature data in the visual coordinate system, i.e., transform the target coordinates;

[0078] Calculate the distance between the transformed target coordinates and the target coordinates with the second feature data;

[0079] The target with the smallest gap is selected for matching to obtain the matched target, which has first feature data and second feature data.

[0080] Based on the characteristic data of the fused target, a warning command is issued by comparing it with the pre-stored control threshold.

[0081] In this embodiment, it should be noted that the target contour shape extraction method includes, but is not limited to, extraction using the Canny edge detection algorithm. Obtaining the target's speed information is crucial for analyzing the target's motion state, determining whether it is speeding, and dynamically evaluating subsequent traffic flow. It is one of the important indicators for describing the target's dynamic characteristics. Extracting the target contour shape helps to intuitively distinguish different types of targets. By comparing the fused target feature data with a preset control threshold, it is determined whether there are abnormal situations in the traffic scene (such as speeding, abnormal acceleration or deceleration, etc.), and then it is decided whether to issue a warning message to achieve real-time monitoring of traffic order and timely warning of potential dangers, thereby ensuring road traffic safety.

[0082] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0083] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A roadside perception control system based on radar and vision fusion, characterized in that, The system comprises a sensor unit (10), a data processing unit (20) and a pre-warning control unit (30), wherein: The sensor unit (10) is used to acquire data information of a target, which comprises lidar data, video data and positioning information data; The data processing unit (20) is used to match the lidar data and the video data in space, extract first feature data and second feature data based on the lidar data and the video data respectively, and fuse the feature data extracted by the lidar and vision, and is connected with the sensor unit (10); The data processing unit (20) comprises a second communication module (21), a coordinate conversion module (22) and a feature extraction module (23), wherein: The second communication module (21) is used to realize information interaction between the data processing unit (20) and the sensor unit (10) and the pre-warning control unit (30); The coordinate conversion module (22) is used to correspond the coordinates of the calibration object acquired under the lidar coordinate system with the coordinates under the vision coordinate system, determine the conversion relationship between the lidar coordinate system and the vision coordinate system, and realize the conversion between the lidar coordinates and the vision coordinates based on the calibration object acquired under the lidar and the calibration object acquired under the vision, and is connected with the second communication module (21); The feature extraction module (23) is used to extract the first feature data based on the lidar data, and extract the second feature data based on the video data, and is connected with the coordinate conversion module (22); The data processing unit (20) further comprises a feature fusion module (24), a threshold comparison module (25) and a database module (26), wherein: The feature fusion module (24) is used to fuse the feature data extracted by the lidar and the vision to obtain a matching target with the first feature data and the second feature data, and is connected with the feature extraction module (23); The threshold comparison module (25) is used to compare the fused feature data of the target with the pre-stored control threshold value to determine whether to issue a corresponding pre-warning instruction, and is connected with the feature fusion module (24) The database module (26) is used to store the acquired sensor data and the pre-set control threshold information, and is connected with the second communication module (21) and the threshold comparison module (25); The pre-warning control unit (30) is used to upload the pre-set control threshold value, receive a pre-warning instruction, and issue a pre-warning information according to the received pre-warning instruction, and is connected with the data processing unit (20); The process of extracting the first feature data is as follows: Speed feature extraction: obtain the emission signal frequency of the laser radar to the target , the receiving signal frequency , and the laser radar propagation speed , and calculate the speed of the target ; Acceleration feature extraction: Obtain velocity data of the target at several consecutive moments, and calculate the target's velocity at... The acceleration at time t is In the formula, for The velocity at time +1 for The velocity at time -1 The update cycle for the lidar is given; then, the values ​​at each stage are calculated. The average value of several consecutive accelerations within a +n period is the acceleration within that period. The process of extracting the second feature data is as follows: Extract the target contour shape from the video data, and identify the extracted target contour shape to obtain classification information of the target.

2. The radar and vision fusion based roadside perception control system of claim 1, wherein, The sensor unit (10) comprises a laser radar module (11), a position positioning module (12), a video acquisition module (13) and a first communication module (14), wherein: The data processing unit (20) is used for real-time laser radar data of the target; The position positioning module (12) is used for acquiring real-time positioning information of the target; The video acquisition module (13) is used for acquiring real-time video information of the target; The first communication module (14) is used for realizing information interaction between the sensor unit (10) and the data processing unit (20).

3. The weather fusion based roadside perception control system of claim 1, wherein, The early warning control unit (30) comprises a third communication module (31), an early warning issuing module (32) and a threshold setting module (33), wherein: The third communication module (31) is used for realizing information interaction between the early warning control unit (30) and the data processing unit (20); The early warning issuing module (32) is used for receiving a warning instruction and issuing a warning information according to the received warning instruction, and the early warning issuing module (32) is connected with the third communication module (31); The threshold setting module (33) is used for uploading the preset control threshold, and the threshold setting module (33) is connected with the third communication module (31).

4. The method for road side perception control based on radar and vision fusion, applied to the system for road side perception control based on radar and vision fusion according to any one of claims 1-3, characterized in that, The method comprises the following steps: Acquiring sensor data, wherein the sensor data comprises laser radar data and video data; Matching the laser radar data and the video data in space; Extracting first feature data based on the laser radar data; Extracting second feature data based on the video data; Fusing the feature data extracted by the laser radar and vision; According to the fused feature data of the target, comparing with the pre-stored control threshold to issue a warning instruction. 5.The method of claim 4, wherein, The matching process is as follows: Based on the calibration object acquired by the laser radar and the calibration object acquired by vision, the calibration object coordinates acquired by the laser radar coordinate system are one-to-one corresponding to the coordinates in the vision coordinate system; According to the calibration object coordinates, a conversion relationship between a laser radar coordinate system and a vision coordinate system is determined, , wherein, is a coordinate of the target in the laser radar coordinate system, is a coordinate of the target in the vision coordinate system, is a conversion matrix. 6.The method of claim 4, wherein, The process of fusing the feature data extracted by the laser radar and vision is as follows: Matching the target coordinates of the first feature data to the vision coordinate system to obtain the target coordinates with the first feature data in the vision coordinate system, that is, converting the target coordinates; Calculating the distance between the converted target coordinates and the target coordinates with the second feature data; Selecting the target with the smallest distance from the above to match, obtaining the matching target, and the matching target has the first feature data and the second feature data.

Images