Road condition detection method based on millimeter wave radar and camera

Abstract

The invention discloses a road condition detection method based on millimeter-wave radars and cameras. The method comprises the following specific steps: S1, installing millimeter-wave radars with different working frequencies and cameras with different precisions at corresponding fixed relative positions; s2, firstly, the millimeter wave radar with the lowest working frequency is adopted to acquire millimeter wave radar point cloud data corresponding to the road condition in real time, and then whether the acquired millimeter wave radar point cloud data meet requirements is analyzed; s3, acquiring an optimal millimeter wave radar as a radar data detection basis; acquiring optimal millimeter wave radar point cloud data in real time; s4, the camera with the lowest precision is adopted, and data of the current camera corresponding to the road condition is obtained in real time; then analyzing whether the acquired camera data meets the requirements or not; s5, acquiring an optimal camera as a camera data detection basis; and S6, matching and fusing the millimeter wave radar data and the camera data according to a preset time rule. The detection precision is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to road condition detection, and in particular relates to a road condition detection method based on millimeter wave radar and a camera. Background Art

[0002] Road condition monitoring is a key technology that uses sensors, cameras, radars and other equipment as well as data analysis and artificial intelligence algorithms to collect and analyze data and information on the road to assess and monitor road conditions such as traffic flow, congestion, and accident risks. It provides real-time road condition information and warnings for traffic management and drivers.

[0003] The technology of road condition detection mainly covers three core links: data collection, data processing and data analysis. In the data collection stage, through the use of advanced equipment such as sensors and cameras, a wide range of multi-dimensional information on the road is collected. This information includes key indicators such as traffic flow, vehicle speed, vehicle density and accident occurrence. In the data processing stage, the collected raw data is carefully preprocessed and cleaned to ensure that the accuracy and integrity of the data are retained to the greatest extent. In the data analysis stage, with the help of efficient algorithms and models, the processed data is deeply analyzed to extract valuable information and features, such as congestion conditions, accident risk warnings, etc., to provide strong decision-making support for road traffic management.

[0004] Millimeter-wave radar is generally used in the process of road condition detection. The applications of millimeter-wave radar in road conditions mainly include obstacle avoidance, driving assistance, anti-theft, adaptive cruise control, collision warning system, lane change assistance, active braking system, etc. The operating frequency range of millimeter-wave radar ranges from 24GHz to 77GHz. Among them, 24GHz radar is often used for parking assistance and blind spot monitoring, while 77GHz radar is mainly installed in the front to detect obstacles in the distance. The application of this technology improves the driver's field of vision and reaction speed, ensuring the stability and safety of driving.

[0005] For example: Patent application No.: 201910075572.9 discloses a method and device for target fusion of vehicle millimeter-wave radar and camera, which is used to solve the problem of target consistency confirmation in target-level data fusion of vehicle millimeter-wave radar and camera sensors, so as to improve the accuracy of target detection by vehicle environment perception sensors. The method includes: obtaining data information of a first target point and a second target point, the first target point represents the target object detected by the millimeter-wave radar sensor, and the second target point represents the target object detected by the camera; determining that the distance between the first target point and the second target point is less than a first preset threshold, at which time the first target point and the second target point are considered to be the same target object; combining the distance and speed of the target object detected by the millimeter-wave radar, and the category and lateral position of the target object detected by the camera as the target-level data information of the target object, but the structure has the following problems:

[0006] 1. The use of millimeter-wave radar sensors and cameras for separate detection results in the inability to complement and fuse the data of the two, and the information detection is not comprehensive, which ultimately affects the detection accuracy; 2. The millimeter-wave radar sensor uses a single frequency and cannot be replaced with millimeter-wave radar sensors of different frequencies according to actual conditions, which ultimately affects the accuracy of detection and therefore needs to be improved. Summary of the invention

[0007] The purpose of the present invention is to provide a road condition detection method based on millimeter-wave radar and camera, which solves the problem that the existing road condition detection uses millimeter-wave radar sensors and cameras for separate detection, resulting in the inability to complement and fuse the data of the two, and the information detection is not comprehensive, which ultimately affects the detection accuracy; and the millimeter-wave radar sensor uses a single frequency and cannot be replaced with a millimeter-wave radar sensor of a different frequency according to actual conditions, which ultimately affects the detection accuracy.

[0008] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0009] The present invention is a road condition detection method based on millimeter wave radar and camera, which specifically includes the following steps:

[0010] S1. Install millimeter-wave radars with different operating frequencies and cameras with different precisions at corresponding fixed relative positions;

[0011] S2, first use the millimeter-wave radar with the lowest operating frequency to obtain millimeter-wave radar point cloud data relative to the road condition in real time, and then analyze whether the obtained millimeter-wave radar point cloud data meets the requirements. If it meets the requirements, go to step S4; if it does not meet the requirements, go to step S3;

[0012] S3, selecting and judging the remaining millimeter-wave radars of other operating frequencies in turn, obtaining the best millimeter-wave radar as a basis for radar data detection; obtaining the best millimeter-wave radar point cloud data in real time; and proceeding to step S4;

[0013] S4, using the camera with the lowest accuracy and obtaining the data of the current camera relative to the road condition in real time; then analyzing whether the obtained camera data meets the requirements, if it meets the requirements, proceeding to step S6; if it does not meet the requirements, proceeding to step S5;

[0014] S5, selecting and judging the remaining cameras of other precisions in turn, obtaining the best camera as the basis for camera data detection; obtaining the detection data of the best camera in real time; and proceeding to step S6;

[0015] S6. Match and fuse the millimeter-wave radar data and the camera data according to a preset time regulation to obtain a data frame that is closest to the millimeter-wave radar data and the camera data to form a pairing group;

[0016] S7. Perform the following steps for each pair of paired groups:

[0017] S7.1, projecting the data points of each pairing group to the pixel position in the image space according to the relative position of the millimeter wave radar and the camera and the data information of the camera;

[0018] S7.2. If the pixel position corresponding to the data point of the paired group is within the pixel area of ​​a certain target, the data point is considered to have occurred and the data is saved. If it is not successfully judged to be outside the range, it means that the data point information has not occurred and no processing is performed.

[0019] Preferably, in step S2, the specific method of acquiring millimeter-wave radar point cloud data is as follows: taking frames as units, each frame of millimeter-wave radar point cloud data contains an acquisition time and a set of data points, wherein the set of data points is N acquired radar data points within the current acquisition time; and each radar data point contains a lateral position, a longitudinal position, a lateral velocity, and a longitudinal velocity.

[0020] Preferably, in step S4, each frame of camera data is a time and a two-dimensional image (t q ,Z), where t q is the previous frame time of the current camera data, Z is the two-dimensional matrix of the image, and the length and width of Z are the image resolution.

[0021] Preferably, when it is considered that the data point occurs in step S7.2, a pre-trained artificial intelligence (AI) neural network model is used to detect whether the target data point has a traffic jam or a traffic accident problem. If a traffic jam or a traffic accident is detected, the corresponding panoramic image and target trajectory data are stored to provide evidence for the corresponding traffic accident when a query request is received.

[0022] Preferably, in step S6, the specific method of matching and fusing the millimeter-wave radar data and the camera data is: during the fusion, ensure that the scanning period of the camera and the scanning period of the millimeter-wave radar have the same time interval, and select a frame as a reference marker frame at each corresponding time segment to achieve complete temporal fusion of the two sensors.

[0023] Preferably, after the camera acquires the data information in step S4, it is necessary to adjust the clarity of the data in advance.

[0024] Preferably, the millimeter wave radar includes a millimeter wave antenna, a radio frequency processing front end, and a digital-to-analog signal conversion module, and multiple millimeter wave radars use the same millimeter wave antenna.

[0025] Preferably, in step S7.2, a certain target refers to: detecting and classifying the target in advance, and then classifying the target into several categories such as pedestrians, bicycles, cars, and trucks through big data recognition, and marking the pixel area occupied by the target on the image in advance.

[0026] Preferably, after the targets are classified, simulated road condition analysis is performed on different targets in advance to generate simulated road condition modeling for different targets.

[0027] Preferably, when acquiring millimeter-wave radar point cloud data, the millimeter-wave radar needs to extract features from multi-frame point cloud data using time domain integration based on the sparsity of the point cloud, and estimate the three-dimensional corner points using chessboard measurement based on the reflection intensity distribution constraints of the chessboard point cloud. Then, combined with the two-dimensional corner points detected in the camera image, the two types of corner points are sequentially fused, and a two-dimensional image matrix for external calibration is generated by solving a nonlinear optimization problem.

[0028] The present invention has the following beneficial effects: the present invention adopts pre-setting of multiple millimeter-wave radars and multiple cameras for data detection to further improve the detection accuracy, and adopts the method of effectively integrating the millimeter-wave radar and the camera to further realize data complementary fusion and realize comprehensive information detection, which ultimately affects the detection accuracy and precision. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of a road condition detection method based on millimeter-wave radar and camera in Example 1. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0031] Example 1

[0032] See also Figure 1 As shown, the present embodiment discloses a road condition detection method based on millimeter wave radar and camera, which specifically includes the following steps:

[0033] S1. Install millimeter-wave radars with different operating frequencies and cameras with different precisions at corresponding fixed relative positions;

[0034] S2, first use the millimeter-wave radar with the lowest operating frequency to obtain millimeter-wave radar point cloud data relative to the road condition in real time, and then analyze whether the obtained millimeter-wave radar point cloud data meets the requirements. If it meets the requirements, go to step S4; if it does not meet the requirements, go to step S3;

[0035] S3, selecting and judging the remaining millimeter-wave radars of other operating frequencies in turn, obtaining the best millimeter-wave radar as a basis for radar data detection; obtaining the best millimeter-wave radar point cloud data in real time; and proceeding to step S4;

[0036] S4, using the camera with the lowest accuracy and obtaining the data of the current camera relative to the road condition in real time; then analyzing whether the obtained camera data meets the requirements, if it meets the requirements, proceeding to step S6; if it does not meet the requirements, proceeding to step S5;

[0037] S5, selecting and judging the remaining cameras of other precisions in turn, obtaining the best camera as the basis for camera data detection; obtaining the detection data of the best camera in real time; and proceeding to step S6;

[0038] S6. Match and fuse the millimeter-wave radar data and the camera data according to a preset time regulation to obtain a data frame that is closest to the millimeter-wave radar data and the camera data to form a pairing group;

[0039] S7. Perform the following steps for each pair of paired groups:

[0040] S7.1, projecting the data points of each pairing group to the pixel position in the image space according to the relative position of the millimeter wave radar and the camera and the data information of the camera;

[0041] S7.2. If the pixel position corresponding to the data point of the paired group is within the pixel area of ​​a certain target, the data point is considered to have occurred and the data is saved. If it is not successfully judged to be outside the range, it means that the data point information has not occurred and no processing is performed.

[0042] Preferably, in step S2, the specific method of acquiring millimeter-wave radar point cloud data is as follows: taking frames as units, each frame of millimeter-wave radar point cloud data contains an acquisition time and a set of data points, wherein the set of data points is N acquired radar data points within the current acquisition time; and each radar data point contains a lateral position, a longitudinal position, a lateral velocity, and a longitudinal velocity.

[0043] Preferably, in step S4, each frame of camera data is a time and a two-dimensional image (t q ,Z), where t q is the previous frame time of the current camera data, Z is the two-dimensional matrix of the image, and the length and width of Z are the image resolution.

[0044] Preferably, when it is considered that the data point occurs in step S7.2, a pre-trained artificial intelligence (AI) neural network model is used to detect whether the target data point has a traffic jam or a traffic accident problem. If a traffic jam or a traffic accident is detected, the corresponding panoramic image and target trajectory data are stored to provide evidence for the corresponding traffic accident when a query request is received.

[0045] Preferably, in step S6, the specific method of matching and fusing the millimeter-wave radar data and the camera data is: during the fusion, ensure that the scanning period of the camera and the scanning period of the millimeter-wave radar have the same time interval, and select a frame as a reference marker frame at each corresponding time segment to achieve complete temporal fusion of the two sensors.

[0046] Preferably, after the camera acquires the data information in step S4, it is necessary to adjust the clarity of the data in advance.

[0047] Preferably, the millimeter wave radar includes a millimeter wave antenna, a radio frequency processing front end, and a digital-to-analog signal conversion module, and multiple millimeter wave radars use the same millimeter wave antenna.

[0048] Preferably, in step S7.2, a certain target refers to: detecting and classifying the target in advance, and then classifying the target into several categories such as pedestrians, bicycles, cars, and trucks through big data recognition, and marking the pixel area occupied by the target on the image in advance.

[0049] Preferably, after the targets are classified, simulated road condition analysis is performed on different targets in advance to generate simulated road condition modeling for different targets.

[0050] Preferably, when acquiring millimeter-wave radar point cloud data, the millimeter-wave radar needs to extract features from multi-frame point cloud data using time domain integration based on the sparsity of the point cloud, and estimate the three-dimensional corner points using chessboard measurement based on the reflection intensity distribution constraints of the chessboard point cloud. Then, combined with the two-dimensional corner points detected in the camera image, the two types of corner points are sequentially fused, and a two-dimensional image matrix for external calibration is generated by solving a nonlinear optimization problem.

[0051] The specific working process of the present invention is as follows: install millimeter-wave radars of different working frequencies and cameras of different precisions at corresponding fixed relative positions; then use the millimeter-wave radar with the lowest working frequency to obtain millimeter-wave radar point cloud data relative to the road condition in real time, and then analyze whether the obtained millimeter-wave radar point cloud data meets the requirements. If it meets the requirements, use the camera with the lowest precision and obtain the data of the current camera relative to the road condition in real time; then analyze whether the obtained camera data meets the requirements; then proceed to the next step 1: if it does not meet the requirements, select and judge the remaining millimeter-wave radars of other working frequencies in turn, and obtain the best millimeter-wave radar as the basis for radar data detection; obtain the best millimeter-wave radar point cloud data in real time; then proceed to the next step 1:

[0052] Next step 1: If the camera with the lowest accuracy is used and the data of the current camera relative to the road condition is obtained in real time; then analyze whether the obtained camera data meets the requirements. If it meets the requirements, match and fuse the millimeter-wave radar data and the camera data according to the preset time regulations to obtain the data frame closest to the millimeter-wave radar data and the camera data to form a pairing group; then proceed to next step 2:

[0053] If it does not meet the requirements, the remaining cameras with other accuracies are selected and judged in turn to obtain the best camera as the basis for camera data detection; the detection data of the best camera is obtained in real time; the millimeter-wave radar data and the camera data are matched and fused according to the preset time regulations to obtain the data frame closest to the millimeter-wave radar data and the camera data to form a pairing group, and then proceed to the next step 2:

[0054] Next step 2: For each pair of pairs, perform the following steps:

[0055] S7.1, projecting the data points of each pairing group to the pixel position in the image space according to the relative position of the millimeter wave radar and the camera and the data information of the camera;

[0056] S7.2. If the pixel position corresponding to the data point of the paired group is within the pixel area of ​​a certain target, the data point is considered to have occurred and the data is saved. If it is not successfully judged to be outside the range, it means that the data point information has not occurred and no processing is performed.

[0057] Therefore, the present invention adopts pre-setting multiple millimeter-wave radars and multiple cameras for data detection to further improve the detection accuracy, and adopts the method of effectively integrating the millimeter-wave radar and the camera to further realize data complementary fusion, realize comprehensive information detection, and ultimately affect the detection precision and accuracy.

[0058] The above are only preferred embodiments of the present invention and do not limit the present invention. Any modification to the technical solutions recorded in the aforementioned embodiments, any equivalent replacement of some of the technical features therein, and any modification, equivalent replacement, and improvement made shall fall within the protection scope of the present invention.

Claims

1. A road condition detection method based on millimeter wave radar and camera, characterized in that: The specific steps include: S1. Install millimeter-wave radars with different operating frequencies and cameras with different precisions at corresponding fixed relative positions; S2, first use the millimeter-wave radar with the lowest operating frequency to obtain millimeter-wave radar point cloud data relative to the road condition in real time, and then analyze whether the obtained millimeter-wave radar point cloud data meets the requirements. If it meets the requirements, go to step S4; if it does not meet the requirements, go to step S3; S3, selecting and judging the remaining millimeter-wave radars of other operating frequencies in turn, obtaining the best millimeter-wave radar as a basis for radar data detection; obtaining the best millimeter-wave radar point cloud data in real time; and proceeding to step S4; S4, using the camera with the lowest accuracy and obtaining the data of the current camera relative to the road condition in real time; then analyzing whether the obtained camera data meets the requirements, if it meets the requirements, proceeding to step S6; if it does not meet the requirements, proceeding to step S5; S5, selecting and judging the remaining cameras of other precisions in turn, obtaining the best camera as the basis for camera data detection; obtaining the detection data of the best camera in real time; and proceeding to step S6; S6. Match and fuse the millimeter-wave radar data and the camera data according to a preset time regulation to obtain a data frame that is closest to the millimeter-wave radar data and the camera data to form a pairing group; S7. Perform the following steps for each pair of paired groups: S7.1, projecting the data points of each pairing group to the pixel position in the image space according to the relative position of the millimeter wave radar and the camera and the data information of the camera; S7.

2. If the pixel position corresponding to the data point of the paired group is within the pixel area of ​​a certain target, the data point is considered to have occurred and the data is saved. If it is not successfully judged to be outside the range, it means that the data point information has not occurred and no processing is performed.

2. The road condition detection method based on millimeter wave radar and camera according to claim 1, characterized in that: In step S2, the specific method of acquiring millimeter-wave radar point cloud data is as follows: taking frames as units, each frame of millimeter-wave radar point cloud data contains an acquisition time and a set of data points, wherein the set of data points is N acquired radar data points within the current acquisition time; and each radar data point contains a lateral position, a longitudinal position, a lateral velocity, and a longitudinal velocity.

3. The road condition detection method based on millimeter wave radar and camera according to claim 1, characterized in that: In step S4, each frame of camera data is a time and a two-dimensional image (t q ,Z), where t q is the previous frame time of the current camera data, Z is the two-dimensional matrix of the image, and the length and width of Z are the image resolution.

4. The road condition detection method based on millimeter wave radar and camera according to claim 1, characterized in that: When it is considered in step S7.2 that the data point occurs, a pre-trained artificial intelligence (AI) neural network model is used to detect whether the target data point has traffic congestion or a traffic accident problem. If traffic congestion or a traffic accident is detected, the corresponding panoramic image and target trajectory data are stored to provide evidence for the corresponding traffic accident when a query request is received.

5. The road condition detection method based on millimeter wave radar and camera according to claim 1, characterized in that: The specific method of matching and fusing the millimeter-wave radar data and the camera data in step S6 is: during the fusion, ensure that the camera's sweep frequency cycle and the millimeter-wave radar's sweep frequency cycle have the same time interval, select a frame as a reference marker frame at each corresponding time interval, and achieve complete temporal fusion of the two sensors.

6. The road condition detection method based on millimeter wave radar and camera according to claim 1, characterized in that: After the camera acquires the data information in step S4, it is necessary to adjust the clarity of the data in advance.

7. The road condition detection method based on millimeter wave radar and camera according to claim 1, characterized in that: The millimeter wave radar includes a millimeter wave antenna, a radio frequency processing front end, and a digital-to-analog signal conversion module, and multiple millimeter wave radars use the same millimeter wave antenna.

8. The road condition detection method based on millimeter wave radar and camera according to claim 1, characterized in that: In step S7.2, a certain target refers to: detecting and classifying the target in advance, and then classifying the target into several categories such as pedestrians, bicycles, cars, and trucks through big data recognition, and marking the pixel area occupied by the target on the image in advance.

9. The road condition detection method based on millimeter wave radar and camera according to claim 8, characterized in that: After the targets are classified, simulated road condition analysis is performed on different targets in advance to generate simulated road condition modeling for different targets.

10. The road condition detection method based on millimeter wave radar and camera according to claim 8, characterized in that: When acquiring millimeter-wave radar point cloud data, the millimeter-wave radar needs to use time domain integration to extract features from multi-frame point cloud data based on the sparsity of the point cloud, and use chessboard measurement to estimate the three-dimensional corner points based on the reflection intensity distribution constraints of the chessboard point cloud. Then, combined with the two-dimensional corner points detected in the camera image, the two types of corner points are sequentially fused, and the two-dimensional image matrix of external calibration is generated by solving a nonlinear optimization problem.

Images