A road obstacle detection method based on millimeter-wave radar and visual signals

Abstract

A road obstacle detection method based on millimeter-wave radar and visual signals uses millimeter-wave radar and a visual sensor to detect obstacles ahead. The method then fuses the millimeter-wave radar and visual image signals to determine the target object's type. The method first preprocesses the millimeter-wave radar signal, converting the millimeter-wave radar message into a target point position and time-synchronizing it with the visual image signal. The millimeter-wave radar target point is mapped onto the visual image signal based on a coordinate matching relationship, achieving a fusion of the two modalities. Target detection is then performed on the millimeter-wave radar information mapped onto the visual image signal. Finally, the detection results are scaled and refined to refine the obstacle information. This method achieves multimodal, fine-grained obstacle detection, offering advantages in accuracy and efficiency over traditional single-modality methods, and possesses high practical value.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and machine learning technology, and relates to convolutional neural network and knowledge base technology, specifically a road obstacle detection method based on millimeter-wave radar and visual signals. Background Art

[0002] With the advancement of artificial intelligence (AI), autonomous driving technology is increasingly gaining traction. Autonomous driving integrates numerous technologies, including automatic control, intelligent environmental perception, and optimization, and is a product of highly advanced AI. Intelligent environmental perception plays a particularly crucial role in autonomous driving, as obstacle perception provides a crucial basis for vehicle control planning. Only real-time and accurate obstacle perception and detection can enable autonomous vehicles to effectively avoid obstacles, prevent collisions, and ensure the safety of autonomous driving.

[0003] At present, obstacle perception is mainly based on optical sensors, visual sensors or millimeter-wave radar sensors.

[0004] Optical sensors use an optical ranging instrument mounted on the vehicle body to detect distances around the vehicle. However, these sensors only provide distance information, not specific category information. Furthermore, the ranging device is fixed and can only detect a fixed direction, resulting in low reliability. Furthermore, the device is exposed to the outside of the vehicle and is susceptible to damage.

[0005] Radar detection involves extracting features reflecting target attributes from the radar signal echo signature of a target. A machine then determines the target's category or model based on specific criteria. Millimeter-wave short-range detection radars have a wide range of applications due to their small size, light weight, ease of integration, wide bandwidth, high resolution, strong anti-interference performance, and good all-weather operation. However, the dense distribution, large number, and diverse types of targets in road scenes for autonomous driving applications, along with their diverse and changing motion states and complex clutter, significantly reduce the robustness of target feature extraction. Furthermore, the incomplete and low-reliability of millimeter-wave radar target characteristic data, as well as the complex environments and poor adaptability of manually designed recognition features, further limit the engineering application and performance improvement of target recognition algorithms. The complexity of the clutter environment and the inherent technical bottlenecks of target recognition make the application of millimeter-wave radar detection alone in autonomous driving scenarios difficult.

[0006] Another common approach is to detect objects around a vehicle using vision. Research on vision-based object detection methods focuses on optimizing detection algorithms and improving performance, aiming for more accurate and efficient object recognition. However, in practical applications, visual object detection methods struggle to obtain accurate information such as the target's location and are susceptible to interference from factors such as lighting and weather that can affect visual images. Summary of the Invention

[0007] The problem to be solved by the present invention is that the existing technology mainly relies on a single signal source in environmental perception obstacle detection, which cannot avoid the low recognition accuracy caused by the limitations of sensor signals and the inability to effectively monitor the target, and cannot meet the monitoring needs of actual complex road scenes.

[0008] The technical solution of the present invention is: a road obstacle detection method based on millimeter-wave radar and visual signals. The method detects obstacles in front of the road using millimeter-wave radar and visual sensors, and detects the type of target object by fusing the millimeter-wave radar and visual image signals. First, the millimeter-wave radar signal is preprocessed, and the millimeter-wave radar message is converted into the form of target point position and time-synchronized with the visual image signal. The millimeter-wave radar target point is mapped onto the visual image signal based on a coordinate matching relationship to achieve the fusion of the two modalities. Target detection is performed on the millimeter-wave radar information mapped to the visual image signal. Finally, the detection result is scale-estimated to refine the obstacle information.

[0009] Furthermore, the millimeter-wave radar information mapped onto the visual image signal is first subjected to similarity detection. Structures below the similarity threshold are judged as clutter and no longer undergo the next step of detection. Structures above the similarity threshold are then subjected to target detection through a feature extraction network.

[0010] Furthermore, the present invention specifically comprises the following steps:

[0011] 1) Obtain the millimeter-wave radar sensor message signal stream and the visual image signal stream, and use multi-threaded processing based on the message timestamp to synchronize the two signal streams;

[0012] 2) Taking the millimeter-wave radar signal and visual image signal at the same time, the synchronized millimeter-wave radar signal is converted and mapped onto the visual image signal to obtain a mapped point set S. S contains a set of pixel coordinate points on the image, which is the target position mapping of the millimeter-wave detection object in the image;

[0013] 3) For each point in the point set S obtained in step 2), perform edge box local similarity estimation on the image at that location;

[0014] 4) Based on the edge box similarity estimation results, the millimeter-wave radar signal is clutter filtered. Millimeter-wave detection points with edge box similarity above the threshold are retained, while detection points with similarity below the threshold are considered irrelevant clutter and filtered out.

[0015] 5) Using convolutional neural networks to extract local features of the target position of the filtered millimeter-wave radar signal;

[0016] 6) Performing feature pyramid feature fusion on the multi-scale features output in step 5);

[0017] 7) Classify and regress the fused features to obtain the detection results;

[0018] 8) Integrate the detection results of each millimeter-wave radar mapping point and perform non-maximum suppression;

[0019] 9) The prediction category, confidence level, location results and other information are formatted in a standard manner, output in descending order of confidence level, and the obstacle detection results are displayed.

[0020] The present invention also provides a road obstacle detection device based on millimeter wave radar and visual signals. The device is provided with a computer-readable storage medium, and the computer-readable storage medium is configured with a computer program. When the computer program is executed, the above-mentioned road obstacle detection method is implemented.

[0021] The present invention accurately and efficiently uses millimeter-wave radar sensor information and visual sensor information to detect road obstacle targets, and includes heterogeneous signal synchronous matching technology, millimeter-wave radar signal mapping technology, similarity estimation technology, image feature extraction technology, feature pyramid feature fusion technology, image feature target detection technology and scale estimation technology.

[0022] The beneficial effect of the present invention is to provide an efficient and accurate obstacle detection and recognition technology. Compared with the existing technology, it has the following advantages.

[0023] (1) The present invention includes a computer vision target detection function based on deep learning, extracts the semantic information of sensor data, and effectively improves the accuracy and efficiency of target detection.

[0024] (2) The present invention also combines millimeter-wave radar sensor signals to improve the stability of the system when the visual image effect is poor under abnormal conditions such as rainy and snowy weather, and more comprehensively covers long-distance targets that are easily overlooked in visual detection.

[0025] (3) The structure of the present invention can also be expanded to other types of detection, such as road driving detection, personnel flow detection, etc., with extremely strong generalization ability and broad market application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is the overall flow chart of the present invention.

[0027] Figure 2 This is the flow chart of millimeter wave radar clutter filtering. DETAILED DESCRIPTION

[0028] The present invention proposes a road obstacle detection method based on millimeter wave radar and visual signals, which combines two modal signals, millimeter wave radar and visual signals, to identify road obstacles. The overall flow chart of the method of the present invention is as follows: Figure 1 The technical solution of the present invention specifically includes the following steps:

[0029] (1) Input the millimeter-wave radar sensor message signal stream and the visual image signal stream. For two input streams from different sources, synchronize and match the two inputs based on their message timestamps.

[0030] Separate millimeter-wave radar message reading threads and visual image reading threads are established. The video frames read by the visual image reading thread are stored in a data structure dictionary using their timestamps as keys. Simultaneously, the millimeter-wave radar message reading thread reads the latest millimeter-wave radar message and uses the timestamp of the read millimeter-wave radar message to index the dictionary storing the visual image. If the index is successful, this set of messages is matched with the visual image for subsequent processing. If the index is unsuccessful, the timestamp is compared with the earliest and latest timestamps in the dictionary. If the millimeter-wave radar message timestamp is earlier than the earliest visual data, the visual image reading thread is put into hibernation until the millimeter-wave radar message is read at the same time as the image data. Similarly, if the millimeter-wave radar message is later than the latest visual data, the millimeter-wave radar message reading thread is put into hibernation until the visual image data is read at the same time.

[0031] The image reading thread stores up to two minutes of visual signals. As new data is read in, the oldest data is cleared in chronological order. The two-minute buffer window is established to prevent data mismatches caused by thread scheduling issues due to the uncertainty inherent in multi-threaded execution.

[0032] (2) Take a set of millimeter-wave radar signals and visual image signals. The millimeter-wave radar signal is the millimeter-wave radar detection result at a certain moment, and the visual image signal is the monitoring image of the vehicle-mounted image acquisition device at the same moment. Transform the synchronized millimeter-wave radar signal and map it onto the visual image signal to obtain the mapped point set S, which contains a set of pixel coordinates on the image.

[0033] The mapping of millimeter-wave radar signals to visual image signals is essentially a conversion between the real-world coordinate system and the pixel coordinate system. The real-world coordinate system must first be converted to the image coordinate system, and then the image coordinate system must be converted to the pixel coordinate system. The pixel coordinate system refers to the upper left corner of the visual image signal as the origin, with the u axis pointing right and the v axis pointing downward. The image coordinate system refers to the center of the image as the origin, with the x axis pointing right and the y axis pointing downward. The conversion relationship between the pixel coordinate system and the image coordinate system is:

[0034]

[0035] Where u0, v0 are the coordinates of the origin of the image coordinate system in the pixel coordinate system, u, v are the coordinates of the target point in the pixel coordinate system, and x, y are the coordinates of the target point in the image coordinate system. x Indicates the actual physical size of each pixel along the x-axis; d y Represents the actual physical size of each pixel along the y-axis.

[0036] The above formula is expressed in homogeneous coordinates as:

[0037]

[0038] The correspondence between the image coordinate system and the real coordinate system has the following conversion relationship:

[0039]

[0040] Where f is the focal length of the camera, X c ,Y c ,Z c The x, y, and z axis coordinates of the target position in the real coordinate system.

[0041] Combining the above formula, the conversion relationship between the pixel coordinate system and the real coordinate system is as follows:

[0042]

[0043] in, It is the intrinsic parameter matrix of the camera, which is determined by the camera itself and is expressed as M C The above formula can be expressed as:

[0044]

[0045] From this, we can get the coordinate conversion formula between the real coordinate system and the pixel coordinate system. The signal provided by the millimeter wave radar sensor is the real coordinate of the target. According to this formula, the millimeter wave radar detection signal is converted into pixel coordinates and mapped onto the visual image.

[0046] (3) For the millimeter-wave radar detection result mapping point set S obtained in step 2, take each point in S and perform edge box local similarity estimation on the image at that location.

[0047] The edge box method uses the contour information generated by the edges of objects in an image to calculate the number of contours contained within the box and those that overlap with the box's edges. The more contour information a target box contains, the more likely it is to contain the target. All potential target boxes are then scored and ranked to achieve a rough estimate of the target area. Among existing object similarity estimation methods, edge boxes are highly capable of detecting objects of various shapes, exhibit a certain degree of scale adaptability, and offer high computational speed.

[0048] Based on the edge box's object-likeness estimation results, the millimeter-wave radar signal is clutter filtered. Millimeter-wave radar detection points with high edge box object-likeness are retained, while detection points with low object-likeness are treated as irrelevant clutter and filtered.

[0049] (4) Extract local features of the target position of the filtered millimeter-wave radar signal.

[0050] Local feature extraction is to inversely infer the pixel size in the visual signal by the target size, and then intercept the local image signal for feature extraction.

[0051] The pixel scale estimation of the obstacle is based on the internal and external parameters of the camera, the target distance and orientation measured by the millimeter-wave radar, and the actual size of the expected target. First, the value of each pixel in the visual image is obtained by the light intensity at the corresponding position on the camera's visual sensor. Let the width and height of the visual sensor be W respectively. s ,H s , the pixel width and height of the target image is w p ,h p , the image resolution is W p ,H p Then there is where w s ,h s is the width and height of the target image on the visual sensor. Then, the target distance and orientation measured by the millimeter wave radar can be used to obtain the vertical distance d from the target to the camera through the lens and the perpendicular line of the optical axis. According to the perspective principle, from the focal length f, we can get where w t ,h t is the actual width and height of the target. According to sampling statistics, the length and width of the target to be detected are concentrated within 2 meters. According to the above formula, let w t =2,h t=2, the distance detected by the millimeter wave radar can be used to calculate the pixel scale of the target in the visual signal. With the detection point as the center, w s ,h s The length and width are used to intercept the local image and input it into the subsequent target detection neural network.

[0052] Feature extraction uses CSP-Darknet, which primarily consists of CBL and CSP modules. The CBL module is a convolutional neural network architecture consisting of a convolutional layer, a batch normalization layer, and an activation function layer. Leaky ReLU is used as the activation function. The network input is a local image of the target area of ​​the millimeter-wave radar signal. After multiple convolutional layers are applied, feature outputs of varying scales are captured at different depths within the network.

[0053] (5) Perform feature pyramid feature fusion on the three features output by the step.

[0054] Feature fusion combines the outputs of convolutional neural networks at different depths, allowing shallow features to incorporate deeper semantic information, and deep features to incorporate more shallow positional information. Multi-scale feature fusion utilizes a feature pyramid structure, dividing the features extracted from the previous level into three different scales through bilinear upsampling and convolutional neural network downsampling. Larger-scale features have finer feature centers, allowing them to predict smaller objects in the image. Smaller-scale features have sparser feature centers and a larger receptive field, allowing them to predict larger objects. Side connections within the feature pyramid structures ensure the fusion of deep semantic information and shallow positional information, improving the accuracy of subsequent prediction branches for object classification and position regression. Furthermore, a bottom-up feature pyramid is added to the top-down feature pyramid. This includes two PAN structures. The PAN structure uses convolutional layers for secondary downsampling. This combination allows the feature pyramid to convey strong semantic features from the top down, while the PAN conveys strong localization features from the bottom up. Parameters of different detection layers are aggregated from different backbone layers.

[0055] (6) Classify and regress the features to obtain the detection results.

[0056] For each pixel position of the multiple scale features after feature fusion, the convolution layer is used to reduce the dimension, and the features are further abstracted and passed into the classification prediction branch and the detection frame position regression branch respectively. The classification branch is responsible for predicting the possible object category of the pixel position. In the present invention, the confidence of each category is calculated according to the pre-trained classification network, and the category with the highest confidence at that position is output. The recognizable categories include pedestrians, bicycles, cars, card pools, street lights, dogs, light signs, and a large number of object categories that may appear on the road. The detection frame position regression predicts the length, width and offset of the detection frame based on the pixel position. The pixel coordinates of the detection frame can be calculated based on the prediction results, combined with the feature scale and pixel position information.

[0057] (7) Integrate the detection results of each millimeter-wave radar mapping point and perform non-maximum suppression.

[0058] The detection frame coordinates obtained in step 7 are based on the local image of each millimeter-wave radar mapping point. Therefore, after obtaining the detection results of all mapping points, the detection frame coordinates need to be converted into pixel coordinates in the original visual image. The conversion method is to add the x and y coordinates of the local pixel coordinates to the x and y pixel coordinates of the upper left corner of the local image in the original image. After the conversion is completed, a series of detection frame pixel coordinates in the original image are obtained, which contains a large number of duplicate detections. In order to remove duplicate detections, the results need to be non-maximum suppressed. The specific method is to sort in descending order according to the classification confidence. If the overlap ratio of a detection frame with another detection frame ranked before it is higher than a fixed threshold, it is removed from the detection results.

[0059] (8) The prediction category, confidence, location, scale estimation results and other information are formatted in a standard way and output in descending order of confidence.

[0060] The above method of the present invention is implemented by a computer program, and provides a road obstacle detection device based on millimeter-wave radar and visual image signals. The device is provided with a computer-readable storage medium, and the computer-readable storage medium is configured with a computer program to implement the above detection method. The computer program is divided into functional modules and includes a signal preprocessing module, a clutter filtering module, a modal mapping module and a multimodal detection module. The signal preprocessing module is used to preprocess the millimeter-wave radar signal, convert the millimeter-wave radar message into the form of the target point position, and synchronize it with the visual signal in time; the modal mapping module maps the millimeter-wave radar target point onto the visual signal to achieve the fusion of the two modes; the clutter filtering module is used for similarity judgment and filters the clutter generated by environmental noise in the millimeter-wave radar signal; the multimodal detection module performs target detection on the millimeter-wave radar information mapped to the image information collected by the visual sensor; the scale estimation module performs scale estimation on the detection result and refines the obstacle information.

Claims

1. A road obstacle detection method based on millimeter wave radar and visual signals, characterized by Obstacles ahead of the road are detected using millimeter-wave radar and visual sensors. The target object category is detected by fusing the millimeter-wave radar and visual image signals. First, the millimeter-wave radar signal is preprocessed, and the millimeter-wave radar message is converted into the form of target point position and time-synchronized with the visual image signal. The millimeter-wave radar target point is mapped onto the visual image signal based on the coordinate matching relationship to achieve the fusion of the two modalities. Target detection is performed on the millimeter-wave radar information mapped to the visual image signal, and finally the scale of the detection result is estimated to refine the obstacle information. Among them, the millimeter-wave radar information mapped to the visual image signal is first tested for similarity. If it is below the similarity threshold, it is judged as clutter and no longer undergoes the next step of detection. Structures above the similarity threshold are then subjected to target detection through the feature extraction network. The specific steps include: 1) Obtain the millimeter-wave radar sensor message signal stream and the visual image signal stream, and use multi-threaded processing based on the message timestamp to synchronize the two signal streams; 2) Taking the millimeter-wave radar signal and visual image signal at the same time, the synchronized millimeter-wave radar signal is converted and mapped onto the visual image signal to obtain a mapped point set S. S contains a set of pixel coordinate points on the image, which is the target position mapping of the millimeter-wave detection object in the image; 3) For each point in the point set S obtained in step 2), perform edge box local similarity estimation on the image at that location; 4) Based on the edge box similarity estimation results, the millimeter-wave radar signal is clutter filtered. Millimeter-wave detection points with edge box similarity above the threshold are retained, while detection points with similarity below the threshold are considered irrelevant clutter and filtered out. 5) Using convolutional neural networks to extract local features of the target position of the filtered millimeter-wave radar signal; 6) Performing feature pyramid feature fusion on the multi-scale features output in step 5); 7) Classify and regress the fused features to obtain the detection results; 8) Integrate the detection results of each millimeter-wave radar mapping point and perform non-maximum suppression; 9) The prediction category, confidence level, and location result information are formatted in a standard manner, output in descending order of confidence level, and the obstacle detection results are displayed.

2. The method for detecting road obstacles based on millimeter-wave radar and visual signals according to claim 1, characterized in that When the millimeter-wave radar signal and the visual image signal are time-synchronized, multi-threaded heterogeneous data synchronization matching is adopted. Specifically, a millimeter-wave radar message reading thread and a visual image reading thread are established respectively, and the video frame read by the visual image reading thread is saved in a data structure dictionary with its timestamp as the key. At the same time, the millimeter-wave radar message reading thread reads the latest millimeter-wave radar message and uses the read millimeter-wave radar message timestamp to index the dictionary storing the visual image. If the index is successful, this group of messages is matched with the visual image for subsequent processing. If the index is not successful, its timestamp is compared with the earliest and latest timestamps in the dictionary. If the millimeter-wave radar message timestamp is earlier than the earliest visual image data, the visual image reading thread is put into sleep until the millimeter-wave radar message is read at the same time as the visual image data; similarly, if the millimeter-wave radar message is later than the latest visual image data, the millimeter-wave radar message reading thread is put into sleep until the visual image data is read at the same time.

3. The method for detecting road obstacles based on millimeter-wave radar and visual signals according to claim 1, characterized in that The millimeter wave radar target point is mapped to the visual image signal as follows: where X c ,Y c ,Z c is the x, y, and z axis coordinates of the millimeter wave radar target point in the real coordinate system, M C is the internal parameter matrix of the visual sensor, which is determined by the properties of the visual sensor itself. u and v are the coordinates of the target point in the pixel coordinate system.

4. The method for detecting road obstacles based on millimeter wave radar and visual signals according to claim 1, characterized in that Adaptive local visual signal extraction is used for target detection. The specific method is as follows: where w t ,h t is the actual width and height of the target, w s ,h s is the width and height of the target image on the visual sensor, d is the vertical distance from the target to the vertical line of the optical axis through the lens of the visual sensor, and f is the focal length of the visual sensor. According to the above formula, the corresponding relationship between the scale in the image and the length and width of the actual target is obtained. If the length and width of the detected target are concentrated within n meters, then let w t =n,h t =n, the distance detected by the millimeter wave radar, calculate the pixel scale of the target in the visual image signal, with the detected target point as the center, w s ,h s The length and width are used to intercept the local image and input it into the subsequent target detection neural network for target detection.

5. A road obstacle detection device based on millimeter wave radar and visual signals, characterized by The device is provided with a computer-readable storage medium, and the computer-readable storage medium is configured with a computer program, and when the computer program is executed, the road obstacle detection method according to any one of claims 1 to 4 is implemented.

Images