Non-line-of-sight ghost detection and warning method based on millimeter-wave radar and vision equipment

Abstract

The present invention discloses a non-line-of-sight ghost detection warning method based on a vehicle-mounted millimeter-wave radar and a visual sensor, which is applied to the field of non-line-of-sight target detection. It aims to solve the problems of high detection cost, low detection accuracy and low efficiency in the existing technology for detecting the "ghost" problem. The present invention first obtains information about the surrounding environment through a millimeter-wave radar sensor and a visual sensor, synchronously processes the data of the two sensors, and then calibrates the visual sensor. If there is a posture change, it performs posture inverse processing. Then, interpolation is performed to obtain the target position and target tracking results output by the millimeter-wave radar sensor, and at the same time, the target detection, classification results and target position processed based on the visual sensor data are obtained. Finally, based on the data association method of the present invention, the two types of data are associated, denoised, and displayed to achieve the effect of warning of "ghost" accidents and locating "ghosts".

Description

Technical Field

[0001] The present invention belongs to the technical field of non-line-of-sight target detection, and in particular relates to a non-line-of-sight ghost detection early warning technology. Background Art

[0002] Pedestrian detection, as a key component of safe driving, is an important guarantee for driving safety and intelligence. A single detection method or sensor makes it difficult to robustly perceive pedestrians in complex scenarios. Using multiple sensors, however, can obtain more comprehensive and compatible pedestrian information, thus meeting the reliability and accuracy requirements of safe driving systems. In real life, it's not uncommon for pedestrians to suddenly appear out of the driver's blind spot due to non-compliance with traffic rules, causing many serious traffic accidents. To combat these "ghosting," traditional approaches have proposed a method based on real-time detection of the road surface and intersections to alert drivers. This method is typically implemented using devices such as visual sensors and millimeter-wave radars.

[0003] Millimeter-wave radar is a sensor that obtains target-related information by emitting a series of FMCW waveforms and receiving electromagnetic waves reflected from the target. Compared to other radars, millimeter-wave radar has higher range and velocity resolution. Moreover, because it is not affected by weather, light, temperature, and haze, compared to visual sensors, millimeter-wave radar is very suitable for detecting targets in more complex environments such as roads. In addition, millimeter-wave radar has another major advantage, which is that it can achieve non-line-of-sight detection, which is difficult for traditional sensors to achieve. Traditional visual sensors can only detect and identify targets within the direct line of sight, but millimeter-wave radar can detect targets within non-line-of-sight through electromagnetic wave reflection phenomena such as road surfaces, walls, and other obstacles. If millimeter-wave radar is used in combination with sensors such as cameras and lidar, it can also achieve target detection and identification within the entire range.

[0004] In the field of traditional technical methods, there is a large amount of research on using sensors at intersections to warn drivers of "ghosting" targets. However, this method requires the installation of a large number of sensors, which is difficult to reduce costs. At the same time, this method does not provide early warning of "ghosting" targets for every car, every intersection, and every scenario. Therefore, it is difficult to adapt to different environments and different types of "ghosting" problems by simply installing sensors at intersections. Summary of the Invention

[0005] To solve the above technical problems, the present invention proposes a non-line-of-sight ghost detection warning method based on vehicle-mounted millimeter-wave radar and visual sensor. The vehicle-mounted millimeter-wave radar and visual sensor are used to collect data, and the data is used for correlation filtering and time alignment. Finally, the targets within the line of sight and the non-line-of-sight targets are fused to mark the places where "ghost" targets may exist.

[0006] The technical solution adopted by the present invention is: a non-line-of-sight ghost detection warning method based on vehicle-mounted millimeter-wave radar and visual sensor. The "ghost detection" data acquisition terminal includes: a visual sensor and two millimeter-wave radars. The visual sensor is set on the top of the vehicle in front, and the two millimeter-wave radars are set at both ends of the vehicle's front bumper. The warning method includes the following steps:

[0007] S1. Acquire millimeter-wave radar data and process the acquired millimeter-wave radar data to obtain the position of the target within the detection area relative to the "ghost probe" data acquisition terminal; the detection area includes a line-of-sight area and a non-line-of-sight area;

[0008] S2, tracking the target in the detection area relative to the "ghost probe" data acquisition terminal according to the position obtained in step S1 to obtain a tracking point;

[0009] S3. Obtaining visual sensor data and processing the obtained visual sensor data to obtain the position of the target in the viewing range relative to the "ghost probe" data acquisition terminal;

[0010] S4, adjusting and synchronizing the position of the tracking point in step S2 and the target in the sight range in step S3 relative to the "ghost probe" data acquisition terminal;

[0011] S5. Associating the tracking points processed in step S4 with the positions of the targets in the line of sight area relative to the "ghost peeking" data acquisition terminal. When there is no pedestrian target in the visual sensor, and the millimeter-wave radar has confirmed that the target is detected, and the target detected by the millimeter-wave radar can be associated with the object that will block the pedestrian target in the visual sensor, the target detected by the millimeter-wave radar is marked as a "ghost peeking" target.

[0012] Beneficial effects of the present invention: The present invention provides a non-line-of-sight ghost detection warning method based on vehicle-mounted millimeter-wave radar and visual sensor, which can realize active detection of pedestrian targets within the line-of-sight and non-line-of-sight range on the vehicle, and can effectively implement warnings for pedestrian targets within the detection range of the system, and mark them on the driver's HUD (head-up display) to prompt that there is a risk of "ghosting" in the designated area. At the same time, the present invention adopts a multi-recognition framework, which can provide different platforms with a YOLO deep learning recognition framework with high selection accuracy but poor real-time performance, and also provides a HOG+SVM recognition framework with poor accuracy but can be used on any platform, and further adopts a millimeter-wave radar data association framework, which can not only remove the noise data of recognition errors through relevant associations, but also further detect areas where "ghosting" may exist through the framework; at the same time, in order to avoid the problem that the recognition area range is limited by the vehicle and the accuracy of single radar sensors and single visual sensors is insufficient, the present invention adds a multi-position, multi-posture calibration scheme, and the correction of its recognition results can be achieved by solving the transformation of its inverse posture matrix. Therefore, this method has a wider recognition range and is more stable, providing more options to adapt to different platforms. It can be directly applied to vehicles equipped with visual sensors and millimeter-wave radar sensors, and can also be directly applied to facilities that need to prevent "ghosting" accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the instrument in the present invention;

[0014] in, Figure 1 (a) is a visual sensor equipped with a pan-tilt system; Figure 1 (b) is a millimeter-wave radar sensor equipped with a single-axis servo gimbal; Figure 1 (c) with Figure 1 (d) is a schematic diagram of the vehicle-mounted sensor designed in the present invention.

[0015] Figure 2 It is a schematic diagram of the scene in the present invention;

[0016] in, Figure 2 (a) is a schematic diagram of a virtual scene of the present invention; Figure 2 (b) is a scenario designed to test the effectiveness of the algorithm designed by the present invention.

[0017] Figure 3 It is a schematic diagram of the data synchronization algorithm.

[0018] Figure 4 This is the processing flow of visual sensor data.

[0019] Figure 5 This is the processing flow of millimeter wave radar data.

[0020] Figure 6 This is the structural block diagram of millimeter wave radar tracking target;

[0021] Figure 7 It is a structural block diagram of data fusion, data correlation, and noise removal.

[0022] Figure 8 It is a data processing result diagram in the present invention;

[0023] in, Figure 8 (a) shows the detection results of pedestrians and vehicles in the direct viewing area of the present invention; Figure 8 (b) shows the position of the “pedestrian” obscured by the vehicle; Figure 8 (c) Figure 8 (d) shows the locations of surrounding objects from the vehicle’s top-down perspective. DETAILED DESCRIPTION

[0024] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0025] like Figure 1 (a) shows a visual sensor equipped with a gimbal. The position of the target relative to the sensor within the visual range can be obtained by using the attitude inverse calculation method, as shown in Figure 1 (b) shows a millimeter-wave radar sensor equipped with a single-axis servo gimbal. The single-axis gimbal can achieve different obstacle reflection effects, thereby adjusting the non-line-of-sight measurement effect. Figure 1 (c) with Figure 1 (d) is a schematic diagram of the vehicle-mounted sensors designed in the present invention; specifically, the visual sensor is installed on the top of the front section of the vehicle, and two millimeter-wave radar sensors are installed on both sides of the front bumper of the vehicle.

[0026] The present invention first acquires information about the surrounding environment through a millimeter-wave radar sensor and a visual sensor, synchronously processing the data from both sensors. The visual sensor is then calibrated and, if any posture changes occur, subjected to posture inversion processing. Frames are then inserted to obtain the target position and target tracking results output by the millimeter-wave radar sensor, while simultaneously obtaining target detection and classification results and target position based on the visual sensor data. Finally, based on the data association method proposed in the present invention, the two types of data are correlated, denoised, and displayed to achieve the effects of warning and locating "ghosting" accidents.

[0027] like Figure 2 (a) is a schematic diagram of a virtual scene of the present invention, in which a warning is given to the detection range (line of sight and non-line of sight range) by a "ghost head" warning system. Figure 2(a) shows the vehicle's direct view area, non-direct view area, "ghosting" target, and pedestrian target. The present invention sets the simulation scene of the "ghosting" accident as follows: Figure 2 (b) shows the scene, Figure 2 In the scenario shown in (b), the pedestrian target directly crosses the direct view area and the non-direct view area of the "ghost head" warning system.

[0028] The present invention can simultaneously detect and warn pedestrian targets in the direct line of sight area and "ghost peeking" targets in the non-direct line of sight area. The "ghost peeking" targets refer to human targets in the non-line of sight area and where accidents may occur.

[0029] In the present invention, the millimeter-wave radar sensor and the visual sensor are used as a whole as the "ghost probe" data acquisition end, and the center of the "ghost probe" data acquisition end is used as the coordinate origin, the direction along the front of the vehicle is used as the positive direction of the y-axis of the coordinate axis, and the direction along the right side of the vehicle is used as the positive direction of the x-axis of the coordinate axis.

[0030] The present invention requires the tester to walk horizontally at a distance of about 9.3 meters from the center of the "ghost head" data acquisition terminal in the y-axis direction, and at the same time pass by the car and the open space at a distance of 4.1 meters from the center of the "ghost head" data acquisition terminal in the y-axis direction. The 9.3-meter distance set in this scene is roughly the braking distance of a car at a speed of 40 km / h, which still leaves some reaction time. In this way, the tester will walk from the position (-3.3 meters, 9.3 meters) to the position (1.0 meter, 9.3 meters). The radar will be adjusted by the "pan-tilt platform" to a position with a pitch angle of 0° and a height of 13 cm. At the same time, the visual sensor will be controlled by the "pan-tilt platform" at a pitch angle of 0°, a yaw angle of 0°, and a roll angle of 0°, and a height of 100 cm, and the two sides will be kept the same in the other position parameters to conduct the experiment.

[0031] The present invention will Figure 2 Taking the actual test scenario in (b) as an example, the specific implementation steps are introduced:

[0032] Step 1: Radar signal processing, such as Figure 5 As shown, the specific processing process is:

[0033] Step 1-1: Radar data processing

[0034] Considering that the vehicle-mounted millimeter-wave radar is usually a MIMO antenna and transmits linear frequency modulation signals in a time-division alternating manner, assuming that the radar's carrier frequency is f0, the frequency modulation period is T, and the frequency modulation bandwidth is B, the frequency modulation slope is The initial phase is Ignoring the amplitude parameter and noise, the expression of the transmitted signal is:

[0035]

[0036] Assume that the initial distance between the target and the radar is R0, and the target moves at a constant speed relative to the radar, with a speed of v r , the Doppler frequency is f d =2v r / λ,λ=c / f0,c is the speed of light. The relationship between the distance of the target relative to the radar and time is: R(t)=R0-v r t, the delay of the received signal is:

[0037]

[0038] Where τ0=2R0 / c,k=f d / f0.

[0039] The received signal is:

[0040]

[0041] In the above formula, is random phase noise. The difference frequency signal obtained by mixing the transmit signal and the receive signal is:

[0042]

[0043]

[0044] Continuing to substitute τ(t) into the arrangement, we can get:

[0045]

[0046] Among them, f b0 ≈Sτ0-f d +(τ0+B / 2)k,S b =-2kS-k 2 S,

[0047] It can be seen that the intermediate frequency signal S IF Is f b0 is the center frequency, S b The linear frequency modulation signal of the frequency modulation slope. Combining the speed of the general actual target and the carrier frequency of the radar, we know that k<<1, and we get f b0 ≈Sτ0-f d , S b ≈0, and substituting it into the difference frequency signal expression, the target echo signal is obtained as follows:

[0048]

[0049] It can be seen from the above formula that the intermediate frequency signal S IFThe frequency information contains the distance and speed information of the target. The present invention is mainly aimed at human targets, and the Doppler frequency f generated by human targets during movement is d is smaller, so f d can be ignored, and the above formula can be rewritten as:

[0050]

[0051] By performing Fast Fourier Transform (FFT) on the above equation, the distance information of the target can be obtained, and the distance image of different time segments can be obtained.

[0052] At this time, the received range image contains a lot of clutter generated by stationary objects such as the road surface, so it is necessary to perform stationary target clutter suppression processing. The present invention adopts the mean cancellation algorithm to remove static information in the received signal. n (t l ) can be expressed as:

[0053]

[0054] Where L is the time length of the signal, t l represents the lth cycle;

[0055] In order to further improve the signal-to-noise ratio of non-line-of-sight targets, the present invention performs non-coherent accumulation on the range image, and the target signal after non-coherent accumulation is called Z p [n]∈C 1×N , and the lth period range image in each frame is represented by Y p [l]∈C 1×N , then the expression of non-coherent accumulation is:

[0056]

[0057] Wherein, C represents the complex number domain in mathematical notation, and N represents the length of the target signal.

[0058] Considering that reflective surfaces such as the road surface may introduce a large number of multipath signals, these echo signals may generate false targets. To eliminate these false targets, the present invention adopts the ordered statistical constant false alarm detection (OS-CFAR) method and simultaneously adopts the unit average constant false alarm detection (CA-CFAR) method to eliminate the angular false targets described in the next step.

[0059] Furthermore, the MVDR (Minimum Variance Distortionless Response) method is used to measure the angle of each distance unit in the current scene. The spatial spectrum expression at the azimuth angle θ is:

[0060]

[0061] Where a(θ) in the above formula represents the direction vector, and R represents the signal autocorrelation matrix. Select the angle unit with the largest value as the current estimated angle to obtain the azimuth angle θ of the candidate target. At this point, the following formula is used to obtain the two-dimensional plane information of the candidate target and calculate the plane position of the target, where It is the distance of the target relative to the "ghost probe" data collection end.

[0062]

[0063] The radar signal processing part of the present invention ends here, and the target distance can be directly obtained based on the range image And the angle obtained by trigonometric operation

[0064] Step 1-2: Radar target tracking, the specific tracking process is as follows Figure 6 shown.

[0065] At this time, the target candidate points obtained still have false targets caused by clutter and noise. Therefore, the present invention adopts a moving target tracking method to estimate the true target position by correlating the positioning results of multiple consecutive frames with Kalman filtering, and predict the target position when the measurement is missing. At the same time, the DBSCAN (density-based clustering method with noise) clustering method is used to remove noise points when establishing a new track, in order to improve the track reliability.

[0066] Specifically, the system variables are represented by X(k), the system process noise is defined as V(k), and the system input is u(k). Then the general form of the target state equation can be expressed as:

[0067] X(k+1)=FX(k)+Gu(k)+V(k)

[0068] Where F is the state transition matrix of the system, G is the input control term matrix, and V(k) is zero-mean, white Gaussian noise with covariance Q(k).

[0069] The measurement equation is an assumption of the radar measurement process. The measurement vector is defined as Z(k+1) and the measurement noise is defined as W(k+1). For a linear system, the measurement equation can be expressed as:

[0070] Z(k+1)=H(k+1)X(k+1)+W(k+1)

[0071] Where H(k+1) is the measurement matrix, W(k+1) is zero-mean, white Gaussian noise, and its covariance is R(k+1).

[0072] The state estimate at time k is The corresponding covariance is P(k|k). Then the prediction of the state and measurement at time k+1 is:

[0073]

[0074]

[0075] The prediction of the covariance is:

[0076] P(k+1|k)=F(k)P(k|k)F'(k)+Q(k)

[0077] The innovation and innovation covariance are:

[0078]

[0079] S(k+1)=H(k+1)P(k+1|k)H'(k+1)+R(k+1)

[0080] The gain of the Kalman filter is obtained as follows:

[0081] K(k+1)=P(k+1|k)H'(k+1)S -1 (k+1)

[0082] The state and covariance update equations are:

[0083]

[0084] P(k+1|k+1)=P(k+1|k)-K(k+1)S(k+1)K'(k+1)

[0085] The Kalman filter described above is only one part of the overall target tracking process. In the event of missed detections, the tracking filter uses the prediction results to update the trajectory.

[0086] In practice, multiple targets are detected and tracked in each frame. Therefore, it is necessary to associate the measurement results with the existing trajectory, identify the measurement results generated by the target being tracked, and then use them to update the trajectory.

[0087] In order to achieve data association, this system uses the nearest neighbor (NN) algorithm based on Mahalanobis distance. Represents the distance between a point and a distribution. The specific calculation formula is:

[0088]

[0089] The basic idea of the NN algorithm is to calculate the Mahalanobis distance and select the measurement point closest to the predicted position of the tracked target as the measurement associated with the track.

[0090] If a measurement during the association process fails to correlate with an existing track, it is considered a new target and its track needs to be initialized to establish a new trajectory for subsequent tracking. After observing for a certain period of time, tracks that meet certain conditions are output as real targets, while those that do not meet certain conditions are considered false targets.

[0091] Step 2: Visual data processing, such as Figure 4 As shown, the specific processing process is:

[0092] Step 2-1: Vision sensor calibration

[0093] To achieve line-of-sight target position estimation, it is necessary to achieve target angle estimation and position estimation separately. Among them, target angle estimation is relatively simple. The visual angle of the visual sensor can be calibrated in advance to achieve angle estimation of the target in the image.

[0094] After the target detection is completed, the pixel value P of the target bottom on the imaging plane can be obtained. bottom Then, after camera calibration, the focal length f of the camera and the pixel coordinate P of the optical center of the lens in the y direction in the pixel coordinate system can be solved. center If the camera installation height H is known, the distance Dis between the target and the camera can be obtained as:

[0095]

[0096] Among them, the focal length is (f x ,f y ), the pixel coordinates of the optical center are (c x ,c y ), since this system uses the y-axis direction data as the calculation basis, thus:

[0097]

[0098] In order to estimate the focal length f of the camera, it is necessary to calibrate the camera and estimate its intrinsic parameter matrix K.

[0099] Let oxyz be the camera coordinate system, and assume that the image point P in the real world, after being projected through the pinhole, is P' on the physical imaging plane o'-x'-y'-z'. Assume that the coordinates of P and P' are (X, Y, Z) and (X', Y', Z') respectively. Then, according to the triangle similarity relationship, we have:

[0100]

[0101] There are many photosensitive electronic components near the physical imaging plane, each of which represents a pixel of the output image. Therefore, it is assumed that the pixel coordinate system on the photosensitive plane is ouv, where the origin is located in the upper left corner of the image, the u axis is parallel to the x axis to the right, and the v axis is parallel to the y axis downward. Therefore, the difference between the pixel coordinate system and the physical imaging coordinate system is the origin translation (c x ,c y ) and scaling (α, β). Then the spatial coordinate transformation of point P' between the physical imaging plane and the photosensitive element can be expressed as:

[0102]

[0103] Substituting into the formula, we have:

[0104]

[0105] Expressing it in matrix form, we have:

[0106]

[0107] Thus, key parameters such as the focal length of the visual sensor can be obtained.

[0108] Step 2-2: Visual Object Detection

[0109] This paper will directly use the more accurate YOLO deep learning target detection framework to directly estimate the position ROI (x, y, w, h) of the target of interest. Similarly, based on the intrinsic parameter matrix of the above process, the focal length f can be approximately estimated as:

[0110]

[0111] Based on the target location ROI (x, y, w, h), the center of mass of the target can be further calculated using the following formula:

[0112]

[0113]

[0114]

[0115]

[0116] Among them, M is the geometric moment of the image, the subscript x, y of M is the relative coordinate of the image, M 00 =∑ x,y∈I x 0 y 0 I(x,y),M 10 =∑ x,y∈I x 1 y 0 I(x,y),M 01 =∑ x,y∈I x 0 y 1 I(x,y), I represents the grayscale value of the image data, are the left and right coordinates of the centroid of the annotated image. are the upper and lower coordinates of the centroid of the labeled image.

[0117] Furthermore, the target distance The calculation formula can be expressed as:

[0118]

[0119] h camera is the height of the camera relative to the ground, is the area of the visual sensor data where the target is located, is the height of the visual sensor data area where the target is located, Represents the centroid position of the target area, and the target angle The formula can be expressed as:

[0120]

[0121] That is, the minimum absolute value difference of the center of mass position of the target in the visual sensor on the calibration axis. Therefore, the information D of the visual sensor can be obtained by this step. v , and specify dynamic personnel information as ROI Dynamic (x, y, w, h), static obstacle information is ROI obstacle (x,y,w,h). Considering that the human information is usually blocked, the estimated It is not accurate, so the ROI of personnel information Dynamic (x,y,w,h), only estimates the angle information.

[0122] At this step, the visual sensor can directly calculate the distance of the target within the line of sight angle Vision sensor information D v Including distance and angles

[0123] Step 3: Data Processing

[0124] Step 3-1: Data Synchronization

[0125] The present invention is based on Figure 3 Data synchronization is performed in this way. In order to compare the data volume of millimeter wave radar and visual sensor, the data is collected in advance and then compared, and then interpolation synchronization is performed. Figure 3 The algorithm shown will not be executed unless the number of frames of millimeter wave radar data and visual sensor data is the same. Otherwise, synchronization will be performed based on the larger data volume.

[0126] Step 3-2: Data Adjustment

[0127] After the calibration of each sensor is completed, considering that each sensor will be partially fine-tuned to obtain different measurement ranges, the rotation matrix R and displacement matrix T of the millimeter-wave radar and optical sensor can be directly calculated by converting the current posture and the actual position:

[0128]

[0129]

[0130]

[0131] T x (x)=(x,0,0)

[0132] T y (y)=(0,y,0)

[0133] T z (z)=(0,0,z)

[0134] Thus, the offset position O of the current sensor can be calculated by the above formula and its inverse matrix. Assume that the j-th millimeter wave radar sensor MMW i With the jth visual sensor VI j The pitch angle, roll angle, and yaw angle differ from the original position by θ i , μ i Degrees and θ j , μ j degrees, and offset by x i ,y i ,z i Meter and x j ,y j ,z j m, then restore its posture by calculating its inverse matrix O* , as follows:

[0135]

[0136]

[0137] By restoring its posture, the deviation of the detected target relative to the original calibrated posture can be further calculated.

[0138] Step 3-3: Data association, the specific implementation process is as follows Figure 7 shown.

[0139] The present invention focuses on processing three types of data: newly added visual data New radar data And historical data D h , and set the associated parameters at the same time: associated tolerance Th a , time noise margin Th t , spatial distance tolerance Th r , and frame loss tolerance Th l .

[0140] according to Figure 7 The present invention firstly sets all data D that meet the initialization conditions. v 、D r All are initialized to the initial state of Kalman filter:

[0141] The state of the target is estimated to be

[0142] The covariance of the target is P(k|k);

[0143] And set the equations for state transfer and state update:

[0144] The state transition equation of the target is

[0145] The update equation of the target at time k+1 is: Where F' represents the transpose of F, Q(k) is the process noise variance matrix;

[0146] Next, update the parameters of each historical data. After obtaining the new information Z(k), use Z(k+1)=H(k+1)X(k+1)+W(k+1) to update, where H(k+1) is the measurement matrix, which is generally W(k+1) is zero-mean, white Gaussian noise with covariance R(k+1).

[0147] The innovation v(k+1) and innovation covariance S(k+1) are:

[0148]

[0149] S(k+1)=H(k+1)P(k+1|k)H′(k+1)+R(k+1)

[0150] Where H′ represents the transpose of H;

[0151] From the above data, the Kalman gain can be calculated:

[0152] K(k+1)=P(k+1|k)H'(k+1)S -1 (k+1)

[0153] The state and covariance update equations are:

[0154]

[0155] P(k+1|k+1)=P(k+1|k)-K(k+1)S(k+1)K'(k+1)

[0156] Where K' represents the transpose of K;

[0157] After the update is completed, traverse each new information and calculate the value according to the associated threshold Th a Calculate the newly added visual data D v 、Add radar data D r The difference is calculated based on the NN algorithm, and the distance difference Err of each data is calculated. r , if and only if the distance difference Err r Less than or equal to the distance tolerance Th r , number of associations Num a Increment by 1, otherwise skip this step. a Greater than the associated tolerance Th a When the data is added to the historical information cycle; in this embodiment, the distance tolerance Th r The value is 2.5 meters, and the associated tolerance Th a The value is 3 frames;

[0158] Similarly, for the remaining newly added visual data D that cannot be associated v 、Add radar data D r Continue traversing, join the temporary loop, and keep Th in the temporary loop t times, and calculate its number of occurrences Num t Differences in distance and angle Similarly, only when the number of occurrences Num t Greater than or equal to the time noise margin Th t , and the distance or angle difference Less than or equal to Th r, added to the historical information loop; in this embodiment, the time noise tolerance Th t The value is 3 frames;

[0159] The distance and angle differences calculated in this embodiment The distance difference value and the angle difference value are weighted and then added together. In this embodiment, the weight of the distance difference value is set to 0.8, and the weight of the angle difference value is set to 0.2.

[0160] Once new data is added to the historical information loop, each data in the historical loop information is traversed and the estimated position information of the new data is found. Or estimate angle information and a data in the historical cycle information The position information or angle information difference value Err r Less than or equal to Th r , use the new data to update the data And the number of dropped frames Num l Set to 0. Otherwise, create it as a new data node in the history information loop And the number of dropped frames Num l Set to 0; if a data in the historical information loop is not updated by the new data, the number of frame drops Num l Increment by 1.

[0161] By traversing the historical information loop, detect the number of frame drops Num l , when the number of frame loss is greater than the frame loss tolerance Th l , automatically remove the historical information cycle, so as to achieve the purpose of updating each data. In this embodiment, the frame loss tolerance Th l Set to 10 frames.

[0162] Step 4: "Ghost Heading" Warning

[0163] Based on the latest historical data D h The target position predicted by each "ghost peeking" warning system designed by the present invention is obtained. When the human body and vehicle data recognized by the visual sensor can be associated with the moving target and static target data obtained by the millimeter-wave radar, it is marked as a pedestrian target within the line of sight. Conversely, when there is no pedestrian target in the visual sensor and the millimeter-wave radar detects a moving target and can associate it with the vehicle target in the visual sensor, it is marked as a "ghost peeking" target in the non-line-of-sight area.

[0164] The effect of the present invention is further illustrated by the following experimental verification:

[0165] Experimental results:

[0166] Experimental scenario such as Figure 2As shown in (b), a pedestrian target behind the vehicle is detected simultaneously using millimeter-wave radar and camera sensors, and the parameters are consistent with the above steps.

[0167] According to the above method, the results are Figure 8 As shown, Figure 8 (a) shows that in the direct viewing area, the system can simultaneously identify vehicles and pedestrians, and mark the angle and distance of the vehicle relative to the visual sensor; in the case of the risk of "ghosting", Figure 8 (b) shows the results of non-line-of-sight detection, and uses a thick frame to mark the area where the driver needs to pay attention.

[0168] contrast Figure 8 (c) Figure 8 The result of (d) shows the position of surrounding objects from the vehicle's bird's-eye view. It can be found that the corresponding radar data detection results basically correspond to the results of the simulated driver's HUD. It can be found that the "ghost head" warning system designed by the present invention has the ability to perceive people within the line of sight and those outside the line of sight, and its accuracy is high and its performance is relatively stable.

[0169] Those skilled in the art will appreciate that the embodiments described herein are intended to aid the reader in understanding the principles of the present invention, and it should be understood that the scope of the present invention is not limited to such specific descriptions and embodiments. Various modifications and variations are readily apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims.

Claims

1. A non-line-of-sight ghost detection warning method based on vehicle-mounted millimeter-wave radar and visual sensor, characterized in that: The "ghost probe" data acquisition terminal includes: a visual sensor and two millimeter-wave radars. The visual sensor is set on the top of the vehicle in front, and the two millimeter-wave radars are set at both ends of the vehicle's front bumper. The early warning method includes the following steps: S1. Acquire and process millimeter-wave radar data to determine the distance and angle of targets within the detection area relative to the "ghost probe" data acquisition terminal; the targets within the detection area include pedestrian targets within the line-of-sight area and pedestrian targets within the non-line-of-sight area; S2. Track the target within the detection area obtained in step S1 relative to the "ghost probe" data acquisition terminal to obtain a tracking point; S3. Obtaining and processing visual sensor data to obtain the distance and angle of targets within the viewing area relative to the "ghost probe" data acquisition terminal; targets within the viewing area include pedestrian targets and vehicle targets within the viewing area; S4, adjusting and synchronizing the distance and angle between the tracking point in step S2 and the target in the sight range in step S3 relative to the "ghost probe" data acquisition terminal; the data adjustment is specifically as follows: By converting the current posture and the real position, the rotation matrix R and displacement matrix T of the millimeter wave radar and the visual sensor are calculated. Assuming that the i-th millimeter wave radar MMW i With the jth visual sensor VI j The pitch angle, roll angle, and yaw angle differ from the original position by θ i , μ i Degrees and θ j , μ j degrees, and offset by x i ,y i ,z i Meter and x j ,y j ,z j m, calculate x i ,y i ,z i with x j ,y j ,z j Their respective displacement matrices T, calculate θ i , μ i and θ j , μ j The inverse matrix of each rotation matrix R; Based on x i ,y i ,x i with x j ,y j ,z j Their respective displacement matrices T,θ i , μ i and θ j , μ j The inverse matrix of each rotation matrix R is used to restore the posture of the millimeter wave radar and visual sensor; S5. Correlate the tracking point processed in step S4 with the distance of the target in the sight area relative to the "ghost probe" data acquisition terminal. Successful correlation includes the following two situations: If the pedestrian target within the line of sight detected by the millimeter-wave radar is successfully associated with the pedestrian target within the line of sight detected by the visual sensor, the target will be marked as a pedestrian target within the line of sight; If a pedestrian target in the non-line-of-sight area detected by the millimeter-wave radar is successfully associated with a vehicle target in the line-of-sight area detected by the visual sensor, the pedestrian target in the non-line-of-sight area will be marked as a "ghost" target.

2. The non-line-of-sight ghost detection warning method based on vehicle-mounted millimeter-wave radar and visual sensor according to claim 1 is characterized in that: The process of determining whether the association is successful in step S5 is as follows: Based on the nearest neighbor algorithm, the distance difference Err between the target position detected by the millimeter wave radar and the target detected by the visual sensor is calculated. r , when continuous Th a The distance difference Err obtained by the frame r Are less than or equal to the tolerance Th r , it is considered that the target detected by the millimeter-wave radar is successfully associated with the target detected by the visual sensor.

3. The non-line-of-sight ghost detection warning method based on vehicle-mounted millimeter-wave radar and visual sensor according to claim 2 is characterized in that: Step S5 also includes: after the association is successful, putting the target detected by the millimeter wave radar and the target detected by the visual sensor into historical loop information.

4. The non-line-of-sight ghost detection warning method based on vehicle-mounted millimeter-wave radar and visual sensor according to claim 3 is characterized in that: Step S5 also includes: if the association is not successful, the target detected by the millimeter wave radar and the target detected by the visual sensor are added to a temporary loop, and the position and angle differences between the target detected by the millimeter wave radar and the target detected by the visual sensor are calculated according to the nearest neighbor algorithm. like Satisfy continuous Th t The frames are less than or equal to the tolerance Th r , then the targets detected by the millimeter-wave radar and the targets detected by the visual sensor in the temporary cycle are added to the historical cycle information; Th t Represents the temporal noise margin.

5. The non-line-of-sight ghost detection warning method based on vehicle-mounted millimeter-wave radar and visual sensor according to claim 4 is characterized in that: The temporary loop will temporarily store Th t Targets detected by sub-millimeter wave radar and targets detected by visual sensors.

6. The non-line-of-sight ghost detection warning method based on vehicle-mounted millimeter-wave radar and visual sensor according to claim 5 is characterized in that: Step S5 also includes: updating the historical cycle information: For each data added with historical cycle information, each corresponds to a number of frame drops Num l ; When new data is added to the historical cycle information, the existing data in the historical cycle is updated and matched. For the data that successfully matches the update, the new data is used to update it, and the number of frame drops Num is set. l Set to 0, for data that failed to update the match, the number of frame drops Num l Increment by 1; If the new data does not successfully match any data update in the historical cycle information, the new data is treated as a new data in the historical cycle information, and the number of frame drops of the current data Num l Set to 0; When the number of frame drops of a certain data in the historical cycle information is Num l Greater than the frame loss tolerance Th l , the data will be deleted from the historical cycle information.

7. The non-line-of-sight ghost detection warning method based on vehicle-mounted millimeter-wave radar and visual sensor according to claim 6 is characterized in that: The judgment process for successful update matching is as follows: Based on the nearest neighbor algorithm, calculate the distance or angle difference between the new data and the existing data in the historical cycle information. If there is a difference value corresponding to a certain data Continuous Th a The frames are less than or equal to the tolerance Th r , then the update match is considered successful.

8. The non-line-of-sight ghost detection warning method based on vehicle-mounted millimeter-wave radar and visual sensor according to claim 7 is characterized in that: Distance and angle difference The calculation method is as follows: calculate the distance difference value and the angle difference value respectively, and then add the weighted values of the distance difference value and the angle difference value respectively to obtain the distance difference value and the angle difference value. The sum of the weights of the distance difference value and the angle difference value is 1.

Images