A pedestrian detection method under cloud and fog conditions

Abstract

The present invention discloses a pedestrian detection method under cloud and fog conditions, and the steps are as follows: preprocess the target pedestrian information under cloud and fog conditions collected in real time by an in-vehicle millimeter-wave radar; preprocess the point cloud data under cloud and fog conditions collected by an in-vehicle lidar; synchronize the preprocessed data in terms of time and space, and then perform fusion using a multiple Kalman filtering algorithm. The present invention fuses a lidar and a millimeter-wave radar. The millimeter-wave radar has more stable detection performance, a longer operating range, strong penetration ability, and the characteristics of all-weather and all-time. After fusion, it can improve the pedestrian target detection rate, reduce the false detection rate, and greatly improve driving safety.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving perception, and particularly relates to a method for detecting pedestrians ahead in a cloud and fog condition. Background Art

[0002] According to statistics, the main form of road traffic accidents is collision accidents. Such accidents not only occur between vehicles, but also often occur between vehicles and pedestrians. When colliding with a pedestrian, compared with the passengers in the vehicle, the pedestrian has no safety devices such as seat belts and airbags. In the event of a vehicle-pedestrian collision accident, not only is there a property loss, but serious personal injuries are often caused, and the accident result is more serious. Therefore, based on the traditional passive safety technology, more manpower, material resources, etc. need to be invested in researching the vehicle's active safety system to avoid it before the accident occurs, which will make the development of intelligent assisted driving faster.

[0003] A millimeter-wave radar is an active sensor that determines the distance of an object by emitting millimeter waves. The resolution and accuracy of the millimeter-wave radar are relatively low, and it is prone to missed detection and false detection due to background clutter and the limitations of the sensor itself, making it difficult to meet the requirements of pedestrian detection. Therefore, it is necessary to simultaneously use multiple sensors for information collection and correct the final result according to the information collected by each. Fusing other sensors with the millimeter-wave radar can effectively improve the vehicle's perception ability. Fusing lidar with the millimeter-wave radar can not only obtain the contour information of obstacles but also get accurate point cloud data, overcoming the problems that a vision camera cannot obtain accurate depth information and the point cloud of the millimeter-wave radar is too sparse and interfered by noise.

[0004] At present, certain achievements have been made in the research on pedestrian detection. For example, in the Chinese invention patent application with the application number CN202010741321.2 and the title "Vehicle and Pedestrian Detection Method Based on On-vehicle Infrared Thermal Imager in Complex Scenarios", an infrared image dataset is constructed and divided into training samples and test samples; the infrared image training samples are augmented; two 3×3 convolutional layers are added in parallel above and below the 103rd layer of the YOLOv3 network feature map to work in parallel with the 3×3 convolutional layer and 1×1 convolutional layer after the 103rd layer of the TOLOv3 network feature map to form a new TOLOv3 network; the new TOLOv3 network is trained using the infrared image training samples; and the infrared images in the test sample set are detected using the trained new YOLOv3 network model. This method has a large amount of calculation and is cumbersome; in the Chinese invention patent application with the application number CN201810027242.8 and the title "Automobile Driving Environment Detection Method Based on Millimeter-wave Radar", information on target objects in the surrounding environment where the vehicle is currently driving is collected using a millimeter-wave radar, and calculations are performed with the downloaded static and dynamic targets for real-time positioning and the production of a high-precision map to expand road condition information. However, due to the complex and changeable road environment where pedestrians are located, and the low visibility under cloud and fog conditions, the danger of pedestrians increases, which also greatly increases the difficulty of real-time pedestrian detection and judgment. When only a single sensor is used, limited by its characteristics and accuracy, the detection results have a high degree of uncertainty.

[0005] In summary, the infrared imaging pedestrian detection method using deep learning has a large amount of calculation, low resolution, is easily interfered by the environmental temperature and items carried by pedestrians, the millimeter-wave radar has low resolution and accuracy, large atmospheric transmission loss, and cannot identify pedestrians; the lidar has sparse point clouds under cloud and fog conditions, which is prone to missed detection; the pedestrian postures are variable and the ability to reflect electromagnetic waves is weak, so the safety of pedestrians on the road cannot be guaranteed. Due to the complex and changeable road environment where pedestrians are located, the difficulty of real-time pedestrian detection and judgment is increased, and the detection results also have a high degree of uncertainty. Summary of the Invention

[0006] Aiming at the deficiencies of the above-mentioned existing technologies, the purpose of the present invention is to provide a pedestrian detection method under cloud and fog conditions to solve the problems in the existing technologies, such as the variable postures of pedestrian targets, weak ability to reflect electromagnetic waves, and easy missed detection of pedestrian detection based on a single sensor; the present invention fuses lidar and millimeter-wave radar. The millimeter-wave radar has more stable detection performance, a longer working distance, strong penetration ability, and has the characteristics of all-weather and all-time. After fusion, it can improve the pedestrian target detection rate, reduce the false detection rate, and greatly improve driving safety.

[0007] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0008] A pedestrian detection method under cloud and fog conditions of the present invention comprises the following steps:

[0009] (1) Preprocess the target pedestrian information collected by the vehicle-mounted millimeter-wave radar in real time under cloud and fog conditions. Convert the millimeter-wave radar data in the time domain and frequency domain to remove invalid targets, stationary targets, and non-dangerous targets, determine the valid targets, and then use a filtering algorithm to clean the data of the valid targets to obtain smooth data. The target pedestrian information includes the distance and angle information of the pedestrian relative to the vehicle itself.

[0010] (2) Preprocess the point cloud data collected by the vehicle-mounted lidar under cloud and fog conditions through secondary echo, coordinate conversion, region segmentation, clustering, and feature extraction and classification. The point cloud data includes the three-dimensional coordinates in the lidar coordinate system.

[0011] (3) Synchronize the data preprocessed in steps (1) and (2) in time and space, and then use the multiple Kalman filtering algorithm for fusion to obtain the target information of the pedestrian to be detected.

[0012] Further, the time domain and frequency domain conversion method in step (1) adopts the fast Fourier transform, and the specific steps are as follows:

[0013] (11) Divide the data point sequence obtained by the millimeter-wave radar into two sequences of even and odd numbers, that is:

[0014]

[0015]

[0016] where x1(·) is the even sequence; x2(·) is the odd sequence; N is the total number of data points.

[0017] (12) Perform a discrete Fourier transform on the data sequence x(n) to obtain:

[0018]

[0019] where X(κ) is the data sequence after the discrete Fourier transform, and it is defined as e is the natural constant, i is the imaginary unit, and it has symmetry and periodicity; n is a natural number, 2r represents an even number, and 2r + 1 represents an odd number.

[0020] Combined with the symmetry and periodicity of the parameter W N m that is, W N 2κr = W N / 2 κr According to equations (1)-(3), rewrite equation (3) as:

[0021]

[0022] Considering that there must be an even number M among the integers in the neighborhood of N / 2, the discrete Fourier transform of point M is further decomposed to obtain:

[0023]

[0024]

[0025] In the formula, DFT[·] represents the discrete Fourier transform.

[0026] Furthermore, the preprocessing of the point cloud data by the lidar in the case of clouds and fog in step (2) includes the following steps:

[0027] (21) Using the second echo to weaken the influence of clouds and fog;

[0028] (22) Converting the polar coordinate system of the lidar to the Cartesian coordinate system:

[0029]

[0030] In the formula, D0 is the distance between the effective pedestrian target in front and the vehicle itself in the polar coordinate; D cor is the distance correction coefficient, representing the distance deviation; D r is the actual distance from the effective pedestrian target to the origin in the polar coordinate system; θ is the horizontal correction angle, representing the angle between the laser beam and the y-axis in the xoy projection plane; β is the vertical correction angle, representing the angle between the laser beam and the xoy plane; V0 is the vertical offset, representing the offset of the laser emission point to the lidar coordinate origin on the xoz projection plane; P x is the position in the x direction in the Cartesian coordinate system; P y is the position in the y direction in the Cartesian coordinate system; P z is the position in the z direction in the Cartesian coordinate system; D xy is the distance from the projection point of the end point of the laser beam on the horizontal plane to the coordinate origin in the Cartesian coordinate system; H0 is the horizontal offset, representing the offset of the laser emission point to the lidar coordinate origin on the xoy projection plane;

[0031] (23) Performing regional segmentation on the obtained point cloud data: After the coordinate system calibration in step (22) is completed, for the three points selected from the point cloud data each time, a simple linear model is used for plane model estimation, and the equation is:

[0032] ax + by + cz + d = 0 (8)

[0033] Wherein, a is the fitting accuracy, and b, c, and d are constants; points with a distance from the fitting plane less than or equal to a are all inliers (ground points). After using the inliers to re-estimate the plane model parameters, the non-ground point cloud can be screened out by removing the inliers and retaining the outliers, thus completing the ground segmentation;

[0034] (24) Cluster the point cloud data after region segmentation;

[0035] (25) Use a support vector machine to extract and classify point cloud features.

[0036] Furthermore, the clustering of the point cloud in step (24) includes the following steps:

[0037] (241) Define the measure M ι of each sample point p ι :

[0038]

[0039] Wherein, Ψ is the measure coefficient; the point p ι with the largest measure value is used as the initial clustering center; when calculating the next clustering center, the clustering center obtained for the first time is deleted to avoid oscillation;

[0040] (242) Calculate the next clustering center:

[0041]

[0042] The closer the sample point is to the initial clustering center, the smaller the measure value M ι . Select the sample point with the highest measure value as the next clustering center;

[0043] (243) After repeating step (242) j times, the required number of clustering categories σ and the initial clustering center are obtained:

[0044] (244) Select m = 2 as the fuzzy index of iteration, and the number of iteration times L = 0;

[0045] (245) Calculate the matrix U(L) according to the following formula:

[0046]

[0047] Wherein, ρ is the row number of the matrix; q is the column number of the matrix;

[0048] (246) Modify the membership matrix U(L), and after modification, the membership degree of the sample point p ι in the point cloud data to the s-th class is:

[0049] μ sw = μ sw +(1 - α)μtw (12)

[0050] In the formula, α is the correction coefficient; μ sw = max 1≤w≤σ μ ρw ; μ tw = max 1≤w≤σ,w≠s μ ρw ;

[0051] After correction, the membership degree of the sample point p in the point cloud data to the t-th class is ι The membership degree of the sample point p in the point cloud data to the remaining classes remains unchanged;

[0052] μ tw = αμ tw (13)

[0053] Among them, for the sample point p in the point cloud data ι The membership degrees to the remaining classes remain unchanged;

[0054] (247) Calculate the next clustering center C using the corrected membership degrees (L+1) :

[0055]

[0056] In the formula, c υ is the clustering center, is the membership degree function of the w-th sample in the point cloud data to the υ-th class;

[0057] (248) Modify the clustering center. For each center point c in the clustering center C λ , λ = 1, 2,..., n, calculate the distance d(c λ to each sample point p in the sample set ι (c λ , p ι ), where ι = 1, 2,..., N, and reselect a new clustering center to reduce the sensitivity to isolated points. The new iterative clustering center point at this time is the sample point p corresponding to the minimum distance d(c λ , p ι ) to replace the existing center point c ι ; When and only when the maximum number of loops or the given threshold ε2 is reached, that is λ , exit the loop, otherwise return to the above step (245), update the relevant parameters, at this time L = L + 1, and continue the calculation. When, exit the loop, otherwise return to the above step (245), update the relevant parameters, at this time L = L + 1, and continue the calculation.

[0058] Furthermore, the specific steps of the step (3) include the following steps:

[0059] (31)Obtain the position and speed of the pedestrian, establish a pedestrian motion model, and take the longitudinal and lateral components of the pedestrian's motion speed and position as the state variables of the pedestrian motion model. Then the pedestrian motion model is expressed as:

[0060]

[0061] In the formula, p′ x is the predicted value of the pedestrian's position in the x direction; p x is the measured value of the pedestrian's position in the x direction; v′ x is the predicted value of the pedestrian's speed in the x direction; v x is the measured value of the pedestrian's speed in the x direction; v px is the position noise of the pedestrian in the x direction; v py is the position noise of the pedestrian in the y direction; v vx is the speed noise of the pedestrian in the x direction; v y ' is the predicted value of the pedestrian in the y direction; v y is the measured value of the pedestrian's speed in the y direction; v vy is the speed noise of the pedestrian in the y direction; p′ y is the predicted value of the pedestrian's position in the y direction; p y is the measured value of the pedestrian's position in the y direction;

[0062] (32)Estimate the data of the lidar through Kalman filtering. The measurement model of the lidar is:

[0063]

[0064] In the formula, z li is the measurement value vector; H is the lidar measurement matrix; w li is the lidar measurement noise; R li is the lidar measurement noise covariance matrix;

[0065] Lidar parameter update model: The parameter update process is to update the covariance matrix of the real-time state x of the pedestrian and output the real-time estimated value of the lidar, which is expressed as:

[0066]

[0067] In the formula, y li is the lidar measurement error; z li is the lidar measurement value; H is the lidar measurement matrix; x(k) is the state at time k; S li is the lidar parameter update process matrix; K li is the lidar estimation gain; x lio is the estimated value, that is, the output value of the lidar Kalman filter; P p(k) is the predicted covariance matrix of the state x at time k; P li (k) is the actual covariance matrix of the state x at time k updated by the lidar parameter update module; I is the identity matrix;

[0068] (33) Estimate the data of the millimeter-wave radar through the extended Kalman filter. The measurement model of the millimeter-wave radar is expressed as:

[0069]

[0070] where w ra is the measurement noise of the millimeter-wave radar; R ra is the millimeter-wave radar measurement noise covariance matrix, and h(x) is the mapping from the system state space to the measurement space;

[0071] Perform a first-order Taylor expansion on the nonlinear function in the measurement model of the millimeter-wave radar to linearize the nonlinear part, that is:

[0072]

[0073] The linearized measurement model of the millimeter-wave radar is expressed as:

[0074]

[0075] where is the linearized h(x) function, and its Jacobian matrix is expressed as:

[0076]

[0077] where H j is the Jacobian matrix of the linearized h(x) function;

[0078] The measurement error covariance matrix of the millimeter-wave radar is expressed as:

[0079]

[0080] The matrix R ra is the uncertainty of the position measurement result received at the millimeter-wave radar;

[0081] Millimeter-wave radar parameter update model: The parameter update process is to update the covariance matrix of the real-time state x of the pedestrian and output the real-time estimated value of the millimeter-wave radar, expressed as:

[0082]

[0083] where y ra is the millimeter-wave radar measurement error; h -1 is the inverse function of h(x); z rais the measurement value of the millimeter-wave radar; S ra is the millimeter-wave radar parameter update process matrix; K ra the millimeter-wave radar estimation gain; x rao is the estimated value, i.e., the output value of the millimeter-wave radar Kalman filter; P ra (k) is the actual covariance matrix of the state x at time k updated by the millimeter-wave radar parameter update module;

[0084] (34) Fuses the data obtained by the millimeter-wave radar and the lidar through multiple Kalman filters.

[0085] Furthermore, the step (34) specifically includes the following four cases:

[0086] (341) For the case of dual-sensor synchronous update, after performing the parameter update of the first sensor and obtaining the preliminary estimated value, this value is used as the predicted value in the parameter update process of the second sensor for estimation. The final result obtained is the sensor information fusion result, expressed as:

[0087]

[0088] In the formula, x(k) is the state at time k; F is the update equation, I is the identity matrix, T is the sampling time; v is the system noise; P p (k) is the predicted covariance matrix of the state x at time k; P li (k - 1) is the actual covariance matrix of the state x at time k - 1 updated by the lidar parameter update module; Q is the covariance matrix; y li is the lidar measurement error; z li is the lidar measurement value; H is the lidar measurement matrix; S li is the lidar parameter update process matrix; R li the measurement value noise covariance matrix of the lidar; K li the lidar estimation gain; x lio is the estimated value, i.e., the output value of the lidar Kalman filter; P ra (k - 1) is the actual covariance matrix of the state x at time k - 1 updated by the millimeter-wave radar parameter update module; y ra is the millimeter-wave radar measurement error; z ra is the linearized millimeter-wave radar measurement model; S ra is the millimeter-wave radar parameter update process matrix; H j is the Jacobian matrix of the linearized h(x) function; x rao is the estimated value, i.e., the output value of the millimeter-wave radar Kalman filter; K ra is the millimeter-wave radar estimation gain;

[0089] (342) For the fusion in the case of asynchronous update of dual sensors, when updating the parameters, only the respective parameter update equations of different sensors need to be called to obtain the final output result; when the first sensor is updated, the predicted value of the measurement value of the second sensor is calculated based on the measurement value of the second sensor at the previous moment as the measurement value of the second sensor at the current moment, which is expressed as the following two cases;

[0090] Lidar is updated, millimeter-wave radar is not updated:

[0091]

[0092] In the formula, is the predicted value of the measurement value of the millimeter-wave radar; is the predicted value of the measurement value of the lidar;

[0093] Millimeter-wave radar is updated, lidar is not updated:

[0094]

[0095] (343) For the fusion in the case of synchronous non-update of dual sensors, at this time, the data lost simultaneously by the two sensors due to the influence of cloud and fog factors. At this time, the dual-sensor measurement value synchronous prediction fusion method is used for output, which is expressed as:

[0096]

[0097] According to the final output, the target information of the pedestrian to be detected can be obtained.

[0098] Advantages of the present invention:

[0099] Compared with the traditional fusion of lidar and millimeter-wave radar, the present invention suppresses cloud and fog during preprocessing and sets different fusion algorithms according to different synchronous situations of sensor information during fusion, solving the problems of overly sparse data and noise interference under cloud and fog conditions; improving the robustness of the vehicle perception system in bad weather and solving the problems of missed detection by a single sensor and incomplete information acquisition under cloud and fog conditions. Description of the Drawings

[0100] Figure 1 is the front pedestrian detection flow chart under cloud and fog conditions;

[0101] Figure 2 is the millimeter-wave radar data processing flow chart;

[0102] Figure 3 is the lidar preprocessing flow chart;

[0103] Figure 4It is a clustering flow chart. Specific implementation manners

[0104] For the convenience of those skilled in the art to understand, the present invention will be further described below in conjunction with embodiments and the accompanying drawings. The content mentioned in the implementation manners does not limit the present invention.

[0105] Refer to Figure 1 As shown, a pedestrian detection method under cloud and fog conditions of the present invention is as follows:

[0106] (1) Preprocess the target pedestrian information collected by the vehicle-mounted millimeter-wave radar in real time under cloud and fog conditions, perform conversions in the time domain and frequency domain on the millimeter-wave radar data to remove invalid targets, stationary targets, and non-dangerous targets, determine valid targets, and then use a filtering algorithm to clean the data of the valid targets to obtain smooth data. The target pedestrian information includes the distance and angle information of the pedestrian relative to the vehicle itself;

[0107] Refer to Figure 2 As shown, the time domain and frequency domain conversion method in step (1) adopts the fast Fourier transform, and the specific steps are as follows:

[0108] (11) Divide the data point sequence obtained by the millimeter-wave radar into two sequences of odd and even numbers, that is:

[0109]

[0110]

[0111] In the formula, x1(·) is the even sequence; x2(·) is the odd sequence; N is the total number of data points;

[0112] (12) Perform discrete Fourier transform on the data sequence x(n) to obtain:

[0113]

[0114] In the formula, X(κ) is the data sequence after discrete Fourier transform, and it is defined as

[0115] e is the natural constant, i is the imaginary unit, which has symmetry and periodicity; n is a natural number, 2r represents an even number, and 2r + 1 represents an odd number;

[0116] Combined with the symmetry and periodicity of the parameter W N m Namely W N 2κr = W N / 2 κr According to formulas (1)-(3), rewrite formula (3) as:

[0117]

[0118] Considering that there must be an even integer M in the neighborhood of N / 2, the discrete Fourier transform of point M is further decomposed to obtain:

[0119]

[0120] Wherein, DFT[·] is the discrete Fourier transform.

[0121] (2) Preprocess the point cloud data collected by the vehicle-mounted lidar under cloud and fog conditions through secondary echo, coordinate conversion, region segmentation, clustering, and feature extraction and classification. The point cloud data includes three-dimensional coordinates in the lidar coordinate system;

[0122] Refer to Figure 3 As shown, the preprocessing of the point cloud data by the lidar under cloud and fog conditions in step (2) includes the following steps:

[0123] (21) Use secondary echo to weaken the influence of cloud and fog;

[0124] (22) Convert the polar coordinate system of the lidar to the Cartesian coordinate system:

[0125]

[0126] Wherein, D0 is the distance between the effective pedestrian target in front and the host vehicle in the polar coordinate; D cor is the distance correction coefficient, indicating the distance deviation; D r is the actual distance from the effective pedestrian target to the origin in the polar coordinate system; θ is the horizontal correction angle, indicating the angle between the laser beam and the y-axis in the xoy projection plane; β is the vertical correction angle, indicating the angle between the laser beam and the xoy plane; V0 is the vertical offset, indicating the offset of the laser emission point to the lidar coordinate origin on the xoz projection plane; P x is the position in the x direction in the Cartesian coordinate system; P y is the position in the y direction in the Cartesian coordinate system; P z is the position in the z direction in the Cartesian coordinate system; D xy is the distance from the projection point of the end point of the laser beam on the horizontal plane to the coordinate origin in the Cartesian coordinate system; H0 is the horizontal offset, indicating the offset of the laser emission point to the lidar coordinate origin on the xoy projection plane;

[0127] (23) Perform region segmentation on the obtained point cloud data: After completing the coordinate calibration in step (22), for the three points selected from the point cloud data each time, a simple linear model is used for plane model estimation, and the equation is:

[0128] ax + by + cz + d = 0 (8)

[0129] Wherein, a is the fitting accuracy, and b, c, and d are constants; points with a distance from the fitting plane less than or equal to a are all inliers (ground points). After using the inliers to re - estimate the plane model parameters, the non - ground point cloud can be screened out by removing the inliers and retaining the outliers, thus completing the ground segmentation;

[0130] (24) Cluster the point cloud data after region segmentation;

[0131] (25) Use a support vector machine for feature extraction and classification of the point cloud.

[0132] Specifically, referring to Figure 4 as shown, the clustering of the point cloud in step (24) includes the following steps:

[0133] (241) Define the measure M ι of each sample point p ι :

[0134]

[0135] Wherein, Ψ is the measure coefficient; the point p ι with the maximum measure value is used as the initial clustering center; when calculating the next clustering center, the first - obtained clustering center is deleted to avoid oscillation;

[0136] (242) Calculate the next clustering center:

[0137]

[0138] The closer the sample point is to the initial clustering center, the smaller the measure value M ι is. Select the sample point with the highest measure value as the next clustering center;

[0139] (243) After repeating step (242) j times, the required number of clustering categories σ and the initial clustering center are obtained:

[0140] (244) Select m = 2 as the fuzzy exponent of iteration, and the number of iteration times L = 0;

[0141] (245) Calculate the matrix U(L) according to the following formula:

[0142]

[0143] Wherein, ρ is the row number of the matrix; q is the column number of the matrix;

[0144] (246) Modify the membership matrix U(L), and after modification, the sample point p in the point cloud dataι The membership degree to the sth class is as follows:

[0145] μ sw = μ sw +(1 - α)μ tw (12)

[0146] In the formula, α is the correction coefficient; μ sw = max 1≤w≤σ μ ρw ; μ tw = max 1≤w≤σ,w≠s μ ρw ;

[0147] After correction, the membership degree of the sample point p in the point cloud data to the tth class is ι as follows

[0148] μ tw = αμ tw (13)

[0149] Among them, the membership degrees of the sample point p in the point cloud data to the remaining classes remain unchanged; ι (247) Calculate the next clustering center C using the corrected membership degrees

[0150] : (L+1)

[0151]

[0152] In the formula, c υ is the clustering center, is the membership function of the wth sample in the point cloud data to the υth class;

[0153] (248) Correct the clustering center. For each center point c in the clustering center C λ , λ = 1, 2, =, n, calculate the distance d(c λ to each sample point p in the sample set ι (c λ , p ι ), where ι = 1, 2,..., N, and re - select a new clustering center to reduce the sensitivity to isolated points. The new iterative clustering center point at this time is the sample point p corresponding to the minimum distance d(c λ , p ι ) to replace the existing center point c ι ; When and only when the maximum number of loops or the given threshold ε2 is reached, that is λ , exit the loop, otherwise return to the step (245), update the relevant parameters, at this time L = L + 1, and continue the calculation.

[0154] ​​(3) Synchronize the preprocessed data in steps (1) and (2) in terms of time and space, and then use the multiple Kalman filtering algorithm for fusion to obtain the target information of the pedestrians to be detected; specifically, it includes the following steps:

[0155] (31) Obtain the position and speed of the pedestrian, establish a pedestrian motion model, and take the longitudinal and lateral components of the pedestrian's motion speed and position as the state variables of the pedestrian motion model. Then, the pedestrian motion model is expressed as:

[0156]

[0157] In the formula, p′ x is the predicted value of the pedestrian's position in the x direction; p x is the measured value of the pedestrian's position in the x direction; v′ x is the predicted value of the pedestrian's speed in the x direction; v x is the measured value of the pedestrian's speed in the x direction; v px is the position noise of the pedestrian in the x direction; v py is the position noise of the pedestrian in the y direction; v vx is the speed noise of the pedestrian in the x direction; v y ' is the predicted value of the pedestrian in the y direction; v y is the measured value of the pedestrian's speed in the y direction; v vy is the speed noise of the pedestrian in the y direction; p′ y is the predicted value of the pedestrian's position in the y direction; p y is the measured value of the pedestrian's position in the y direction;

[0158] (32) Estimate the data of the lidar through Kalman filtering. The measurement model of the lidar is:

[0159]

[0160] In the formula, z li is the measurement value vector; H is the lidar measurement matrix; w li is the lidar measurement noise; R li is the lidar measurement noise covariance matrix;

[0161] Lidar parameter update model: The parameter update process is to update the covariance matrix of the real-time state x of the pedestrian and output the real-time estimated value of the lidar, which is expressed as:

[0162]

[0163] In the formula, y li is the lidar measurement error; z li is the lidar measurement value; H is the lidar measurement matrix; x(k) is the state at time k; S liis the process matrix for lidar parameter update; K li Lidar estimation gain; x lio is the estimated value, i.e., the output value of the lidar Kalman filter; P p (k) is the predicted covariance matrix of the state x at time k; P li (k) is the actual covariance matrix of the state x at time k updated by the lidar parameter update module; I is the identity matrix;

[0164] (33) Estimate the data of the millimeter-wave radar through the extended Kalman filter. The measurement model of the millimeter-wave radar is expressed as:

[0165]

[0166] where, w ra is the measurement noise of the millimeter-wave radar; R ra is the millimeter-wave radar measurement noise covariance matrix, and h(x) is the mapping from the system state space to the measurement space;

[0167] Perform a first-order Taylor expansion on the non-linear function in the measurement model of the millimeter-wave radar to linearize the non-linear part, i.e.:

[0168]

[0169] The linearized measurement model of the millimeter-wave radar is expressed as:

[0170]

[0171] where, is the linearized h(x) function, and its Jacobian matrix is expressed as:

[0172]

[0173] where, H j is the Jacobian matrix of the linearized h(x) function;

[0174] The measurement error covariance matrix of the millimeter-wave radar is expressed as:

[0175]

[0176] Matrix R ra is the uncertainty of the position measurement result received at the millimeter-wave radar;

[0177] Millimeter-wave radar parameter update model: The parameter update process is to update the covariance matrix of the real-time state x of the pedestrian and output the real-time estimated value of the millimeter-wave radar, which is expressed as:

[0178]

[0179] where y ra is the measurement error of the millimeter-wave radar; h -1 is the inverse function of h(x); z ra is the measurement value of the millimeter-wave radar; S ra is the process matrix for updating the parameters of the millimeter-wave radar; K ra is the estimation gain of the millimeter-wave radar; x rao is the estimated value, i.e., the output value of the Kalman filter of the millimeter-wave radar; P ra (k) is the actual covariance matrix of the state x at time k updated by the millimeter-wave radar parameter update module;

[0180] (34) Fuses the data obtained by the millimeter-wave radar and the lidar through multiple Kalman filters.

[0181] Specifically, the step (34) specifically includes the following four cases:

[0182] (341) For the case of synchronous update of the two sensors, after updating the parameters of the first sensor and obtaining a preliminary estimated value, this value is used as the predicted value in the parameter update process of the second sensor for estimation, and the finally obtained result is the sensor information fusion result, expressed as:

[0183]

[0184] where x(k) is the state at time k; F is the update equation, I is the identity matrix, T is the sampling time; v is the system noise; P p (k) is the predicted covariance matrix of the state x at time k; P li (k - 1) is the actual covariance matrix of the state x at time k - 1 updated by the lidar parameter update module; Q is the covariance matrix; y li is the measurement error of the lidar; z li is the measurement value of the lidar; H is the lidar measurement matrix; S li is the process matrix for updating the parameters of the lidar; R li is the measurement value noise covariance matrix of the lidar; K li is the estimation gain of the lidar; x lio is the estimated value, i.e., the output value of the Kalman filter of the lidar; P ra (k - 1) is the actual covariance matrix of the state x at time k - 1 updated by the millimeter-wave radar parameter update module; y ra is the measurement error of the millimeter-wave radar; z ra is the linearized measurement model of the millimeter-wave radar; S ra is the process matrix for updating the parameters of the millimeter-wave radar; H jis the Jacobian matrix of the h(x) function after linearization; x rao is the estimated value, i.e., the output value of the millimeter-wave radar Kalman filter; K ra is the estimated gain of the millimeter-wave radar;

[0185] (342) For the fusion in the case of asynchronous update of the dual sensors (millimeter-wave radar and lidar), when updating the parameters, only the respective parameter update equations of different sensors need to be called to obtain the final output result; when the first sensor (millimeter-wave radar or lidar) is updated, the predicted value of the measurement value of the second sensor is calculated based on the measurement value of the second sensor at the previous moment as the measurement value of the second sensor at the current moment, which is expressed as the following two cases;

[0186] Lidar is updated and millimeter-wave radar is not updated:

[0187]

[0188] In the formula, is the predicted value of the measurement value of the millimeter-wave radar; is the predicted value of the measurement value of the lidar;

[0189] Millimeter-wave radar is updated and lidar is not updated:

[0190]

[0191] (343) For the fusion in the case of synchronous non-update of the dual sensors, at this time, the data lost simultaneously by the two sensors due to the influence of cloud and fog factors. At this time, the synchronous prediction fusion method of the measurement values of the dual sensors is used for output, which is expressed as:

[0192]

[0193] According to the final output, the target information of the pedestrian to be detected can be obtained.

[0194] The specific application ways of the present invention are numerous. The above description is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements can still be made, and these improvements should also be regarded as the protection scope of the present invention.

Claims

1. A pedestrian detection method under cloud and fog conditions, characterized in that, The steps are as follows: (1) Preprocess the target pedestrian information collected by the vehicle-mounted millimeter-wave radar in cloud and fog conditions. Convert the millimeter-wave radar data in the time domain and frequency domain to remove invalid targets, stationary targets, and non-dangerous targets, determine the valid targets, and then use a filtering algorithm to clean the data of the valid targets to obtain smooth data; (2) Preprocess the point cloud data collected by the vehicle-mounted lidar in cloud and fog conditions through secondary echo, coordinate transformation, region segmentation, clustering, and feature extraction and classification; (3) Synchronize the data preprocessed in steps (1) and (2) in time and space, and then use the multiple Kalman filtering algorithm for fusion to obtain the target information of the pedestrian to be detected; Step (3) specifically includes the following steps: (31) Obtain the position and speed of the pedestrian, establish a pedestrian motion model, and take the longitudinal and lateral components of the pedestrian's motion speed and position as the state variables of the pedestrian motion model. Then the pedestrian motion model is expressed as: where p x ′ is the predicted value of the pedestrian's position in the x - direction; p x is the measured value of the pedestrian's position in the x - direction; v x ′ is the predicted value of the pedestrian's velocity in the x - direction; v x is the measured value of the pedestrian's velocity in the x - direction; v px is the position noise of the pedestrian in the x - direction; v py is the position noise of the pedestrian in the y - direction; v vx is the velocity noise of the pedestrian in the x - direction; v y ' is the predicted value of the pedestrian in the y - direction; v y is the measured value of the pedestrian's velocity in the y - direction; v vy is the velocity noise of the pedestrian in the y - direction; p′ y is the predicted value of the pedestrian's position in the y - direction; p y is the measured value of the pedestrian's position in the y - direction; (32) Estimate the lidar data through Kalman filtering. The measurement model of the lidar is: where z li is the measurement value vector; H is the lidar measurement matrix; w li is the lidar measurement noise; R li is the lidar measurement noise covariance matrix; Lidar parameter update model: The parameter update process is to update the covariance matrix of the pedestrian's real-time state x and output the real-time estimate value of the lidar, which is expressed as: where y li is the measurement error of the lidar; z li is the lidar measurement value; H is the lidar measurement matrix; x(k) is the state at time k; S li is the lidar parameter update process matrix; K li is the lidar estimation gain; x lio is the estimated value, i.e., the output value of the lidar Kalman filter. P p (k) is the predicted covariance matrix of the state x at time k; P li (k) is the actual covariance matrix of the state x at time k updated by the lidar parameter update module; I is the identity matrix; (33) Estimate the millimeter-wave radar data through the extended Kalman filtering. The measurement model of the millimeter-wave radar is expressed as: where w ra is the measurement noise of the millimeter-wave radar; R ra is the covariance matrix of the millimeter-wave radar measurement noise, and h(x) is the mapping from the system state space to the measurement space; Perform a first-order Taylor expansion on the nonlinear function in the measurement model of the millimeter-wave radar to linearize the nonlinear part, that is: The linearized measurement model of the millimeter-wave radar is expressed as: wherein, is the linearized h(x) function, and its Jacobian matrix is expressed as: where H j is the Jacobian matrix of the h(x) function after linearization; The measurement error covariance matrix of the millimeter-wave radar is expressed as: Matrix R ra is the uncertainty of the position measurement result received at the millimeter-wave radar; Millimeter-wave radar parameter update model: The parameter update process is to update the covariance matrix of the pedestrian's real-time state x and output the real-time estimate value of the millimeter-wave radar, which is expressed as: where y ra is the millimeter-wave radar measurement error; h -1 is the inverse function of h(x); z ra is the millimeter-wave radar measurement value; S ra is the millimeter-wave radar parameter update process matrix; K ra is the millimeter-wave radar estimation gain; x rao is the estimated value, i.e., the output value of the millimeter-wave radar Kalman filter; P ra (k) is the actual covariance matrix of the state x at time k updated by the millimeter-wave radar parameter update module; (34) Fuse the data obtained by the millimeter-wave radar and the lidar through multiple Kalman filtering; Step (34) specifically includes the following four situations: (341) For the case of synchronous update of the two sensors, perform parameter update of the first sensor. After obtaining the preliminary estimate value, use this value as the predicted value in the parameter update process of the second sensor for estimation. The finally obtained result is the sensor information fusion result, which is expressed as: Where x(k) is the state at time k; F is the update equation, I is the identity matrix, T is the sampling time; v is the system noise; P p (k) is the predicted covariance matrix of the state x at time k; P li (k - 1) is the actual covariance matrix of the state x at time k - 1 updated by the lidar parameter update module; Q is the covariance matrix; y li is the lidar measurement error; z li is the lidar measurement value; H is the lidar measurement matrix; S li is the lidar parameter update process matrix; R li is the measurement value noise covariance matrix of the lidar; K li is the lidar estimation gain; x lio is the estimated value, i.e., the output value of the lidar Kalman filter; P ra (k - 1) is the actual covariance matrix of the state x at time k - 1 updated by the millimeter-wave radar parameter update module; y ra is the millimeter-wave radar measurement error; z ra is the linearized millimeter-wave radar measurement model; S ra is the millimeter-wave radar parameter update process matrix; H j is the Jacobian matrix of the linearized h(x) function; x rao is the estimated value, i.e., the output value of the millimeter-wave radar Kalman filter; K ra is the millimeter-wave radar estimation gain; (342) For the fusion in the case of asynchronous update of the two sensors, only need to call the respective parameter update equations of different sensors to obtain the final output result when updating the parameters; when the first sensor is updated, calculate the predicted value of the measurement value of the second sensor based on the measurement value of the second sensor at the previous moment as the measurement value of the second sensor at the current moment, which is expressed as the following two situations; Lidar is updated, millimeter-wave radar is not updated: Wherein, is the predicted value of the millimeter-wave radar measurement; is the predicted value of the lidar measurement; Millimeter-wave radar is updated, lidar is not updated: (343) For the fusion in the case of synchronous non-update of the two sensors, at this time, the data of both sensors are lost due to the influence of cloud and fog factors. At this time, use the synchronous prediction fusion method of the two-sensor measurement values for output, which is expressed as: According to the final output, the target information of the pedestrian to be detected can be obtained.

2. The pedestrian detection method under cloud and fog conditions according to claim 1, wherein, The time-domain and frequency-domain conversion method in step (1) uses the fast Fourier transform, and the specific steps are as follows: (11) Divide the data point sequence obtained by the millimeter-wave radar into two sequences of odd and even numbers, that is: x2(r) = x(2r + 1) (15) In the formula, x1(·) is the even sequence; x2(·) is the odd sequence; N is the total number of data points; (12) Perform discrete Fourier transform on the data sequence x(n) to obtain: where X(κ) is the data sequence after discrete Fourier transform, defined e is the natural constant, i is the imaginary unit, which has symmetry and periodicity; n is a natural number, 2r represents an even number, and 2r + 1 represents an odd number; Combined with the symmetry and periodicity of parameter W N m , that is, W N 2κr = W N / 2 κr , rewrite Equation (3) according to Equations (1)-(3) as: Considering that there must be an even number M among the integers in the neighborhood of N / 2, the discrete Fourier transform of M points is further decomposed to obtain: In the formula, DFT[·] represents the discrete Fourier transform.

3. The pedestrian detection method under cloud and fog conditions according to claim 2, wherein, The preprocessing of the point cloud data by the lidar in the case of clouds and fog in step (2) includes the following steps: (21) Use the second echo to weaken the influence of clouds and fog; (22) Convert the polar coordinate system of the lidar to the Cartesian coordinate system: Where, D0 is the distance between the effective pedestrian target in front and the host vehicle in polar coordinates; D cor is the distance correction coefficient, representing the distance deviation; D r is the actual distance from the effective pedestrian target to the origin in the polar coordinate system; θ is the horizontal correction angle, representing the angle between the laser beam and the y-axis in the xoy projection plane; β is the vertical correction angle, representing the angle between the laser beam and the xoy plane; V0 is the vertical offset, representing the offset of the laser emission point to the origin of the lidar coordinates on the xoz projection plane; P x is the position in the x direction in the Cartesian coordinate system; P y is the position in the y direction in the Cartesian coordinate system; P z is the position in the z direction in the Cartesian coordinate system; D xy is the distance from the projection point of the end point of the laser beam on the horizontal plane to the origin in the Cartesian coordinate system; H0 is the horizontal offset, representing the offset of the laser emission point to the origin of the lidar coordinates on the xoy projection plane; (23) Perform region segmentation on the obtained point cloud data: After the coordinate calibration in step (22) is completed, for the three points selected from the point cloud data each time, a simple linear model is used for plane model estimation, and the equation is: ax + by + cz + d = 0 (21) In the formula, a is the fitting accuracy, and b, c, and d are constants; the points with a distance from the fitting plane less than or equal to a are all inliers. After using the inliers to re-estimate the plane model parameters, the inliers are removed and the outliers are retained to screen out the non-ground point cloud and complete the ground segmentation; (24) Cluster the point cloud data after region segmentation; (25) Use a support vector machine to extract and classify the point cloud features.

4. The pedestrian detection method under cloud and fog conditions according to claim 3, characterized in that For the clustering of the point cloud in step (24), it includes the following steps: (241) Define the measure M of each sample point p ι ι :​ where Ψ is the measurement coefficient; the point p with the largest measurement value ι is used as the initial clustering center; when calculating the next clustering center, the clustering center obtained for the first time is deleted to avoid oscillation phenomena; (242) Calculate the next clustering center: The closer the sample point is to the initial clustering center, the measure value M ι is smaller, and the sample point with the highest measure value is selected as the next clustering center; (243) After repeating step (242) j times, the required number of clustering categories σ and the initial clustering center are obtained: (244) Select m = 2 as the fuzzy index of iteration, and the number of iteration times L = 0; (245) Calculate the matrix U(L) according to the following formula: In the formula, ρ is the row number of the matrix; q is the column number of the matrix; (246) Membership degree correction matrix U(L), after correction, the sample point p in the point cloud data is obtained ι The membership degree for the s-th class is: μ sw = μ sw + (1 - α)μ tw (25) where α is the correction coefficient; μ sw = max 1≤w≤σ μ ρw ; μ tw = max 1≤w≤σ,w≠s μ ρw ; After correction, the sample point p in the point cloud data is obtained ι The membership degree to the t-th class is μ tw = αμ tw (26) Among them, the sample point p in the point cloud data ι The membership degrees to the rest of each category remain unchanged; (247) Calculate the next clustering center C using the corrected membership degree (L+1) : where c υ is the clustering center, is the membership function of the w-th sample in the point cloud data for the υ-th class; (248) Modify the cluster center, for each center point c in the cluster center C λ ,λ=1,2,…,n, calculate the center point c λ To each sample point p in the sample set ι The distance d(c λ ,p ι ), where ι = 1, 2, ..., N, and reselect new cluster centers to reduce the sensitivity of isolated points. At this time, the new iterative cluster center point is the corresponding distance d (c λ ,p ι )The smallest sample point p ι To replace the existing center point c λ ; If and only if the maximum number of cycles or the given threshold ε2 is reached, that is When L=L+1, exit the loop, otherwise return to step (245) to update relevant parameters, and then L=L+1, and continue calculating.

Images