Multi-target fusion detection method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-target fusion detection method and device, electronic equipment and a storage medium. The multi-target fusion detection method comprises the following steps: acquiring collection data of a sensor of a target vehicle; performing preprocessing on the acquired data by adopting filtering and plane fitting to obtain detection data; associating the millimeter-wave radar detection data with the laser radar detection data by adopting a Hungary matching algorithm based on mahalanobis distance to obtain associated data; and carrying out multi-target detection on the associated data by adopting unscented Kalman filtering and covariance intersection to obtain a multi-target detection result. The beneficial effects of the invention are that the fusion multi-target detection method based on the unscented Kalman filtering and covariance intersection combines the laser radar and the millimeter wave radar, and achieves the high-precision, multi-target and rapid real-time detection in a severe environment.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to a multi-object fusion detection method, apparatus, electronic device, and storage medium. Background Art

[0002] At present, the environmental perception part of autonomous vehicles mainly relies on a variety of sensor devices such as cameras, millimeter-wave radars, and lidar. A single sensor device cannot meet the requirements of high timeliness and high robustness of autonomous vehicles, and it has been proven in practice that autonomous vehicles using a single sensor do not have the ability to completely and reliably perceive the external environment. Multi-sensor data fusion can integrate the data of each sensor in an autonomous vehicle for fusion, increasing the reliability and robustness of the system, expanding the observation range in both space and time, and enhancing the credibility of the data and the identification ability of the perception system. However, most of the existing multi-object detections based on the fusion of cameras and lidar use deep learning algorithms for fusion, which are affected by weather factors (such as bad weather like rain, snow, fog, and haze), the computing power of the algorithm, and computer hardware devices. For multi-object detection based on the fusion of cameras and millimeter-wave radars, although the all-weather available millimeter-wave radar is used for detection, the detection accuracy of the millimeter-wave radar is lower than that of the lidar and it is not sensitive enough to detect pedestrians. Summary of the Invention

[0003] The main purpose of the embodiments of the present invention is to propose a multi-object fusion detection method, apparatus, electronic device, and storage medium, which improve the accuracy of target detection of autonomous vehicles.

[0004] One aspect of the present invention provides a multi-object fusion detection method, including:

[0005] According to a detection request, obtain the acquisition data of the sensors of the target vehicle, where the sensors include a millimeter-wave radar and a lidar sensor, and the acquisition data includes millimeter-wave radar acquisition data and lidar acquisition data;

[0006] Perform preprocessing on the acquisition data by filtering and plane fitting to obtain detection data, where the detection data is used to represent point cloud data including detection identifiers and detection targets, and the detection data includes millimeter-wave radar detection data and lidar detection data;

[0007] Associate the millimeter-wave radar detection data and the lidar detection data using the Hungarian matching algorithm based on the Mahalanobis distance to obtain associated data;

[0008] Perform multi-object detection on the associated data using unscented Kalman filtering and covariance intersection to obtain a multi-object detection result.

[0009] According to the multi-object fusion detection method, before the step of acquiring the acquisition data of the sensors of the target vehicle according to the detection request, the following steps are further included:

[0010] Determine the kinematic model of the target vehicle by using a constant turn rate and speed model;

[0011] According to the kinematic model, taking the center of the rear axle of the target vehicle as the reference point, determine the translation matrix of the acquisition sensor relative to the vehicle coordinate system, and determine the rotation matrix of the acquisition sensor relative to the vehicle coordinate system. Through the translation matrix and the rotation matrix, the space of the sensor and the vehicle coordinate system is unified. The vehicle coordinate system takes the center of the rear axle as the origin, the forward direction of the origin as the x-axis, the direction perpendicular to the vehicle body to the left from the origin as the y-axis, and the direction perpendicular to the vehicle body upward from the origin as the z-axis;

[0012] Perform time synchronization on sensors with different scanning frequencies at a unified sampling frequency. The time synchronization includes marking the scanning frames and synchronizing the timestamps of the marked frames with the closest scanning time;

[0013] Determine the first motion state quantity of the target vehicle at the current moment according to the kinematic model, and perform state transition calculation on the motion state of the target vehicle according to the first motion state quantity to obtain the second motion state quantity.

[0014] According to the multi-object fusion detection method, the method further includes:

[0015] When calculating the motion state of the target vehicle, calculate the second motion state quantity of the target vehicle by adding process noise, where the process noise includes speed noise and angular velocity noise.

[0016] According to the multi-object fusion detection method, the preprocessing of the acquisition is performed by filtering and plane fitting to obtain the detection data, including:

[0017] Perform statistical filtering on the point cloud information in the millimeter-wave radar acquisition data, including calculating the average distance between any point in the point cloud information and its adjacent points, comparing the adjacent points with a preset distance, and removing the adjacent points not within the preset distance range;

[0018] Use the RANSAC plane fitting algorithm to separate the target vehicle from the ground for the millimeter-wave radar acquisition data and the lidar acquisition data after filtering;

[0019] Cluster the separated acquisition data by using the Euclidean clustering algorithm and based on distance partitioning to generate the detection boxes of the detection targets.

[0020] According to the multi-target fusion detection method described above, the RANSAC plane fitting algorithm is used to separate the target vehicle and the ground from the millimeter-wave radar acquisition data and the lidar acquisition data after filtering, including:

[0021] Taking the point cloud after statistical filtering as the original seed point set, and selecting a subset including three points from the original point set;

[0022] Calculating the ground model parameters based on the ground model constructed by the subset and using the equivalent coefficient method;

[0023] Testing the ground model parameters with the remaining point cloud in the original seed point set except the subset, updating the ground model parameters according to the test results, and repeating the ground model parameters until the ground model parameters meet the requirements, where the number of repetitions does not exceed the iteration threshold.

[0024] According to the multi-target fusion detection method described above, the Hungarian matching algorithm based on Mahalanobis distance is used to associate the millimeter-wave radar detection data and the lidar detection data to obtain associated data, including:

[0025] Determining the distance relationship between any detection target in the lidar sensor data and any target in the millimeter-wave radar sensor data through the Mahalanobis distance matrix;

[0026] Using the optimal assignment problem to analyze the correlation of the elements with assignments in the Mahalanobis distance matrix, including determining the objective function and constraint conditions according to whether any element in the Mahalanobis distance matrix is assigned and the minimum assigned Mahalanobis distance;

[0027] Performing correlation analysis on the elements of the Mahalanobis distance matrix that meet the constraint conditions, including calculating the correlation quality. When the correlation quality is greater than the set threshold, associating the two detection target data as the same detection target;

[0028] Using the UKF algorithm to generate local estimates for target tracking and then performing weighted fusion on the associated detection target data to obtain associated data.

[0029] According to the multi-target fusion detection method described above, using unscented Kalman filtering and covariance intersection to perform multi-target detection on the associated data to obtain multi-target detection results, including:

[0030] Initializing the initial state of the detection target and the initial value of the error covariance matrix, where the error covariance matrix includes velocity noise and angular velocity noise;

[0031] Perform state prediction on the detected target according to the kinematic model and the error covariance matrix, including using the first Sigma point as a sampling point to obtain a first Sigma point set, and performing prediction through the first Sigma point set to obtain a state prediction value;

[0032] Perform unscented transformation on the state prediction value to obtain a second Sigma point set, process the second Sigma point set using the millimeter-wave radar observation equation to obtain predicted observation values, the average value of measurement predictions, the covariance matrix of measurement values, and the cross-covariance matrix of measurement and prediction;

[0033] Calculate the Kalman gain according to the average value of measurement predictions, the covariance matrix of measurement values, and the cross-covariance matrix of measurement and prediction to obtain the state prediction estimate and the state covariance;

[0034] Use the covariance intersection algorithm to fuse the state prediction estimate and the state covariance to obtain the detection result of the detected target. Among them, when fusing, according to the optimality of the mean square error, weight factors are attached to the lidar sensor data and the millimeter-wave radar sensor data, and the weight factors are determined by the diagonal terms of the sub-optimal method.

[0035] Another aspect of the embodiments of the present invention provides a detection device for fusing multiple targets, including:

[0036] A first module for obtaining the acquisition data of the sensors of the target vehicle according to the detection request, where the sensors include a millimeter-wave radar and a lidar sensor, and the acquisition data includes millimeter-wave radar acquisition data and lidar acquisition data;

[0037] A second module for performing preprocessing on the acquisition data by filtering and plane fitting to obtain detection data, where the detection data is used to characterize the point cloud data including detection identifiers and detected targets, and the detection data includes millimeter-wave radar detection data and lidar detection data;

[0038] A third module for associating the millimeter-wave radar detection data and the lidar detection data using the Hungarian matching algorithm based on the Mahalanobis distance to obtain associated data;

[0039] A fourth module for performing multi-target detection on the associated data using unscented Kalman filtering and covariance intersection to obtain a multi-target detection result.

[0040] Another aspect of the embodiments of the present invention provides an electronic device, including a processor and a memory;

[0041] The memory is used to store programs;

[0042] The processor executes the program to implement the method described above.

[0043] An embodiment of the present invention also discloses a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method described above.

[0044] The beneficial effects of the present invention are as follows: A fusion multi-target detection method based on unscented Kalman filter and covariance intersection combines the advantages of lidar and millimeter-wave radar. It uses millimeter-wave radar and lidar to collect data, calculates the position similarity of the measurement data of the two sensors, and establishes a corresponding Mahalanobis distance matrix for multiple targets collected by the two sensors. The Hungarian algorithm is used for target matching to achieve data association. Finally, in a post-fusion manner, the unscented Kalman algorithm is used to perform local estimation on the targets detected by lidar and millimeter-wave radar, and then the covariance intersection algorithm is used to fuse the obtained local estimations to generate a fused state estimation, so as to achieve high-precision, multi-target, and fast real-time detection in harsh environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present invention will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, where:

[0046] Figure 1 is a schematic diagram of the detection process of fusing multiple targets according to an embodiment of the present invention.

[0047] Figure 2 is a schematic diagram of the generation process of the motion scenario and coordinate system of the target vehicle according to an embodiment of the present invention.

[0048] Figure 3 is a schematic diagram of the vehicle motion model according to an embodiment of the present invention.

[0049] Figure 4 is a schematic diagram of the sampling frequencies and time synchronization of lidar and millimeter-wave radar according to an embodiment of the present invention.

[0050] Figure 5 is a schematic diagram of the preprocessing process of the data collected by the sensor according to an embodiment of the present invention.

[0051] Figure 6 is a schematic diagram of the separation process of the RANSAC plane fitting algorithm according to an embodiment of the present invention.

[0052] Figure 7 is a schematic diagram of the Euclidean clustering process according to an embodiment of the present invention.

[0053] Figure 8Schematic diagram of the data association process of the Hungarian matching algorithm based on Mahalanobis distance according to an embodiment of the present invention.

[0054] Figure 9 Schematic diagram of the process of multi-target detection using unscented Kalman filter and covariance intersection according to an embodiment of the present invention.

[0055] Figure 10 Schematic diagram of the detection device for fusing multi-targets according to an embodiment of the present invention. Detailed implementation manners

[0056] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions from beginning to end. In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only for the convenience of the description of the present invention, and they have no specific meaning by themselves. Therefore, "module", "component" or "unit" can be used interchangeably. "First", "second", etc. are only used to distinguish technical features and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of the indicated technical features or implicitly indicating the sequence relationship of the indicated technical features. In the subsequent description of the present invention, the consecutive numbering of the method steps is for the convenience of review and understanding. Combining the overall technical solution of the present invention and the logical relationship between each step, adjusting the implementation order between steps will not affect the technical effect achieved by the technical solution of the present invention. The embodiments described below with reference to the drawings are exemplary and are only used to explain the present invention and cannot be understood as a limitation of the present invention.

[0057] Refer to Figure 1 , where Figure 1 is the schematic diagram of the detection process for fusing multi-targets according to an embodiment of the present invention. It includes but is not limited to steps S200 to S500:

[0058] S200, according to the detection request, obtain the acquisition data of the sensors of the target vehicle. The sensors include millimeter-wave radar and lidar sensors, and the acquisition data includes millimeter-wave radar acquisition data and lidar acquisition data.

[0059] In some embodiments, before collecting the sensor data of the target vehicle, it further includes step S100: constructing the motion scenario and coordinate system of the target vehicle;

[0060] In some embodiments, refer to Figure 2 the schematic diagram of the generation process of the motion scenario and coordinate system of the target vehicle shown, which includes but is not limited to steps S110 to S140:

[0061] S110, use the constant turn rate and speed model to determine the kinematic model of the target vehicle;

[0062] S120. According to the kinematic model, taking the center of the rear axle of the target vehicle as the reference point, determine the translation matrix of the acquisition sensor relative to the vehicle coordinate system and the rotation matrix of the acquisition sensor relative to the vehicle coordinate system. Unify the space of the sensor and the vehicle coordinate system through the translation matrix and the rotation matrix. The vehicle coordinate system takes the center of the rear axle as the origin, the forward direction of the origin as the x-axis, the direction perpendicular to the body and to the left from the origin as the y-axis, and the direction perpendicular to the vehicle body and upward from the origin as the z-axis;

[0063] S130. Synchronize the time of sensors with different scanning frequencies at the same sampling frequency. The time synchronization includes marking the scanning frames and synchronizing the timestamps of the marked frames with the closest scanning time;

[0064] S140. Determine the first motion state quantity of the target vehicle at the current moment according to the kinematic model, and perform state transition calculation on the motion state of the target vehicle according to the first motion state quantity to obtain the second motion state quantity.

[0065] In some embodiments, a kinematic model of an autonomous vehicle is established based on the Constant Turn Rate and Velocity (CTRV) model, and the space of the lidar and millimeter-wave radar is unified by integrating the rotation and translation matrices. At the same time, time unification is performed according to the different sampling frequencies of the two sensors. Taking the position of the center of the rear axle of the vehicle body as the coordinate origin O, the forward direction of the origin pointing to the vehicle as the positive direction of the x-axis, the direction perpendicular to the body and to the left from the origin as the positive direction of the y-axis, and the direction perpendicular to the vehicle body and upward from the origin as the positive direction of the z-axis to define the vehicle coordinate system OXYZ; The lidar is installed on the roof of the vehicle, and the lidar coordinate system is defined as O l X l Y l Z l The detection of the vehicle's surrounding environment information is relative to the lidar coordinate system. The lidar coordinate system takes the center of the lidar rotation axis as the coordinate origin O l ,O l X l Taking the forward direction of the vehicle as the positive direction, O l Y l Taking the direction perpendicular to the body and to the left as the positive direction, O l Z l Taking the direction perpendicular to O l X l and O l Y lThe plane formed is vertically upward as the positive direction. The point cloud data measured by the lidar is saved in the spherical coordinate system. Therefore, the lidar first needs to convert the point cloud coordinates into the local Cartesian coordinate system of the lidar. According to the elevation angle ω, azimuth angle α, and the measured distance r, the (x, y, z) in the local Cartesian coordinate system can be calculated as follows:

[0066] x = r * cos(ω) * sinα

[0067] y = r * cos(ω) * cosα

[0068] z = r * sin(ω)

[0069] The millimeter-wave radar is installed around the vehicle. The long-range millimeter-wave radar is installed at the front and rear of the vehicle, and the short-range millimeter-wave radar is installed on the left and right of the vehicle. Define the millimeter-wave radar coordinate system as O r1 X r1 Y r1 Z r1 With the millimeter-wave radar signal transmitter as the origin O r1 X r1 Taking the vehicle forward direction as the positive direction, Y r1 Taking the perpendicular to the vehicle body to the left as the positive direction, Z r1 Taking the plane formed by perpendicular to O r1 X r1 and O r1 Y r1 and vertically upward as the positive direction. When the millimeter-wave radar measures the target, the position of the target is usually placed in a two-dimensional space, and its measurement coordinate system is the polar coordinate system which are respectively represented as distance, deflection angle, and radial velocity. In addition, the deflection angle refers to the angle between the line connecting the target and the origin of the coordinate system and the Y r axis. This angle will change positively and negatively as the position of the target changes. When the target is in the positive direction of the coordinate system X r axis, the angle is positive, otherwise it is negative. The following formula can be used to calculate the position of the measured value in the Cartesian coordinate system:

[0070]

[0071]

[0072]

[0073] In some embodiments, refer to Figure 3The schematic diagram of the vehicle motion model shown is, to conform to the vehicle operation state in the actual situation, the motion model of the embodiments of the present invention is the CTRV model. The CTRV assumes that the target object moves straight forward, and can also move at a fixed turning rate and a constant speed. During the motion process of the autonomous vehicle at time k, the motion state quantity can be expressed as x k =[x, y, s, θ, ω] T , where x and y represent the vehicle position information, s represents the vehicle speed, θ represents the vehicle yaw angle, and ω represents the vehicle angular velocity. After differentiating the motion state quantity, the differential equation of the state quantity can be obtained as That is, the change of the vehicle state from time k to time k + 1 can be obtained as:

[0074]

[0075] In some embodiments, when calculating the motion state of the target vehicle, the second motion state quantity of the target vehicle is calculated by adding process noise, where the process noise includes speed noise and angular velocity noise

[0076] Among them, considering the influence of the external environment and the sensor's own noise, that is, the process noise V k , is represented by a two-dimensional vector to represent the speed noise and the angular velocity noise. Adding the noise to the state transition equation, when the time interval Δt = t k+1 -t k , the state transition equation including noise can be obtained as:

[0077]

[0078] Among them, is the longitudinal acceleration noise, with a mean of 0 and a variance of Gaussian white noise; is the angular acceleration noise, with a mean of 0 and a variance of Gaussian white noise. The covariance matrix of the state noise

[0079] In some embodiments, all the data collected by the sensors in the embodiments of the present invention are referenced to the center of the rear axle of the vehicle. Each sensor has a translation matrix and a rotation matrix with respect to the vehicle coordinate system and the position of the center of the rear axle. Translation matrix:

[0080]

[0081] The above formula represents the translation amounts of each coordinate axis of the sensor relative to the center of the rear axle.

[0082] In some embodiments, if the target tracked by lidar is converted into the vehicle coordinate system, the sensor coordinate axes also need to be rotated by a certain angle. R1(α) represents a rotation of angle α about the x-axis, represents a rotation of angle R3(ε) represents a rotation of angle ε about the z-axis, that is, the rotation matrix is as follows:

[0083]

[0084]

[0085]

[0086] where the target position detected in the millimeter-wave radar sensor coordinate system O r1 X r1 Y r1 Z r1 is T r1 , and the included angles with the vehicle coordinate system OXYZ are respectively The translation amount is T, then the coordinates of the target detected by the millimeter-wave radar in the vehicle coordinate system OXYZ are:

[0087]

[0088] The target information of other sensors is converted into the vehicle coordinate system OXYZ by the same method.

[0089] In some embodiments, since the targets detected by in-vehicle sensors are often traffic participants and static obstacles, such as: motorcycles, bicycles, pedestrians, vehicles, tree shrubs, road traffic signs, etc., these detected targets often have no movement in the z-axis direction or have a very small velocity in the z-axis direction. Therefore, the movement of the target in the z-direction is ignored, and the spatial movement of the target is simplified to the movement in a two-dimensional plane.

[0090] In some embodiments, referring to Figure 4Schematic diagram of the sampling frequencies and time synchronization of the lidar and millimeter-wave radar shown, where the lidar and millimeter-wave radar data fusion is used to solve the real-time target detection problem. Therefore, the data collected by the two sensors must be time-synchronized before fusion. A sampling frequency of 20 Hz is adopted, that is, one frame of data is collected every 50 ms, and the lidar information is 20 frames per second. After every 9 samples, the 10th frame is marked; the millimeter-wave radar has a sampling frequency of 10 Hz, that is, one frame of data is obtained every 100 ms, and the millimeter-wave radar information is 10 frames per second. After every 5 samples, the 6th frame is marked. Due to the difference in the scanning frequencies of the two sensors, it is difficult to keep the time stamps of the marked frames consistent. Therefore, in the embodiments of the present invention, based on the sampling frequency of the millimeter-wave radar, the frame closest to the lidar marked frame in the millimeter-wave scanning frames is selected as the marked frame, and the marked frame is selected once every 0.5 seconds. In addition, the time stamps of the millimeter-wave radar marked frames and the lidar time stamps are unified.

[0091] S300, perform preprocessing on the collected data by filtering and plane fitting to obtain detection data, where the detection data is used to characterize the point cloud data including detection identifiers and detection targets, and the detection data includes millimeter-wave radar detection data and lidar detection data.

[0092] In some embodiments, refer to Figure 5 Schematic diagram of the preprocessing process of the sensor collected data shown, which includes but is not limited to steps S310 to S330:

[0093] S310, perform statistical filtering on the point cloud information in the millimeter-wave radar collected data, including calculating the average distance between any point in the point cloud information and its neighboring points, comparing the neighboring points with a preset distance, and removing the neighboring points that are not within the preset distance range.

[0094] In some embodiments, for a vehicle perception system, the richer and more sufficient the information collected by the sensors, the more perfect the information processed and analyzed, and the more correct and accurate the analysis of the entire vehicle operating environment. However, due to the movement of the vehicle and the installation angle during driving, the on-vehicle sensors may collect invalid information. Therefore, preprocessing the collected sensor data has become an important part. The preprocessed data is beneficial to accelerating the data association speed of different sensors detecting the same target and the estimation speed of the front and rear states during target tracking.

[0095] In some embodiments, due to problems such as vehicle movement or its own installation angle when the sensor collects data, unusable data will be collected. A statistical filtering algorithm is used to filter the data and eliminate noise points. The statistical filter performs statistical analysis on the neighborhood of each point in the point cloud map and eliminates some points that do not meet the standards. In the collected data, this method mainly calculates the distance distribution from each point to the point closest to it, and calculates the average distance from each point to all n adjacent points. In addition, assuming that the obtained result conforms to a Gaussian distribution, its shape is determined by the mean and standard deviation, and the points with the average distance outside the standard range are defined as discrete points and deleted from the data.

[0096] In some embodiments, the point cloud output by the millimeter-wave radar in the scene is relatively sparse, and statistical filtering processing is performed on the point cloud information collected by the millimeter-wave radar. For the point P(x, y, z) in the point cloud map and its n adjacent points p1, p2,......, p n The average distance is d, and the distribution of the distance d follows a Gaussian distribution f(d) with a mean of μ d , and a variance of σ d , as follows:

[0097]

[0098]

[0099] Where when d ∈ (μ d - 3σ d , μ d + 3σ d ), the point cloud is not a discrete point. When d is not within this range, the point cloud is a discrete point and is excluded. Subsequently, the RANSAC plane fitting algorithm is used to separate the ground from the vehicle, which speeds up the clustering of the target point cloud and the generation of the detection box for subsequent detection.

[0100] S320, Use the RANSAC plane fitting algorithm to separate the target vehicle from the ground for the millimeter-wave radar collected data and lidar collected data that have undergone filtering processing.

[0101] In some embodiments, referring to Figure 6 The schematic diagram of the separation process of the RANSAC plane fitting algorithm shown, which includes but is not limited to steps S321~S323:

[0102] S321, Use the point cloud after statistical filtering processing as the original seed point set, and select a subset including three points from the original point set.

[0103] In some embodiments, selecting a subset including three points from the original point set includes using the point cloud after statistical filtering processing as the original seed point set Pr , select a subset P containing three points from the original point set P r in which, p1(x1, y1, z1), p2(x2, y2, z2), and p3(x3, y3, z3) are the three points of the subset P s s

[0104] S322, construct a ground model based on the subset and calculate the ground model parameters using the equivalent coefficient method.

[0105] In some embodiments, construct a parametric model and solve: The general expression of the ground model composed of the subset P s is Ax + By + Cz + D = 0. The normal vector of the plane formed by p1, p2, and p3 is used to solve the parameters A, B, and C of the ground model using the equivalent coefficient method.

[0106] S323, test the ground model parameters using the remaining point clouds in the original seed point set except for the subset, update the ground model parameters based on the test results, and repeat the ground model parameters until the ground model parameters meet the requirements, where the number of repetitions does not exceed the iteration threshold.

[0107] In some embodiments, model inspection and parameter update include, after obtaining the initial ground model parameters, using the remaining point clouds P r in the original seed point set P s1 to test the model and complete the update of the model parameters. The test method is to calculate the orthogonal projection distance d s1 from the points in the point cloud subset P p to the ground model. Use the standard deviation of the distance from the points to the ground model as the distance threshold d f . Determine whether the projection distance d p is less than the set ground distance threshold d f . If d p < d f , then the point is a ground point; otherwise, mark the point as a non-ground point to complete the segmentation of the ground point cloud. The distance threshold d f considers the allowable error range of the actual point cloud data and can effectively eliminate abnormal points and error points. Finally, calculate the number of base points under the ground model, that is, the number of inliers of the Ransac algorithm. If the number of inliers is greater than the set threshold, recalculate the ground model parameters using all the inliers. If it is less than the set threshold, then repeat the steps of S321~S323. If the number of repetitions is greater than the set iteration threshold k, stop the calculation of the parameter model.

[0108] In some embodiments, the determination of the number of iterations includes, taking M as the original seed point set P r ​​The probability that all n points in are inliers, where ω is the probability that a randomly selected point in the original seed point set P each time r is an inlier. ω is expressed as:

[0109]

[0110] Assume ω n represents the probability that each selection in n random selections is an inlier, then 1 - ω n is the probability that at least one of the points selected in these n random selections is an outlier. This model can be considered an error model, so the error probability that occurs in k iterations can be expressed as:

[0111] p k = (1 - ω n ) k

[0112] Since p k in the above formula is equal to 1 - M, we can get:

[0113] 1 - M = (1 - ω n ) k

[0114] Taking the logarithm of both sides of the above equation, we can get:

[0115]

[0116] S330. For the separated acquisition data, use the Euclidean clustering algorithm and cluster based on distance partitioning to generate the detection box of the detection target.

[0117] In some embodiments, referring to Figure 7 the schematic diagram of the Euclidean clustering process shown, it clusters the preprocessed point cloud data through the Euclidean clustering algorithm based on distance partitioning and generates a detection box. It includes a point P in three-dimensional space. Through KD-tree nearest neighbor search, K points closest to P are obtained. If the distance between them and P is less than the set threshold, they are clustered into the subset Q. If the number in Q no longer increases, the clustering ends. Otherwise, other points in the Q set are selected to repeat the above steps until the number of elements in Q no longer increases. Using the Euclidean clustering algorithm based on distance partitioning can effectively solve the problem of sparse near and dense far caused by the collection of point clouds by vehicle-mounted radar sensors.

[0118] S400. Use the Hungarian matching algorithm based on Mahalanobis distance to associate the millimeter-wave radar detection data and the lidar detection data to obtain the associated data.

[0119] In some embodiments, referring to Figure 8Schematic diagram of the data association process of the Hungarian matching algorithm based on Mahalanobis distance, including but not limited to steps S410 to S440:

[0120] S410. Determine the distance relationship between any detection target in the lidar sensor data and any target in the millimeter-wave radar sensor data through the Mahalanobis distance matrix.

[0121] In some embodiments, sensor data association is the basis for realizing information fusion and multi-target detection. Using the common position information of lidar data and millimeter-wave radar data, the distance relationship between the detection targets of the two sensors is established, and the measurement data that meets the threshold requirements is the detection of the same target. According to the Mahalanobis distance formula, define the Mahalanobis distance matrix of n l targets in the lidar sensor and n r targets in the millimeter-wave radar sensor as:

[0122]

[0123] S420. Use the optimal assignment problem to analyze the correlation of the elements with assignments in the Mahalanobis distance matrix, including determining the objective function and constraint conditions according to whether any element in the Mahalanobis distance matrix is assigned and the minimum of the assigned Mahalanobis distance.

[0124] Convert the multi-sensor data association problem into an optimal assignment problem. If the element is assigned, then x ij = 1, otherwise x ij = 0, x ij = {0, 1}. At most one non-zero element exists in each row and each column of its matrix, and the sum of the Mahalanobis distances of the assigned elements is the smallest. Then the objective function and constraint conditions are:

[0125]

[0126]

[0127] S430. Conduct correlation analysis on the elements in the Mahalanobis distance matrix that meet the constraint conditions, including calculating the correlation quality. When the correlation quality is greater than the set threshold, associate the two detection target data as the same detection target;

[0128] In some embodiments, at time k, if target i and target j are assigned, then determine whether they are target-associated. The correlation quality is m lr (k) + 1. When the correlation quality is greater than the set threshold, determine that target i and j data are associated as the same target.

[0129] S440. Use the UKF algorithm for the associated detection target data to generate a local estimate for target tracking and then perform weighted fusion to obtain the associated data.

[0130] In some embodiments, the fused target state of the target is obtained by weighted fusion of local estimates generated by lidar and millimeter-wave radar sensors using the UKF algorithm for target tracking. Only the state of the target at the previous moment is required to predict and update the state of the target at the next moment, and it is not necessary to calculate by bringing in historical fusion estimates.

[0131] S500. Perform multi-target detection on the associated data using unscented Kalman filtering and covariance intersection to obtain multi-target detection results.

[0132] In some embodiments, referring to Figure 9 the schematic flow diagram of multi-target detection using the unscented Kalman filtering and covariance intersection shown, which includes but is not limited to steps S510 to S550:

[0133] S510. Initialize the initial state of the detection target and the initial value of the error covariance matrix, where the error covariance matrix includes velocity noise and angular velocity noise.

[0134] In some embodiments, the initialization includes setting the initial state of the target and the initial value of the initial error covariance matrix.

[0135]

[0136] Since the two-dimensional process noise vector v k is also non-linear, for the convenience of calculation, in the embodiments of the present invention, v k is incorporated into x k ; in the embodiments of the present invention, the state vector will also be increased According to the adopted system state model being the CTRV model, Q is the state noise covariance.

[0137]

[0138] S520. Perform state prediction on the detection target according to the kinematic model and the error covariance matrix, including using the first Sigma point as the sampling point to obtain the first Sigma point set, and performing prediction through the first Sigma point set to obtain the state prediction value;

[0139] In some embodiments, calculate 2N + 1 Sigma points as the sampling points, where N represents the dimension of the state.

[0140]

[0141] In the formula: x a,k|k and P a,k|k are respectively the estimated value of the sensor at the k-th moment and the estimated error covariance matrix; It is a Sigma point set; λ is a scaling factor, and its optimal value is λ - 3N; the weights of the sampling points are as follows:

[0142]

[0143] Adopt the prediction of a set of Sigma point sets, and calculate their weighted average to obtain the predicted value x of the system state quantity a,k+1|k , and update the predicted error covariance P of the system state a,k+1|k .

[0144]

[0145]

[0146] S530, perform unscented transformation on the state predicted value to obtain the second Sigma point set, and process the second Sigma point set using the millimeter-wave radar observation equation to obtain the predicted observation value, the average value of the measurement prediction, the covariance matrix of the measurement value, and the cross-covariance matrix of the measurement and the prediction;

[0147] In some embodiments, according to the system state predicted value in step S520, use the unscented transformation again to generate a new Sigma point set.

[0148]

[0149] By performing a linear transformation on the millimeter-wave radar observation equation, substitute the new Sigma point set into the observation equation to obtain the predicted observable quantity, i = 1, 2,..., 2n + 1.

[0150]

[0151]

[0152]

[0153] Among them, Z k+1|k is the predicted observation value obtained by substituting the new Sigma point set into the observation equation; z k+1|k is the average value of the measurement prediction obtained by weighted summation; P z,k+1|k is the covariance matrix of the measurement value; P x,z is the cross-covariance matrix of the measurement and the prediction.

[0154] S540, calculate the Kalman gain according to the average value of the measurement prediction, the covariance matrix of the measurement value, and the cross-covariance matrix of the measurement and the prediction to obtain the state predicted value and the state covariance;

[0155] In some embodiments, the calculated Kalman gain is:

[0156]

[0157] The update of the system state value and the covariance update are as follows:

[0158] x k+1|k+1 = x k+1|k + K k+1|k (z k+1 - z k+1|k )

[0159]

[0160] Among them, x k+1|k+1 represents the estimated value of the system state update; P k+1|k+1 represents the updated system state covariance.

[0161] S550 uses the covariance intersection algorithm to fuse the state prediction value and the state covariance to obtain the detection result of the detection target. Among them, when fusing, according to the optimality of the mean square error, a weight factor is attached to the lidar sensor data and the millimeter-wave radar sensor data, and the weight factor is determined by the diagonal term of the sub-optimal method.

[0162] In addition, when fusing multi-sensor information, it is usually necessary to know that the errors of each sensor's target state estimation are uncorrelated or the cross-covariance between sensors needs to be calculated. In actual situations, due to the different sampling and transmission rates of each sensor in the multi-sensor fusion technology, the calculation of the cross-covariance matrix in the state estimation is extremely complex and cannot meet the real-time requirements of the perception part of autonomous vehicles. Therefore, a method of information fusion based on the covariance intersection (Covariance Intersection, CI) algorithm is proposed. The CI algorithm only uses the state estimation and the state estimation covariance to generate the fusion estimation and does not need to calculate the cross-covariance matrix, which can reduce the computational complexity and improve the efficiency of the system. And during the fusion process, according to the optimality in the sense of mean square error, a weight factor is attached to the fusion of lidar and millimeter-wave radar, providing specific limitations on the uncertainty of the covariance in the actual scenario and making the fusion estimation closer to the real traffic scenario.

[0163] During the fusion time interval [t k , t k+1 , the fusion estimation and covariance are:

[0164]

[0165]

[0166] Among them, X k+1,b = {1, 2} represents the weight coefficients of the millimeter-wave radar and the lidar; Xk+1,1 represents a millimeter-wave radar, X k+1,2 represents a lidar. According to the optimality of the mean square error, i.e., X k+1,b : min{tr[P k+1 , b = 1, 2}.

[0167]

[0168] The weight coefficient is determined by the relevant diagonal terms in the sub-optimal method:

[0169]

[0170] Reference Figure 10 , where Figure 10 is the diagram of the detection and analysis device for fusing multiple targets in the embodiments of the present invention. The device includes a first module 1010, a second module 1020, a third module 1030, and a fourth module 1040.

[0171] Among them, the first module is used to obtain the acquisition data of the sensors of the target vehicle according to the detection request. The sensors include a millimeter-wave radar and a lidar sensor, and the acquisition data includes millimeter-wave radar acquisition data and lidar acquisition data; the second module is used to perform preprocessing on the acquisition data by filtering and plane fitting to obtain detection data, and the detection data is used to characterize the point cloud data including detection identifiers and detection targets, and the detection data includes millimeter-wave radar detection data and lidar detection data; the third module is used to associate the millimeter-wave radar detection data and the lidar detection data by using the Hungarian matching algorithm based on the Mahalanobis distance to obtain associated data; the fourth module is used to perform multi-target detection on the associated data by using the unscented Kalman filter and covariance intersection to obtain multi-target detection results.

[0172] Exemplarily, with the cooperation of the first module in the device, the device in the embodiment can implement any one of the foregoing fusion multi-target detection methods, i.e., in response to. The beneficial effects of the present invention are as follows: The fusion multi-target detection method based on the unscented Kalman filter and covariance intersection combines the advantages of both lidar and millimeter-wave radar, uses the acquisition data of the millimeter-wave radar and lidar, calculates the position similarity of the measurement data of the two sensors, and establishes a corresponding Mahalanobis distance matrix for multiple targets collected by the two sensors, and uses the Hungarian algorithm to perform target matching to achieve data association. Finally, in a post-fusion manner, the unscented Kalman algorithm is used to perform local estimation on the targets detected by the lidar and millimeter-wave radar, and then the covariance intersection algorithm is used to fuse the obtained local estimations to generate a fused state estimation, so as to achieve high-precision, multi-target, and fast real-time detection in a harsh environment.

[0173] An embodiment of the present invention further provides an electronic device, which includes a processor and a memory;

[0174] The memory stores a program;

[0175] The processor executes the program to execute the foregoing method for detecting fused multi - targets; this electronic device has the function of carrying and running the software system for detecting fused multi - targets provided by the embodiment of the present invention. For example, a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or communicating with charged particle tools or other imaging devices, etc.

[0176] An embodiment of the present invention further provides a computer - readable storage medium, and the storage medium stores a program, and the program is executed by a processor to implement the method for detecting fused multi - targets as described above.

[0177] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order mentioned in the operation diagrams. For example, depending on the functions / operations involved, two consecutive blocks shown may actually be executed substantially simultaneously or the blocks can sometimes be executed in the reverse order. In addition, the embodiments presented and described in the flowcharts of the present invention are provided by way of example for the purpose of providing a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and the sub - operations described as part of a larger operation are executed independently.

[0178] An embodiment of the present invention also discloses a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in a computer - readable storage medium. The processor of the computer device can read the computer instructions from the computer - readable storage medium, and the processor executes the computer instructions, so that the computer device executes the foregoing method for detecting fused multi - targets.

[0179] In addition, although the present invention has been described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the described functions and / or features may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It should also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. Rather, given the attributes, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the modules will be understood within the ordinary skills of an engineer. Thus, those skilled in the art can implement the present invention as set forth in the claims without undue experimentation. It should also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0180] If the described function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0181] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch instructions from the instruction execution system, apparatus, or device and execute the instructions), or in conjunction with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0182] More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection (electronic device) having one or more wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, then editing, interpreting, or otherwise processing it as appropriate, and then storing it in a computer memory.

[0183] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

[0184] In the description of this specification, the descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0185] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the claims and their equivalents.

[0186] The above has specifically described the preferred embodiments of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art can also make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A detection method for fusing multiple targets, characterized in that, Including: Obtain the acquisition data of the sensors of the target vehicle according to the detection request. The sensors include a millimeter-wave radar and a lidar sensor, and the acquisition data includes millimeter-wave radar acquisition data and lidar acquisition data; Perform preprocessing on the acquisition data by filtering and plane fitting to obtain detection data, which is used to characterize the point cloud data including detection identifiers and detection targets. The detection data includes millimeter-wave radar detection data and lidar detection data; Associate the millimeter-wave radar detection data and the lidar detection data using the Hungarian matching algorithm based on Mahalanobis distance to obtain associated data; Perform multi-target detection on the associated data using unscented Kalman filtering and covariance intersection to obtain multi-target detection results.

2. The detection method for fusing multiple targets according to claim 1, wherein Before the step of obtaining the acquisition data of the sensors of the target vehicle according to the detection request, it further includes: Determine the kinematic model of the target vehicle using a constant turn rate and speed model; According to the kinematic model, taking the center of the rear axle of the target vehicle as the reference point, determine the translation matrix of the acquisition sensor relative to the vehicle coordinate system, and determine the rotation matrix of the acquisition sensor relative to the vehicle coordinate system. Unify the space of the sensor and the vehicle coordinate system through the translation matrix and the rotation matrix. The vehicle coordinate system takes the center of the rear axle as the origin, the forward direction of the origin as the x-axis, the direction perpendicular to the vehicle body to the left from the origin as the y-axis, and the direction perpendicular to the vehicle body upward from the origin as the z-axis; Synchronize the time of sensors with different scanning frequencies at a unified sampling frequency. The time synchronization includes marking the scanning frames and synchronizing the timestamps of the marked frames with the closest scanning time; Determine the first motion state quantity of the target vehicle at the current moment according to the kinematic model, and perform state transition calculation on the motion state of the target vehicle according to the first motion state quantity to obtain the second motion state quantity.

3. The detection method for fusing multiple targets according to claim 2, wherein The method further includes: When calculating the motion state of the target vehicle, calculate the second motion state quantity of the target vehicle by adding process noise, where the process noise includes speed noise and angular velocity noise.

4. The detection method for fusing multiple targets according to claim 2, characterized in that, The step of performing preprocessing on the acquisition by filtering and plane fitting to obtain detection data includes: Perform statistical filtering on the point cloud information in the millimeter-wave radar acquisition data, including calculating the average distance between any point in the point cloud information and its adjacent points, comparing the adjacent points with a preset distance, and removing the adjacent points not within the preset distance range; Use the RANSAC plane fitting algorithm to separate the target vehicle from the ground for the millimeter-wave radar acquisition data and the lidar acquisition data after filtering; Cluster the separated acquisition data using the Euclidean clustering algorithm and based on distance partitioning to generate detection frames for the detection targets.

5. The detection method for fusing multiple targets according to claim 4, characterized in that The step of using the RANSAC plane fitting algorithm to separate the target vehicle from the ground for the millimeter-wave radar acquisition data and the lidar acquisition data after filtering includes: Taking the point cloud after statistical filtering as the original seed point set, and selecting a subset including three points from the original point set; Construct a ground model based on subsets and calculate the ground model parameters using the equivalent coefficient method; Test the ground model parameters with the remaining point clouds in the original seed point set except for the subsets, update the ground model parameters based on the test results, and repeat the ground model parameters until the ground model parameters meet the requirements, where the number of repetitions does not exceed the iteration threshold.

6. The detection method for fusing multiple targets according to claim 2, wherein The correlation of the millimeter-wave radar detection data and the lidar detection data using the Hungarian matching algorithm based on Mahalanobis distance to obtain correlation data includes: Determine the distance relationship between any detection target in the lidar sensor data and any target in the millimeter-wave radar sensor data through the Mahalanobis distance matrix; Use the optimal assignment problem to analyze the correlation of the elements with assignments in the Mahalanobis distance matrix, including determining the objective function and constraints according to whether any element in the Mahalanobis distance matrix is assigned and the minimum assigned Mahalanobis distance; Conduct correlation analysis on the elements of the Mahalanobis distance matrix that meet the constraints, including calculating the correlation quality. When the correlation quality is greater than the set threshold, associate the two detection target data as the same detection target; Use the UKF algorithm for the correlated detection target data to generate local estimates for target tracking and then perform weighted fusion to obtain correlation data.

7. The detection method for fusing multiple targets according to claim 6, wherein The multi-target detection of the correlation data using unscented Kalman filtering and covariance intersection to obtain multi-target detection results includes: Initialize the initial state of the detection target and the initial value of the error covariance matrix, where the error covariance matrix includes velocity noise and angular velocity noise; Perform state prediction on the detection target according to the kinematic model and the error covariance matrix, including using the first Sigma points as sampling points to obtain the first Sigma point set and performing prediction through the first Sigma point set to obtain the state prediction value; Perform unscented transformation on the state prediction value to obtain the second Sigma point set, and process the second Sigma point set using the millimeter-wave radar observation equation to obtain the predicted observation value, the average value of the measurement prediction, the covariance matrix of the measurement value, and the cross-covariance matrix of the measurement and prediction; Calculate the Kalman gain according to the average value of the measurement prediction, the covariance matrix of the measurement value, and the cross-covariance matrix of the measurement and prediction to obtain the state predicted value and the state covariance; Use the covariance intersection algorithm to fuse the state predicted value and the state covariance to obtain the detection result of the detection target. When fusing, according to the optimality of the mean square error, attach weight factors to the lidar sensor data and the millimeter-wave radar sensor data, where the weight factors are determined by the diagonal terms of the suboptimal method.

8. A detection device integrating multiple targets, characterized in that, Include: The first module is used to obtain the acquisition data of the sensors of the target vehicle according to the detection request. The sensors include millimeter-wave radar and lidar sensors, and the acquisition data includes millimeter-wave radar acquisition data and lidar acquisition data; The second module is used to perform preprocessing on the acquisition data using filtering and plane fitting to obtain detection data. The detection data is used to represent the point cloud data including detection identifiers and detection targets, and the detection data includes millimeter-wave radar detection data and lidar detection data; The third module is used to associate the millimeter-wave radar detection data and the lidar detection data by using the Hungarian matching algorithm based on Mahalanobis distance to obtain associated data; The fourth module is used to perform multi-target detection on the associated data by using unscented Kalman filtering and covariance intersection to obtain multi-target detection results.

9. An electronic device, characterized in that, It includes a processor and a memory; The memory is used to store programs; The processor executes the program to implement the multi-target fusion detection method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a program, and the program is executed by the processor to implement the multi-target fusion detection method according to any one of claims 1-7.

Images