Vehicle and pedestrian detection and tracking method and system based on multi-source sensor fusion

Abstract

The present invention discloses a vehicle and pedestrian detection and tracking method and system based on multi-source sensor fusion, comprising: dedistorting and instance segmenting image data; obtaining a sparse point cloud depth map based on LiDAR point cloud coordinates; densifying the depth map to obtain a dense point cloud depth map; voxelizing and encoding the dense point cloud and performing feature mapping to obtain a mapped two-dimensional dense tensor; obtaining the two-dimensional target center point position based on a heat map and a Gaussian distribution function, performing a one-stage feature regression on the dense tensor to obtain a preliminary target detection bounding box; obtaining refined features and confidence prediction values for the bounding box size and orientation; and tracking vehicles and pedestrians based on nearest neighbor matching and the preliminary target detection bounding box. The present invention solves the problems of missed detection and misclassification in single sensors, the problems of scale reduction and magnification in the sensor field of view, and the problems of missed detection and mislocalization caused by compression and distortion of depth features during fusion, thereby reducing the average scale error.

Description

Technical Field

[0001] The present invention belongs to the field of environmental perception technology, and in particular relates to a vehicle and pedestrian detection and tracking method and system based on multi-source sensor fusion. Background Art

[0002] Autonomous vehicles not only alleviate rush-hour congestion and reduce traffic accidents caused by human error, but also promote the development of the technology industry and the upgrading of the mobility services sector. The government has begun issuing policies to promote the development of the autonomous driving industry, and automotive-related companies have also begun adjusting their industry R&D strategies. The advent of 5G technology is expected to trigger a new wave of academic research on autonomous driving technology, a large-scale, complex topic spanning multiple disciplines. The environmental perception layer primarily utilizes sensors such as cameras, lidar, millimeter-wave radar, integrated navigation systems, and ultrasonic radar to implement their functions. Only a highly reliable and accurate environmental perception layer can ensure the effectiveness and security of the two subsequent layers. Therefore, how to utilize sensors to more accurately and reliably acquire environmental information has become a hot topic of research. Current environmental perception technologies primarily encompass three areas: simultaneous localization and mapping, and object detection and tracking. Simultaneous localization and mapping technology addresses the positioning and mapping challenges of autonomous vehicles. This invention exclusively utilizes this technology to construct a campus map. Therefore, this invention focuses on addressing the challenges of existing algorithms for object detection, object tracking, and multi-sensor fusion, while also improving the corresponding evaluation metrics.

[0003] Target detection technology is mainly divided into obstacle detection and traffic signal detection. The main research objects of obstacle detection are obstacles such as vehicles and pedestrians. However, since the environmental features described by a single sensor are different in their respective spaces and dimensions, missed detections and misclassifications may occur during detection, resulting in large average scale errors and average attribute errors. Moreover, when the posture of the detected object changes, the existing algorithm using axis-aligned bounding boxes is difficult to enumerate and fit all target states, resulting in a large average direction error. The traditional lidar point cloud feature extraction algorithm has a large computational complexity and poor real-time performance, resulting in a large average translation error.

[0004] Object tracking technology primarily maintains the stable identity of objects detected by sensors, tracking their positions over data frames to provide semantic information for system decision-making. This technology is a prerequisite for intelligent vehicles to make accurate decisions and is primarily used to implement autonomous driving features such as adaptive cruise control and automatic emergency braking. However, as the pose of the tracked object changes, its scale within the sensor's field of view also changes. When the scale decreases, the scene's background is contained within the 3D object bounding box, resulting in a decrease in the number of tracking trajectories where the target is correctly tracked in at least 80% of the video frames. When the scale increases, the 3D object bounding box becomes less able to fully represent the target information, increasing the number of erroneous changes or swaps in the identity of the tracked object. Existing algorithms that use 3D object bounding boxes for tracking require a large computational load, resulting in short tracking times and poor real-time performance. Furthermore, depth maps generated by projecting sparse point clouds onto images cannot fully represent the target's features, leading to low multi-object tracking accuracy.

[0005] To better realize the environmental perception function of autonomous driving, multi-source sensor fusion technology can complement the advantages of various sensors and better obtain the attributes and state estimation of environmental targets. However, since the heat map used in the fusion of point clouds and images is obtained based on the model prediction results, the sparse distribution of three-dimensional point clouds in space will cause compression and distortion of deep features, making the center positions of different detection frames of the same target object closer, which in turn leads to problems such as missed detection and incorrect positioning. Existing fusion solutions do not handle the amount of information from multi-source sensors well, making it difficult to balance the real-time and accuracy of fusion-based target detection and tracking. Predicting the shape and orientation of the entire object through the limited number of point clouds in the three-dimensional target bounding box may not contain enough information, which leads to poor fusion effect. Summary of the Invention

[0006] The purpose of the present invention is to provide a vehicle and pedestrian detection and tracking method and system based on multi-source sensor fusion to solve the problems existing in the above-mentioned prior art.

[0007] On the one hand, to achieve the above-mentioned object, the present invention provides a vehicle and pedestrian detection and tracking method based on multi-source sensor fusion, comprising:

[0008] Acquire image data, perform dedistortion and instance segmentation processing on the image data, and obtain a detection object range;

[0009] Acquire a sparse point cloud depth map based on the detection object range and the laser radar point cloud coordinates;

[0010] Densifying the sparse point cloud depth map to obtain a dense point cloud depth map;

[0011] voxelize and encode the dense point cloud in the dense point cloud depth map and perform feature mapping to obtain a dense tensor of the dense point cloud feature in a two-dimensional mapping;

[0012] Performing a one-stage feature regression on the dense tensor to obtain a preliminary object detection bounding box;

[0013] Performing a two-stage feature regression on the preliminary object detection bounding box to obtain refined features and confidence prediction values of the size and orientation of the preliminary object detection bounding box;

[0014] Vehicle and pedestrian tracking is performed based on the preliminary object detection bounding box and nearest neighbor matching.

[0015] On the other hand, to achieve the above-mentioned purpose, the present invention provides a vehicle and pedestrian detection and tracking system based on multi-source sensor fusion, comprising:

[0016] Image processing module, multi-source sensor feature-level fusion module, target detection module, target tracking module;

[0017] The image processing module is used to obtain image data, and perform dedistortion and instance segmentation processing on the image data to obtain the detection object range;

[0018] The multi-source sensor feature-level fusion module is used to obtain a dense point cloud feature map according to the detection object range;

[0019] The target detection module performs one-stage and two-stage feature regression on the dense point cloud feature map and obtains the target detection result through a multi-layer perceptron;

[0020] The target tracking module is used to track the target according to the feature regression result.

[0021] The technical effects of the present invention are:

[0022] (1) Target detection. To address the problems of missed detection and misclassification that occur when a single sensor is used for detection, the present invention fuses the lidar point cloud and the camera image at the feature level. After the different sensors complete the feature extraction in the data, the feature information is uniformly input into the fusion model for comprehensive processing to reduce the amount of processing tasks required for fusion, thereby reducing the average scale error and average attribute error. Secondly, to address the problem that the target state is difficult to fit due to the change in the posture of the detected object, the present invention abandons the anchor-based approach of exhaustively enumerating potential targets and then performing post-processing, and adopts an algorithm based on the center point of a heat map without intrinsic direction to improve the detection accuracy of targets with orientation angles, thereby reducing the average scale error. Finally, to address the problem that the lidar point cloud processing consumes resources and leads to poor real-time performance, the present invention abandons the method of direct convolution processing of the point cloud. After image segmentation, the point cloud is projected into the divided area of interest to generate a depth map, and then the depth map is encoded by point cloud to extract features to generate a feature map, thereby achieving the goal of balancing real-time performance and accuracy as much as possible, thereby reducing the average translation error. The nuScenes detection score NDS value of the algorithm of the present invention is 64.9%, the mean average precision mAP value is 81.5%, the vehicle detection accuracy is 84.1%, and the pedestrian detection accuracy is 78.9%.

[0023] (2) Target tracking. First, in order to solve the scale reduction and magnification problem in the sensor field of view caused by the change of the tracking object's posture, the present invention adopts a target detection method without an intrinsic direction of the center point of the heat map to effectively avoid the target model update error problem caused by the change of the object scale, thereby increasing the number of tracking trajectories in which at least 80% of the video frames of the target can be correctly tracked, while reducing the number of identity exchanges of the tracked target. Secondly, the present invention combines the two modules of target detection and data association into one, by projecting the target center of the current frame to the previous frame, using the center point positioning loss function to calculate the position deviation of the two frames, and then calculating the negative velocity estimate between the two frames to reversely infer the position of the target center in the previous frame, and looking for the object closest to the position in the previous frame, so as to achieve the goal of improving real-time performance and tracking duration. Finally, to address the problem that sparse point clouds lead to low multi-target tracking accuracy after being projected onto an image, the present invention uses these sparse points and their depth values within the instance segmentation foreground entity range to generate a cone of interest region, randomly samples points with depth information in the two-dimensional foreground entity segmentation region, uses nearest neighbor retrieval to estimate the depth, and then projects the matched randomly sampled two-dimensional points around the sparse points back into three-dimensional space to obtain a virtual point cloud. The obtained dense point cloud is voxelized and encoded, and the obtained columnar unit features are extracted to generate a dense point cloud feature map, completing the point cloud densification work to achieve the goal of improving the accuracy of multi-target tracking. The multi-target tracking accuracy MOTA of the algorithm of the present invention is 0.668, the multi-target tracking precision MOTP is 0.250, the number of tracking trajectories MT for which at least 80% of the video frames of each target can be correctly tracked is 4851, the number of tracking trajectories ML for which at most 20% of the video frames of each target can be correctly tracked is 1405, the number of negative samples predicted as positive samples FP is 15769, and the number of positive samples predicted as negative samples FN is 18643.

[0024] (3) Feature-level fusion of multi-source sensors. First, in order to address the problems of missed detection and mis-positioning caused by compression and distortion of deep features during fusion, the present invention designs a Gaussian scattering kernel by classifying and discussing the solution to the minimum Gaussian radius, and spreads the target's feature information as close as possible to the Gaussian peak of each true target center, so as to achieve the goal of not deleting the prediction box with an intersection-over-union ratio greater than 0.7 with the true target detection box by default, thereby reducing the problems of missed detection and mis-positioning. Secondly, in order to address the problem that existing multi-source sensor fusion algorithms are difficult to balance real-time performance and accuracy, the present invention adopts a feature-level fusion scheme based on a center point without intrinsic direction, first completing the preliminary fusion of multi-source sensors through dense point cloud feature mapping, reducing the search space of the detector, and enabling the fusion layer and the decision layer to make full use of computing resources to improve accuracy, so as to achieve the goal of balancing the real-time performance and accuracy of the system. Finally, to address the problem of insufficient target features due to the limited number of point clouds in the target frame, the present invention uses bilinear interpolation from the three-dimensional target detection bounding box and feature map information output from the first stage to extract the required features from the center points of each face of the three-dimensional target bounding box and the located heat map center point. All target frame features are then concatenated and passed to a multi-layer perceptron to obtain the second-stage target bounding box attribute refinement and confidence score prediction, thereby achieving the goal of improving the fusion effect. Based on the evaluation indicators of target detection and tracking, the vehicle and pedestrian detection and tracking results obtained from the dataset test and actual vehicle experiments were analyzed and compared from the perspective of accuracy and error. Then, compared with the results of other algorithms, the effectiveness and feasibility of the target detection and tracking algorithm of the present invention were verified. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0026] Figure 1 1 is a flow chart of an embodiment of a vehicle and pedestrian detection and tracking algorithm based on multi-source sensor fusion in an embodiment of the present invention;

[0027] Figure 2 Target detection result diagrams of the vehicle and pedestrian detection and tracking algorithm based on multi-source sensor fusion in an embodiment of the present invention, where (a) is the vehicle detection result diagram of the dataset test, (b) is the vehicle detection result diagram of the actual vehicle experiment, (c) is the pedestrian detection result diagram of the dataset test, and (d) is the pedestrian detection result diagram of the actual vehicle experiment;

[0028] Figure 3These are the target tracking result diagrams of the vehicle and pedestrian detection and tracking algorithm based on multi-source sensor fusion in an embodiment of the present invention, where (a) is the MT result diagram of the dataset test, (b) is the MT result diagram of the actual vehicle experiment, (c) is the ML result diagram of the dataset test, and (d) is the ML result diagram of the actual vehicle experiment. DETAILED DESCRIPTION

[0029] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] Example 1

[0032] This embodiment provides a vehicle and pedestrian detection and tracking method and system based on multi-source sensor fusion, wherein the method includes:

[0033] Obtaining image data from a camera, performing dedistortion and instance segmentation processing on the image data, and obtaining an approximate range of the segmented detection object;

[0034] Within the segmented approximate range of the detected object, the LiDAR point cloud coordinates are converted to the segmented range to obtain a sparse point cloud depth map. A virtual point cloud is then generated based on the sparse point cloud within the foreground entity segmented within the instance and the frustum of interest of the two-dimensional randomly sampled points. This is then back-projected back into three-dimensional space to obtain a dense point cloud. The point cloud is then voxelized and encoded, and the resulting columnar unit features are extracted to obtain a dense point cloud feature map.

[0035] A heat map-based center point positioning method is used for the dense point cloud feature map. The minimum Gaussian radius value is obtained through classification discussion. Then, the Gaussian distribution function is combined to solve the true value of the center point positioning. The preliminary target detection bounding box feature is obtained through center point feature regression and stored on the center feature of the object.

[0036] The preliminary target detection bounding box features are further refined through a two-stage target detection network. Bilinear interpolation is used to extract the center points of each face of the three-dimensional bounding box and the heat map center point features located previously. These are concatenated into a vector containing the entire three-dimensional target bounding box and passed to a multi-layer perceptron to obtain feature refinement and confidence prediction of the preliminary target detection bounding box size and orientation.

[0037] After locating the center point of the target using heat maps and Gaussian distribution functions, the target velocity features obtained from the first-stage network regression are first used to calculate the negative velocity estimate between the two frames, thereby reversely inferring the position of the center of this target in the previous frame. Then, after extracting the target bounding box motion features of the current frame, similarity calculation and nearest neighbor matching are used to determine the probability that the objects in the previous and next frames are the same target. Finally, the successfully tracked target in the current frame inherits the identity number of the previous frame, and newly appearing targets are reassigned new identity numbers. For targets that have not been tracked for three consecutive frames due to trajectory mismatch, this patent uses the last known velocity to estimate and update the position of these targets in the current frame.

[0038] In some embodiments, the process of dedistorting and instance segmenting the image data includes: first performing coordinate calibration of radial and tangential distortions, and then dividing the image into regions according to features, so that the features of these regions are basically different in each place, but basically the same in the same place, to obtain target detection boxes and pixel regions with labels.

[0039] In some embodiments, the process of converting the laser radar point cloud coordinates into the segmentation range to obtain a sparse point cloud depth map and densifying the point cloud includes: starting from the laser radar space coordinate system C1, converting it to C after rotation and translation c Coordinate system, converted to image coordinate system C according to the pinhole imaging principle pic , and finally converted to pixel coordinate system C pix , thereby matching the 2D image pixel coordinates with the 3D point cloud coordinates.

[0040] In some embodiments, the point cloud densification process includes: randomly sampling points in the two-dimensional foreground entity segmentation area with depth information, using nearest neighbor retrieval depth estimation around the cone of interest area, taking the associated point cloud depth as the depth of the current pixel, and then projecting the matched randomly sampled two-dimensional points back to the three-dimensional space to obtain a virtual point cloud, so that the virtual point cloud at this time and the sparse point cloud obtained by the original projection both have the category information in the instance segmentation.

[0041] In some embodiments, the process of voxelization, encoding and feature mapping of point clouds includes: aggregating irregular point clouds through voxels by means of columnar unit encoding, using a sparse convolutional network to process voxel information of fixed length, width and height, and converting it into a two-dimensional map view to avoid the use of time-consuming three-dimensional convolution. Columnar segmentation and point cloud encoding are performed on three-dimensional unordered point cloud data to convert the three-dimensional point cloud coordinates into the form of network input. This patent only uniformly encodes the point cloud into a grid with a height of H and a width of W in a certain step size on the Cartesian coordinate system of the xy top view, and then stretches the grid containing the point cloud data along the z axis to obtain a columnar unit set P. The points in the columnar unit are then encoded as a 9-dimensional feature vector {x, y, z, r, xc ,y c ,z c ,x p ,y p}, where x, y, z are the spatial coordinates of the point, r is the reflectivity, x c ,y c ,z c is the deviation between the coordinates of the point and all points in the cylindrical unit, x p =xx c ,y p =yy c To offset the point in space to the center of the columnar unit, a frame of point cloud data is encoded into a (D, P, N) dense tensor with a dimension of D, a number of columnar units P, and a number of points in the columnar unit N.

[0042] In some embodiments, the process of point cloud feature mapping includes: first, processing each of the above-mentioned dense tensors containing D-dimensional features with an activation function to generate a dense tensor (C, P, N) with C channels, P columnar units, and N points in the columnar unit, then performing maximum pooling on each columnar unit to obtain a dense tensor of dimension (C, P), and finally generating a map view in a stacking manner. According to the (x, y) coordinate index value corresponding to each columnar unit recorded in the process of constructing the stacked columnar units, the dense tensor (C, P) generated in the previous step is scattered back to its original columnar unit position to create a map view (C, H, W) with a width W and a height H of C categories. In summary, the conversion relationship is Finally, the dense tensor (C, H, W) of the two-dimensional mapping of the point cloud features is obtained, whose number of channels is C, height is H, width is W, and it meets the conditions of two-dimensional convolution.

[0043] In some embodiments, the process of locating the center point of the target in one stage includes: adopting a heat map-based center point positioning method for the dense point cloud feature mapping, and classifying and discussing the three situations that the prediction box is tangent to a circle with a radius of r1 centered on the two corner points of the true value box, the prediction box is intangent to a circle with a radius of r1 centered on the two corner points of the true value box, and the prediction box is intangent to a circle with a radius of r3 centered on the left corner point of the true value box and tangent to a circle with a radius of r3 centered on the right corner point of the true value box. The minimum Gaussian radius value is obtained to design a Gaussian scattering kernel, and the Gaussian distribution function is combined to solve the true value of the center point positioning. The preliminary target detection bounding box feature is obtained by center point feature regression, and the target feature information is spread as much as possible near the Gaussian peak of the center of each true value target.

[0044] In some embodiments, the process of one-stage target detection frame feature regression includes: obtaining the center point positioning true value Y xykAfter that, the six object attributes, including center point heat map information, center point positioning offset, target size, height from the ground, target orientation, and target speed, can be stored in the center feature of the object. Combining the object attributes output by the above six one-stage feature regression outputs, a preliminary three-dimensional target bounding box with complete state information can be obtained.

[0045] In some embodiments, the process of two-stage target detection box feature regression includes: the second-stage network extracts the required additional point features from the three-dimensional target bounding box and feature map information output by the first-stage network, uses bilinear interpolation to extract the corresponding features of these five points from the two-dimensional dense point cloud feature map obtained in Chapter 2, and then connects the extracted point features to obtain a vector containing the overall three-dimensional target bounding box features, which is passed to the multi-layer perceptron MLP to obtain the refinement of the size and orientation prediction parameters of the preliminary target bounding box in the first stage and the final confidence score prediction combining the two-stage target detection.

[0046] In some embodiments, the target tracking process includes: after locating the center point of the target through a heat map and a Gaussian distribution function, first using the target speed features obtained by the first-stage network regression to calculate the negative speed estimate between the two frames before and after, thereby reversely inferring the position of the center of this target in the previous frame, and then extracting the target bounding box motion features of the current frame, using similarity calculation and nearest neighbor matching to determine the probability that the objects in the previous and next frames are the same target, and finally the successfully tracked current frame target inherits the identity number of the previous frame, and the newly appearing target is reassigned a new identity number. For targets that are no longer tracked due to trajectory mismatch for three consecutive frames, this patent uses the last known speed to estimate and update the position of these targets in the current frame. This allows targets containing many peak centers to be analyzed in the same heat map, simplifying three-dimensional target tracking to the nearest target center point search and matching to achieve speed prediction and tracking.

[0047] Vehicle and pedestrian detection and tracking systems based on multi-source sensor fusion include:

[0048] Multi-source sensor feature-level fusion module: This module converts the LiDAR point cloud coordinates to the segmentation range to obtain a sparse point cloud depth map. It then generates a virtual point cloud based on the sparse point cloud within the foreground entity segmented by the instance and the frustum of interest of the 2D randomly sampled points. This is then back-projected back into 3D space to obtain a dense point cloud. The point cloud is then voxelized and encoded, and the resulting columnar unit features are extracted to obtain a dense point cloud feature map.

[0049] The object detection module combines the Gaussian distribution function to solve the true value of the center point positioning. The initial object detection bounding box features are obtained through center point feature regression and stored in the center feature of the object. The features are further refined through a two-stage object detection network. The center points of each face of the 3D bounding box and the center point features of the heat map are extracted using bilinear interpolation. These are concatenated into a vector containing the entire 3D object bounding box. The vector is passed to the multi-layer perceptron to obtain the feature refinement and confidence prediction of the initial object detection bounding box size and orientation.

[0050] Target tracking module: It uses the target velocity features obtained from the first-stage network regression to calculate the negative velocity estimate between the two frames, thereby reversely inferring the position of the target center in the previous frame. It then extracts the target bounding box motion features of the current frame and uses similarity calculation and nearest neighbor matching to determine the probability that the objects in the previous and next frames are the same target. Finally, the successfully tracked target in the current frame inherits the identity number of the previous frame, and the newly appeared target is re-assigned a new identity number.

[0051] Example 2

[0052] like Figure 1-3 As shown, this embodiment provides a specific implementation of a vehicle and pedestrian detection and tracking method based on multi-source sensor fusion, including:

[0053] 2.1 Image Preprocessing

[0054] First, after dedistorting and instance segmenting the camera image data, the approximate range of the detected object can be obtained. The lidar point cloud coordinates are converted to the segmentation range to obtain a sparse point cloud depth map. Secondly, based on the sparse point cloud within the foreground entity range of the target segmentation and the cone of interest area of the two-dimensional random sampling points, a virtual point cloud is generated and back-projected back to the three-dimensional space. The obtained dense point cloud is then voxelized and encoded, and the obtained columnar unit features are extracted to generate a dense point cloud feature map, which provides effective data for subsequent fusion with the center point based on the heat map.

[0055] (1) Image dedistortion

[0056] The images captured by the camera will be distorted, mainly in the form of pincushion, barrel, and tangential distortions. The following formulas are used to calibrate the coordinates of the radial and tangential distortions to obtain the image coordinates for projection:

[0057]

[0058] where x pic_rec 、y pic_rec Represent the coordinates of the corrected image coordinate system, x pic and y picRepresent the coordinates of the original image coordinate system, k1, k2, k3, p1, p2 represent their respective distortion coefficients, r is the distance from the point to the imaging center, and k is used alone c =[k1 k2 p1 p2 k3] to represent the distortion.

[0059] (2) Instance Segmentation

[0060] In computer vision, instance segmentation divides an image into regions based on features, so that the features within these regions are generally different in different places, but similar in the same place. Instance segmentation performs object detection and semantic segmentation on an image, generating object detection boxes and labeled pixel regions.

[0061] 2.2 Coordinate transformation and densification of point clouds

[0062] In order to match the two-dimensional image pixel coordinates with the three-dimensional point cloud coordinates, this patent introduces four coordinate systems, namely the laser radar space coordinate system, the camera space coordinate system, the image coordinate system and the pixel coordinate system. The overall conversion idea starts from the laser radar space coordinate system C1, and then converts it to C after rotation and translation. c Coordinate system, converted to image coordinate system C according to the pinhole imaging principle pic , and finally converted to pixel coordinate system C pix .

[0063] (1) Conversion between the LiDAR space coordinate system and the camera space coordinate system

[0064] The transformation is mainly completed by the rotation matrix and the translation matrix. Suppose there is a point M whose coordinates in C1 are (x1, y1, z1). c The coordinates in (x c ,y c ,z c ), use the translation matrix T to transform the origin of C1 to C c Axis origin position, and then around the Z1 axis through the rotation matrix R z Convert C1 to c X c 、Y c The position parallel to the axis, similarly around the X1 axis through the rotation matrix R x Convert C1 to c Y c 、Z c The position parallel to the axis is rotated around the Y1 axis by the rotation matrix R y Convert C1 to c X c 、Z c Axis-parallel position.

[0065] Below is R z As an example, let the rotation angle be θ, and we can get the following from the trigonometric relationship:

[0066]

[0067] Combining and simplifying the above formulas yields:

[0068]

[0069] Written in matrix form:

[0070]

[0071] R z Complement to a 3×3 matrix:

[0072]

[0073] Assume that the rotation angles of the x-axis and y-axis are α and β respectively, then solve the rotation matrix R by the same logic. x and R y , we can get C c The coordinates of the rotation matrix R1 are:

[0074]

[0075] After obtaining the rotation matrix R1, the translation matrix T1 is solved. Its function is to transform the laser radar space coordinate system C1 and the camera space coordinate system C c Perform coordinate transformation, set x d 、y d 、z d is the distance between the two XYZ axes, then:

[0076]

[0077] In summary, the rotation matrix R1 and translation matrix T1 can be used to transform C1 to C c :

[0078]

[0079] Complement C1 to a 4×1 matrix:

[0080]

[0081] Conversion between camera space coordinate system and image coordinate system

[0082] Suppose there exists a point M in C c The coordinates in (x c ,y c ,z c), in C pic The coordinates are (x pic ,y pic ), according to the pinhole imaging principle and the triangle similarity principle, point M is projected to the image coordinate system through the focal position, and the focal length is f. Then the two cameras in the figure are at X c The displacement in the direction is b, then:

[0083]

[0084] Its matrix expression is:

[0085]

[0086] in, That is the transformation matrix K1.

[0087] (2) Conversion between image coordinate system and pixel coordinate system

[0088] Since the units in the image coordinate system and the pixel coordinate system are different, conversion is also required.

[0089] Let the coordinate X in the image coordinate system be pic and Y pic The corresponding pixel coordinates are u0 and v0, and the corresponding length is d x d y ,but:

[0090]

[0091] It can be expressed as a matrix:

[0092]

[0093] in, That is the transformation matrix K2.

[0094] (3) Conversion between the LiDAR spatial coordinate system and the pixel coordinate system

[0095] To sum up, the projection matrix P of the ideal transformation is:

[0096]

[0097] (4) Generation of dense point cloud depth map

[0098] The near clipping plane is closer to the camera, and the far clipping plane is farther away. These represent the minimum and maximum distances the camera can draw, respectively. The range between the two is the field of view, called the frustum of interest (FOI), which can be derived from similar triangles. However, the FOI derived from this alone is too large, so a tolerance in the depth direction is introduced to constrain the LiDAR point cloud column within the region to obtain a more accurate ROI.

[0099] Since the sparse point cloud projection points within the three projected image instance segmentation ranges in the sparse point cloud depth map cannot fully reflect the characteristics of the target, this patent uses sparse point clouds and their depth values to generate a cone of interest. Since the point cloud depths are basically consistent in the same foreground target object, points in the two-dimensional foreground entity segmentation area with depth information can be randomly sampled. Nearest neighbor retrieval depth estimation is used around the cone of interest area, and the depth of the associated point cloud is taken as the depth of the current pixel. The matched randomly sampled two-dimensional points are then projected back into three-dimensional space to obtain a virtual point cloud, so that the virtual point cloud at this time and the sparse point cloud obtained by the original projection both have the category information in the instance segmentation.

[0100] 2.3 Processing of dense point clouds

[0101] (1) Voxelization and encoding

[0102] Because direct neural network convolution processing of 3D point clouds requires high hardware requirements, is inefficient, and suffers from poor real-time performance, this patent introduces the concept of volume pixels, similar to 2D pixels. Using columnar unit encoding, irregular point clouds are aggregated into voxels. A sparse convolutional network is then used to process voxel information with fixed length, width, and height, converting it into a 2D map view to avoid the time-consuming 3D convolution. Although this results in a certain amount of information loss, the information within each voxel is aggregated to reduce this loss.

[0103] After pre-processing the point cloud into voxels, columnar segmentation and point cloud encoding are performed on the 3D unordered point cloud data to convert the 3D point cloud coordinates into a network input format. This patent uniformly encodes the point cloud into a grid with a height of H and a width of W using a fixed step size in the Cartesian coordinate system of the xy top view. The grid containing the point cloud data is then stretched along the z-axis to obtain a columnar unit set P.

[0104] Then the points in the columnar unit are encoded as a 9-dimensional feature vector {x, y, z, r, x c ,y c ,z c ,x p ,y p}, where x, y, z are the spatial coordinates of the point, r is the reflectivity, x c ,yc ,z c is the deviation between the coordinates of the point and all points in the cylindrical unit, x p =xx c ,y p =yy c is the offset of the midpoint in space to the center of the cylindrical unit.

[0105] However, due to the sparsity of point clouds in space, not all columnar units contain enough points. Therefore, it is necessary to use the methods of truncation and padding to select dense columnar units for convolution. Truncation means that when the number of points in each columnar unit exceeds the threshold N, random downsampling is performed to reduce the number of points. Padding means that when the number of points in each columnar unit is lower than the threshold N, the number of points is first cleared and then filled with 0. Through the above method, a frame of point cloud data is encoded into a (D, P, N) dense tensor with a dimension of D, a number of columnar units P, and a number of points in the columnar unit N.

[0106] (2) Dense point cloud feature mapping

[0107] First, each of the above dense tensors containing D-dimensional features is processed by an activation function to generate a dense tensor (C, P, N) with C channels, P columns, and N points in each column. Then, each column is max-pooled to obtain a dense tensor of dimension (C, P). Finally, a stacking method is used to generate a map view. According to the (x, y) coordinate index value corresponding to each column recorded during the construction of the stacked column, the dense tensor (C, P) generated in the previous step is scattered back to its original column position to create a map view (C, H, W) with a width W and a height H of C categories.

[0108] In summary, the conversion relationship is Finally, the dense tensor (C, H, W) of the two-dimensional mapping of the point cloud features is obtained, whose number of channels is C, height is H, width is W, and it meets the conditions of two-dimensional convolution.

[0109] 2.4 First-stage target regression

[0110] The one-stage network uses a heat map-based center point positioning method for dense point cloud feature mapping, obtains the minimum Gaussian radius value through classification discussion, and then combines the Gaussian distribution function to solve the true value of the center point positioning, and obtains the preliminary target detection bounding box feature through center point feature regression.

[0111] (1) Target center point positioning based on heat map

[0112] The dense point cloud feature map obtained above is mapped to the dense tensor (C, H, W) of the map view and input into the two-dimensional object detection neural network for processing to perform feature-level fusion of point cloud and image, where C is the number of channels, H and W are the height and width of the two-dimensional map.

[0113] ① Center point positioning prediction value

[0114]

[0115] in Represents the value at the kth channel (x, y) in the heat map, and the number of channels k needs to correspond to the number of channels C in the dense point cloud feature map. R represents the step size of the network output and is set to R=4 by default.

[0116] ②True value of center point positioning

[0117] When the predicted value When , it means that the (x, y) of the heat map is a key point, and its position represents the center point of a target detection box; when the predicted value The (x,y) position of the heat map is the background;

[0118] In the more general case, when the predicted value Take the true value p of the center position of the original image of any key point and divide it by the sampling multiple, that is, the step size R, and round it down to get the center position of the object.

[0119]

[0120] Distribute all the real key points on a heat map and use the Gaussian convolution kernel to generate a two-dimensional normal distribution. Then the value at (x, y) of any k-th channel of the heat map is the center of the target bounding box, so the center point locates the true value Y xyk It can be expressed as a Gaussian function:

[0121]

[0122] Among them, the target scale adaptive variance σ p It is related to the width W and height H of the object. The Gaussian distribution function is used to calculate σ. p and Y xyk Perform detailed solution.

[0123] (2) Gaussian distribution function

[0124] ①Classification discussion on solving Gaussian radius

[0125] Define the intersection of the predicted target box and the true target box as the overlap area. First, the calculation of the Gaussian radius is divided into three types according to the tangency of the predicted box and the true value box. The internal and external tangency cases refer to the relationship between the predicted box and the two circles:

[0126] Step 1: The predicted box is tangent to a circle with a radius of r1 and the two corner points of the true value box as the center

[0127] First calculate the critical value of the Gaussian radius r:

[0128]

[0129] Arranged as a quadratic equation in r:

[0130] 4*overlap*r 2 +2*overlap*(h+w)*r+(overlap-1)*(h*w)=0 (3-5)

[0131] This is equivalent to finding the quadratic equation about r.

[0132] Let a=4*overlap, b=2*overlap*(h+w), c=(overlap-1)*(h*w)

[0133] According to the root discriminant formula and r needs to be greater than 0, then:

[0134] Denoted as r1.

[0135] Step 2: The predicted box is inscribed in a circle with the two corner points of the true value box as the center and r1 as the radius

[0136] First calculate the critical value of the Gaussian radius r:

[0137]

[0138] Arranged into a quadratic equation of r:

[0139] 4r 2 -2(h+w)r+(1-overlap)hw=0 (3-7)

[0140] So let a = 4, b = -2(h + w), c = (1-overlap)(h*w)

[0141] According to the root discriminant formula and r needs to be greater than 0, then:

[0142] Denoted as r2.

[0143] Step 3: The predicted box is inscribed in a circle with a radius of r3 and a center at the left corner of the true value box, and is also circumscribed in a circle with a radius of r3 and a center at the right corner of the true value box.

[0144] First calculate the critical value of the Gaussian radius r:

[0145]

[0146] Arrange the equation to get r:

[0147]

[0148] So let a=1,b=-(w+h),

[0149] get:

[0150] Denoted as r3.

[0151] ② Solve the true value of the center point positioning based on the Gaussian radius

[0152] Take the minimum value of the three Gaussian radii r = min (r1, r2, r3) as the Gaussian kernel radius f (w, h) depending on the width w and height h of the target detection box, that is,

[0153] f(w,h)=r=min(r1,r2,r3) (3-10)

[0154] Finally, the Gaussian distribution function is used to represent the target scale adaptive variance σ of any center point p in category k p for:

[0155] σ p =max(f(w,h),τ) (3-11)

[0156] Among them, τ = 2 is the minimum Gaussian radius within the allowable range. Therefore, the center point of formula (3-2) locates the true value Y xyk Expressed as:

[0157]

[0158] In summary, by classifying and discussing the minimum Gaussian radius to design a Gaussian scattering kernel, the target's feature information is spread as close as possible to the Gaussian peak of each true target center, so that the prediction box with an intersection-over-union ratio of less than 0.7 with the true target detection box is not deleted by default, and the problems of missed detection and incorrect positioning are also reduced.

[0159] (3) One-stage target detection box feature regression

[0160] Get the true value Y at the center point xykAfter that, six object attributes, including center point heat map information, center point positioning offset, target size, ground clearance, target orientation, and target speed, are stored in the object's central feature. Combining these six object attributes output by the first-stage feature regression, a 3D bounding box result with complete state information is obtained. Ground clearance helps locate objects in three dimensions and can also add missing altitude information that was removed during feature mapping. The target orientation rotation angle uses the sine and cosine of the yaw angle to continuously regress the target and predict its directionality. Both involve fewer parameters and are relatively simple to solve. Target speed provides an effective basis for subsequent target tracking, so this section only analyzes the first three items and does not elaborate on the remaining features.

[0161] ①Center point heat map loss function

[0162] Construct the Focal loss function in the classification task loss function to solve the problem of extremely unbalanced number of positive and negative samples, then the center point heat map loss function can be expressed as:

[0163]

[0164] Among them, N is the number of targets in the input image, which is also the number of key points in the heat map, and xyk represents the coordinate points of k-type targets on all heat maps. is the predicted value of the test, Y xyk is the true value, and is the cross entropy loss function, and It is used to limit the problem that the function is dominated by easy samples, and (1-Y xyk ) β It is used to adjust the points around the center point to prevent false detection. Generally, the hyperparameters α = 2 and β = 4 are set to balance the samples.

[0165] In the formula, when Y xyk =1, if When the value is close to 1, that is, the sample is easy to classify, The term makes the loss function very small, so the training ratio, that is, the loss value, can be appropriately reduced at this point; and when When it is close to 0, it means that the sample is difficult to classify. The term makes the loss function very large, so the training proportion of the center point that has not been learned should be increased.

[0166] In the formula, when Y xyk ≠1, if When it is close to 1, that is, closer to the actual center point, The training weight becomes larger, that is, the loss value becomes larger, but because it is closer to the center point, it is necessary to use (1-Y xyk )β Reduce the training weight to make the training balanced; and when When it is close to 0, that is, far away from the actual center point, (1-Y xyk ) β The training weight becomes larger, that is, the loss value becomes larger, but because it is far away from the center point, it is necessary to use Reduce the training weight a little to make the training balanced.

[0167] ②Center point positioning offset loss function

[0168] When analyzing the Gaussian distribution of the center point of any position in the previous center point positioning, it can be seen from formula (3-2) that the center position of the real object The center position p of the object in the original image is obtained by downsampling with a step size of R and then approximating it to an integer. Therefore, there will be a positional deviation of the center point. In order to compensate for the quantization error caused by the output step size and thus more accurately predict the position of the center point of the bounding box in the input image, this patent additionally calculates the offset of the center point positioning. The formula is as follows:

[0169]

[0170] Construct the Loss function in the regression task loss function for supervised training, that is, use the mean absolute error to measure the relationship between the predicted value and the true value. The loss function of the center point offset is expressed as follows:

[0171]

[0172] Among them, N is the number of targets in the input image, which is also the number of key points in the heat map, and p is the coordinate of the center point of the target in the image. represents the offset predicted by the network,

[0173] ③Target size loss function

[0174] make is the bounding box coordinate of target k, then its center point is:

[0175]

[0176] use To predict the center point and then regress the length and width of each target k, use Indicates the target size after downsampling. In order to reduce the computational burden, this patent adopts a two-channel output with a width of W and a length of H. To predict the target, it can be expressed as:

[0177]

[0178] Construct the Loss function in the regression task loss function for supervised training, that is, use the mean absolute error to measure the relationship between the predicted value and the true value. The target length and width prediction loss function is expressed as follows:

[0179]

[0180] Among them, N is the number of targets in the input image, which is also the number of key points in the heat map. The center point coordinates P k The target size at .

[0181] ④ Overall objective loss function

[0182] The overall loss function is a combination of the above three, and different weights are assigned, namely:

[0183] L det =L k +λ offset L offset +λ size L size (3-19)

[0184] Among them, λ offset =1,λ size =0.1.

[0185] 2.5 Two-stage object detection frame feature regression

[0186] In the first stage of obtaining target features, the number of point clouds within each 3D detection is actually small. Predicting features such as the shape and orientation of the entire object using these limited point clouds may not contain enough information to obtain complete target features. For example, a single sensor often only sees a portion of the target, not its center. Therefore, to address this issue, a second refinement stage of the lightweight point cloud feature extractor is used to perform feature regression again.

[0187] The second stage extracts the required additional point features from the three-dimensional target bounding box and feature map information output by the first stage. Theoretically, the three-dimensional target bounding box should have 6 center points, but since the projections of the top and bottom bounding box centers are all at a central position in the two-dimensional dense point cloud feature map, only the center points of the four faces and the target center point obtained previously need to be considered. Use bilinear interpolation to extract the corresponding features of these 5 points from the two-dimensional dense point cloud feature map obtained in Chapter 2, and then connect the extracted point features to obtain a vector containing the overall three-dimensional target bounding box features, which is passed to the multi-layer perceptron MLP to obtain the refinement of the size and orientation prediction parameters of the preliminary target bounding box in the first stage and the final confidence score prediction of the target detection combined with the two stages. Use IoU tTo represent the intersection-over-union ratio between the t-th 3D target boundary prediction box and the true 3D target boundary prediction box, I t is the second-stage true confidence score of the t-th 3D target boundary prediction box, which can be expressed as:

[0188] I t =min(1,max(0,2×IoU t -0.5)) (3-20)

[0189] Constructing the binary classification cross entropy loss function in the classification task loss function for supervised training can provide an unbiased estimated proxy loss to measure the distance between the predicted value and the true value:

[0190]

[0191] in, The second stage prediction confidence score for the t-th 3D object bounding box prediction.

[0192] During the reasoning process, the target box confidence of the first stage network is taken into account From formula (3-1), we can get Therefore, the confidence scores of the two stages are geometrically averaged to obtain the final prediction confidence score:

[0193]

[0194] 3.0 Vehicle and pedestrian tracking based on feature-level fusion of multi-source sensors

[0195] Unlike most existing target tracking algorithms that primarily use target detection followed by data association, this patent integrates both into a single neural network, representing each target using a center point without intrinsic orientation. Like other regression targets, velocity estimation is supervised and trained using the L1 loss function at the true target's position at the current time step. Because the tracking process is greatly simplified after obtaining the center point, after locating the target's center using a heat map and Gaussian distribution function, the algorithm first uses the target velocity features obtained from the first-stage network regression to calculate a negative velocity estimate between the two previous frames, thereby inferring the target's center position in the previous frame. After extracting the target's bounding box motion features in the current frame, similarity calculation and nearest neighbor matching are used to determine the probability that objects in the previous and next frames are the same target. Finally, successfully tracked targets in the current frame inherit the identity number of the previous frame, while newly appearing targets are assigned new identity numbers. For targets that were not tracked for three consecutive frames due to trajectory mismatches, this patent uses the last known velocity to estimate and update their positions in the current frame. This allows targets with multiple peak centers to be analyzed within the same heat map, simplifying 3D target tracking to a search and match of the nearest target center point for velocity prediction and tracking, significantly reducing computational complexity.

[0196] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A vehicle and pedestrian detection and tracking method based on multi-source sensor fusion, characterized in that: The following steps are involved: Acquire image data, perform dedistortion and instance segmentation processing on the image data, and obtain a detection object range; Acquire a sparse point cloud depth map based on the detection object range and the laser radar point cloud coordinates; Densifying the sparse point cloud depth map to obtain a dense point cloud depth map; voxelize and encode the dense point cloud in the dense point cloud depth map and perform feature mapping to obtain a dense tensor of the dense point cloud feature in a two-dimensional mapping; Performing a one-stage feature regression on the dense tensor to obtain a preliminary object detection bounding box; Performing a two-stage feature regression on the preliminary object detection bounding box to obtain refined features and confidence prediction values of the size and orientation of the preliminary object detection bounding box; Tracking vehicles and pedestrians based on the preliminary target detection bounding box and nearest neighbor matching; The process of performing one-stage feature regression on the dense tensor includes: A heat map-based center point positioning method is used to perform feature mapping on the dense point cloud. The minimum Gaussian radius value is obtained based on classification discussion. The true value of the center point positioning is solved based on the Gaussian distribution function. Feature regression is performed on the center point to obtain feature information of a preliminary target detection bounding box. Object attributes are stored at the center point of the detected object, and a preliminary three-dimensional target bounding box is obtained based on the object attributes; the object attributes include center point heat map information, center point positioning offset, target size, height above the ground, target orientation, and target speed. The process of performing two-stage feature regression on the feature information of the preliminary object detection bounding box includes: Based on the preliminary three-dimensional target bounding box and feature information obtained by the first-stage feature regression, additional point features are extracted, and corresponding features are extracted from the mapped dense tensor of the dense point cloud features based on the bilinear interpolation method; Connecting the corresponding features to obtain a feature vector of the overall three-dimensional object bounding box; Passing the overall 3D object bounding box feature vector to a multi-layer perceptron (MLP) to obtain refined features and confidence predictions of the preliminary 3D object bounding box size and orientation; The process of tracking vehicles and pedestrians based on the preliminary target detection bounding box and nearest neighbor matching includes: Based on the feature information obtained from the first-stage feature regression, the position of the center of the target detected in the previous frame is calculated and predicted, and the three-dimensional target bounding box motion features of the current frame are extracted. The probability that the detected targets of the previous and next frames are the same target is calculated by similarity calculation and nearest neighbor matching. The successfully tracked target in the current frame inherits the identity number of the previous frame, and the newly appearing target is re-assigned a new identity number. The three-dimensional target tracking is simplified to the search and matching of the nearest target center point for speed prediction and tracking.

2. The vehicle and pedestrian detection and tracking method based on multi-source sensor fusion according to claim 1 is characterized in that: The process of performing dedistortion and instance segmentation processing on the image data includes: The image data is subjected to coordinate calibration for radial and tangential distortion, the image data is divided into regions, a target detection frame and a pixel region with a label are obtained, and a detection object range is obtained based on the target detection frame and the pixel region.

3. The vehicle and pedestrian detection and tracking method based on multi-source sensor fusion according to claim 1 is characterized in that: The process of obtaining a sparse point cloud depth map based on the detection object range and the laser radar point cloud coordinates includes: The two-dimensional image pixel coordinates are matched with the three-dimensional point cloud coordinates based on the lidar space coordinate system, camera space coordinate system, image coordinate system and pixel coordinate system.

4. The vehicle and pedestrian detection and tracking method based on multi-source sensor fusion according to claim 1 is characterized in that: The process of densifying the sparse point cloud depth map includes: By randomly sampling points in the 2D foreground entity segmentation area with depth information, the nearest neighbor retrieval depth estimation is performed around the frustum of interest; The depth of the associated point cloud is taken as the depth of the current pixel, and the matched randomly sampled two-dimensional points are projected back into the three-dimensional space to obtain a virtual point cloud, so that the virtual point cloud at this time and the sparse point cloud obtained by the original projection have both the category information in instance segmentation.

5. The vehicle and pedestrian detection and tracking method based on multi-source sensor fusion according to claim 1 is characterized in that: The process of voxelizing and encoding the dense point cloud in the dense point cloud depth map and performing feature mapping includes: The irregular point cloud in the dense point cloud depth map is aggregated through voxels using columnar unit encoding, and the voxel information of fixed length, width and height is processed based on a sparse convolutional network to convert the voxel information into a two-dimensional map view.

6. A vehicle and pedestrian detection and tracking system based on multi-source sensor fusion that implements the method of claim 1, characterized in that: include: Image processing module, multi-source sensor feature-level fusion module, target detection module, target tracking module; The image processing module is used to obtain image data, and perform dedistortion and instance segmentation processing on the image data to obtain the detection object range; The multi-source sensor feature-level fusion module is used to obtain a dense point cloud feature map according to the detection object range; The target detection module performs one-stage and two-stage feature regression on the dense point cloud feature map and obtains the target detection result through a multi-layer perceptron; The target tracking module is used to track the target according to the feature regression result.

Images