A traffic target detection and tracking method based on vehicle-mounted multi-sensor fusion

Abstract

The application discloses a traffic target detection and tracking method based on vehicle-mounted multi-sensor fusion, which comprises the following steps: obtaining semantic information of an image by using a semantic segmentation algorithm on the image collected by a vehicle-mounted camera; projecting point cloud information collected by a laser radar to an image plane to obtain semantic information of a corresponding position; fusing and correcting semantic information according to left and right visual angles; adding the corrected semantic information to the point cloud to obtain extended point cloud data with semantic information; detecting the position, speed, width, height and direction information of a traffic target by using a point cloud target detector; on the basis of obtaining the traffic target information, associating a track and detected information observed by a sensor by using a Hungarian algorithm; and updating the state of a vehicle target according to Kalman filtering by using the associated detected information, so as to obtain shape, motion and track information of the traffic target, thereby improving the accuracy and real-time performance of environment perception of an autonomous vehicle.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of automatic driving perception, and relates to a traffic target detection and tracking method based on vehicle-mounted multi-sensor fusion. BACKGROUND

[0002] A plurality of sensors are often installed on an automatic driving vehicle, and different sensors have respective advantages and disadvantages in terms of sensing capability. For example, an RGB-based visual sensor such as a monocular camera has been widely used, and data thereof can be efficiently and accurately processed for target detection. However, this type of sensor is susceptible to the surrounding environment, such as weather, light, and the like, and has low accuracy in speed and distance sensing. A 3D sensor such as a laser radar can overcome the disadvantage of being susceptible to weather and light, and has high accuracy in distance and speed measurement of a target. However, they cannot sense some visual attributes such as color and texture. Therefore, each type of sensor has limited sensing capability and respective advantages and disadvantages. At the same time, multi-sensor fusion can maximize the respective advantages and effectively avoid the disadvantages. Specifically, multi-sensor fusion has the following advantages: 1. enhanced system survivability, the measurement information of multiple sensors has a certain redundancy; 2. expanded spatial coverage; 3. expanded time coverage; 4. improved reliability and reduced information ambiguity; 5. improved detection performance and increased spatial resolution. In the automatic driving scene, it is necessary to accurately perceive the traffic targets and trajectories around the vehicle in real time to make correct and reasonable decisions. That is, the information of the traffic targets and the driving trajectories must be accurately detected, and therefore effective fusion of multi-sensor information is a necessary condition for accurate perception and tracking of traffic targets. In order to achieve the above-mentioned goals, it is necessary to reasonably extract multi-sensor data information and implement an efficient fusion and tracking method to meet the real-time requirement.

[0003] Nowadays, a lot of research results have been achieved in the research of multi-source sensor fusion perception for automatic driving. However, there are still the following problems: 1. the multi-source sensor fusion perception method has high computational complexity, and real-time performance is difficult to meet; 2. the process of multi-sensor fusion perception is too complex, and has poor scalability and robustness, which is not convenient for actual deployment. SUMMARY

[0004] To solve the above problems, the application discloses a traffic target detection and tracking method based on vehicle-mounted multi-sensor fusion, semantic segmentation algorithm is used to obtain semantic information of images collected by a vehicle-mounted camera, point cloud information collected by a laser radar is projected onto the semantic information to obtain information of corresponding positions, semantic information obtained from different perspectives is corrected, and the information is expanded to point cloud data, which is used as input of a point cloud target detector to detect position, speed, width, height and direction information of traffic targets.

[0005] To achieve the above purpose, the technical scheme of the application is as follows:

[0006] A traffic target detection and tracking method based on vehicle-mounted multi-sensor fusion, the method comprises the following steps:

[0007] 1. A semantic segmentation network DeepLab V3+ is used to segment images collected by left and right vehicle-mounted cameras respectively to obtain class information corresponding to each pixel on the images;

[0008] 2. Point clouds collected by a laser radar are projected onto image planes of left and right cameras respectively to obtain class confidence of corresponding projection positions in left and right images;

[0009] 3. Evidence theory is used as a fusion correction method, different view class information exists deviation, according to the pixel class and its confidence of left and right view, design evidence framework and basic belief assignment, using Dempster fusion rule to realize image pixel fusion correction, obtain more accurate class information;

[0010] 4. According to the PointPainting fusion process, class semantic information obtained from the image plane is added to the information of the point cloud, and a point cloud target detection network PointPillar is used to detect position, speed, width, height and direction information of traffic targets;

[0011] 5. Kalman filter is used to predict state information corresponding to the next time of the existing trajectory;

[0012] 6. Based on the traffic target information obtained in step 4 and the trajectory information obtained in step 5, the Hungarian algorithm is used to correlate the trajectory with the sensor observations. Uncorrelated trajectories are deleted if they exceed a time threshold, and new trajectories are created for uncorrelated detections. The correlated trajectories and observations are used to update the state parameters of the vehicle target Kalman filter, thereby deriving information about the shape, motion, and trajectory of the traffic target, thereby improving the accuracy and real-time performance of the autonomous vehicle's environmental perception.

[0013] Furthermore, in step 1, the semantic information of each pixel of the left and right view images is obtained from the semantic segmentation network DeepLab V3+. The main steps are as follows:

[0014] 1) First, find two pictures with corresponding timestamps from the picture queues collected by the left and right front cameras on the vehicle;

[0015] 2) Use the KITTI dataset to train the DeepLab V3+ model and fine-tune the model to make it suitable for traffic target image segmentation;

[0016] 3) Take the left front view and right front view images with the same timestamp as network input to obtain the segmented image. The process is expressed as:

[0017] S=Seg(l img ) (1)

[0018] where l img ∈R W,H,3 For an RGB image with width W and height H, S∈R W,H,C is the output segmentation map, the segmentation map has the same width and height as the original image, and C is the total number of traffic target categories.

[0019] Furthermore, in step 2, the point clouds collected by the lidar at the same timestamp are projected onto the image planes of the left and right cameras respectively to obtain the category confidence of the corresponding projection positions in the left and right images. The main steps are as follows:

[0020] 1) Select the point cloud data with the corresponding timestamp from the point cloud queue collected by the lidar;

[0021] 2) Project the point cloud data onto the corresponding image plane. Specifically, first transform the point cloud points from the lidar coordinate system to the camera coordinate system, then rotate the camera within 15° based on the specific viewing angle, and finally project the points in 3D space onto the camera plane. The complete projection process is expressed as follows:

[0022] p c (i) = P rect R rect T L2c p L(i) (2)

[0023] Among them, p L (i)=(x i ,y i ,z i ,r i ), For any point among the N points in the point cloud, T L2c is the transformation matrix from the vehicle-mounted lidar to the camera, R rect is the rotation matrix of the camera with different viewing angles, P rect is the projection matrix for projecting onto the camera plane;

[0024] 3) According to the mapping position obtained in 2), the category confidence of the corresponding position in the semantic graph is indexed. The indexing process is expressed as:

[0025] s(i)=S[p c (i)[0],p c (i)[1]] (3)

[0026] The category confidence s(i) is the possibility that the target of the pixel after projection of the i-th point in the point cloud is background, car, pedestrian, non-motor vehicle, etc.

[0027] Furthermore, in step 3, a fusion correction method based on evidence theory is used. In the case where the confidence of the category information of the left front view and the right front view images is inconsistent, an evidence framework and basic confidence allocation are designed according to the pixel categories and confidence of the left and right views, and the Dempster fusion rule is used to implement image pixel fusion correction. The specific steps are as follows:

[0028] 1) For all possible class hypotheses, construct the evidence hypothesis Ω = {background, motor vehicle, pedestrian, non-motor vehicle} and the basic belief assignments (BBA). The BBA of the i-th point is defined as:

[0029]

[0030] 2) Consider the pixel semantic information from the left front view and the right front view cameras as two sets of evidence m l (·) and m r (·), based on visual similarity, we consider the complex categories with uncertainty: motor vehicle ∪ non-motor vehicle, pedestrian ∪ non-motor vehicle, and thus determine the recognition framework as

[0031] 3) Using Dempster's fusion rule, the two sets of evidence m l (·) and m r(·) Fusion correction is performed to obtain the fused BBA, as follows:

[0032]

[0033] where K is a discount factor, and is calculated as follows:

[0034]

[0035] Further, in step 4, according to the PointPainting fusion process, the category semantic information corresponding to each point in the point cloud is added to the information of the point cloud, and then the point cloud target detection network PointPillar is used to detect the position, speed, width, height, and orientation information of the traffic target. Specifically, the following steps are included:

[0036] 1) The result after evidence theory fusion correction is connected with the original point cloud point vector to expand the dimension of the original point cloud data. The expanded point cloud is denoted as p painted (·) Any point p painted (i) in the expanded point cloud is represented as

[0037] p painted (i) = (p L (i)[0], …, s(i)[0], …) (7)

[0038] 2) The PointPillar network is used to detect the expanded point cloud to obtain the position, speed, width, height, and orientation information of the traffic target. The specific implementation of PointPillar mainly includes three steps: (a) converting the point cloud into a pseudo image; (b) using a 2D backbone network to learn high-level representations; and (c) using a task network to implement the 3D box detection and regression task.

[0039] For the conversion of the point cloud into a pseudo image, first, the point cloud is divided into grids according to the X and Y axis positions of the point cloud. All point cloud points divided into the same grid form a pillar. The expanded point cloud needs to be further processed and represented as a D = 13-dimensional vector (x, y, z, r, x c y c z c x p y p s), where x c y c z c is the geometric center of all points of the pillar where the point cloud is located, and x p = x-x c y p = y-yp , represents the offset between a point cloud point and the geometric center of all points in the pillar, and s is a 4-dimensional vector containing the category semantic information carried by the expanded point cloud. Assuming that each point cloud frame has P non-empty pillars and each pillar has N point cloud data, the point cloud frame can be represented as a tensor of (D, P, N). To ensure that each pillar has N point cloud data, a strategy of selecting more and filling less is adopted. When the number of points in each pillar exceeds N, N points are randomly sampled; when the number of points in each pillar is less than N, the remaining points are padded with zero vectors. This achieves the conversion from irregular point cloud data to stacked pillars of a fixed shape. A simplified PointNet is then used to extract features from the pillars, resulting in a tensor of shape (C, P, N). A MaxPooling operation is then performed on the pillar dimension to obtain a feature map of shape (C, P). To implement pseudo-image features, P is converted to (H, W), resulting in a pseudo-image of shape (C, H, W).

[0040] For the backbone module of the detection network, progressive downsampling is mainly used to form pyramid features, and then the features at different levels are upsampled to the same size and finally spliced.

[0041] The task network uses SSD for 3D detection. Each 3DBBox is represented by a 7-dimensional vector (x, y, z, w, h, l, θ). (x, y, z) represents the center, w, h, l represent the object's dimensions, and θ represents the azimuth angle. The task network's regression task is to learn these seven parameters. The network's loss consists of three sub-losses: positioning loss, orientation loss, and classification loss.

[0042] The positioning loss is trained using the SmoothL1 loss function, which is defined as

[0043]

[0044] The specific position and shape parameters are calculated as follows:

[0045]

[0046]

[0047]

[0048] where d a The diagonal of the anchor.

[0049] To avoid the direction discrimination error, the Softmax loss is introduced to learn the direction of the traffic target, which is denoted as L dir . The classification of the traffic target adopts the Focal Loss, which is expressed as follows:

[0050] L cls = -α a (1-p a ) γ log p a (9)

[0051] In the detection network, the parameters are set as α = 0.25, γ = 2.

[0052] The total loss is defined as:

[0053]

[0054] Further, in step 5, the Kalman filter is used to predict the state information corresponding to the next time of the existing trajectory, which is specifically divided into the following steps:

[0055] 1) Use an 11-dimensional vector T = (x, y, z, θ, l, w, h, s, v x , v y , v z ) to describe the state of the traffic target trajectory, and the state of the associated trajectory at t-1 time is expressed as where m t-1 is the total number of trajectories at t-1 time;

[0056] 2) Establish a traffic target motion model to estimate the state at t time from the state at t-1 time. Since the time interval is short, the motion model is assumed to be a uniform speed model, so that

[0057] x est = x + v x , (11)

[0058] y est = y + v y , (12)

[0059] z est = z + v z (13)

[0060] Therefore, the i-th trajectory at t time is predicted as where i ∈ {1, 2, …, m t-1}.

[0061] Furthermore, step 6 uses the Hungarian algorithm to associate the trajectory and the detection information of the sensor. Different processing is performed according to the association result. For unassociated trajectories, if the time threshold is exceeded, it indicates that the tracked object has left the observation range, and the previously stored and maintained associated trajectory is deleted. For unassociated detection information, it indicates that the traffic target is a new target object that has just entered the observation range, so a new trajectory object is created and maintained. The associated trajectory and observation are used to update the state parameters of the vehicle target Kalman filter, so as to obtain the latest shape, motion, and trajectory information of the traffic target, and save the latest matched trajectory and output it. It is specifically divided into the following steps:

[0062] 1) Use the Hungarian algorithm to associate the maintained trajectory with the results of the target detector. The Hungarian algorithm creates a bipartite graph. One part of the bipartite graph is the trajectory predicted at the current moment in step 5, and the other part is the target detection result at the current moment. The cost metric for the edge between the trajectory and the detection result needs to be determined during graph construction. The main methods for determining this cost metric are: (a) Euclidean distance calculation. This is done by calculating the distance between the center point of the target detection box and the center point of the box in the trajectory state; (b) Intersection over Union (IoU) matching. This is done by calculating the intersection over union (IoU) of the target detection box and the box maintained in the trajectory state to further determine the edge weight; (c) Appearance similarity. In the first two methods, when the target overlaps and blocks objects, the shape of the target box will also change, resulting in inaccurate weights in the matching calculation. Appearance similarity uses a convolutional network to extract features and compare them with the features in the trajectory box. Combining IoU matching with visual similarity improves the accuracy of the weights. After the bipartite graph is established, the Hungarian matching algorithm is used to find a maximum match for this bipartite graph. The result is expressed as:

[0063]

[0064]

[0065]

[0066]

[0067] Where T match and D match is the matched trajectory set and detection set, T unmatch is the set of unmatched trajectories, D unmatch is the set of unmatched detections.

[0068] 2) The result of matching the trajectory and detection is used to update the current state value of the Kalman filter. From 1), the matching trajectory result obtained after data matching is According to the Kalman filter update formula, and After weighted average, the updated kth trajectory is represented as where k∈{1,2,…,w t}.

[0069] 3) For the trajectory T unmatch that has no match, there are two cases: (a) the traffic target has left the observation area; (b) the traffic target is still in the observation area, but the detector missed it or the traffic target is blocked. Therefore, in order to distinguish the two cases, a time threshold is set, when the time of the trajectory without match exceeds the threshold, the trajectory is deleted, indicating that the traffic target has left the observation area.

[0070] For the detection D unmatch that has no match, there are also two cases: (a) the traffic target has just entered the observation area, and there is no record of the trajectory before; (b) the traffic target has been in the observation area, but the detector missed it this time. Therefore, in order to distinguish the two cases, a frame number threshold is set, when the number of frames exceeds the threshold, there is still a detection without match, a new trajectory object is created and maintained, indicating that the traffic target starts to enter the observation area.

[0071] The beneficial effects of the present application are:

[0072] 1. The present application provides a method for improving automatic driving traffic target detection and tracking under the premise of accuracy and real-time;

[0073] 2. In view of the problem of high computational complexity and weak scalability of multi-source information fusion detection algorithm, the present application provides a method of using image semantics to expand point cloud information, which effectively reduces the fusion calculation complexity and improves the scalability;

[0074] 3. In view of the problem of traffic target tracking, the present application uses the Hungarian algorithm for data association, and Kalman filter for prediction and update, which ensures real-time while improving the accuracy of tracking. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 is a schematic diagram of various coordinate systems involved in the present application and the conversion relationship between them;

[0076] Figure 2 is a schematic diagram of the Hungarian algorithm for associating trajectories and target detection observation results in the present application;

[0077] Figure 3 is a result map of the detection result of the fusion target detection module in the present application;

[0078] Figure 4 is a result map of the tracking result of the target tracking module in the present application;

[0079] Figure 5 It is a structural block diagram of the present invention. DETAILED DESCRIPTION

[0080] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings and examples, so that the present invention can fully understand how to apply technical means to solve technical problems and achieve technical effects, and thus implement the invention accordingly. It should be noted that, as long as no conflict exists, the various embodiments of the present invention and the various features of the embodiments are combined with each other to form technical solutions that fall within the scope of protection of the present invention.

[0081] A traffic target detection and tracking method based on vehicle-mounted multi-sensor fusion, such as Figure 5 As shown, the following steps are included:

[0082] 1. Use the semantic segmentation network DeepLab V3+ to segment the images captured by the left and right onboard cameras, respectively, and obtain the category information corresponding to each pixel in the image;

[0083] 2. Project the point cloud collected by the lidar onto the image planes of the left and right cameras respectively to obtain the category confidence of the corresponding projection position in the left and right images;

[0084] 3. Using evidence theory as a fusion correction method, we address the problem of bias in category information from different viewpoints. We design an evidence framework and basic confidence allocation based on the pixel categories and confidence levels of the left and right viewpoints. We then use the Dempster fusion rule to achieve image pixel fusion correction, resulting in more accurate category information.

[0085] 4. According to the PointPainting fusion process, the category semantic information obtained from the image plane is added to the point cloud information, and then the point cloud object detection network PointPillar is used to detect the position, speed, width, height, and orientation information of the traffic object;

[0086] 5. Use Kalman filtering to predict the state information corresponding to the next moment trajectory for the existing trajectory;

[0087] 6. Based on the traffic target information from step 4 and the trajectory information from step 5, the Hungarian algorithm is used to correlate the trajectory with the sensor observations. Uncorrelated trajectories are deleted if they exceed a time threshold. New trajectories are created for uncorrelated detections. Correlated trajectories and observations are used to update the state parameters of the vehicle target Kalman filter, thereby deriving information about the shape, motion, and trajectory of the traffic target, thereby improving the accuracy and real-time performance of the autonomous vehicle's environmental perception.

[0088] Step 1: Obtain the semantic information of each pixel of the left and right view images from the semantic segmentation network DeepLab V3+. The main steps are as follows:

[0089] 1) First, find two pictures with corresponding timestamps from the picture queues collected by the left and right front cameras on the vehicle;

[0090] 2) Use the KITTI dataset to train the DeepLab V3+ model and fine-tune the model to make it suitable for traffic target image segmentation;

[0091] 3) Take the left front view and right front view images with the same timestamp as network input to obtain the segmented image. The process is expressed as:

[0092] S=Seg(l img ) (1)

[0093] where l img ∈R W,H,3 For an RGB image with width W and height H, S∈R W,H,C is the output segmentation map, the segmentation map has the same width and height as the original image, and C is the total number of traffic target categories.

[0094] Step 2: Project the point clouds collected by the lidar at the same timestamp onto the image planes of the left and right cameras to obtain the category confidence of the corresponding projection positions in the left and right images. The main steps are as follows:

[0095] 1) Select the point cloud data with the corresponding timestamp from the point cloud queue collected by the lidar;

[0096] 2) Project the point cloud data onto the corresponding image plane. Specifically, first transform the point cloud points from the lidar coordinate system to the camera coordinate system, then rotate the camera within 15° based on the specific viewing angle, and finally project the points in 3D space onto the camera plane. The complete projection process is expressed as follows:

[0097] p c (i) = P rect R rect T L2c p L (i) (2)

[0098] Among them, p L (i)=(x i ,y i ,z i ,r i ), For any point among the N points in the point cloud, T L2c is the transformation matrix from the vehicle-mounted lidar to the camera, R rectis the rotation matrix of the camera with different viewing angles, P rect is the projection matrix for projecting onto the camera plane;

[0099] 3) According to the mapping position obtained in 2), the category confidence of the corresponding position in the semantic graph is indexed. The indexing process is expressed as:

[0100] s(i)=S[p c (i)[0],p c (i)[1]] (3)

[0101] The category confidence s(i) is the possibility that the target of the pixel after projection of the i-th point in the point cloud is background, car, pedestrian, non-motor vehicle, etc.

[0102] Step 3: Using the fusion correction method based on evidence theory, in the case of inconsistent confidence levels of category information between the left and right front view images, we design an evidence framework and basic confidence allocation based on the pixel categories and confidence levels of the left and right views, and implement image pixel fusion correction using the Dempster fusion rule. The specific steps are as follows:

[0103] 1) For all possible class hypotheses, construct the evidence hypothesis Ω = {background, motor vehicle, pedestrian, non-motor vehicle} and the basic belief assignments (BBA). The BBA of the i-th point is defined as:

[0104]

[0105] 2) Consider the pixel semantic information from the left front view and the right front view cameras as two sets of evidence m l (·) and m r (·), based on visual similarity, we consider the complex categories with uncertainty: motor vehicle ∪ non-motor vehicle, pedestrian ∪ non-motor vehicle, and thus determine the recognition framework as

[0106] 3) Using Dempster's fusion rule, the two sets of evidence m l (·) and m r (·) performs fusion correction and obtains the fused BBA as shown below:

[0107]

[0108] Where K is the discount factor, which is calculated as follows:

[0109]

[0110] Step 4: According to the PointPainting fusion process, the category semantic information corresponding to each point in the point cloud is added to the point cloud information. Then, the point cloud object detection network PointPillar is used to detect the position, speed, width, height, and orientation information of the traffic object. Specifically, it is divided into the following steps:

[0111] 1) Connect the result after fusion correction using evidence theory with the point vector of the original point cloud to expand the dimension of the original point cloud data. The expanded point cloud is recorded as p painted (·), any point p in the expanded point cloud painted (i) Expressed as

[0112] p painted (i)=(p L (i)[0],…,s(i)[0],…) (7)

[0113] 2) Use the PointPillar network to perform object detection on the expanded point cloud, obtaining information such as the location, speed, width, height, and orientation of traffic targets. The specific implementation of PointPillar is mainly divided into three steps: (1) converting the point cloud into a pseudo-image; (2) using the 2D backbone network to learn high-level representations; and (3) the task network implements the detection and regression tasks of 3DBBox.

[0114] For the conversion of point cloud to pseudo image, we first divide the point cloud into grids according to the X and Y axis positions of the point cloud. All the point cloud data divided into the same grid form a pillar. The expanded point cloud needs to be further processed to represent D = 13-dimensional vectors (x, y, z, r, x c ,y c ,z c ,x p ,y p ,s), where x c ,y c ,z c is the geometric center of all points of the pillar where the point cloud is located, x p =xx c ,y p =yy p, represents the offset between a point cloud point and the geometric center of all points in the pillar, and s is a 4-dimensional vector containing the category semantic information carried by the expanded point cloud. Assuming that each point cloud frame has P non-empty pillars and each pillar has N point cloud data, the point cloud frame can be represented as a tensor of (D, P, N). To ensure that each pillar has N point cloud data, a strategy of selecting more and filling less is adopted. When the number of points in each pillar exceeds N, N points are randomly sampled; when the number of points in each pillar is less than N, the remaining points are padded with zero vectors. This achieves the conversion from irregular point cloud data to stacked pillars of a fixed shape. A simplified PointNet is then used to extract features from the pillars, resulting in a tensor of shape (C, P, N). A MaxPooling operation is then performed on the pillar dimension to obtain a feature map of shape (C, P). To implement pseudo-image features, P is converted to (H, W), resulting in a pseudo-image of shape (C, H, W).

[0115] For the backbone module of the detection network, progressive downsampling is mainly used to form pyramid features, and then the features at different levels are upsampled to the same size and finally spliced.

[0116] The task network uses SSD for 3D detection. Each 3DBBox is represented by a 7-dimensional vector (x, y, z, w, h, l, θ). (x, y, z) represents the center, w, h, l represent the object's dimensions, and θ represents the azimuth angle. The task network's regression task is to learn these seven parameters. The network's loss consists of three sub-losses: positioning loss, orientation loss, and classification loss.

[0117] The positioning loss is trained using the SmoothL1 loss function, which is defined as

[0118]

[0119] The specific position and shape parameters are calculated as follows:

[0120]

[0121]

[0122]

[0123] where d a The diagonal of the anchor.

[0124] In order to avoid direction discrimination errors, the Softmax loss is introduced to learn the direction of the traffic target, and the loss is recorded as Ldir The classification of traffic targets uses Focal Loss, which is expressed as follows:

[0125] L cls =-α a (1-p a ) γ log p a (9)

[0126] In the detection network, the parameters are set to α = 0.25, γ = 2.

[0127] The total loss is defined as:

[0128]

[0129] Step 5 uses Kalman filtering to predict the state information corresponding to the trajectory at the next moment. The specific steps are as follows:

[0130] 1) Using an 11-dimensional vector T = (x, y, z, θ, l, w, h, s, v x ,v y ,v z ) is used to describe the state of the traffic target trajectory, and the state of the associated trajectory at time t-1 is expressed as where m t-1 is the total number of trajectories at time t-1;

[0131] 2) Establish a traffic target motion model and estimate the state at time t from time t-1. Due to the short time interval, the motion model is assumed to be a uniform speed model, thus obtaining

[0132] x est =x+v x , (11)

[0133] y est =y+v y , (12)

[0134] z est =z+v z (13)

[0135] Therefore, the i-th trajectory prediction at time t is where i∈{1,2,…,m t-1}.

[0136] In step 6, the Hungarian algorithm is used to associate the trajectory and the detection information of the sensor. Different processing is performed according to the association result. For unassociated trajectories, if the time threshold is exceeded, it indicates that the tracked object has left the observation range, and the previously stored and maintained associated trajectory is deleted. For unassociated detection information, it indicates that the traffic target is a new target object that has just entered the observation range, so a new trajectory object is created and maintained. The associated trajectory and observation are used to update the state parameters of the vehicle target Kalman filter, so as to obtain the latest shape, motion, and trajectory information of the traffic target, and save the latest matched trajectory and output it. It is divided into the following steps:

[0137] 1) Use the Hungarian algorithm to associate the maintained trajectory with the results of the object detector. The Hungarian algorithm builds a bipartite graph. One part of the bipartite graph is the trajectory predicted at the current moment in step 5), and the other part is the result of the object detection at the current moment. The cost metric for the edge between the trajectory and the detection result needs to be determined during graph construction. The main methods for determining this cost metric are: (a) Euclidean distance calculation. This is done by calculating the distance between the center point of the target detection box and the center point of the box in the trajectory state; (b) Intersection over Union (IoU) matching. This is done by calculating the intersection over union (IoU) of the target detection box and the box maintained in the trajectory state to further determine the edge weight; (c) Appearance similarity. In the first two methods, if the target overlaps or blocks, the shape of the target box will also change, resulting in inaccurate matching weights. Appearance similarity uses a convolutional network to extract features and compare them with the features in the trajectory box. Combining IoU matching with visual similarity improves the accuracy of the weights. After the bipartite graph is built, the Hungarian matching algorithm is used to find a maximum matching of this bipartite graph. The result is expressed as:

[0138]

[0139]

[0140]

[0141]

[0142] Where T match and D match is the matched trajectory set and detection set, T unmatch is the set of unmatched trajectories, D unmatch is the set of unmatched detections.

[0143] 2) The result of matching the trajectory and detection is used to update the current state value of the Kalman filter. From 1), the matching trajectory result obtained after data matching is According to the Kalman filter update formula, and After weighted averaging, the updated k-th trajectory is expressed as where k∈{1,2,…,w t}.

[0144] 3) For the trajectory T without matching unmatch There are two situations: (a) the traffic target has left the observation area; (b) the traffic target is still in the observation area, but the detector missed it or the traffic target is blocked. Therefore, to distinguish between these two situations, a time threshold is set. When the duration of the unmatched track exceeds the threshold, the track is deleted, indicating that the traffic target has left the observation area.

[0145] For detection D without a match unmatch There are two cases: (a) the traffic target has just entered the observation area and there is no previous trajectory record; (b) the traffic target is already in the observation area, but the detector detected it incorrectly. Therefore, to distinguish these two cases, a frame number threshold is set. If there is still no matching detection after exceeding this threshold, a new trajectory object is created and maintained, indicating that the traffic target has entered the observation area.

[0146] Figure 3 and Figure 4 This is a diagram of the experimental results of the present invention.

[0147] Figure 3 This image shows the result of using the PointPainting fusion object detection process. A semantic segmentation algorithm is used to obtain semantic information from images captured by the vehicle's camera. The point cloud information collected by the lidar is projected onto the image plane to obtain semantic information about the corresponding locations. This results in expanded point cloud data with semantic information. A point cloud object detector is then used to detect the position, speed, width, height, and orientation of traffic objects. The resulting image shows that the detector based on the PointPainting fusion process can efficiently and accurately detect traffic objects.

[0148] Figure 4 This graph shows the results of using the Hungarian algorithm to correlate the trajectory with sensor observations and then using the correlated detection information to update the vehicle's trajectory state using a Kalman filter. The graph shows that the 3D Kalman filter tracking method is effective for tracking traffic targets.

[0149] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A traffic target detection and tracking method based on vehicle-mounted multi-sensor fusion, characterized by: The steps include: S1, using the semantic segmentation network DeepLabV3+, segmented the images captured by the left and right onboard cameras respectively to obtain the category information corresponding to each pixel in the image; S2, projects the point cloud collected by the lidar onto the image planes of the left and right cameras respectively to obtain the category confidence of the corresponding projection position in the left and right images; S3 uses evidence theory as a fusion correction method to address the problem of bias in category information from different perspectives. It designs an evidence framework and basic confidence allocation based on the pixel categories and confidence levels of the left and right perspectives, and uses the Dempster fusion rule to achieve image pixel fusion correction, resulting in more accurate category information. S4, according to the PointPainting fusion process, adds the category semantic information obtained from the image plane to the point cloud information, and then uses the point cloud object detection network PointPillar to detect the position, speed, width, height and orientation information of the traffic object; S5, use Kalman filtering to predict the state information corresponding to the next moment trajectory for the existing trajectory; S6, based on the traffic target information obtained in step S4 and the trajectory information obtained in step S5, the Hungarian algorithm is used to associate the trajectory and the detection information observed by the sensor; for unassociated trajectories, if they exceed the time threshold, they are deleted; for unassociated detection information, new trajectories are created; the associated trajectories and observations are used to update the state parameters of the vehicle target Kalman filter, thereby deriving the shape, motion and trajectory information of the traffic target to improve the accuracy and real-time performance of the autonomous driving vehicle's environmental perception.

2. The method for traffic target detection and tracking based on vehicle-mounted multi-sensor fusion according to claim 1, characterized in that: In step S1, the semantic information of each pixel of the left and right view images is obtained from the semantic segmentation network DeepLabV3+. The specific steps are as follows: 1) First, find two pictures with corresponding timestamps from the picture queues collected by the left and right front cameras on the vehicle; 2) Use the KITTI dataset to train the DeepLabV3+ model and fine-tune the model to make it suitable for traffic target image segmentation; 3) Take the left front view and right front view images with the same timestamp as network input to obtain the segmented image. The process is expressed as: S=Seg(l img )#(1) in For an RGB image with width W and height H, S∈R W，H，C is the output segmentation map, the segmentation map has the same width and height as the original image, and C is the total number of traffic target categories.

3. The method for traffic target detection and tracking based on vehicle-mounted multi-sensor fusion according to claim 1, characterized in that: In step S2, the point clouds collected by the lidar at the same timestamp are projected onto the image planes of the left and right cameras respectively to obtain the category confidence of the corresponding projection positions in the left and right images. The specific steps are as follows: 1) Select the point cloud data with the corresponding timestamp from the point cloud queue collected by the lidar; 2) Project the point cloud data onto the corresponding image plane. Specifically, first transform the point cloud points from the LiDAR coordinate system to the camera coordinate system, then rotate the camera by an angle of less than 15° based on the specific viewing angle, and finally project the points in 3D space onto the camera plane. The complete projection process is expressed as follows: p c (i)=P rect R rect T L2c p L (i)#(2) in For any point among the N points in the point cloud, T L2c is the transformation matrix from the vehicle-mounted lidar to the camera, R rect is the rotation matrix of the camera with different viewing angles, P rect is the projection matrix for projecting onto the camera plane; 3) According to the mapping position obtained in 2), the category confidence of the corresponding position in the semantic graph is indexed. The indexing process is expressed as: s(i)=S[p c (i)[0],p c (i)[1]]#(3) The category confidence s(i) is the probability that the target of the pixel after projection of the i-th point in the point cloud is background, car, pedestrian, or non-motor vehicle.

4. The method for traffic target detection and tracking based on vehicle-mounted multi-sensor fusion according to claim 1, characterized in that: In step S3, a fusion correction method based on evidence theory is used. In the case where the confidence of the category information of the left front view and the right front view images is inconsistent, an evidence framework and basic confidence allocation are designed according to the pixel categories and confidence of the left and right views, and the Dempster fusion rule is used to implement image pixel fusion correction. The specific steps are as follows: 1) For all possible category hypotheses, construct the evidence hypothesis Ω = {background, motor vehicle, pedestrian, non-motor vehicle} and the basic confidence distribution; the basic confidence distribution of the i-th point is defined as: 2) Consider the pixel semantic information from the left front view and the right front view cameras as two sets of evidence m l (·) and m r (·), based on visual similarity, we consider the complex categories with uncertainty: motor vehicle ∪ non-motor vehicle, pedestrian ∪ non-motor vehicle, and thus determine the recognition framework as 3) Using Dempster's fusion rule, the two sets of evidence m l (·) and m r (·) performs fusion correction and obtains the basic reliability distribution after fusion, as shown below: Where K is the discount factor, which is calculated as follows:

5. The method for traffic target detection and tracking based on vehicle-mounted multi-sensor fusion according to claim 1, characterized in that: In step S4, according to the PointPainting fusion process, the category semantic information corresponding to each point in the point cloud is added to the point cloud information, and then the point cloud object detection network PointPillar is used to detect the position, speed, width, height, and orientation information of the traffic object. Specifically, it is divided into the following steps: 1) Connect the result after fusion correction using evidence theory with the point vector of the original point cloud to expand the dimension of the original point cloud data. The expanded point cloud is recorded as p painted (·), any point in the expanded point cloud Expressed as p painted (i)=(p L (i)[0],...,s(i)[0],...)#(7) 2) Use the PointPillar network to perform object detection on the expanded point cloud to obtain the location, speed, width, height, and orientation information of traffic targets. The specific implementation of PointPillar is divided into three steps: (a) converting the point cloud into a pseudo-image; (b) using the 2Dbackbone network to learn high-level representations; (c) the task network implements the 3DBBox detection and regression tasks; For the conversion of point cloud to pseudo image, the point cloud is first divided into grids according to the X and Y axis positions of the point cloud. All the point cloud data divided into the same grid constitute a Pillar. The expanded point cloud needs to be further processed to represent a D = 13-dimensional vector (x, y, z, r, x c ,y c ,z c ,x p ,y p ,s), where x c ,y c ,z c is the geometric center of all points of the pillar where the point cloud is located, x p =xx c and y p =yy p It represents the offset between the point cloud point and the geometric center of all points in the Pillar, and s is a 4-dimensional vector of the category semantic information carried by the extended point cloud. Assuming that each frame of the point cloud has P non-empty pillars and each pillar has N point cloud data, the point cloud of this frame is represented as a tensor of (D, P, N). In order to ensure that each pillar has N point cloud data, the strategy of selecting more and supplementing less is adopted. When the number of point clouds in each pillar exceeds N, N points are randomly sampled. When the number of point clouds in each pillar is less than N, the remaining points are filled with zero vectors. This achieves the conversion from irregular point cloud data to stacked pillars with a fixed shape. Then, a simplified PointNet is used to extract features from the Pillars, obtaining a tensor with a shape of (C, P, N). Then, a MaxPooling operation is performed on the dimension where the pillar is located to obtain a feature map with a shape of (C, P). In order to realize the pseudo-image feature, P is converted to (H, W), thereby obtaining a pseudo-image with a shape of (C, H, W). For the backbone module of the detection network, progressive downsampling is mainly used to form pyramid features, and then the features at different levels are upsampled to the same size and finally spliced; For the task network, SSD is used for 3D detection. Each 3DBBox is represented by a 7-dimensional vector (x, y, z, w, h, l, θ), where (x, y, z) is the center, w, h, l are the dimensions of the target, and θ is the azimuth. The regression task of the task network is to learn these seven parameters. The network loss consists of three sub-losses: positioning loss, orientation loss, and category loss. The positioning loss is trained using the SmoothL1 loss function, which is defined as The specific position and shape parameters are calculated as follows: Δθ=sin(θ gt -θ a ) where d a is the diagonal of the anchor; In order to avoid direction discrimination errors, the Softmax loss is introduced to learn the direction of the traffic target, and the loss is recorded as L dir ; The classification of traffic targets uses FocalLoss, which is expressed as follows: L cls =-a a (1-p a ) γ log p a #(9) In the detection network, the parameters are set to α = 0.25, γ = 2; The total loss is defined as:

6. The method for traffic target detection and tracking based on vehicle-mounted multi-sensor fusion according to claim 1, characterized in that: In step S5, the Kalman filter is used to predict the state information corresponding to the next moment trajectory based on the existing trajectory. The specific steps are as follows: 1) Using an 11-dimensional vector T = (x, y, z, θ, l, w, h, s, v x , v y , v z ) is used to describe the state of the traffic target trajectory. The state of the associated trajectory at time t-1 is expressed as where m t-1 is the total number of trajectories at time t-1; 2) Establish a traffic target motion model and estimate the state at time t from time t-1; due to the short time interval, the motion model is assumed to be a uniform speed model, thus obtaining x est =x+v x ,#(11) and est =y+v y ,#(12) with est =with+in z ,#(13) Therefore, the i-th trajectory prediction at time t is where i∈{1, 2, ..., m t-1 }.

7. The method for traffic target detection and tracking based on vehicle-mounted multi-sensor fusion according to claim 1, characterized in that: In step S6, the Hungarian algorithm is used to associate the trajectory with the sensor detection information. Different processing is performed based on the association result. For unassociated trajectories, if the time threshold is exceeded, indicating that the unassociated trajectory has left the observation range, the previously stored and maintained associated trajectory is deleted. For unassociated detection information, it indicates that the traffic target has just entered the observation range, so a new trajectory object is created and maintained. The associated trajectories and observations are used to update the state parameters of the vehicle target Kalman filter, thereby obtaining the latest shape, motion, and trajectory information of the traffic target, and saving the latest matched trajectory and outputting it. The specific steps are as follows: 1) Use the Hungarian algorithm to associate the maintained trajectory with the results of the target detector. The Hungarian algorithm creates a bipartite graph. One part of the bipartite graph is the trajectory predicted at the current moment in step S5, and the other part is the target detection result at the current moment. The weight CostMetric of the edge between the trajectory and the detection result needs to be determined during the graph construction. The methods for determining it are: (a) calculating the Euclidean distance; by calculating the distance between the center point of the target detection box and the center point of the box in the trajectory state; (b) IoU matching; by calculating the intersection over union ratio of the target detection box and the box maintained in the trajectory state, the edge weight is further determined. (c) Appearance similarity. When the target overlaps and blocks the target, the shape of the target box will change, resulting in inaccurate weights for the matching calculation. Appearance similarity uses a convolutional network to extract features and compare them with the features in the trajectory box. Combining IoU matching and visual similarity improves the accuracy of the weights. After building a bipartite graph, the Hungarian matching algorithm is used to find a maximum match for this bipartite graph. The result is expressed as: Where T match and D match is the matched trajectory set and detection set, T unmatch is the set of unmatched trajectories, D unmatch is the set of unmatched detections; 2) The result of matching the trajectory and detection is used to update the current state value of the Kalman filter; According to 1), the matching trajectory result obtained after data matching is According to the Kalman filter update formula, and After weighted averaging, the updated k-th trajectory is expressed as where k∈{1, 2, ..., w t }; 3) For the trajectory T without matching unmatch There are two situations: (a) the traffic target has left the observation area; (b) the traffic target is still in the observation area, but the detector misses it or the traffic target is blocked. Therefore, in order to distinguish these two situations, a time threshold is set. When the time without a matching track exceeds the threshold, the track is deleted, indicating that the traffic target has left the observation area. For detection D without a match unmatch There are also two situations: (a) the traffic target has just entered the observation area and there is no previous trajectory record; (b) the traffic target is already in the observation area, but the detector detects it incorrectly this time; therefore, in order to distinguish these two situations, a frame number threshold is set. When this threshold is exceeded and there is still no matching detection, a new trajectory object is created and maintained, indicating that the traffic target begins to enter the observation area.

Images