A multi-target tracking method based on multi-modal data

Abstract

The application belongs to the technical field of target tracking, and is a multi-target tracking method based on multi-modal data, comprising the following steps: (1) collecting point clouds and camera images of each frame and fusing them into color point clouds, (2) constructing a height-intensity-density image by using the color point clouds of each frame, (3) performing three-dimensional target detection on the color point clouds of each frame, (4) performing target position prediction of the t+1 frame by using a rotation kernel correlation filter, (5) performing data association on the detected targets and the predicted targets of the t+1 frame, (6) updating the tracking targets of the t+1 frame, (7) performing target reservation on the unmatched predicted targets of the t+1 frame, (8) performing target re-identification on the unmatched detected targets of the t+1 frame, and (9) repeating steps 4-8 to process the next frame. The application establishes a multi-target tracking framework by using three-dimensional laser point clouds and camera image multi-modal information, solves the target occlusion problem, and improves the accuracy of the multi-target tracking algorithm.

Description

TECHNICAL FIELD

[0001] The application relates to a multi-target tracking method based on multi-modal data and belongs to the technical field of target tracking. BACKGROUND

[0002] Target tracking technology is a hot research topic in computer vision and one of the most critical technologies in intelligent driving, which provides the position and motion trajectory of the target of interest in the traffic environment and is the premise of subsequent target recognition, target behavior analysis and understanding, and lays a foundation for obstacle avoidance, path planning and adaptive cruise.

[0003] According to the number of tracked objects, target tracking is divided into single-target tracking and multi-target tracking. Since most application scenarios contain a large number of objects of interest, multi-target tracking has become an important research field. With the substantial improvement in the performance of target detection technology, the tracking-by-detection (TBD) framework based on detection has become mainstream, which takes the target of interest obtained by the target detector as input and designs a target matching mechanism to associate the same target in the subsequent frame to the track.

[0004] Most existing multi-target tracking methods are based on images, but images are easily affected by light and occlusion, resulting in tracking drift and tracking errors. The rapid improvement of three-dimensional laser scanning technology can accurately and effectively digitize the shape and surface features of objects. The three-dimensional laser point cloud obtained by the laser scanner provides geometric information of the sampling points on the surface of the physical object and is not affected by light, which is an effective supplement to images and helps to achieve accurate positioning and tracking. Therefore, the use of multi-modal information such as three-dimensional laser point cloud and camera image can avoid the above problems, effectively realize information complementation, and improve the accuracy of multi-target tracking. SUMMARY

[0005] The purpose of the present application is to provide a multi-target tracking method based on multi-modal data, which uses multi-modal information such as three-dimensional laser point cloud and camera image to establish an efficient and stable multi-target tracking framework, solves the problem of target occlusion, and further improves the accuracy of the multi-target tracking algorithm.

[0006] In order to achieve the above-mentioned application purpose and solve the problems existing in the prior art, the technical scheme adopted by the present application is as follows: a multi-target tracking method based on multi-modal data, comprising the following steps:

[0007] Step 1, fuse the point cloud and camera image of each frame into a color point cloud, which comprises the following sub-steps:

[0008] (a) In each frame, laser point cloud and camera image are collected by laser scanner and camera respectively, the geometric mapping relationship between laser point cloud and camera image is described by formula (1),

[0009]

[0010] wherein, denotes a scaling factor, A denotes an intrinsic matrix, [R, t] denotes an extrinsic matrix between laser scanner and camera, which is obtained by laser scanner and camera calibration according to pinhole camera model, denotes the coordinates of a three-dimensional laser point in the point cloud coordinate system, denotes the coordinates of the three-dimensional point after being projected to the camera image in the image pixel coordinate system, and denote their homogeneous coordinates;

[0011] (b) The field of view ranges of laser scanner and camera are different, some laser points beyond the field of view of the camera cannot be projected to the imaging plane, forming an achromatic point cloud; the laser points located within the field of view of the camera can be projected to the imaging plane and colored, forming a chromatic point cloud wherein, denotes the cth chromatic laser point, (x c , y c , z c ) denotes the coordinates of the cth chromatic laser point in the point cloud coordinate system, denotes the reflection intensity of the cth chromatic laser point, R c = R(u c , v c ), G c = G(u c , v c ), B c = B(u c , v c ) denotes the three primary colors of the projection e c = (u c , v c ) of p c in the camera image, n c denotes the number of chromatic laser points;

[0012] Step 2, for each frame, a height-intensity-density (HID) image is constructed using the chromatic point cloud, specifically including the following sub-steps:

[0013] (a) A two-dimensional grid is established in the point cloud coordinate system xoy plane, the chromatic point cloud P is projected into the two-dimensional grid cells, the points in the two-dimensional grid cells are defined as, described by formula (2),

[0014] C(u, v) = {p c|p c ∈P, x min +(u-1)q≤x c ≤x min +uq,

[0015] y min +(v-1)q≤y c ≤ym in +vq} (2)

[0016] where u and v represent the column number and row number of the two-dimensional grid cell respectively, x min and y min represent the minimum value of the x coordinate and y coordinate of the laser point respectively, q represents the size of the two-dimensional grid cell, the value range of u and v is defined as where x max and y max represent the maximum value of the x coordinate and y coordinate of the laser point respectively, the operator is defined as the upward rounding, and represent the numerical lower bound of the u-axis and v-axis of the two-dimensional grid cell respectively;

[0017] (b) calculating the height feature h(u, v) of each two-dimensional grid cell, which is described by formula (3),

[0018]

[0019] where z max and z min are the maximum value and minimum value of the z coordinate of the laser point respectively;

[0020] (c) calculating the reflection intensity feature i(u, v) of each two-dimensional grid cell, which is described by formula (4),

[0021]

[0022] (d) calculating the density feature d(u, v) of each two-dimensional grid cell, which is described by formula (5),

[0023]

[0024] where N uv represents the number of points in the two-dimensional grid cell C(u, v);

[0025] (e) setting r h , r i , r d represent the weight of the height feature, the reflection intensity feature, and the density feature respectively, according to r h: r i : r d The above three features are processed with weights of 1:2:1 to balance the feature intensity, and are normalized to the range of 0-255, described by equations (6) to (10),

[0026] hid max = max{[h(u, v)r h , i(u, v)r i , d(u, v)r d ]} (6)

[0027] hid min = min{[h(u, v)r h , i(u, v)r i , d(u, v)r d ]} (7)

[0028]

[0029]

[0030]

[0031] Finally, three feature maps are obtained: respectively representing the height map, the intensity map and the density map, and the three feature maps are combined into a new three-channel feature map, called height-intensity-density (HID) image;

[0032] Step 3, three-dimensional target detection is performed on each frame using the color point cloud. In each frame, the color point cloud P is input into the Point-GNN target detector for three-dimensional target detection to obtain a three-dimensional detection box of the detected target, and the three-dimensional detection box is projected onto the xoy plane to obtain a two-dimensional detection box in the HID image. Specifically, the two-dimensional detection box in the t-th frame HID image is represented as wherein, Bt represents the i-th two-dimensional detection box in the t-th frame HID image, Nt represents the number of two-dimensional detection boxes in the t-th frame HID image;

[0033] Step 4, the RKCF (Rotated Kernel Correlation Filter) is used to predict the target position in the t+1-th frame. The RKCF is trained using the t-th frame HID image to predict the target position, and the predicted target in the t+1-th frame HID image is obtained. Specifically, the following sub-steps are included:

[0034] (a) Initialize the tracking target: the two-dimensional tracking box in the t-th frame HID image Ω t is defined as B t = {bts = (u ts , v ts , l ts , w ts , γ ts ) | 1≤s≤n t}, wherein b ts denotes the s-th two-dimensional tracking box in the t-th frame of HID image, (u ts , v ts ), (l ts , w ts ) and γ ts denote the center coordinates, size and direction angle of the two-dimensional tracking box respectively, n t denotes the number of two-dimensional tracking boxes in the t-th frame of HID image, and if t = 1, the two-dimensional detection box in the t-th frame is used initializing the two-dimensional tracking box in the t-th frame;

[0035] (b) rotating the HID image in the t-th frame: for each two-dimensional tracking box b ts , taking the tracking box center (u ts , v ts ) as the rotation center point and θ ts as the rotation angle, the HID image Ω t is rotated to obtain the rotated HID image Ω wherein if 0° < γ ts ≤ 90°, then θ ts = 90°- γ ts , Ω t is counterclockwise rotated; if 90° < γ ts ≤ 180°, then θ ts = 90°- γ ts , Ω t is clockwise rotated; if -180°≤ γ ts ≤ 90°, then θ ts = -90°- γ ts , Ω t is counterclockwise rotated; if -90° < γ ts ≤ 0°, then θ ts = -90°- γ ts , Ω t is clockwise rotated;

[0036] (c) training the classifier in the t-th frame: for each two-dimensional tracking box b ts , according to the 2.5 times of the center coordinates (u ts , v ts ) and size (l ts , w ts ) of the two-dimensional tracking box, the classifier is trained Image cropping is performed to obtain training base samples. The base samples are cyclically shifted using a permutation matrix to obtain training samples. The base samples are positive samples, and the rest are negative samples. Each sample is associated with an output that follows a two-dimensional Gaussian distribution. A classifier is trained using the training samples and the corresponding output.

[0037] (d) Rotate the HID image at frame t+1: At frame t+1, obtain the HID image Ω according to step 2. t+1 For each two-dimensional tracking box b (t+1)s , with (u ts v ts Let θ be the center of rotation. ts For the rotation angle, Ω t+1 Rotate to obtain a rotated HID image.

[0038] (e) Predict the target position in frame t+1: based on the two-dimensional tracking box b in frame t. ts center coordinates (u ts v ts ) and size (l ts w ts 2.5 times that of Image cropping is performed to obtain the test base sample. The sample is then cyclically shifted to obtain the test sample. The predicted response is calculated using the classifier trained in sub-step (c). The row number a and column number b where the response peak is located represent the distance the tracked target moves along the u-axis and v-axis in the two adjacent frames of rotated HID images, respectively.

[0039] (f) Rotate the HID image The predicted coordinates are then restored to the original HID image Ω. t+1 In the middle, the s-th two-dimensional tracking box b in the (t+1)-th frame (t+1)s The calculation of the center coordinates is described by formula (11).

[0040]

[0041] The tracked target in frame t+1 predicted by RKCF is denoted as the predicted target, and its two-dimensional prediction bounding box is represented as follows: in, For the j-th two-dimensional prediction box in the (t+1)-th HID image, The number of two-dimensional prediction boxes in the (t+1)th frame of the HID image;

[0042] Step 5: Perform data association between the detected and predicted targets in frame t+1: use the detection bounding boxes in frame t+1. With prediction box Establish the association cost matrix in, the number of two-dimensional detection boxes in the t+1th frame of HID image, the number of two-dimensional prediction boxes in the t+1th frame of HID image, m (t+1)i，j is determined by and IOU(i, j), which is described by formula (12),

[0043]

[0044] wherein, IOU(i, j) represents the intersection over union of the i-th detection box and the j-th prediction box in the t+1th frame, which is described by formula (13),

[0045]

[0046] wherein, represents the prediction score of the j-th prediction box in the t+1th frame, and is set as 1, and is based on the association cost matrix M t+1 , the detection target and the prediction target in the t+1th frame are associated by using the Kuhn-Munkres data association algorithm, and the detection target and the prediction target are divided into unmatched prediction targets, unmatched detection targets, and prediction-detection target matching pairs;

[0047] Step 6, updating the tracking target in the t+1th frame: for the prediction-detection target matching pair in the t+1th frame, the two-dimensional detection box of the detection target is used to update the tracking target corresponding to the prediction target;

[0048] Step 7, target reservation for the unmatched prediction target in the t+1th frame, specifically including the following sub-steps:

[0049] (a) obtaining the camera image clip of the unmatched prediction target: defining the unmatched prediction target in the t+1th frame as wherein, represents the two-dimensional prediction box of the i-th unmatched prediction target in the t+1th frame of HID image, represents the number of unmatched prediction targets in the t+1th frame of HID image, and the three-dimensional bounding box of the i-th unmatched prediction target in the t+1th frame is projected onto the t+1th frame of camera image and cropped to obtain the camera image clip the three-dimensional bounding box corresponding to the target in the t-th frame is projected onto the t-th frame of camera image and cropped to obtain the camera image clip

[0050] (b) target reservation by appearance similarity score: the appearance similarity score of and is calculated by using the mean hash algorithm ​​To determine whether the prediction of the unmatched target is correct, if the appearance similarity score is... This indicates the prediction is correct. The two targets in frame t and frame t+1 belong to the same object. The correctly predicted target in frame t+1 is retained, and its 2D prediction bounding box is used. Update the tracking target; if the appearance similarity score This indicates a prediction error; the target is moved to the list of untracked targets and awaits target re-identification. Where T... s To retain an appearance score threshold for the target, a predefined T is used. s =120;

[0051] (c) Prediction score calculation: A prediction score is given for each prediction target. For a correctly predicted target, the initial prediction score is set to 0.5. As the number of predictions increases, the prediction score is gradually reduced, as described by formula (14).

[0052]

[0053] Where δ is the descent factor, predefined as δ = 0.05. This represents the prediction score of the j-th prediction box in frame t;

[0054] Step 8: Re-identify unmatched targets in frame t+1, which includes the following sub-steps:

[0055] (a) Obtain a crop of the camera image of the unmatched detected target in frame t+1: Define the unmatched detected target in frame t+1 as... in, Represents the (t+1)th HID image in the t-th frame. A two-dimensional detection box that does not match the target. Let be the number of unmatched detected targets in the (t+1)th frame of the HID image. The 3D bounding boxes of each unmatched detected target are projected onto the camera image and cropped to obtain a cropped image of the camera image.

[0056] (b) Obtain a cropped image of the camera that failed to track the target: Define the 2D tracking bounding box before the target failed to be tracked. in, For the k-th untracked target, the 2D tracking bounding box is... This represents the number of targets that were not successfully tracked. The corresponding camera image crop is

[0057] (c) Target re-identification using appearance similarity score and spatial distance score: Calculated using the mean hash algorithm. and Appearance similarity score if This indicates that the two targets belong to the same object; if the appearance similarity score Then use the spatial distance fraction between the two. Further analysis reveals that T... h T l These represent the highest and lowest thresholds for appearance scores in target re-identification, respectively, with T predefined. h =200, T l =160; if This indicates that the two targets belong to the same object, and the distance threshold is met. Among them l m The maximum distance a target can move in two consecutive frames is predefined as l. m =25, The number of frames between the two targets;

[0058] (d) If it is determined that two targets belong to the same object, then the unmatched detected target is considered a correctly detected target or a reappearing target, and its two-dimensional detection box is used. Update the tracked target to continue prediction in the next frame; otherwise, indicate it is a newly appearing target and use its 2D detection box. Initialize a new tracking target. In addition, if a target fails to be tracked for 5 consecutive frames, it is considered to have permanently disappeared and the target is deleted.

[0059] Step 9: Repeat steps 4-8 above to process the next frame until the tracking ends.

[0060] The present application has the advantages that: a multi-target tracking method based on multi-modal data, comprising the following steps: (1) collecting point cloud and camera image of each frame and fusing into color point cloud, (2) constructing height-intensity-density (HID) image by using color point cloud for each frame, (3) performing three-dimensional target detection by using color point cloud for each frame, (4) performing target position prediction of the t+1 frame by using a rotated kernel correlation filter (RKCF), (5) performing data association on the detected target and the predicted target of the t+1 frame, (6) updating the tracking target of the t+1 frame, (7) performing target reservation on the unmatched predicted target of the t+1 frame, (8) performing target re-identification on the unmatched detected target of the t+1 frame, (9) repeating the above steps 4-8 to process the next frame until the tracking ends. Compared with the prior art, the present application has the following advantages: first, the present application uses laser point cloud to construct HID image, effectively overcoming the influence of object occlusion in target tracking; second, the present application uses camera image to establish a target reservation module, checks whether the prediction is correct and reserves the correctly predicted object, so that in the case of missed detection, the prediction can still be continued, and the tracking sequence integrity and the reliability of the prediction result are ensured; third, the present application uses camera image to establish a target re-identification module, avoiding repeated creation of new trajectories and improving tracking performance. It not only can compensate for the wrong prediction, but also can handle the case that the tracked object disappears temporarily and then reappears, which is conducive to restoring the interrupted tracking sequence; fourth, the present application makes full use of laser point cloud and camera image information, so that the multi-modal information is effectively complementary. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a method step flow chart of the present application.

[0062] Figure 2 is a multi-modal information schematic diagram.

[0063] In the figure, (a) is a camera image diagram, (b) is a laser point cloud diagram, (c) is a colorless point cloud diagram, and (d) is a color point cloud diagram.

[0064] Figure 3 is a pinhole camera model schematic diagram of laser scanner and camera calibration.

[0065] Figure 4 is a HID image construction schematic diagram.

[0066] In the figure, (a) is a height diagram, (b) is a reflection intensity diagram, (c) is a density diagram, and (d) is a HID image diagram.

[0067] Figure 5 is a HID image rotation schematic diagram.

[0068] In the figure, (a) is 0°<γ tsFigure (a) is a rotation method diagram when γ ≤ 90°, (b) is 90° < γ ts Figure (c) is a rotation method diagram when γ ≤ 180°, (d) is 180° < γ ts Figure (a) is a rotation method diagram when γ ≤ 90°, (b) is -90° < γ ts Figure (a) is a rotation method diagram when γ ≤ 0°. DETAILED DESCRIPTION

[0069] The application will be further described below with reference to the drawings.

[0070] As Figure 1 shown, a multi-target tracking method based on multi-modal data includes the following steps:

[0071] Step 1, acquire point cloud and camera image for each frame and fuse them into color point cloud, which specifically includes the following sub-steps:

[0072] (a) In each frame, laser point cloud and camera image are acquired by laser scanner and camera respectively, as shown in Figure 2 (b), the geometric mapping relationship of laser point cloud and camera image is described by formula (1),

[0073]

[0074] wherein, denotes a scaling factor, A denotes an intrinsic matrix, [R, t] denotes an extrinsic matrix between laser scanner and camera, which is obtained by laser scanner and camera calibration according to pinhole camera model, denotes the coordinates of a three-dimensional laser point in the point cloud coordinate system, denotes the coordinates of the three-dimensional point after being projected to the camera image in the image pixel coordinate system, and denote their homogeneous coordinates; the intrinsic matrix A and the extrinsic matrix [R, t] are obtained by laser scanner and camera calibration according to pinhole camera model, and the pinhole camera model according to which the calibration is performed is shown in Figure 3 .

[0075] (b) The field of view ranges of the laser scanner and the camera are different, and some laser points beyond the field of view of the camera cannot be projected to the imaging plane, forming achromatic point cloud, as shown in Figure 2 (c). While the laser points within the field of view of the camera can be projected to the imaging plane and colored by the corresponding camera image pixel points, forming color point cloud as shown in Figure 2 (d). denotes the cth color laser point, (x c , y c , z c ) denotes the coordinates of the cth color laser point in the point cloud coordinate system, R represents the reflection intensity of the c-th colored laser point. c =R(u c v c ), G c =G(u c v c ), B c =B(u) c v c ) represents p c Projection e in camera image c =(u c v c The three primary colors of n c Indicates the number of colored laser dots;

[0076] Step 2: Construct a height-intensity-density (HID) image for each frame using the color point cloud, specifically including the following sub-steps:

[0077] (a) Establish a two-dimensional grid in the xoy plane of the point cloud coordinate system, and project the colored point cloud P onto the two-dimensional grid cells. The points in the two-dimensional grid cells are defined as follows, and described by formula (2).

[0078] C(u, v) = {p c |p c ∈P, x min +(u-1)q≤x c ≤x min +uq,

[0079] y min +(v-1)q≤y c ≤y min +vq} (2)

[0080] Where u and v represent the column number and row number of the two-dimensional mesh cell, respectively, and x min and y min Let x and y represent the minimum values ​​of the laser point's x and y coordinates, respectively; q represents the size of the two-dimensional mesh cell; and the ranges of u and v are defined as follows: in x max and y max These represent the maximum values ​​of the x and y coordinates of the laser point, respectively. The operator is defined as rounding up. and These represent the lower bounds of the u-axis and v-axis values ​​for the two-dimensional mesh element, respectively.

[0081] (b) Calculate the height feature h(u, v) of each two-dimensional mesh cell, which is described by formula (3).

[0082]

[0083] where z max and z min are the maximum and minimum values of the z coordinate of the laser points, respectively;

[0084] (c) Calculate the reflectance intensity feature i(u, v) of each two-dimensional grid cell, described by equation (4),

[0085]

[0086] (d) Calculate the density feature d(u, v) of each two-dimensional grid cell, described by equation (5),

[0087]

[0088] where N uv represents the number of points within the two-dimensional grid cell C(u, v);

[0089] (e) Set r h , r i , r d represent the weights of the height feature, reflectance intensity feature, and density feature, respectively, and process the above three features according to the weight ratio of r h ∶ r i ∶ r d = 1 ∶ 2 ∶ 1 to balance the feature intensity, and normalize them to the range of 0-255, described by equations (6) to (10),

[0090] hid max = max{[h(u, v) r h , i(u, v) r i , d(u, v) r d ]} (6)

[0091] hid min = min{[h(u, v) r h , i(u, v) r i , d(u, v) r d ]} (7)

[0092]

[0093]

[0094]

[0095] Finally, three feature maps are obtained: respectively represent the height map, the reflectance intensity map and the density map, as Figure 4 (a) (b) (c) as shown. These three feature maps are merged into a new three-channel feature map, called height-intensity-density (HID) image map, as shown in Figure 4 (d).

[0096] Step 3, three-dimensional target detection is performed on each frame using the color point cloud. In each frame, the color point cloud P is input into the Point-GNN target detector for three-dimensional target detection to obtain a three-dimensional detection box of the detected target, and the three-dimensional detection box is projected onto the xoy plane to obtain a two-dimensional detection box in the HID image. Specifically, the two-dimensional detection box in the t-th frame HID image is represented as wherein, represents the i-th two-dimensional detection box in the t-th frame HID image, represents the number of two-dimensional detection boxes in the t-th frame HID image;

[0097] Step 4, target position prediction is performed on the t+1-th frame using the rotation kernel correlation filter (RKCF). The RKCF is trained using the t-th frame HID image to perform target position prediction to obtain a predicted target in the t+1-th frame HID image. The specific steps include the following sub-steps:

[0098] (a) initialization of the tracking target: the t-th frame HID image Ω t is defined as B t = {b ts = (u ts , v ts , l ts , w ts , γ ts ) | 1≤s≤n t ), wherein b ts represents the s-th two-dimensional tracking box in the t-th frame HID image, (u ts , v ts ), (l ts , w ts ) and γ ts represent the center coordinates, size and direction angle of the two-dimensional tracking box respectively, n t represents the number of two-dimensional tracking boxes in the t-th frame HID image, and if t = 1, the two-dimensional detection box of the t-th frame is used as the two-dimensional tracking box of the t-th frame. the two-dimensional tracking box of the t-th frame is initialized;

[0099] (b) rotation of the t-th frame HID image: for each two-dimensional tracking box b ts , the tracking box center (u ts , v ts ) is taken as the rotation center point, and θ tsrotating the HID image Ω t to obtain a rotated HID image wherein if 0° < γ ts ≤ 90°, then θ ts = 90° - γ ts , Ω t is rotated counterclockwise; if 90° < γ ts < 180°, then θ ts = 90° - γ ts , Ω t is rotated clockwise; if -180° ≤ γ ts ≤ 90°, then θ ts = -90° - γ ts , Ω t is rotated counterclockwise; if -90° < γ ts ≤ 0°, then θ ts = -90° - γ ts , Ω t is rotated clockwise;

[0100] (c) training the classifier at the t-th frame: for each two-dimensional tracking box b ts , image cropping is performed on Ω ts according to its center coordinates (u ts , v ts ) and 2.5 times of its size (l ts , w t+1 ) to obtain a training base sample, and a training sample is obtained by cyclically shifting the base sample using a permutation matrix, wherein the base sample is a positive sample and the rest are negative samples, each sample is associated with an output following a two-dimensional Gaussian distribution, and a classifier is trained using the training sample and the corresponding output;

[0101] (d) rotating the HID image at the t+1-th frame: at the t+1-th frame, the HID image Ω (t+1)s is obtained according to step 2, and for each two-dimensional tracking box b ts , Ω ts is rotated with (u ts , v t+1 ) as the rotation center and θ ts as the rotation angle to obtain a rotated HID image

[0102] (e) predicting the target position at the t+1-th frame: according to the center coordinates (u ts , v ts ) and 2.5 times of the size (l ts , w ts ) of the two-dimensional tracking box b t+1 at the t-th frame, image cropping is performed on Ω (t+1)s to obtain a training base sample, and a training sample is obtained by cyclically shifting the base sample using a permutation matrix, wherein the base sample is a positive sample and the rest are negative samples, each sample is associated with an output following a two-dimensional Gaussian distribution, and a classifier is trained using the training sample and the corresponding output;The image is cropped to obtain the test base sample, which is cyclically shifted to obtain the test sample, the prediction response is calculated using the classifier trained in sub-step (c), and the row number a and column number b where the response peak is located represent the distance of the tracking target moving along the u-axis and v-axis in the adjacent two frames of the rotated HID image;

[0103] (f) The predicted coordinates in the rotated HID image are restored to the original HID image Ω t+1 , the center coordinates of the s-th two-dimensional tracking frame b (t+1)s in the t+1-th frame are calculated by formula (11),

[0104]

[0105] The tracking target in the t+1-th frame predicted by RKCF is recorded as the predicted target, and the two-dimensional prediction frame is represented as wherein, is the j-th two-dimensional prediction frame in the t+1-th frame of the HID image, is the number of two-dimensional prediction frames in the t+1-th frame of the HID image;

[0106] Step 5, data association of the detection target and the predicted target in the t+1-th frame: using the detection frame and the prediction frame to establish an association cost matrix wherein, is the number of two-dimensional detection frames in the t+1-th frame of the HID image, is the number of two-dimensional prediction frames in the t+1-th frame of the HID image, m (t+1)i，j is determined by and IOU(i, j), which is described by formula (12),

[0107]

[0108] wherein, IOU(i, j) represents the intersection over union of the i-th detection frame and the j-th prediction frame in the t+1-th frame, which is described by formula (13),

[0109]

[0110] wherein, represents the prediction score of the j-th prediction frame in the t+1-th frame, and the initial value is set to 1, based on the association cost matrix M t+1 , the detection target and the predicted target in the t+1-th frame are associated using the Kuhn-Munkres data association algorithm, and the detection target and the predicted target are divided into unmatched predicted targets, unmatched detection targets, and prediction-detection target matching pairs;

[0111] Step 6: Update the tracking target in frame t+1: For the prediction-detection target matching pair in frame t+1, update the tracking target corresponding to the prediction target with the two-dimensional detection box of the detection target;

[0112] Step 7: Preserve the unmatched predicted targets in frame t+1, which includes the following sub-steps:

[0113] (a) Obtain camera image crop of unmatched predicted targets: Define the unmatched predicted target in frame t+1 as... in, Represents the (t+1)th HID image in the t-th frame. Two-dimensional prediction boxes that do not match the predicted target. This represents the number of unmatched predicted targets in the (t+1)th frame of the HID image. The 3D bounding boxes of the unmatched predicted targets are projected onto the camera image at frame t+1 and cropped to obtain a cropped camera image. Project the 3D bounding box corresponding to the target in frame t onto the camera image in frame t and crop it to obtain a cropped image of the camera image.

[0114] (b) Target retention based on appearance similarity score: Calculated using mean hash algorithm. and Appearance similarity score To determine whether the prediction of the unmatched target is correct, if the appearance similarity score is... This indicates the prediction is correct. The two targets in frame t and frame t+1 belong to the same object. The correctly predicted target in frame t+1 is retained, and its 2D prediction bounding box is used. Update the tracking target; if the appearance similarity score This indicates a prediction error; the target is moved to the list of untracked targets and awaits target re-identification. Where T... s To retain an appearance score threshold for the target, a predefined T is used. s =120;

[0115] (c) Prediction score calculation: A prediction score is given for each prediction target. For a correctly predicted target, the initial prediction score is set to 0.5. As the number of predictions increases, the prediction score is gradually reduced, as described by formula (14).

[0116]

[0117] Where δ is the descent factor, predefined as δ = 0.05. This represents the prediction score of the j-th prediction box in frame t;

[0118] Step 8: Re-identify unmatched targets in frame t+1, which includes the following sub-steps:

[0119] (a) Obtain a crop of the camera image of the unmatched detected target in frame t+1: Define the unmatched detected target in frame t+1 as... in, Represents the (t+1)th HID image in the t-th frame. A two-dimensional detection box that does not match the target. Let be the number of unmatched detected targets in the (t+1)th frame of the HID image. The 3D bounding boxes of each unmatched detected target are projected onto the camera image and cropped to obtain a cropped image of the camera image.

[0120] (b) Obtain a cropped image of the camera that failed to track the target: Define the two-dimensional tracking bounding box before the target failed to be tracked. in, For the k-th untracked target, the 2D tracking bounding box is... This represents the number of targets that were not successfully tracked. The corresponding camera image crop is

[0121] (c) Target re-identification using appearance similarity score and spatial distance score: Calculated using the mean hash algorithm. and Appearance similarity score if This indicates that the two targets belong to the same object; if the appearance similarity score Then use the spatial distance fraction between the two. Further analysis reveals that T... h T l These represent the highest and lowest thresholds for appearance scores in target re-identification, respectively, with T predefined. h =200, T l =160; if This indicates that the two targets belong to the same object, and the distance threshold is met. Among them l m The maximum distance a target can move in two consecutive frames is predefined as l. m =25, The number of frames between the two targets;

[0122] (d) If it is determined that two targets belong to the same object, then the unmatched detected target is considered a correctly detected target or a reappearing target, and its two-dimensional detection box is used. updating the tracking target to continue the prediction in the next frame; otherwise, it is a new appearing target and its two-dimensional detection box is used initializing a new tracking target, and if a target is not successfully tracked for 5 consecutive frames, the target is considered to disappear permanently and is deleted;

[0123] Step 9, repeating the above steps 4-8 to process the next frame until the tracking ends.

[0124] The present application has the following advantages: first, the present application uses laser point cloud to construct HID image, effectively overcoming the influence of object occlusion in target tracking; second, the present application uses camera image to establish target maintenance module, checks whether the prediction is correct and retains the correct predicted object, so that the prediction can still be continued in the case of missed detection, and ensures the integrity of the tracking sequence and the reliability of the prediction result; third, the present application uses camera image to establish target re-identification module, avoiding repeated creation of new track and improving tracking performance. It not only can compensate for the wrong prediction, but also can handle the situation that the tracked object disappears temporarily and then reappears, which is conducive to the recovery of the interrupted tracking sequence; fourth, the present application makes full use of laser point cloud and camera image information, making the multi-modal information effectively complementary.

Claims

1. A multi-target tracking method based on multi-modal data, characterized in that Comprising the following steps: Step 1, acquiring point cloud and camera image for each frame and fusing into color point cloud, specifically comprising the following sub-steps: (a) in each frame, laser point cloud and camera image are acquired by laser scanner and camera respectively, the geometric mapping relationship between laser point cloud and camera image is described by formula (1), wherein, denotes a scaling factor, A denotes an intrinsic matrix, [R, t] denotes an extrinsic matrix between the laser scanner and the camera, which is obtained by laser scanner and camera calibration according to a pinhole camera model, denotes the coordinates of the three-dimensional laser point in the point cloud coordinate system, denotes the coordinates of the three-dimensional point after being projected to the camera image in the image pixel coordinate system, and denote their homogeneous coordinates; (b) the field of view of the laser scanner and the camera are different, some laser points out of the camera field of view cannot be projected onto the imaging plane, forming a colorless point cloud; laser points within the camera field of view can be projected onto the imaging plane and colored, forming a colored point cloud wherein, represents the cth colored laser point, (x c , y c , z c ) represents the coordinates of the cth colored laser point in the point cloud coordinate system, represents the reflection intensity of the cth colored laser point, R c = R(u c , v c ), G c = G(u c , v c ), B c = B(u c , v c ) represents the three primary colors of the projection e c = (u c , v c ) of the p c th colored laser point in the camera image; n c represents the number of colored laser points; Step 2, constructing height-intensity-density (HID) image for each frame using color point cloud, specifically comprising the following sub-steps: (a) a two-dimensional grid is established in the xoy plane of the point cloud coordinate system, and the color point cloud P is projected into the two-dimensional grid cell, the points in the two-dimensional grid cell are defined as, described by formula (2), C(u, v) = {p c |p c ∈ P, x min + (u - 1)q ≤ x c ≤ x min + uq, y min +(v-1)q≤y c ≤y min +vq} (2) where u and v represent the column number and row number of the two-dimensional grid cell respectively, x min and y min represent the minimum value of the x coordinate and y coordinate of the laser point respectively, q represents the size of the two-dimensional grid cell, and the value range of u and v is defined as where x max and y max represent the maximum value of the x coordinate and y coordinate of the laser point respectively, the operator is defined as rounding up, and represent the numerical lower bound of the u-axis and v-axis of the two-dimensional grid cell respectively. (b) the height feature h(u, v) of each two-dimensional grid cell is calculated, described by formula (3), where z max and z min are the maximum and minimum values of the z coordinate of the laser spot, respectively. (c) the reflection intensity feature i(u, v) of each two-dimensional grid cell is calculated, described by formula (4), (d) the density feature d(u, v) of each two-dimensional grid cell is calculated, described by formula (5), where N uv represents the number of points within a two-dimensional grid cell C(u, v); (e) setting r h , r i , r d respectively represent the weight of the height feature, the reflection intensity feature, the density feature, the above three features are processed according to the weight of r h : r i : r d = 1:2:1 to balance the feature intensity, and they are normalized to the range of 0-255, which is described by formulas (6) to (10), hid max = max{[h(u, v) r h , i(u, v) r i , d(u, v) r d ]} (6) hid min = min{[h(u, v) r h , i(u, v) r i , d(u, v) r d} (7) Three feature maps are finally obtained: respectively represent height map, reflectance intensity map and density map. The three feature maps are merged into a new three-channel feature map, called height-intensity-density (HID) image; Step 3, three-dimensional target detection is performed on each frame using a color point cloud, in each frame, the color point cloud P is input into a Point-GNN target detector for three-dimensional target detection, a three-dimensional detection box of a detected target is obtained, and the three-dimensional detection box is projected to an xoy plane to obtain a two-dimensional detection box in an HID image. Specifically, the two-dimensional detection box in the t-th frame HID image is represented as wherein, represents the i-th two-dimensional detection box in the t-th frame HID image, represents the number of two-dimensional detection boxes in the t-th frame HID image. Step 4, target position prediction for the t+1 frame is performed using the rotation kernel correlation filter (RKCF), RKCF is trained using the t frame HID image to perform target position prediction, and the predicted target in the t+1 frame HID image is obtained, specifically comprising the following sub-steps: (a) initialize tracking targets: the t-th frame HID image Ω t is defined as B t = {b ts = (u ts , v ts , l ts , w ts , γ ts ) | 1≤s≤n t}, where b ts represents the s-th two-dimensional tracking box in the t-th frame HID image, (u ts , v ts ), (l ts , w ts ) and γ ts represent the center coordinates, size and direction angle of the two-dimensional tracking box respectively, n t represents the number of two-dimensional tracking boxes in the t-th frame HID image, and if t = 1, the two-dimensional detection box of the t-th frame is used to initialize the two-dimensional tracking box of the t-th frame; (b) Rotating the HID image in the t-th frame: For each two-dimensional tracking box b ts , take the tracking box center (u ts , v ts ) as the rotation center point, and θ ts as the rotation angle to rotate the HID image Ω t to obtain the rotated HID image Ω , where if 0° < γ ts ≤ 90°, then θ ts = 90° - γ ts , Ω t is counterclockwise rotation; if 90° < γ ts < 180°, then θ ts = 90° - γ ts , Ω t is clockwise rotation; if -180° ≤ γ ts ≤ 90°, then θ ts = -90° - γ ts , Ω t is counterclockwise rotation; if -90° < γ ts < 0°, then θ ts = -90° - γ ts , Ω t is clockwise rotation; (c) training the classifier in the t-th frame: for each two-dimensional tracking box b ts , according to the center coordinates (u ts , v ts ) and the size (l ts , w ts ) of 2.5 times, image cropping is performed on to obtain a training basic sample, and the basic sample is cyclically shifted by using a permutation matrix to obtain a training sample, wherein the basic sample is a positive sample, and the rest are negative samples, each sample is associated with an output following a two-dimensional Gaussian distribution, and a classifier is trained by using the training sample and the corresponding output; (d) Rotating the HID image at the t+1 frame: At the t+1 frame, the HID image Ω is obtained according to step 2 t+1 For each two-dimensional tracking frame b (t+1)s , the HID image Ω is rotated with (u ts , v ts ) as the rotation center and θ ts as the rotation angle to obtain the rotated HID image Ω t+1 ​ (e) Predict the target position in frame t+1: based on the two-dimensional tracking box b in frame t. ts center coordinates (u ts v ts ) and size (l ts w ts 2.5 times that of Image cropping is performed to obtain the test base sample. The sample is then cyclically shifted to obtain the test sample. The predicted response is calculated using the classifier trained in sub-step (c). The row number a and column number b where the response peak is located represent the distance the tracked target moves along the u-axis and v-axis in the two adjacent frames of rotated HID images, respectively. (f) restoring the coordinates predicted in the rotated HID image to the original HID image Ω t+1 The center coordinates of the s-th two-dimensional tracking frame b (t+1)s in the t+1-th frame are calculated by formula (11) The tracking target of the t+1 frame predicted by the RKCF is recorded as a predicted target, and a two-dimensional prediction box thereof is represented as wherein, is the jth two-dimensional prediction box in the t+1 frame HID image, is the number of two-dimensional prediction boxes in the t+1 frame HID image. Step 5, data association for the detected targets and predicted targets of the t+1 frame: using the detection boxes of the t+1 frame with the predicted boxes to establish an association cost matrix wherein, is the number of two-dimensional detection boxes in the HID image of the t+1 frame, is the number of two-dimensional predicted boxes in the HID image of the t+1 frame, m (t+1)i，j the value of is determined by and IOU(i, j), which is described by formula (12), Wherein, IOU(i, j) represents the intersection over union of the i-th detection box and the j-th prediction box in the t+1 frame, described by formula (13), wherein, represents the prediction score of the jth prediction frame of the t+1 frame, and is set as an initial value of 1, and is based on the association cost matrix M t+1 , the detection target of the t+1 frame is associated with the prediction target by using a Kuhn-Munkres data association algorithm, and the detection target and the prediction target are divided into an unmatched prediction target, an unmatched detection target, and a prediction-detection target matching pair. Step 6, updating the tracking target of the t+1 frame: for the prediction-detection target matching pair of the t+1 frame, the two-dimensional detection box of the detection target is used to update the tracking target corresponding to the prediction target; Step 7, target reservation is performed for the unmatched prediction target of the t+1 frame, specifically comprising the following sub-steps: (a) obtaining a camera image crop of the unmatched predicted target: defining the unmatched predicted target of the t+1th frame as wherein, represents the two-dimensional prediction box of the i-th unmatched predicted target in the HID image of the t+1th frame, represents the number of unmatched predicted targets in the HID image of the t+1th frame, projecting the three-dimensional bounding box of the i-th unmatched predicted target of the t+1th frame onto the camera image of the t+1th frame and cropping to obtain the camera image crop projecting the three-dimensional bounding box corresponding to the target of the tth frame onto the camera image of the tth frame and cropping to obtain the camera image crop ​​ (b) Target retention based on appearance similarity score: Calculated using mean hash algorithm. and Appearance similarity score To determine whether the prediction of the unmatched target is correct, if the appearance similarity score is... This indicates the prediction is correct. The two targets in frame t and frame t+1 belong to the same object. The correctly predicted target in frame t+1 is retained, and its 2D prediction bounding box is used. Update the tracking target; if the appearance similarity score This indicates a prediction error; the target is moved to the list of untracked targets and awaits target re-identification. Where T... s To retain an appearance score threshold for the target, a predefined T is used. s =120; (c) prediction score calculation: a prediction score is given to each prediction target, for the correct prediction target, the initial prediction score is set to 0.5, and the prediction score is gradually reduced as the number of predictions increases, described by formula (14), wherein d is a drop factor, predefined as d = 0.05, represents the prediction score of the jth prediction box in the tth frame; Step 8, target re-identification is performed for the unmatched detection target of the t+1 frame, specifically comprising the following sub-steps: (a) Obtain a crop of the camera image of the unmatched detected target in frame t+1: Define the unmatched detected target in frame t+1 as... in, Represents the (t+1)th HID image in the t-th frame. A two-dimensional detection box that does not match the target. Let be the number of unmatched detected targets in the (t+1)th frame of the HID image. The 3D bounding boxes of each unmatched detected target are projected onto the camera image and cropped to obtain a cropped image of the camera image. (b) obtaining a camera image crop of the unsuccessfully tracked object: defining a two-dimensional tracking box of the unsuccessfully tracked object before the tracking failure wherein, is a two-dimensional tracking box of the kth unsuccessfully tracked object, is the number of unsuccessfully tracked objects, the corresponding camera image crop is (c) Target re-identification by appearance similarity score and spatial distance score: the mean hash algorithm is used to calculate with the appearance similarity score If it means that the two targets belong to the same object; if the appearance similarity score then the spatial distance score between the two is used to further determine, where T h , T l represent the highest threshold and the lowest threshold of the target re-identification appearance score respectively, T h = 200 and T l = 160 are predefined; if it means that the two targets belong to the same object, the distance threshold where l m is the maximum distance that the target can move in two consecutive frames, l m = 25 is predefined, is the frame number difference between the two targets; (d) If the two objects are judged to belong to the same object, the unmatched detection object is considered as a correct detection object or a reappeared object, and its two-dimensional detection box is used The tracking object is updated so as to continue the prediction in the next frame; otherwise, it is considered as a newly appeared object, and its two-dimensional detection box is used A new tracking object is initialized. In addition, if a target is not successfully tracked for 5 consecutive frames, the target is considered to be permanently disappeared, and the target is deleted. Step 9, repeat the above steps 4-8 to process the next frame until the tracking ends.

Images