Target tracking method based on intensity image and point cloud data fusion

Abstract

The target tracking method based on intensity image and point cloud data fusion relates to the fields of computer graphics and three-dimensional point cloud target tracking, and solves the problem that the existing target is blocked by obstacles, resulting in tracking accuracy decline or even tracking failure.The method of the present application fuses the HOG feature and the Fourier descriptor feature of the target intensity image formed by the laser radar by adopting a multi-feature fusion mode, and performs kernel correlation filtering tracking; the target distance image formed by the laser radar is converted into a three-dimensional point cloud, and a Kalman tracker is used to predict the trajectory of the point cloud target; the PSR and the ISS are combined to effectively judge the target blocking state, an adaptive factor is proposed according to the different target blocking conditions, and the adaptive factor is used to correct the positions of the kernel correlation and the Kalman tracker to obtain the final tracking position.The method of the present application can stably track the target, and the average time consumption of each frame of data is 39ms.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and 3D point cloud target tracking, specifically to a target tracking method based on the fusion of intensity images and point cloud data. Background Technology

[0002] With the rapid development of technology, unmanned combat vehicles (UCVs) have begun to emerge on the battlefield. The tracking system, as a crucial component of UCVs, directly impacts the ability to accurately track and strike enemy targets, and the tracking algorithm is key to the system's performance. Three-dimensional imaging lidar is a sensor that uses laser beams as a carrier for active detection, acquiring three-dimensional point cloud images of targets and possessing all-weather, high-precision imaging capabilities. Lidar using Geiger-Mode Avalanche PhotoDiode (GM-APD) detectors can perform high-sensitivity single-photon imaging of distant targets, showing broad application prospects in modern military warfare. Due to the complex battlefield environment, enemy vehicles and tanks are easily obstructed and interfered with by buildings, rocks, and other objects, resulting in missing target information acquired by lidar and increasing the difficulty of target tracking. Kernel correlation filter-based tracking algorithms offer high accuracy and speed, but in cases of occlusion, the limited target feature information leads to a significant drop in tracking accuracy. When the target is completely obscured, Kalman filter-based tracking algorithms can predict the target's position. Summary of the Invention

[0003] This invention aims to provide a target tracking method based on the fusion of intensity image and point cloud data to solve the problem of reduced tracking accuracy or even tracking failure caused by target occlusion by obstacles.

[0004] The target tracking method based on the fusion of intensity image and point cloud data is implemented by the following steps:

[0005] Step 1: Fuse the HOG features and Fourier descriptor features of the target intensity image generated by the lidar, and perform kernel correlation filtering for tracking;

[0006] Step 2: Convert the target range image obtained by the lidar into three-dimensional point cloud data, and use a Kalman tracker to predict the trajectory of the point cloud target;

[0007] Step 3: Combine the Peak Sidelobe Ratio (PSR) and Internal Shape Descriptor (ISS) to effectively determine the target occlusion state. Based on the different target occlusion conditions, propose an adaptive factor. Use the adaptive factor to correct the kernel correlation filter tracking and Kalman tracker predicted position to obtain the final tracking position.

[0008] The beneficial effects of this invention are:

[0009] 1. This invention fuses the Histogram of Oriented Gradient (HOG) features of the target intensity image with Fourier descriptors to provide a more comprehensive target description and improve the robustness of the kernel correlation filter tracking algorithm.

[0010] 2. This invention utilizes point cloud data converted from intensity image and range image, combined with peak sidelobe ratio and internal shape description, to effectively determine the target occlusion state. It also proposes an adaptive factor to weight the tracking results of kernel correlation filtering and Kalman filtering according to different target occlusion conditions, thereby obtaining the final target position information.

[0011] 3. Experimental verification of the proposed algorithm on the KITTI dataset shows that the proposed algorithm can accurately track targets in different occlusion states. Compared with the Kalman filter algorithm, the tracking accuracy is improved by 21.67%, and compared with the extended Kalman filter algorithm, the accuracy is improved by 7.94%, with an average processing time of 51ms per frame. In the GM-APD lidar real-world sampling scenario, the method of the present invention can robustly track targets, with an average data processing time of 39ms per frame. Attached Figure Description

[0012] Figure 1 This is a flowchart of the target tracking method based on the fusion of intensity image and point cloud data described in this invention;

[0013] Figure 2 This is a schematic diagram of a Kalman filter.

[0014] Figure 3 The peak response maps are shown below, where (a) is the peak response map when the target is not occluded, and (b) is the peak response map when the target is occluded.

[0015] Figure 4 Image showing the results of key point extraction for ISS;

[0016] Figure 5 The results of the occlusion threshold are shown in the figure; where (a) is the peak sidelobe ratio PSR = 7.69, (b) is the ISS key point N = 9; (c) is the peak sidelobe ratio PSR = 5.62, (d) is the ISS key point N = 6; (e) is the peak sidelobe ratio PSR = 1.77, and (f) is the ISS key point N = 0.

[0017] Figure 6 The following are the tracking results of the algorithm: (a) is intensity image tracking result 1, (b) is final tracking result 1, (c) is intensity image tracking result 2, (d) is final tracking result 2, (e) is intensity image tracking result 3, (f) is final tracking result 3, (g) is intensity image tracking result 4, and (h) is final tracking result 4.

[0018] Figure 7 The figures are error curves and tracking trajectory diagrams; where (a) is the center position error curve diagram, (b) is the Kalman filter tracking trajectory diagram, (c) is the extended Kalman tracking trajectory diagram, and (d) is the fused tracking trajectory diagram.

[0019] Figure 8 This is a scene diagram for outdoor detection.

[0020] Figure 9 The images show the tracking results of the GM-APD lidar; where (a) is intensity image tracking result 1, (b) is final tracking result 1, (c) is intensity image tracking result 2, (d) is final tracking result 2, (e) is intensity image tracking result 3, (f) is final tracking result 3, (g) is intensity image tracking result 4, (h) is final tracking result 4, (i) is intensity image tracking result 5, and (j) is final tracking result 5. Detailed Implementation

[0021] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to the accompanying drawings and embodiments; it should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention.

[0022] Target tracking methods based on the fusion of intensity images and point cloud data, such as Figure 1 As shown, the specific steps include:

[0023] Step S1: Using a multi-feature fusion method, the target intensity image formed by the lidar is fused with the HOG features of the target of interest and the Fourier descriptor features, and then kernel correlation filtering is performed for tracking.

[0024] Step S11: Train the classifier model of the kernel correlation filter tracking algorithm using the ridge regression function;

[0025] Suppose we have a training set {(a1,b1),…(a…b1)} i ,b i For the ridge regression function f(x) i ) = w T a i The goal of training is to make the regression function work on the sample vector a. i and its corresponding tag b i The mean square error between them is minimized, and the optimization formula is as follows:

[0026]

[0027] Where λ is the regularization parameter to prevent overfitting, w is a column vector representing the weight coefficients, and ||w|| 2Let 2 be the norm of w, controlling for overfitting. Equation (1) has a unique optimal solution, which can be calculated by the following formula:

[0028] w = (A T A+λI) -1 A T b (2)

[0029] Replace equation (2) with its complex form:

[0030] w = (A H A+λI) -1 A H b (3)

[0031] Where A represents the sample vector a i The circular matrix b obtained by circular shifting, where each element represents a sample label b i I represents the identity matrix, A H Let A be the Hermitian transpose of A. H =(A * ) T A * It is the complex conjugate of A.

[0032] Step S12: Expand the training samples using a cyclic matrix; obtain the trained kernel correlation filter tracking model;

[0033] Suppose vector a = [a1, a2, ..., a2] n ] T The features of the image patch containing the target of interest are represented and used as the base sample. The base sample is used as the positive sample, and the vector a is shifted by left multiplying it with the permutation matrix P to obtain the negative sample. The permutation matrix P is shown in (4):

[0034]

[0035] Pa = [a N-1 ,a0,a1,…,a N-2 The vector ] is obtained by shifting vector a one unit to the right; here, we use P. N 'a' represents the vector obtained by shifting vector a to the right by N units. If N is negative, vector a is shifted in the opposite direction.

[0036] After performing a cyclic shift operation on vector a, we can obtain the cyclic matrix A:

[0037]

[0038] The circulant matrix can be obtained by performing a Discrete Fourier Transform (DFT) on the diagonal matrix (diag), and it is independent of the fundamental vector 'a'. This property can be expressed by the following equation:

[0039]

[0040]

[0041] Where F represents a constant matrix independent of the fundamental vector a, called the Fourier matrix. For any vector z to be detected, it can be represented by... To obtain its Fourier transform, * is the conjugate operator. It is the discrete Fourier transform of a, and N represents the N×N dimensional sample matrix.

[0042] If the training data is obtained by cyclically shifting the base samples, the Fourier matrix diagonalization method can be used for element-wise operations to obtain the final result. Incorporating diagonalization into the covariance matrix yields:

[0043]

[0044] Due to FF H =I, simplifying equation (8) above to:

[0045]

[0046] Since diagonal matrices are operated on only their diagonal elements, we can obtain:

[0047]

[0048] Where ⊙ represents element-wise multiplication (multiplication of corresponding matrix positions), substitute equations (7) and (10) into equation (3) for calculation:

[0049]

[0050] Performing a Fourier transform on both sides of the result of equation (11) yields the following result:

[0051]

[0052] in, Let be a vector, and λ be a regularization parameter to prevent overfitting. The above equation can be simplified based on the properties of diagonal matrices:

[0053]

[0054] Step S13: Perform rapid target detection and localization based on the kernel correlation filter tracking model;

[0055] In the kernel correlation filter tracking algorithm, in order to introduce the kernel trick, the parameter w to be optimized needs to be transformed from the linear space (original space) to the dual space α.

[0056] Let the sample X = [x1, x2, ..., xn ] T For training, Y = [y1, y2, ..., y] n ] T Let be the sample label value. The nonlinear transformation from the sample space to the Hilbert feature space is: Corresponding kernel function The optimal solution to the optimization problem is:

[0057]

[0058] In the formula, Sample x i The mapping function from x′ to the Hilbert feature space, where x is a training sample and x′ is the sample to be detected; its solution w lies in sample x. i Within Zhang Cheng's subspace, w can be represented as:

[0059]

[0060] At this point, the variable to be sought changes from w to α, and α is called the solution of the original space in the dual space.

[0061] Therefore, the function response f(z) is:

[0062]

[0063] Where κ(x) i ,z i ) represents training samples in high-dimensional space and The inner product of.

[0064] The closed-form solution of the ridge regression nucleated form is as follows:

[0065] α=(K+λI) -1 y (17)

[0066] Where y represents the general meaning of a sample label value, and the kernel matrices corresponding to the polynomial kernel, linear kernel, and Gaussian kernel are all cyclic matrices, denoted as:

[0067] K = C(κ) xx (18)

[0068] In the formula, κ xx This represents the first row of the kernel matrix K. Calculating formula (17) yields:

[0069]

[0070] To detect targets, a classifier needs to be trained on candidate regions to obtain the required regression function. The model for candidate regions is constructed using the following formula:

[0071] K z =C(κ) xz (20)

[0072] Where K z It is the kernel matrix composed of training samples and candidate regions, κ xz It is a cross-correlation matrix. From this, we can obtain the correlation response of the candidate region (the response after function mapping):

[0073] f(z) = K z α (21)

[0074] Utilizing the diagonalization property of the Fourier matrix for rapid target detection, the target vector z and the trained classifier response value are represented in the frequency domain as follows:

[0075]

[0076] in, The location corresponding to the maximum value is the predicted target location.

[0077] Step S14: Multi-feature fusion;

[0078] First, the HOG features of the target intensity image are calculated. The specific steps are as follows:

[0079] a. Extract the image of the tracking area and convert it to grayscale to reduce computation.

[0080] b. The input image is normalized using the Gamma correction method, which makes the extracted features more robust to illumination and background noise. The normalization formula is shown in (23):

[0081] I(h x ,h y )=I(h x ,h y ) Gamma (twenty three)

[0082] In the formula, I(h) x ,h y ) is the image matrix, and Gamma = 5.0 is usually used.

[0083] c. Calculate the gradient information of each pixel to obtain the contour information of the target. First, perform a convolution operation on the original image using the [-1,0,1] gradient operator to obtain the horizontal gradient components. Then, use the [-1,0,1] gradient operator... T The gradient operator performs a convolution operation on the original image to obtain the gradient component in the vertical direction. Its calculation formulas are shown in (24) and (25):

[0084] Gx (h x ,h y )=I(h x+1 ,h y )-I(h x-1 ,h y ) (twenty four)

[0085] G y (h x ,h y )=I(h x ,h y+1 )-I(h x ,h y-1 (25)

[0086] In the formula, G x (h x ,h y ) represents image I(h x ,h y The gradient value in the horizontal direction, G y (h x ,h y ) represents image I(h x ,h y The gradient value in the vertical direction. Therefore, the image (h) x ,h y The gradient value and gradient direction at coordinates can be expressed as:

[0087]

[0088]

[0089] d. Divide the image into many small cells and calculate the gradient information for each cell. Divide the gradient direction of each cell into 9 bins, with each bin having a 20° interval. Then, map the gradient information to the corresponding bins according to the gradient direction.

[0090] e. Statistically analyze the gradient information in the blocks composed of cells, and finally concatenate the gradient histograms of all blocks to obtain the gradient histogram features of the entire image.

[0091] In this embodiment, the Fourier descriptor features are calculated, and the calculation steps are as follows:

[0092] a. First, the intensity image is converted to grayscale and then Gamma correction is performed.

[0093] b. Use the Canny edge detection algorithm to extract the set of contour points of the target to be tracked {(x c ,y c)|c=1,2,…,m}.

[0094] c. Calculate the center coordinates of the contour points:

[0095]

[0096] d. Convert the contour points from the rectangular coordinate system to the polar coordinate system.

[0097] by Let x be the pole of the polar coordinate system, and let the coordinates of the contour points be {(x...} c ,y c Convert |c=1,2,…,m} to the corresponding polar coordinates {(r c ,θ c |c=1,2,…,m}

[0098]

[0099] In the formula r c The distance to the center contour point.

[0100] For r c Sort the points in ascending order to obtain the distance sequence of the center point contour points, denoted as D = [r1, r2, ..., r m ].

[0101] e. Perform a Fast Fourier Transform on the distance sequence of the center contour points.

[0102]

[0103] In the formula, j is the imaginary unit of the Fourier transform, and the final Fourier descriptor is:

[0104]

[0105] In the formula, |·| represents the Fourier spectrum.

[0106] Finally, the HOG features are serially fused with the Fourier descriptors, which can improve the robustness of the kernel correlation filter tracking algorithm.

[0107] Step S2: Convert the target range image obtained by the lidar into a three-dimensional point cloud, and use a Kalman tracker to predict the trajectory of the point cloud target;

[0108] The Kalman filter consists of two stages: prediction and update. Figure 2As shown, in the prediction phase, the optimal estimate of the target state at time t-1 is used to calculate the preliminary estimate of the target state at time t. Since the influence of system noise is ignored in the prediction phase, the obtained preliminary estimate deviates from the actual state value. In the update phase, the preliminary estimate is corrected based on the observed value of the target state at time t and the state observation matrix, thereby obtaining the optimal estimate of the target state. The specific process is as follows:

[0109] Step S21, State prediction;

[0110] The target state prediction process at any time t is as follows:

[0111]

[0112]

[0113] in A is the optimal estimate of the target state at the previous moment. t Here is the state transition matrix. This is a preliminary estimate of the target state at the current moment. This is the covariance matrix corresponding to the optimal estimate of the target state at the previous time step. Q is the covariance matrix corresponding to the estimated value of the target state at the current time. t It is ω t The corresponding noise covariance matrix.

[0114] Step S22: Status update;

[0115] The target state update process at any time t is as follows:

[0116]

[0117] Among them, K t To obtain the Kalman gain matrix of the optimal estimate of the target state, H t Let R be the observation matrix. t To measure noise υ t The covariance matrix, It is the optimal estimate of the target state at the current moment. Z is the covariance matrix corresponding to the optimal estimate of the target state at the current moment. t I represents the observed value of the target state at the current moment. t It is an identity matrix.

[0118] In this invention, the target of interest is represented as (x,y,z,θ,v). x ,v y ,v z A 6-dimensional vector composed of (x, y, z) represents the coordinates of the vehicle's center point, (v...x ,v y ,v z Let x, y, and z be the velocity components in the x, y, and z directions respectively, and θ be the vehicle's orientation. Considering that the orientation does not change significantly during vehicle tracking on adjacent point cloud frames, this invention ignores the target angular velocity and uses a uniform velocity model to describe the inter-frame vehicle motion. The state of a certain vehicle target is denoted as shown in equation (35):

[0119] X = [x,y,z,θ,v] x ,v y ,v z ] T (35)

[0120] The state of the vehicle at time t-1 is X. t-1 The state X at time t can be calculated using kinematic formulas. t As shown in equation (36):

[0121]

[0122] Among them, X t-1 Let X be the target state of the vehicle at time t-1. t The vehicle target state at time t is calculated based on the kinematic model, where Δt is the time interval between the point cloud in frame (t-1) and frame t. t For X t-1 To X t The state transition matrix, ω t It represents noise in the equation of motion and follows a zero-mean Gaussian distribution.

[0123] In this embodiment, a three-dimensional vehicle target motion model is used to estimate the target position. The model output value is used as the state observation value, and the observation equation can be established as shown in equation (37):

[0124]

[0125] Among them, Z t Let H be the vehicle state observation value at time t. t For X t To Z t The observation matrix, υ t Let be the observation noise at time t, and in this invention, let be the observation error of the target model.

[0126] Step S3: Combine the Peak Sidelobe Ratio (PSR) and Intrinsic Shape Signatures (ISS) to effectively determine the target occlusion state. Based on the different target occlusion conditions, propose an adaptive factor ε. Use the adaptive factor ε to correct the kernel correlation and Kalman tracker position to obtain the final tracking position.

[0127] Step S31: Calculate the peak-to-sidelobe ratio;

[0128] The peak-to-sidelobe ratio (PSNR) is a physical quantity that describes the prominence of the main lobe relative to the sidelobes. It can be used to evaluate the matching degree between two target signals in a correlation operation. It is defined as:

[0129]

[0130] In the formula, max(f(z)) is the peak value of the relevant response, and μ and σ are the mean and standard deviation of the side lobes other than the peak value of the maximum response, respectively.

[0131] like Figure 3 As shown, when the kernel correlation filter tracking algorithm tracks the target normally, its response map is close to an ideal two-dimensional Gaussian distribution, with obvious peaks; however, under interference conditions, especially occlusion conditions, such as... Figure 3 In (b), the target features are disrupted, the response map fluctuates, no longer follows a Gaussian distribution, but exhibits multiple peaks, and the contrast between sidelobes and peaks decreases, such as... Figure 3 In (a), when the target is not occluded, the response peak clearly reaches its maximum value.

[0132] This implementation pre-calculates the average PSR μ of the target in the five frames before it is obscured. PSR As shown in equation (39), the peak response standard for a target that is not occluded is used. When the target is completely occluded, the peak response is not 0. After experimental testing, 0.3μ PSR and 0.8μ PSR The low threshold T of the occlusion detection mechanism is defined respectively. L and high threshold T H .

[0133]

[0134] In the formula, n is the total number of image frames, and k is the number of image sequences.

[0135] Step S32: Adaptively update the kernel correlation filter model based on the peak sidelobe ratio;

[0136] During kernel correlation filtering tracking, the target's location information changes with each frame. To adapt to the changing target appearance features as the video frames change, this implementation adaptively updates the appearance model using the peak-to-sidelobe ratio (α). k x k α k-1 x k-1 Let γ represent the parameters and target model of the k-th frame and the (k-1)-th frame, respectively. k The learning rate parameter, β, is a constant value set to 0.025, which adaptively varies based on the calculated PSR value in each frame.

[0137]

[0138] Step S33: The number of key points is used as a criterion for judging the occlusion status based on the different occlusion situations.

[0139] The specific ISS key point extraction process is as follows, and the result is shown in the figure below. Figure 4 As shown:

[0140] Let there be N point cloud targets. p There are points, and any point p is a point. i Coordinates are (x i ,y i ,z i ).

[0141] 1) Plot each point p in the point cloud i Define a local coordinate system and give each point a search radius r. frame ;

[0142] 2) Query each point p in the point cloud data i At radius r frame All points within the surrounding area, and calculate their weights, i.e.:

[0143] w ij =1 / |p i -p j |,|p i -p j |<r frame (41)

[0144] 3) Calculate each point p i Covariance matrix:

[0145]

[0146] 4) Calculate each point p i The covariance matrix cov(p) i eigenvalues Arranged in descending order

[0147] 5) Set thresholds ε1 and ε2, and points that satisfy equation (43) are considered as ISS feature points;

[0148]

[0149] 6) Repeat the above steps until all points are completed;

[0150] Let N be the number of keypoints extracted just before the target in the point cloud is occluded. iss Similarly, 0.3N iss 0.8N iss These are defined as the low threshold (T) of the occlusion detection mechanism. L ) and high threshold (T) H ).

[0151] The final occlusion detection strategy is designed as follows:

[0152]

[0153] Step S34: Based on the different target occlusion conditions, this invention proposes an adaptive factor ε, which is used to correct the kernel correlation and Kalman tracker position.

[0154] The formula for calculating the adaptive factor ε is as follows:

[0155]

[0156] In the formula, PSR k Let N be the peak-to-side-lobe ratio of the k-th frame image. k iss μ is the number of keypoints in the k-th frame image. PSR N is the average PSR of the target in the first five frames before it was not occluded. iss The number of keypoints extracted just before the point cloud target occlusion; the kernel correlation and Kalman tracker position are corrected using an adaptive factor ε to obtain the final tracking position:

[0157]

[0158] In the formula, F op (x,y,z) represents the optimal 3D estimated position of the target by combining Kalman filtering and kernel correlation filtering, F KCF (x,y) represents the two-dimensional position of the target calculated by the kernel correlation filter, F KM (x,y) represents the target's two-dimensional position calculated by the Kalman filter, and F(x,y) represents the target's two-dimensional coordinate position after weighted fusion. KM(z) represents the target's third-dimensional position calculated by the Kalman filter. From the above equation, it can be seen that when the target is unobstructed, ε focuses more on the tracking result of the kernel correlation filter; when the target is completely obstructed, ε focuses more on the tracking result of the Kalman filter.

[0159] To verify the effectiveness of the target tracking algorithm proposed in this invention, experiments were conducted on the KITTI dataset. Color images in the KITTI dataset were converted to grayscale images to simulate the intensity images obtained by the GM-APD LiDAR. Then, kernel correlation filtering and Kalman filtering were used to perform tracking tasks on the images and point cloud data, respectively. Center position error (CLE) was used to evaluate the tracking algorithm performance. CLE refers to the Euclidean distance between the center position of the actual target box and the center position of the predicted target box during tracking; the smaller the Euclidean distance, the higher the tracking accuracy. The CLE calculation formula is as follows:

[0160]

[0161] Where CLE stands for Center Error, p is the predicted center position of the bounding box, and r is the actual center position of the bounding box.

[0162] KITTI dataset tracking experiment:

[0163] The tracking scenario is the KITTI dataset tracking scenario 0001, and the target is a white sedan with license plate number 86. The tracking sequence is the sequence of the sedan's motion from unobstructed to occluded by other cars and back to unobstructed. During target tracking, an occlusion threshold needs to be calculated each time to effectively fuse intensity image and point cloud tracking position information using an adaptive factor. The occlusion thresholds calculated for a portion of the motion sequence are as follows: Figure 5 As shown.

[0164] from Figure 5 As can be seen from (a) and (b), when the target is not occluded, the peak sidelobe ratio (PSR) is 7.69 and the number of ISS keypoints is 9. When the target is partially occluded, as in (c) and (d), the PSR drops to 5.62 and the number of ISS keypoints decreases to 6, which meets the occlusion detection interval set by this invention. When the target is completely occluded, as in (e) and (f), the PSR is not 0 but drops to 1.77, and the number of ISS keypoints is 0, which also meets the threshold interval set by this invention when the target is completely occluded. This proves that the occlusion threshold set by this invention is effective and meets the judgment of the target occlusion status for subsequent tracking tasks.

[0165] Figure 6The images show the tracking results of this invention. (a), (c), (e), and (g) are the intensity image tracking results, respectively; (b), (d), (f), and (h) are the final tracking results, respectively. The red box represents the target's true location, and the yellow box represents the algorithm's tracked location. It can be seen that when the target is unobstructed, the kernel correlation filter based on multi-feature fusion can effectively track the target of interest. When the target is partially occluded, the tracking results based on multi-feature fusion and kernel correlation filter show deviations, while the Kalman tracking algorithm can predict the target's location with a smaller error.

[0166] from Figure 7 As can be seen from the diagram, (a) is the center position error curve, (b) is the Kalman filter tracking trajectory diagram, (c) is the extended Kalman tracking trajectory diagram, and (d) is the fused tracking trajectory diagram.

[0167] The Extended Kalman Filter (EKF) algorithm has a longer runtime than the Kalman Filter (KF) algorithm. However, by combining kernel correlation with a Kalman tracker, the tracking accuracy is superior to both KF and EKF when the target is unobstructed. When the target is partially or completely obstructed, adaptive factor adjustment maintains high tracking accuracy. Table 1 shows that the average center position error of the proposed algorithm is 21.67% higher than that of KF and 7.94% higher than that of EKF, demonstrating that the proposed method can accurately track targets in different obstruction states.

[0168] Table 1

[0169]

[0170] The above analysis and experiments verified the superiority of the algorithm proposed in this invention using the KITTI dataset. The following experiments will further verify this.

[0171] LiDAR real-world scene tracking experiment:

[0172] A fixed GM-APD lidar was used to detect and image a crossroads 600 meters away outdoors. Tracking experiments were conducted on the point cloud data after converting the intensity image and range image formed by the GM-APD lidar. The detection scene is shown in the figure.

[0173] from Figure 9As can be seen from the figures, when the target is relatively complete in the intensity image, as shown in (a) and (c), the kernel correlation filter tracking algorithm can effectively track the target of interest. As the tracked target gradually moves away from the detection field of view, as shown in (e), (g), and (i), the tracking results based on the intensity image gradually become ineffective. Through adaptive factor adjustment, the target position is predicted by Kalman filter. From (b), (d), (f), (h), and (j) of the point cloud results, it can be seen that when the target gradually disappears from the detection field of view from complete information, the algorithm of this invention can effectively track the target. Due to the influence of noise on the detection data, the average processing time of the algorithm of this invention per frame in this experiment is 39ms.

[0174] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0175] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A target tracking method based on the fusion of intensity image and point cloud data, characterized by: This method is implemented by the following steps: Step 1: Fuse the HOG features and Fourier descriptor features of the target intensity image generated by the lidar, and perform kernel correlation filtering for tracking; Step 2: Convert the target range image obtained by the lidar into three-dimensional point cloud data, and use a Kalman tracker to predict the trajectory of the point cloud target; Step 3: Combine the Peak Sidelobe Ratio (PSR) and Internal Shape Descriptor (ISS) to effectively determine the target occlusion state. Based on different target occlusion conditions, propose an adaptive factor. Use this adaptive factor to correct the kernel correlation filter tracking and Kalman tracker predicted positions to obtain the final tracking position. The specific process is as follows: Step 3: Calculate the peak-to-sidelobe ratio; Step 32: Adaptively update the kernel correlation filter tracking model based on the peak sidelobe ratio calculated in Step 31; Step 3: Use the ISS detection algorithm to extract key points of the point cloud target and set an occlusion detection strategy; Steps 3 and 4: Adaptive factors are applied based on the different target occlusion conditions. Correct the nuclear correlation and Kalman tracker position; The adaptive factor The calculation formula is: ； In the formula, For the first Peak-to-side-lob ratio of a frame image For the first The number of key points in a frame image The average PSR of the target in the first five frames when it was not occluded. The number of key points extracted just before the point cloud target is occluded; Using adaptive factors The final tracking position is obtained by correcting the positions of the kernel correlation filter tracker and the Kalman tracker: ； In the formula, To achieve the optimal 3D estimation location of the target by combining Kalman filtering and kernel correlation filtering, The target's two-dimensional position is calculated using the kernel correlation filter. The target's two-dimensional position calculated by the Kalman filter. The target's two-dimensional coordinate position after weighted fusion. The third-dimensional position of the target is calculated using the Kalman filter; from the above equation, it can be seen that when the target is in an unobstructed state... The tracking results of the kernel correlation filter are emphasized; when the target is completely occluded, The focus is on the tracking results of the Kalman filter.

2. The target tracking method based on the fusion of intensity image and point cloud data according to claim 1, characterized in that: The specific process of step one is as follows: Step 11: Train the classifier model of the kernel correlation filter tracking algorithm using the ridge regression function; Steps 1 and 2: Expand the training samples using a cyclic matrix to obtain the trained kernel correlation filter tracking model; Step 13: Use the kernel correlation filter tracking model to detect and locate the target.

3. The target tracking method based on the fusion of intensity image and point cloud data according to claim 1, characterized in that: In step one, the specific process of calculating the HOG features of the target intensity image is as follows: A. Capture the tracking area image, convert the image to grayscale, and normalize the input image using the Gamma correction method; B. Calculate the gradient information of the pixels in the normalized image to obtain the contour information of the target. C. Divide the image into multiple cells, calculate the gradient information of each cell, divide the gradient direction of each cell into 9 bins, that is, the interval between each bin is 20°, and then map the gradient information to the corresponding bin according to the gradient direction. D. Statistically analyze the gradient information in the blocks composed of cells, and finally concatenate the gradient histograms of all blocks to obtain the gradient histogram features of the entire image.

4. The target tracking method based on the fusion of intensity image and point cloud data according to claim 1, characterized in that: In step one, the process of calculating the Fourier descriptor features is as follows: a. Grayscale processing of the intensity image followed by Gamma correction; b. Use the Canny edge detection algorithm to extract the set of contour points of the target to be tracked; c. Calculate the center coordinates of the contour points, convert the contour points from the rectangular coordinate system to the polar coordinate system, and obtain the center point contour point distance sequence. d. Perform a Fast Fourier Transform on the distance sequence of the center contour points to construct a Fourier descriptor.

5. The target tracking method based on the fusion of intensity image and point cloud data according to claim 1, characterized in that: In step 3.2, the kernel phase filter model is adaptively updated based on the peak-to-sidelobe ratio, and the following settings are made: They represent the first Frame parameters and target model, They represent the first and second. Frame parameters and target model, set The learning rate parameter, which adaptively varies according to the calculated PSR value in each frame, is given by the following formula: ； In the formula, T is a constant value. L The threshold is low.

6. The target tracking method based on the fusion of intensity image and point cloud data according to claim 1, characterized in that: In step 33, the specific process of extracting key points of the point cloud target using the ISS detection algorithm is as follows: Setting point cloud targets has any one of the points. Coordinates are ; Each point of the point cloud target Define a local coordinate system and give each point a search radius. ; Query each point In radius Calculate the weights of all points within the surrounding area; Calculate each point The covariance matrix is ​​calculated for each point. eigenvalues ​​of the covariance matrix Arrange them in descending order. ; Set threshold and ,satisfy The points that are not identified are considered as ISS feature points; this process continues until all key points have been extracted. Let the number of keypoints extracted just before the point cloud target is occluded be . and 0.3 0.8 The low threshold T of the occlusion detection mechanism is defined respectively. L and high threshold T H ; The final occlusion detection strategy is as follows: 。

Images