An extended target tracking method based on Swin-transformer

Abstract

The application discloses an extended target tracking method based on a Swin-transformer, and utilizes a visual converter to process a noisy radar point cloud image through a deep learning network of the Swin-transformer and a Kalman filter; wherein the Swin-transformer: realizes inference on a target shape parameter, at least includes one standard multi-head self-attention module based on a shift window, one standard multi-head self-attention module based on a window and a structure internally composed of two layers of multilayer perceptrons, and has a Gaussian error linear unit nonlinearity in the middle; and the Kalman filter: is used for estimating kinematic parameters of a target. The method can simultaneously estimate the kinematic parameters and the shape parameters of a single extended target from a noisy point cloud, so that target tracking is more accurate and efficient.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of navigation, signal processing and target tracking, and mainly relates to an extended target tracking method based on a Swin-transformer. BACKGROUND

[0002] With the development of high-resolution sensors such as automotive radars and lidars, extended target tracking (ETT) has become a hot topic in the field of autonomous driving. ETT usually needs to estimate the motion parameters and shape parameters of the tracked target. Among them, for the estimation of the shape parameters, there are several typical algorithms, such as independent axis estimation (IAE), extended Kalman filter based on multiplication error model (MEM-EKF), random matrix, random hyper-surface model, convolutional neural network (CNN) based method, etc. Although the effectiveness of these methods has been proven in high measurement rate scenarios, their effectiveness will decrease in low measurement rate scenarios. SUMMARY

[0003] The application is exactly aimed at the problem that the existing ETT method has poor performance in the low measurement rate scenario, and provides an extended target tracking method based on a Swin-transformer. Through the deep learning network of the swin-transformer and the Kalman filter, the visual converter is used to process the noisy radar point cloud image. The swin-transformer: realizes the inference of the target shape parameters, and at least includes a standard multi-head self-attention module based on a shift window, a standard multi-head self-attention module based on a window, and a structure internally composed of two layers of multilayer perceptron, which has a Gaussian error linear unit nonlinearity in the middle; the Kalman filter: is used for estimating the kinematic parameters of the target. The method can simultaneously estimate the motion parameters and shape parameters of a single extended target from the noisy point cloud, so that the target tracking is more accurate and efficient.

[0004] In order to achieve the above purpose, the technical scheme adopted by the application is: an extended target tracking method based on a Swin-transformer, through the deep learning network of the swin-transformer and the Kalman filter, the visual converter is used to process the noisy radar point cloud image; wherein,

[0005] The swin-transformer: realizes the inference of the target shape parameters, and at least includes a standard multi-head self-attention module based on a shift window, a standard multi-head self-attention module based on a window, and a structure internally composed of two layers of multilayer perceptron, which has a Gaussian error linear unit nonlinearity in the middle;

[0006] The Kalman filter: is used for estimating the kinematic parameters of the target.

[0007] As an improvement of the present application, an extended target tracking method based on Swin-transformer includes the following steps:

[0008] S1, image conversion: normalize and discretize the noisy point cloud, and convert the point cloud into an image;

[0009] S2, image update: update the image obtained in step S1 according to time evolution, and form an internal intensity measurement image (IIMI);

[0010] S3, shape estimation: use swin-transformer to perform visual conversion processing on the image formed in step S2, and infer the shape parameters of the target;

[0011] S4, kinematics estimation: use Kalman filter to estimate the kinematics parameters based on the original noisy point cloud and the inferred shape parameters for the shape parameters estimated in step S3;

[0012] S5, target tracking: through the shape estimation in step S3 and the kinematics estimation in step S4, the shape parameters and kinematics parameters of the target are obtained, and the tracking of the target is completed.

[0013] As an improvement of the present application, in step S1, the method of normalizing and discretizing the noisy point cloud is as follows:

[0014]

[0015]

[0016] Wherein, Z' k is the Cartesian coordinates of all newly arrived measurements z k centered on the estimated target center, Z" k is the new coordinates after aligning the measurements with the target values, is all newly arrived measurements, represents the rotation matrix with , the measurement matrix H is is the kinematics parameter of the object, expressed as wherein, and represent the horizontal and vertical coordinates of the object, and represent the horizontal and vertical velocities of the object, and T represents the transpose of the matrix.

[0017] As another improvement of the present application, the image update in step S2 is as follows:

[0018]

[0019] wherein, represents the IIMI calculated in the latest time k, 0≤γ≤1 represents a forgetting factor;

[0020] the normalized IIMI k Specifically,

[0021] IIMI k = 255 x IIMI k ÷ max(IIMI k ).

[0022] As another improvement of the present application, in the step S3, the internal intensity measurement image formed in the step S2 is taken as the input of the swin transformer, and after passing through the guided partition module, the image is segmented into non-overlapping blocks, and then the image features are projected to the selected dimension through the linear embedding module, and then the swin transformer is processed; the swin transformer further comprises a patch merging module for reducing the image resolution and using a fully connected layer to generate an estimated shape parameter.

[0023] As another improvement of the present application, the shape parameter in the step S3 comprises and represent the angle of the shape direction, and represent the short semi-axis and the long semi-axis of the ellipse, and for can be predicted as:

[0024]

[0025] wherein, represents the vertical velocity of the object at time k, represents the horizontal velocity of the object at time k.

[0026] As another improvement of the present application, in the step S4, a Kalman filter is used to predict the kinematic parameters of the target:

[0027]

[0028]

[0029] wherein, F is a transition matrix, is the estimated kinematic parameter at time k-1, is the kinematic parameter at time k estimated according to the Kalman filter, is the estimated covariance at k-1 time, Q is noise, after receiving the noise point cloud at time k, the kinematic parameters are updated through the update equation as follows:

[0030]

[0031]

[0032]

[0033]

[0034] P k =(I-G k H)P k|k-1

[0035]

[0036] wherein P k|k-1 is the prediction error covariance, H is the observation matrix, which maps the state space to the observation space, is the covariance matrix of the observation noise, G k is the Kalman gain, which is used to weigh the weights of the prediction error and the measurement error, is the mean of all newly arrived measurement values z k centered on the estimated target center, m k is the number of measurement values , is a term related to the covariance matrix, which is used to expand the target correlation, and represent the short and long semi-axes of the ellipse before updating.

[0037] Compared with the prior art, the present application has the beneficial effects that:

[0038] (1) The method of the present application combines neural network architecture with Kalman filter, fully utilizes the high performance of SwinTransformer in low measurement rate scene, and proposes an extended target tracking method named ST-ETT, which simultaneously estimates the kinematics and shape parameters of a single extended target from the noisy point cloud, greatly improves the accuracy and efficiency of target tracking. The Swin Transformer network makes the scene used by the present architecture more, the robustness stronger, and can cope with various complex scenes, at the same time, the model is more concise, and the training is more efficient.

[0039] (2) The method of the present application proposes an ellipse extended target estimation method based on Swin Transformer, which shows strong performance. In challenging scenes such as low measurement rate, the method of deep learning can mine the general rules of target shape parameters behind sparse measurement, greatly increasing the success rate of shape parameter identification.

[0040] (3) Compared with the neural network of end-to-end target tracking, the architecture combines the advantages of Kalman filter. In the case that some neural networks cannot completely derive the internal model mechanism, the Kalman filter and the modeling method of target shape and kinematics are used to analyze the internal model of the target state, reduce the calculation amount, improve the tracking accuracy, and also increase the interpretability of the neural network.

[0041] (4) The ellipse shape estimation method can be combined with any other kinematic estimation method, which shows that the ellipse estimation method has high portability and modularity. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The step flow chart of the method of the present application is shown in the figure;

[0043] Figure 2 The architecture diagram of the swin transformer in the present application is shown in the figure;

[0044] Figure 3 The architecture diagram of the STB module in the swin transformer architecture is shown in the figure;

[0045] Figures 4(a)-4(d) The comparison diagram of the performance of the ST-ETT method and other methods in the test example is shown in the figure;

[0046] Figure 4(a) shows the average tracking performance diagram of the ST-ETT method over time;

[0047] Figure 4(b) shows the error diagram of the average tracking error of ST-ETT minus IAE under different measurement rates and noises;

[0048] Figure 4(c) shows the error diagram of the tracking error of ST-ETT minus MEM-EKF;

[0049] Figure 4(d) shows the error diagram of the tracking error of ST-ETT minus the CNN-based ETT method. DETAILED DESCRIPTION

[0050] The present application will be further illustrated in conjunction with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0051] Embodiment 1

[0052] A method for extended object tracking based on swin-transformer, by using a swin-transformer deep learning network and a Kalman filter, a visual transformer is used to process noisy radar point cloud images; SwinTransformer, as a high-performance neural network, is widely used in target tracking. This network can directly calculate the correlation between different image blocks without passing through hidden layers, which makes it able to correctly estimate the target using less information. With this ability, Swin Transformer becomes a potential way to improve the performance of extended object tracking in low measurement rate scenarios, and to infer the shape parameters of the target; while the Kalman filter is used to estimate the kinematic parameters of the target.

[0053] A method for extended object tracking based on swin-transformer, as shown in Figure 1 , comprising the following steps:

[0054] Step S1: Normalize and discretize the measurement value, i.e. noisy point cloud, into an image.

[0055] The results of normalizing and discretizing the noisy point cloud are as follows:

[0056]

[0057]

[0058] wherein Z' k is the Cartesian coordinates of all newly arrived measurements z k centered on the estimated target center, and Z" k is the new coordinates after aligning the measurements with the target values. In the subsequent steps, Z k is discretized to form an IIMI image.

[0059] Step S2: Update the image converted in step S1 over time to form an internal intensity measurement image (IIMI).

[0060] Update the formed IIMI image, and calculate as follows:

[0061]

[0062] wherein represents the newly calculated IIMI, 0≤γ≤1 represents the forgetting factor, k represents the latest time, and u is the abbreviation of update, which represents the newly updated image.

[0063] This method can also be used to obtain normalized IIMI k :

[0064] IIMI k = 255 x IIMI k ÷ max(IIMI k )

[0065] Step S3: use swin-transformer to perform visual conversion processing on the image formed in S2, and infer the shape parameters of the target.

[0066] The normalized and updated IIMI k is input into the swin transformer architecture. The entire swin transformer architecture inputs an HxWx2 image, where H and W represent the height and width of the updated IIMI k , and then guides it to the patch partition module. This module divides the image into non-overlapping blocks, and then the linear embedding module projects the image features to the selected dimension, facilitating the subsequent processing of the swin transformer block.

[0067] The swin transformer block (STB), as shown in Figure 3 , includes a shift window-based standard multi-head self-attention (SW-MSA) module, a window-based standard multi-head self-attention (W-MSA) module, and a structure composed of two layers of multi-layer perceptron (MLP) internally with Gaussian error linear unit (GELU) nonlinearity. The window size is set to M = 7. The query dimension of each head is d = 32, and the expansion layer of each MLP is α = 4. Layer normalization (LN) technology is applied before each MSA module and each MLP module, and residual connection is applied after each module. The output dimension of the swin transformer block is set to C = 96. The patch merging module is used to further reduce the resolution of the image, and seamlessly merge adjacent 2x2 pixel regions. After continuous processing through the STB module, one patch merging is completed, and the specific process is shown in Figure 2 .

[0068] After completing all STB modules and patch merging processing, the estimated shape parameters and are generated using a fully connected layer. and are calculated, and the estimate of

[0069]

[0070] is generated. ​

[0071] and the generated together as updated estimated shape parameters, representing the angle of the shape orientation, and representing the minor and major axes of the ellipse.

[0072] Step S4: Estimate the kinematic parameters based on the original noisy point cloud and the inferred shape parameters using a Kalman filter on the estimated shape parameters from step S3.

[0073] Assume that the estimated kinematic parameters at time k-1 have been obtained the corresponding estimated covariance P k-1 and the estimated shape parameters First, the kinematic parameters of the target are predicted using a Kalman filter:

[0074]

[0075]

[0076] Z' k are the Cartesian coordinates of all newly arrived measurements z k centered at the estimated center of the target, Z" k are the new coordinates after aligning the measurements with the target values. are all newly arrived measurements, representing the rotation matrix with the measurement matrix H is

[0077] For the angle of the shape position, it can be predicted as

[0078]

[0079] After receiving the noisy point cloud at time k, the kinematic parameters are updated by the following update equation:

[0080]

[0081]

[0082]

[0083]

[0084] P k = (I - G k H) P k|k-1

[0085] ​

[0086] where P k|k-1 is the predicted error covariance, H is the observation matrix that maps the state space to the observation space, is the covariance matrix of the observation noise, G k is the Kalman gain that balances the weight of the prediction error and the measurement error, is the mean of all newly arrived measurements z k centered at the estimated target center, m k is the number of measurements is the term related to the covariance matrix of the target, which is used to extend the target correlation, and represent the short and long semi-axes of the ellipse before updating.

[0087] Step S5: The shape parameters and kinematic parameters of the target are obtained through the shape estimation of step S3 and the kinematic estimation of step S4, and the tracking of the target is completed.

[0088] Test example

[0089] In the MATLAB simulation environment with the CPU being intel i9 and the GPU being 3060, a process of a target independently moving from the origin is simulated, the target is tracked by using the ST-ETT algorithm and the CNN, IAE and MEM-EKF algorithms respectively, and finally the tracking effects are compared.

[0090] The square Gaussian Wasserstein distance (GWD) is widely used to calculate the error of the position parameters and the shape parameters. The parameters of the GWD are set as follows: the forgetting factor is set as γ=0.95, the covariance of the measurement value is set as R=I·r, r=2, the measurement rate follows a Poisson point process with λ=2, and the process noise covariance Q is set as

[0091]

[0092] The ST-ETT method proposed in the application is further compared with the existing CNN-based ETT method, IAE and MEM-EKF method, as shown in FIG. 4. FIG. 4(a) shows the average tracking performance of the ST-ETT method over time, which is compared in the form of the average square GWD; FIG. 4(b) shows the average tracking error of the ST-ETT minus the error of the IAE under different measurement rates and noises; FIG. 4(c) shows the tracking error of the ST-ETT minus the error of the MEM-EKF; and FIG. 4(d) shows the tracking error of the ST-ETT minus the error of the CNN-based ETT method.

[0093] ​Figure 4(a) shows the comparison of the average squared GWD of different methods over time, and the curve in the image shows that the tracking error of the ST-ETT method is the smallest compared with the existing methods. At the same time, the squared GWD values of these methods under different noise conditions and different measurement rates are calculated and recorded in Table 1. 3000 Monte Carlo (MC) simulations were performed for each scenario, and the final results were averaged.

[0094] Table 1 Squared GWD of different ETT methods

[0095] Measurement rate ST-ETT CNN IAE MEM-EKF λ = 0.75 2.0909 2.4946 5.4344 3.1147 λ = 1 1.997 2.4095 5.4911 3.0815 λ = 1.5 1.848 2.1624 5.3380 3.0401 λ = 2.5 1.7784 2.0224 4.9897 2.8305 λ = 4 1.5588 1.7145 4.0394 2.05731 λ = 6 1.3792 1.2571 3.4535 1.9962

[0096] Figure 4(b) 、 4(c) Figures 4(a), 4(b), 4(c) and 4(d) show the average tracking error of ST-ETT minus AE, MEM-EKF based on CNN ETT method error, respectively. As can be seen from the figure, lower values (blue) indicate that the ST-ETT method is better than other methods, and higher values (red) indicate the opposite. It is obvious that the ST-ETT method proposed in the present case is superior to the existing three measurement methods, especially when the measurement rate λ is less than 6.

[0097] In summary, the core of the method of the present case is to combine a deep learning network of swin-transformer and a Kalman filter, and use a visual converter to process the radar point cloud image from the noise. In this method, the swin-transformer is used to realize the inference of the shape parameters of the target, and the Kalman filter is used to estimate the kinematic parameters of the target. Compared with the existing methods, the proposed ST-ETT method has significant competitiveness in the low measurement rate scenario.

[0098] It should be noted that the above content only illustrates the technical idea of the present application, and cannot be used to limit the protection scope of the present application. For ordinary skilled persons in the technical field, they can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements all fall within the protection scope of the claims of the present application.

Claims

1. A method for extended object tracking based on Swin-transformer, characterized in that The noisy radar point cloud image is processed by a visual converter through a deep learning network of a swin-transformer and a Kalman filter, wherein The swin-transformer: realizes inference of a target shape parameter, at least includes one standard multi-head self-attention module based on a shift window, one standard multi-head self-attention module based on a window, and a structure internally composed of two layers of multi-layer perceptrons, and a Gaussian error linear unit is added between the two layers of multi-layer perceptrons; The Kalman filter: is used for estimating kinematic parameters of the target; The method comprises the following steps: S1, image conversion: normalizing and discretizing the noisy point cloud, and converting the point cloud into an image; S2, image updating: updating the image obtained in step S1 according to time lapse to form an internal intensity measurement image IIMI; S3, shape estimation: use swin-transformer to perform visual transformation processing on the image formed in step S2, and deduce the shape parameters of the target; the shape parameters include 、 and , represent the angle of the shape direction, and represent the short axis and long axis of the ellipse, and for , the prediction is: , )； wherein, represents the vertical velocity of the object at time k, represents the horizontal velocity of the object at time k; S4, kinematic estimation: using a Kalman filter to estimate kinematic parameters based on the original noisy point cloud and the inferred shape parameter of the target shape parameter estimated in step S3; the Kalman filter is used to predict the kinematic parameters of the target: ； +Q； wherein, is a transition matrix, is the estimated kinematic parameter at time k-1, is the kinematic parameter at time k estimated according to the Kalman filter, is the corresponding estimated covariance at time k-1, and Q is the noise; after receiving the point cloud at time k, the kinematic parameter is updated by an update equation as follows: ； ； ； ； ； ； wherein, is the predicted error covariance, is the measurement matrix, is the covariance matrix of the observation noise, is the Kalman gain, is the mean of all newly arrived measurements centered around the estimated target center is the number of measurements is the measurement is the number of measurements is the term related to the covariance matrix of the co-variance, and represent the short and long semi-axes of the ellipse before update; S5, target tracking: the shape parameter and the kinematic parameter of the target are obtained through the shape estimation in step S3 and the kinematic estimation in step S4, and the tracking of the target is completed.

2. The extended object tracking method based on Swin-transformer according to claim 1, wherein: In step S1, the method for normalizing and discretizing the noisy point cloud is as follows: ； ； wherein, is all newly arrived measurements centered at the estimated target center Cartesian coordinates of are new coordinates of the measurements aligned with the target values, is all newly arrived measurements, with a rotation matrix of , the measurement matrix H is are kinematic parameters of the object, in particular wherein, represent horizontal and vertical coordinates of the object, represent horizontal and vertical velocities of the object, and T denotes the transpose of a matrix.​​ 3. The extended object tracking method based on Swin-transformer according to claim 2, wherein: In step S2, the image updating is specifically as follows: ； wherein, represents the newly updated computed IIMI, 0≤ is a forgetting factor, u is an abbreviation for update, representing the most recently updated picture; normalized : 。 4. The extended object tracking method based on Swin-transformer according to claim 2, wherein: In step S3, the internal intensity measurement image formed in step S2 is taken as the input of the swin transformer, is segmented into non-overlapping blocks through a guided partition module, is projected to a selected dimension through a linear embedding module, and is waiting for processing by the swin transformer; the swin transformer further comprises a patch merging module for reducing the image resolution, and a fully connected layer is used to generate the estimated shape parameter.

Images