Karting racing behavior analysis method based on multi-target tracking

Abstract

The present invention discloses a kart racing behavior analysis method based on multi-target tracking. First, the position of the kart in the image is detected, a rectangular detection frame circumscribed to the kart is drawn, and image features are extracted; the motion state of the kart is predicted by Kalman filtering; similarity is calculated using image features and the detection frame, a similarity matrix is ​​output and matched, and multi-target tracking of the kart is completed; after that, the kart track is scene-calibrated to realize the mutual conversion between pixel coordinates and the coordinates of the midpoint in the real world, and the coordinate changes of the kart in the real world are output to form a complete motion trajectory and speed time series change, and then the motion trajectory is input into the network for training to obtain the kart behavior analysis result. The method of the present invention can identify the racing behavior of the kart, and record the pictures of wonderful behaviors such as vehicle collision, overtaking, and entanglement, and can also record the motion trajectories of multiple vehicles in the same picture to analyze the racing behavior between multiple karts.

Description

Technical Field

[0001] The invention belongs to the technical field of visual target tracking, and in particular relates to a kart racing behavior analysis method based on multi-target tracking. Background Art

[0002] Among various automobile sports, karts have the characteristics of high speed, small size, and changeable tracks. People cannot accurately judge the movement behavior of karts, measure the movement speed of karts, and cannot keep the movement track, so they often miss the wonderful scenes during the game. Therefore, it is necessary to detect, track, and analyze karts to obtain some wonderful sports information during the kart game.

[0003] The commonly used coil detector method can only detect the passing kart at a fixed point, has a relatively simple function, and requires the destruction of the kart track during installation and maintenance. Radar, infrared detection and other methods have expensive equipment and can only identify whether a kart has passed and measure its speed. The method using GPS requires the cooperation of other related equipment, and the GPS signal will be affected indoors. The scope of use of the above methods is subject to many restrictions.

[0004] Analyzing the surveillance video of the kart race through computer vision technology can collect a variety of information, and the development and maintenance costs are low, so it can be widely used. Since the monitoring system collects a large number of videos, there are some fragments in the video that have no usable information. Using deep learning target detection technology, redundant video fragments in the surveillance video can be eliminated. Compared with traditional moving target detection technology, the influence of other non-target moving objects can be eliminated. Summary of the invention

[0005] The purpose of the present invention is to provide a karting competition behavior analysis method based on multi-target tracking, so as to solve the problem in the prior art that the movement behavior cannot be accurately identified in karting competitions.

[0006] The technical solution adopted by the present invention is a kart racing behavior analysis method based on multi-target tracking, which is specifically implemented according to the following steps:

[0007] Step 1: Create a kart target detection dataset and train the YOLOv3 network weights required for kart target detection;

[0008] Step 2, select the kart monitoring video to be processed, read the first frame image of the video, use the YOLOv3 target detection network and the network weights trained in step 1 to detect the position of the kart in the video frame image, and draw the circumscribed rectangular detection frame of each kart to extract the image features of the kart;

[0009] Step 3, read the second frame image of the video, use the YOLOv3 target detection network and network weights to detect the position of the kart in the video frame image, draw the circumscribed rectangular detection frame of the kart, and extract the image features of the kart; then predict the motion state of the kart through Kalman filtering;

[0010] Step 4, calculate the similarity between the image features and the detection box obtained in steps 2 and 3, and output the similarity matrix;

[0011] Step 5: Use the Hungarian algorithm and the similarity matrix obtained in step 4 to match the historical frame detection frame ID and the current prediction frame to complete the multi-target tracking of the kart;

[0012] Step 6, using the direct linear transformation method to calibrate the kart track scene, realize the mutual conversion between the pixel coordinates in the video and the coordinates of the midpoint in the real world, and output the coordinate changes of the kart in the real world;

[0013] Step 7, repeat steps 3 to 6 until all the images in the video are read, output the complete motion trajectory and speed time series changes of each kart in the video, and save them in a csv format file;

[0014] Step 8: Input the kart motion trajectory csv file with vehicle collision, overtaking, and fighting in step 7 into the LSTM network for training until the network loss function converges, and save the weights obtained from the training;

[0015] Step 9, read the csv format trajectory and speed data obtained by tracking the kart in step 7 through the trained LSTM network, and output classification labels including collision, overtaking, entanglement or no special behavior to obtain the kart behavior analysis results in the video.

[0016] The present invention is also characterized in that:

[0017] In step 1, the specific process is as follows:

[0018] Step 1.1, split the kart monitoring video into frame images, and use the labelimg target detection and annotation tool to annotate the kart data in each frame image to form a kart detection dataset;

[0019] Among them, the annotated information includes filename: the file name of the image; name: the name of the annotated object; xmin, ymin: the upper left corner coordinates of the kart position; xmax, ymax: the lower right corner coordinates of the kart position;

[0020] Step 1.2: Use the YOLOv3 network to train the kart detection dataset produced in step 1.1 until the loss function converges and save the trained network weight file.

[0021] In step 4, specifically: the cosine distance calculation is performed on the two image features extracted from the detection frames of step 2 and step 3; the Mahalanobis distance calculation is performed on the detection frame predicted by the Kalman filter algorithm in step 3 and the detection frame in step 2, the two distances are fused and input into the matrix of similarity calculation, and the similarity matrix is ​​output.

[0022] In step 8, specifically:

[0023] Step 8.1, storing the kart racing behavior trajectory and speed data obtained by target tracking in a .csv file in chronological order;

[0024] Step 8.2, use Python to read the .csv file and store it in the data structure list; encode the data labels in a one-hot manner, [1,0,0] indicates collision, [0,0,1] indicates overtaking, and [0,1,0] indicates tangle;

[0025] Step 8.3, convert the data and labels into tensor form supported by the tensorflow framework, transfer them to the LSTM network and add attention mechanism training, and save the trained network weights as .h5 files.

[0026] The beneficial effects of the present invention are that the method of the present invention can identify the racing behavior of karts and record images of exciting behaviors such as vehicle collision, overtaking, and fighting, providing drivers and spectators with review, appreciation and souvenirs; it can also identify and record the movement trajectories of multiple vehicles in the same image, and analyze the behaviors between multiple karts; compared with traditional methods, it can make kart detection more accurate, more efficient, and more intelligent, and eliminate interference from other non-target moving objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a kart racing behavior analysis method based on multi-target tracking of the present invention;

[0028] Figure 2 is a kart tracking result diagram in the method of the present invention;

[0029] Figure 3 It is a diagram of the kart behavior analysis result in the method of the present invention. DETAILED DESCRIPTION

[0030] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] The present invention is based on the kart racing behavior analysis method of multi-target tracking, such as Figure 1 As shown, the specific implementation steps are as follows:

[0032] Step 1: Create a kart target detection dataset and train the YOLOv3 network weights required for kart target detection. The specific process is as follows:

[0033] Step 1.1, split the kart monitoring video into frame images, and use the labelimg target detection and annotation tool to annotate the kart data in each frame image to form a kart detection dataset;

[0034] The annotated information includes filename: the file name of the image; name: the name of the annotated object (kart); xmin, ymin: the upper left corner coordinates of the kart position; xmax, ymax: the lower right corner coordinates of the kart position;

[0035] Step 1.2, use the YOLOv3 network to train the kart detection dataset produced in step 1.1 until the loss function converges, and save the trained network weight file;

[0036] Step 2, select the kart monitoring video to be processed, read the first frame image of the video, use the YOLOv3 target detection network and the network weights trained in step 1 to detect the position of the kart in the video frame image, and draw the circumscribed rectangular detection box (ROI) of each kart to extract the image features of the kart;

[0037] Step 3, read the second frame image of the video, use the YOLOv3 target detection network and the network weights trained in step 1 to detect the position of the kart in the video frame image, draw the circumscribed rectangular detection box (ROI) of each kart, and extract the image features of the kart; then predict the motion state of the kart, that is, the position and moving speed of the detection box, through Kalman filtering;

[0038] The Kalman filter algorithm is used to estimate the motion state of the object, that is, the position and speed of the detection frame. Based on the information of the historical frame, the position and estimated value of the detection frame of the current frame can be obtained. Based on the detector of the existing frame, the observed value of the current detection frame position can be obtained. The Kalman filter combines the estimated value and the observed value to obtain a more accurate value.

[0039] (1) Prediction stage: Predict the position of the current box based on the information of the previous bounding box movement.

[0040] Forecast the current best estimate:

[0041] Predict the current covariance matrix: p k =Fk P k-1 F T +Q k

[0042] (2) Correction stage:

[0043] Calculate the Kalman gain:

[0044] Update best estimate:

[0045] Update the covariance matrix: P′ k =P k -K′H k P k

[0046] The input variable x k =[cx,cy,w,h,vx,vy,vw,vh] is the momentum of the four-dimensional detection box related variables and the four-dimensional detection box related variables, cx, cy is the center point of the bounding box; w, h is the width and height of the bounding box, vx, vy, vw, vh is the change rate of the above variables; F is the state transfer matrix, which is used to model the motion of the target; P k represents the position accuracy (i.e., the covariance matrix); Q k is the noise in the environment, i.e., the interference caused in the prediction process; k′ is the filter gain matrix, which is the intermediate calculation result of the filter; H k It is the conversion matrix from state variables to observation values, which converts the m-dimensional measurement values ​​into n-dimensional ones to conform to the mathematical form of the state variables; R is the measurement noise covariance;

[0047] By correcting the current frame, we can get the current estimated value, take the estimated value to the next frame, and predict the value of the next frame. After the prediction, we further iterate through correction. Through continuous iterative Kalman filter estimation, the motion state becomes more and more accurate.

[0048] Step 4, calculate the similarity between the image features and the detection box obtained in steps 2 and 3, and output the similarity matrix;

[0049] Specifically, the cosine distance of the two image features extracted from the detection frames in step 2 and step 3 is calculated; the Mahalanobis distance of the detection frame predicted by the Kalman filter algorithm in step 3 and the detection frame in step 2 is calculated, and the two distances are fused and input into the matrix of similarity calculation, and the similarity matrix is ​​output;

[0050] Step 5, using the Hungarian algorithm and the similarity matrix obtained in step 4 to match the historical frame detection frame ID and the current prediction frame (i.e., it is concluded that the kart 1 detected in the current frame is the kart with ID 1 in the previous frame), completing the multi-target tracking of the kart;

[0051] Step 6, calculate the speed of the kart and extract the motion trajectory of the kart; use the direct linear transformation (DLT) method to calibrate the scene of the kart track, realize the mutual conversion between the pixel coordinates in the video and the coordinates of the midpoint in the real world, that is, the coordinate changes of the kart in the real world can be output;

[0052] The specific process is as follows:

[0053] Step 6.1, place four feature point (dot) markers on the kart track. The four points form a rectangle. Take a point in the rectangle as the origin of the coordinate system to establish the kart track plane coordinate system. Then measure the distance between each point to obtain the coordinates of the four points on the kart track plane coordinate system. The internal and external parameters of the camera can be obtained through the above four points. The mapping relationship between the kart track plane coordinate P = (X, Y, Z) and the image coordinate p = (x, y) is:

[0054]

[0055] Among them, s is a non-zero real number, K is the camera intrinsic parameter matrix; R is the rotation matrix, T is the translation matrix, which belongs to the camera extrinsic parameter;

[0056] Step 6.2, intercept the track image of the four marking points of the surveillance camera, use Hough circle detection to detect the feature point (dot) marks in the image, and output the pixel coordinates of the center of the circle;

[0057] Step 6.3, using the above information, the direct linear transformation (DLT) method is used to find the conversion relationship between the kart track plane coordinate system and the track image coordinate system; according to the formula in step 6.1, the expression of the mapping F from the kart track plane coordinate to the image coordinate is obtained:

[0058]

[0059] Where A = KR = (a ij ), 1≤i,j≤3, B=KT=(b1,b2,b3) T .

[0060] Step 7, repeat steps 3 to 6 until every frame of the video is read, output the complete motion trajectory and speed time series changes of each kart in the video, and save them in a csv format file;

[0061] Step 8: Input the csv file of the kart motion trajectory with vehicle collision, overtaking, and fighting in step 7 into the LSTM network for training until the network loss function converges, and save the weights obtained from the training; specifically:

[0062] Step 8.1, storing the kart racing behavior trajectory and speed data obtained by target tracking in a .csv file in chronological order;

[0063] Step 8.2, use Python to read the .csv file and store it in the data structure list; encode the data labels in a one-hot manner, [1,0,0] indicates collision, [0,0,1] indicates overtaking, and [0,1,0] indicates tangle;

[0064] Step 8.3, convert the data and labels into tensor forms supported by the tensorflow framework, transfer them to the LSTM network and add attention mechanism training, and save the trained network weights as .h5 files;

[0065] Step 9, read the csv format trajectory and speed data obtained by kart tracking in step 7 through the trained LSTM network, and output classification labels including collision, overtaking, entanglement or no special behavior to obtain the kart behavior analysis results.

[0066] The method of the present invention can effectively and accurately identify the kart in the video frame, mark the position of the kart, and is tested on a dataset of 400 kart images obtained from surveillance video frames of 8 different scenes. The results are shown in Table 1, and the average detection accuracy is between 91.38% and 99.76%.

[0067] Table 1 Kart test results

[0068]

[0069] The above evaluation indicators are as follows:

[0070] Precision: The ratio of correctly predicted positive samples to all predicted positive samples, that is, the ratio of the targets detected by the model to the real target objects.

[0071]

[0072] Recall rate: The ratio of correctly predicted positive samples to all actual positive samples, that is, the ratio of all real targets detected by the model.

[0073]

[0074] Average precision is the average of the precision values ​​on the Precision-Recall (PR) curve. It is calculated by integrating the PR curve.

[0075] Figure 2 Tracking results for go-kart, Figure 3 The figure shows the results of kart behavior analysis. It can be seen from the figure that the method can identify the racing behavior of karts. In addition, the method of the present invention can track different vehicles in continuous video frames and draw their motion trajectories. The multi-target tracking accuracy of the method is measured to be 90.6% in a video with a duration of 30s and a frame rate of 25fps.

[0076] The accuracy of MOTA (Multiple Object Tracking Accuracy) multi-target tracking is shown in the formula:

[0077]

[0078] Where, IDSW t Indicates the total number of ID changes within time t, FN t (False Negatives) is the total number of missed detections within time t, that is, the number of true values ​​that were not successfully matched; FP t (False Positives) is called the total number of false positives, which is the number of wrongly predicted tracking results. t (Ground Truth) is the number of tracked targets within time t.

[0079] The method of the present invention can analyze the competition behavior during the kart competition in the monitoring video and record the wonderful pictures of vehicle collision, overtaking, fighting and other behaviors, providing drivers and spectators with review, appreciation and souvenirs. Automatically measure the movement speed of the kart and retain the movement trajectory. The kart detection method using deep learning can make the kart detection more accurate, efficient and intelligent than the traditional method, and eliminate the interference of other non-target moving objects. The kart tracking method using multi-target tracking can identify and record the movement trajectories of multiple vehicles in the same picture, and analyze the behavior between multiple karts. It has low development and maintenance costs, reduces resource waste, has a high degree of automation and is widely applicable to the analysis of kart competition behavior.

Claims

1. A kart racing behavior analysis method based on multi-target tracking, characterized in that: Follow the steps below to implement it: Step 1: Create a kart target detection dataset and train the YOLOv3 network weights required for kart target detection; Step 2, select the kart monitoring video to be processed, read the first frame image of the video, use the YOLOv3 target detection network and the network weights trained in step 1 to detect the position of the kart in the video frame image, and draw the circumscribed rectangular detection frame of each kart to extract the image features of the kart; Step 3, read the second frame image of the video, use the YOLOv3 target detection network and network weights to detect the position of the kart in the video frame image, draw the circumscribed rectangular detection frame of the kart, and extract the image features of the kart; then predict the motion state of the kart through Kalman filtering; Step 4, calculate the similarity between the image features and the detection box obtained in steps 2 and 3, and output the similarity matrix; Step 5: Use the Hungarian algorithm and the similarity matrix obtained in step 4 to match the historical frame detection frame ID and the current prediction frame to complete the multi-target tracking of the kart; Step 6, using the direct linear transformation method to calibrate the kart track scene, realize the mutual conversion between the pixel coordinates in the video and the coordinates of the midpoint in the real world, and output the coordinate changes of the kart in the real world; Step 7, repeat steps 3 to 6 until all the images in the video are read, output the complete motion trajectory and speed time series changes of each kart in the video, and save them in a csv format file; Step 8: Input the kart motion trajectory csv file with vehicle collision, overtaking, and fighting in step 7 into the LSTM network for training until the network loss function converges, and save the weights obtained from the training; Step 9, read the csv format trajectory and speed data obtained by kart tracking in step 7 through the trained LSTM network, output classification labels including collision, overtaking, entanglement or no special behavior, and get the kart behavior analysis results.

2. The karting competition behavior analysis method based on multi-target tracking according to claim 1 is characterized in that: In step 1, the specific process is as follows: Step 1.1, split the kart monitoring video into frame images, and use the labelimg target detection and annotation tool to annotate the kart data in each frame image to form a kart detection dataset; Among them, the annotated information includes filename: the file name of the image; name: the name of the annotated object; xmin, ymin: the upper left corner coordinates of the kart position; xmax, ymax: the lower right corner coordinates of the kart position; Step 1.2: Use the YOLOv3 network to train the kart detection dataset produced in step 1.1 until the loss function converges and save the trained network weight file.

3. The karting competition behavior analysis method based on multi-target tracking according to claim 1 is characterized in that: In the step 4, specifically: the cosine distance calculation is performed on the two image features extracted from the detection frames of step 2 and step 3; the Mahalanobis distance calculation is performed on the detection frame predicted by the Kalman filter algorithm in step 3 and the detection frame in step 2, the two distances are fused and input into the matrix of similarity calculation, and the similarity matrix is ​​output.

4. The karting competition behavior analysis method based on multi-target tracking according to claim 1 is characterized in that: In the step 8, specifically: Step 8.1, storing the kart racing behavior trajectory and speed data obtained by target tracking in a .csv file in chronological order; Step 8.2, use Python to read the .csv file and store it in the data structure list; encode the data labels in a one-hot manner, [1,0,0] indicates collision, [0,0,1] indicates overtaking, and [0,1,0] indicates tangle; Step 8.3, convert the data and labels into tensor form supported by the tensorflow framework, transfer them to the LSTM network and add attention mechanism training, and save the trained network weights as .h5 files.

Images