Sea surface multi-target tracking method and system based on Gaussian distance matching

Abstract

A sea surface multi-target tracking method and system based on Gaussian distance matching, adopt a target detector to obtain the pre-detection result of each frame in the video, after using Kalman filtering to make prior estimation on the track formed by the pre-detection result, in turn ① using IoU distance to associate the track and the high-score detection box, ② using IoU distance to associate the remaining track in step ① and the low-score detection box, and ③ using Gaussian distance to associate the remaining track in step ② and the remaining high-score detection box in step ①, after comprehensively associating the results of the three times, using the Kalman filter centered on observation to perform posterior estimation to obtain the track of the current frame, realizing multi-target tracking. The present application is aimed at real sea surface scene and target characteristics, adopts TBD paradigm and only associates targets through motion information, through being equipped with Gaussian distance cascade matching, Kalman filter module centered on observation, thereby designing a multi-target tracker capable of adapting to sea surface low frame rate, observation platform shaking, target nonlinear motion and other challenges.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a sea surface multi-target tracking method and system based on Gaussian distance matching. BACKGROUND

[0002] The existing multi-target tracking algorithm based on deep learning can be mainly divided into a tracking-by-detection (TBD) paradigm and a joint-detection-embedding (JDE) paradigm. Unlike the field of pedestrian re-identification, the weather conditions of the sea surface scene are relatively complex, the target distance is relatively far, and the position of the sea surface target changes greatly between frames. Whether it is a relatively low video frame rate, a severe shaking of the observation platform caused by the sea waves in the open sea, or a fast-moving sailboat, speedboat, etc., all of them cause the intersection over union (IOU) between the same target in the adjacent two frames to be very small, even 0. As shown in FIG. 1, especially the small target far away is more difficult to associate. The existing matching rule based on IOU will cause a very serious performance decline. In addition, the sea surface target cannot be simply assumed to be uniformly linear motion. Because of the relatively low video frame rate, especially the severe shaking of the observation platform caused by the sea waves in the open sea, the observation value will often deviate greatly from the trajectory. As shown in FIG. 2, the trajectory prediction drawn on the right, especially in the open sea scene, is relatively common. This challenge causes the use of traditional Kalman filtering (based on the assumption of uniform linear motion of the target) to cause a very serious performance decline. Figure 1 SUMMARY Figure 1 The present application is directed to the above-mentioned deficiencies in the prior art, and proposes a sea surface multi-target tracking method and system based on Gaussian distance matching. For real sea surface scenes and target characteristics, the TBD paradigm is adopted and the target is associated only through motion information. By equipping a Gaussian distance cascade matching and a Kalman filter module centered on observation, a multi-target tracker that can adapt to the challenges of low frame rate, shaking of the observation platform, and non-linear motion of the target on the sea surface is designed.

[0003] The present application is realized by the following technical solutions:

[0004] The present application is realized by the following technical solutions:

[0005] The present application relates to a sea surface multi-target tracking method based on Gaussian distance matching. A target detector is used to obtain the pre-detection result of each frame in the video. After the Kalman filter performs prior estimation on the trajectory formed by the pre-detection result, ① the IoU distance is used to associate the trajectory and the high-score detection box, ② the IoU distance is used to associate the remaining trajectory in step ① and the low-score detection box, and ③ the Gaussian distance is used to associate the remaining trajectory in step ② and the remaining high-score detection box in step ①. After the three association results are comprehensively used, the Kalman filter centered on observation is used to perform posterior estimation to obtain the trajectory of the current frame, and multi-target tracking is realized. ​

[0006] The pre-detection result includes: a high-score detection frame with a confidence greater than 0.6 and a low-score detection frame with a confidence greater than 0.1 and less than or equal to 0.6.

[0007] The Kalman filter firstly predicts the position of the track in the next frame before association, and then corrects the position of the track online by directly using the detection frame after association, so as to avoid the large error of the Kalman filter posterior estimation caused by the Gaussian distance association.

[0008] The multi-target tracking method reinitializes the track when there are still remaining high-score detection frames after step ③, that is, the confidence of the remaining high-score detection frames of two consecutive frames is greater than 0.7, and the specific operation is: assigning a new identity to the remaining high-score detection frame and initializing a new track.

[0009] The multi-target tracking method marks a lost state when there is still a remaining track after step ③, that is, there is no detection frame that can be successfully associated with the remaining track for 30 consecutive frames, and the specific operation is: marking the remaining track as a lost state and deleting the remaining track.

[0010] The present application relates to a system for implementing the above method, comprising: a video stream unit, a target detection algorithm unit, a multi-target tracking algorithm unit and an interface display and control unit, wherein: the video stream unit reads the camera or local video stream, and uses the ROS2 topic asynchronous communication mechanism to output the source image to the target detection algorithm unit; the target detection algorithm unit uses the deployed TensorRT model to infer the detection frame result according to the real-time received source image; the multi-target tracking algorithm unit performs target association, track initialization and lost track removal according to the real-time received detection frame, and generates the detection frame and identity result; and the interface display and control unit performs image superposition drawing display according to the real-time received source image, detection frame and identity result.

[0011] Each unit is provided with a functional module and a ROS2 communication module, so that the video stream unit, the target detection algorithm unit and the multi-target tracking algorithm unit perform real-time communication through the way of publishing and subscribing to specific name topics, wherein: the source image, the detection frame, the detection frame and the identity result are the communication interface and data format of the topic; the interface display and control unit communicates with the ROS2 communication module of other units through the signal and slot mechanism of Qt, and finally displays the image and data of the detection tracking result on the interface.

[0012] Technical effects

[0013] The present application solves the problem of large posterior estimation deviation by using the Gaussian distance cascade matching, that is, first using the IoU distance matching, and then using the Gaussian distance matching, significantly improving the tracking accuracy and efficiency: the multi-target tracking accuracy MOTA is improved from 34.6% to 46.9%; the target association capability IDF1 is improved from 30.8% to 49.2%; and the overall performance S is improved from 27.3% to 44.4%. At the same time, using the present technical means only reduces the running speed from 35.4FPS to 35.2FPS, almost no additional calculation amount. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A sea scene diagram is shown for the prior art;

[0015] Figure 2 A system diagram is shown for the present application;

[0016] Figure 3 A flowchart is shown for the present application;

[0017] Figure 4 A Kalman filter failure diagram is shown for the embodiment;

[0018] Figure 5 A training loss function change diagram is shown for the embodiment;

[0019] Figure 6 A training set accuracy change diagram is shown for the embodiment;

[0020] Figure 7 A tracking performance diagram of the present application under different CMGD threshold parameters σ is shown for the embodiment;

[0021] Figure 8 A tracking performance diagram of ByteTrack and the present application under different detection results is shown for the embodiment. DETAILED DESCRIPTION

[0022] As shown in Figure 3 , the present embodiment relates to a sea surface multi-target tracking method based on Gaussian distance matching, comprising:

[0023] Step 1, target detection: using YOLOv7 neural network to detect high-score bounding boxes and low-score bounding boxes according to the input frame, if the confidence of the bounding box is greater than 0.6, it is classified as a high-score bounding box; if the confidence of the bounding box is greater than 0.1 and less than 0.6, it is classified as a low-score bounding box.

[0024] Step 2, observation-centered Kalman filter prior estimation, that is, according to the position and speed of the target in the last frame, the position of the target in the current frame is predicted. Specifically:

[0025] 2.1) In the task of multi-target tracking, a state space is established for each target Where: (u, v), a, h represent the position, aspect ratio, height of the detection box and track in the image pixel coordinate system respectively. represents the rate of change of the corresponding variable (the difference between the variable in the previous and the current frame). Then the state equation of the system is: Where: n ~ N(0, Q), v ~ N(0, R), A is the state transition matrix; H is the measurement matrix; n and v obey Gaussian distribution; Q, R are the system process noise and observation noise.

[0026] 2.2) According to the assumption of linear motion, the prior estimate of the state variable in the tracking process is calculated by the following equation Where: the definition of the state variable is seen in 2.1) state space, the subscript k of the state variable represents the current state, and k-1 represents the previous state.

[0027] Step 3, use IoU distance to associate the track and high-score detection box. Set the high-score detection box set D, a total of m detection targets, the track prediction box set T, a total of n tracks. Then the cost matrix C of the Hungarian matching is m x n, then for any d i ∈D, any t j ∈T, the IoU distance between them is the cost matrix element c i，j , which can be calculated by the following formula: Where: area() represents the area of the bounding box. c i，j is larger, the greater the overlap rate of the two bounding boxes, the maximum is 1, which represents complete overlap, and the minimum is 0, which represents complete non-overlap. After obtaining the cost matrix, the problem can be converted into an optimal bipartite matching problem, and the Hungarian matching is used to realize matching according to the minimum loss criterion.

[0028] Step 4, use IoU distance to associate the remaining tracks of step 3 and low-score detection boxes. Set the remaining tracks after step 3 as T, and set the low-score detection box set D. Take D and T as input, and repeat step 3 to realize the matching of the remaining tracks and low-score detection boxes.

[0029] Step 5, use Gaussian distance to associate the remaining tracks of step 3 and the remaining high-score detection boxes of step 4. Specifically:

[0030] 5.1) Set the remaining detection and tracks after the two IoU associations of step 3 and step 4 as: remain D m ={d1,…d m}, tju ) 2 +(d iv -t iv ) 2 +(d ia -t ja ) 2 +(d ih -t ih ) 2 where (u,v), a and h are the horizontal and vertical coordinates of the target or track in pixel coordinate system, aspect ratio, height.

[0031] 5.2) Calculate the matching cost matrix C={c ij} based on the Gaussian distance, where σ is the maximum Gaussian matching distance. Considering the low frame rate and the shaking of the observation platform, σ is set to 140. Similar to the IoU distance, the Hungarian matching is adopted, and the matching threshold is set to 0.98. Due to the high matching threshold, when the area ratio between the target and the track is > 4 or < 1 / 4, the matching is rejected, thereby greatly improving the tracking performance in the open sea.

[0032] Step 6, integrate the three association results to use the observation-centered Kalman filter to perform posterior estimation to obtain the track of the current frame, specifically: set the target detection result, i.e. the observation value as d k , if successful matching, use the observation value d k to directly replace the posterior estimation. Considering the low frame rate and nonlinear motion, the posterior estimation of the Kalman filter is large, and the present application directly uses the more accurate detection result as the final result.

[0033] The present project adopts common data enhancement methods such as Mosaic, left-right flipping, translation, scale change, HSV domain enhancement, and certain angle rotation enhancement (to simulate ship body shaking).

[0034] The evaluation tracking performance indicators adopted by the present application include multi-target tracking accuracy (MOTA), false positive (FP), false negative (FN), IDS (identity switching), and IDF1 (identity score) and the like. MOTA is calculated from FP, FN and IDS. Considering that the values of FP and FN are much larger than IDS, and MOTA is biased towards the evaluation of detection performance. IDF1 has the ability to evaluate target identity, so more attention is paid to the performance of IDF1 on association performance.

[0035] In addition, in order to reflect the MOT performance of different categories, the following harmonic mean formula is adopted: where MOTA i and IDF1 i represent the indicator value corresponding to category i. S iThe harmonic mean is used. In the calculation process, when MOTA i Less than 0, the corresponding value S i = 0.

[0036] In view of the different number and importance of different categories, a weighted average is used to describe the final evaluation index

[0037] Through specific actual experiments, under the environment of platform: RTX 3090*4; batchsize: 20; iteration period: 100; learning rate: 0.01, cosine decay, the training resolution is set to 1920*1920, the test resolution is set to 1920*1088, and the model size is set to yolov7-w6, the simulation results are as follows:

[0038] Firstly, the present application is compared with various classical algorithms, including SORT, ByteTrack and OC-SORT. For fairness, all algorithms select YOLOv7-W6 as the target detector.

[0039] Table 1-quantitative comparison results of the present application and various classical algorithms

[0040]

[0041]

[0042] ByteTrack and OC-SORT have very limited improvement on SORT when facing bad sea conditions. ByteTrack increases the present application from 34.4 to 36.6, IDF1 from 29.7 to 30.8, and S from 26.7 to 27.3. OC-SORT increases the present application from 34.4 to 35.0, IDF1 from 29.7 to 30.4, and S from 26.7 to 27.8. The present application has the best performance in 7 of the 9 evaluation indexes, which are MOTA, IDF1, MT, ML, FN, IDS and FM. Since the calculation consumption of the data association part is low, the speed of the present application is almost not slowed down (i.e. FPS from 35.5 to 35.2).

[0043] Table 2-harmonic weighted average index S comparison results of the present application and various classical algorithms

[0044]

[0045] As can be seen from Table 2, the present application has greatly improved the tracking performance for the fishing boat and cargo ship categories. This is because they often appear in stormy seas, where there are stronger waves, causing the observation platform to swing. As for the sailboat and speedboat, they often have nonlinear or violent motion, especially in low frame rate videos. Due to the small size of the buoy, it is often not associated in previous algorithms. For example, SORT, ByteTrack, OC-SORT, etc. only use the IoU distance algorithm. The present application uses OCKF to handle nonlinear motion and CMGD to handle the swing of the observation platform or IoU distance association failure. The above experimental results prove the improvement in tracking performance and effectiveness brought by the present application.

[0046] ByteTrack is used as the baseline method for comparison. To clearly show the functions of the CMGD and OCKF modules, the modules are added step by step for comparison.

[0047] Table 3 - Gaussian distance cascade matching + observation-centered Kalman filter ablation experiment

[0048]

[0049] ByteTrack can only achieve a performance of 34.6% MOTA, 30.8% IDF1, and 27.3% S. When only adding the Gaussian distance cascade matching module, IDF1 increases from 30.8% to 41.1%. When only adding the observation-centered Kalman filter module, MOTA increases from 34.6% to 41.3%. Due to the lack of effective Gaussian distance association, performance is often poor in cases of observation platform swing or IoU association failure. After adding both key modules at the same time, MOTA increases from 34.6% to 46.9%, IDF1 increases from 30.8% to 49.2%, and S increases from 27.3% to 44.4%. The time consumption has almost no significant change.

[0050] Table 4 - Comparison table of different Kalman filter selection

[0051]

[0052] As can be seen from Table 4, the observation-centered Kalman filter can improve the association performance better than the KF-free, because the observation-centered Kalman filter preserves the prior estimate, i.e. the position and velocity of the predicted trajectory in the next frame according to the current state space.

[0053] Table 5 - Detection model depth + detection model input image resolution

[0054]

[0055] The model size and input size of the target detector in the application are two key factors affecting the balance between precision and speed. As shown in Table 5, the detection running time increases significantly with the model size and input size, and the tracking performance also improves greatly. Given the large number of small targets and their blurred appearance in real sea surface videos, a large resolution like 1088*1920 is an effective means to ensure good detection results as input for data association. When the performance of the target detector is good enough, a suitable data association algorithm like the application can further improve the tracking performance to a great extent.

[0056] Compared with the prior art, the application adopts YOLOv7 as the detector of the algorithm to provide better detection results as input for the data association part. To adapt to the sea surface scene, the data association part is additionally designed with Gaussian distance cascade matching and observation-centered Kalman filtering module. The application can be applied to the fields of video detection and tracking such as unmanned ship reconnaissance and sea surface target search and rescue.

[0057] The above specific embodiments can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the application. The protection scope of the application is subject to the claims and is not limited by the above specific embodiments. Each implementation scheme within the scope is subject to the constraints of the application.

Claims

1. A method for tracking multiple targets on the sea surface based on Gaussian distance matching, characterized in that: An object detector is used to obtain the pre-detection results of each frame in the video. After using the Kalman filter to perform a priori estimation of the trajectory formed by the pre-detection results, the trajectory is sequentially associated with the high-scoring detection frame using ① the Intersection over Union (IoU) distance, the remaining trajectory in step ① and the low-scoring detection frame using ② the IoU distance, and the remaining trajectory in step ② and the remaining high-scoring detection frame in step ① using ③ the Gaussian distance. After combining the three association results, the observation-centered Kalman filter is used to perform a posteriori estimation to obtain the trajectory of the current frame, thus achieving multi-target tracking. The a priori estimation is to predict the position of the target in the current frame based on the position and speed of the target in the previous frame, specifically: 2.1) In the task of multi-target tracking, a state space is established for each target ,in: 、 、 Represent the position, aspect ratio, and height of the detection frame and trajectory in the image pixel coordinate system, respectively. , , Represents the rate of change of the corresponding variable, then the state equation of the system is: ,in: , A is the state transfer matrix; H is the measurement matrix; n and v obey Gaussian distribution; Q, R are the system process noise and observation noise; 2.2) Based on the assumption of linear motion, during tracking, the prior estimate of the state variables is solved by the following equation , , , , where: the definition of state variables is as in the state space of 2.1), the subscript k of the state variable represents the current state, and k-1 represents the previous state; The use of IoU distance to associate trajectories and high-score detection frames means: according to the high-score detection frame set D, there are m detection targets, the trajectory prediction frame set is T, there are n trajectories, the cost matrix of Hungarian matching is C, and the shape is m×n, then for any , any , then the IoU distance between them is the cost matrix element , where: area() represents the area of ​​the bounding box. After obtaining the cost matrix, the problem can be transformed into an optimal bipartite matching problem, and Hungarian matching is used to achieve matching based on the minimum loss criterion; The use of Gaussian distance to associate the remaining tracks in step ② with the remaining high-score detection frames in step ① refers to: 5.1) The remaining detections and trajectories after two IoU associations are: , then the squared Gaussian distance between each detection and trajectory is: ; 5.2) Calculate the matching cost matrix based on Gaussian distance ,in: , is the maximum Gaussian matching distance. Considering the low frame rate and the shaking of the observation platform, Hungarian matching is used. When the area ratio between the target and the trajectory is >4 or <1 / 4, the matching is rejected.

2. The method for tracking multiple targets on the sea surface based on Gaussian distance matching according to claim 1, wherein: The Kalman filter first predicts the position of the trajectory in the next frame before association, and then corrects the position of the trajectory online by directly using the detection frame after association, thereby avoiding the excessive error of the Kalman filter posterior estimation caused by using Gaussian distance association.

3. The method for tracking multiple targets on the sea surface based on Gaussian distance matching according to claim 1, wherein: In the multi-target tracking method, when there are still remaining high-score detection frames after step ③, that is, the confidence of the remaining high-score detection frames of two consecutive frames is greater than 0.7, the trajectory is re-initialized, specifically: a new identity is assigned to the remaining high-score detection frames and a new trajectory is initialized.

4. The method for tracking multiple targets on the sea surface based on Gaussian distance matching according to claim 1, wherein: In the multi-target tracking method, if there are still remaining tracks after step ③, that is, no detection box can be successfully associated with it for 30 consecutive frames, the tracking state is marked as lost. Specifically, the remaining tracks are marked as lost and deleted.

5. The method for tracking multiple targets on the sea surface based on Gaussian distance matching according to any one of claims 1 to 4, characterized in that: The object detector uses the YOLOv7 neural network to detect the input frame to obtain a high-score detection frame and a low-score detection frame.

6. The method for tracking multiple targets on the sea surface based on Gaussian distance matching according to any one of claims 1 to 4, characterized in that: The aforementioned step ① of using IoU distance to associate the remaining track and the low-score detection frame refers to: according to the track T remaining after the IoU distance is associated with the high-score detection frame and the low-score detection frame set is D, D and T are used as input, and the IoU distance is repeatedly used to associate the track and the high-score detection frame to achieve matching of the remaining track and the low-score detection frame.

7. The method for tracking multiple targets on the sea surface based on Gaussian distance matching according to any one of claims 1 to 4, characterized in that: The posterior estimation is to use the observation-centered Kalman filter to perform posterior estimation based on the three correlation results to obtain the trajectory of the current frame. Specifically, let the target detection result, that is, the observation value, be If a match is successful, the observed value is used Directly replace the posterior estimate.

8. A system for implementing the sea surface multi-target tracking method based on Gaussian distance matching as described in any one of claims 1 to 7, characterized in that: include: Video stream unit, target detection algorithm unit, multi-target tracking algorithm unit and interface display and control unit, among which: the video stream unit reads the camera or local video stream and uses the ROS2 topic asynchronous communication mechanism to output the source image to the target detection algorithm unit; the target detection algorithm unit uses the deployed TensorRT model to infer the detection box result based on the source image received in real time; The multi-target tracking algorithm unit performs target association, trajectory initialization, and removal of lost tracks based on the detection frame received in real time, and generates a detection frame and identity identification result; the interface display and control unit performs image overlay drawing and display based on the source image, detection frame, and identity identification result received in real time.

9. The system according to claim 8, wherein: Each unit is equipped with a functional module and a ROS2 communication module, which enables the video stream unit, target detection algorithm unit and multi-target tracking algorithm unit to communicate in real time by publishing and subscribing to topics with specific names, where the source image, detection frame, detection frame and identification result are the communication interface and data format of the topic; the interface display and control unit communicates with the ROS2 communication modules of other units through the Qt signal and slot mechanism, and finally the images and data of the detection and tracking results are displayed on the interface.

Images