A sea surface multi-target multi-class tracking method based on improved D-IOU matching

Abstract

The application discloses a kind of sea surface multi-target multi-class tracking method based on improved D-IOU matching, to solve the multi-target tracking challenge in sea surface environment.The method of the present application comprises the following key components: first, the present application introduces improved D-IOU matching, which extends the local search field of traditional IOU matching to the global search field.Enables efficient matching of targets under the condition of severe shaking of the field of view of unmanned warships without relying on a separate re-identification network.Second, to better cope with the situation of sea surface object position drift, the present application uses a detection-driven Kalman filter.This strategy focuses on the observation results rather than the prediction results of the traditional Kalman filter, thereby improving the stability and accuracy of tracking.Finally, the present application introduces a soft voting mechanism to correct the incorrect class determination that may be generated by the detector.This mechanism effectively addresses the target class detection problem in complex marine environments.

Description

Technical Field

[0001] This invention belongs to the field of target tracking technology, specifically, it is a multi-target, multi-class tracking method for the sea surface based on improved D-IOU matching. Background Technology

[0002] Unmanned vessels need to navigate safely in various sea conditions, requiring them to possess autonomous environmental awareness capabilities. Surface object tracking is a crucial component of surface vision perception technology, aiming to quickly and accurately detect, identify, and track objects on the sea surface. This technology is vital for unmanned vessels, enabling them to identify other vessels and obstacles on the sea surface, achieving autonomous obstacle avoidance. Multi-object tracking has consistently been one of the most active research directions in computer vision. Its goal is to detect and track specific categories of objects frame by frame. With the rapid development of deep learning-based object detection algorithms in recent years, mainstream multi-object tracking methods follow a detection-then-track paradigm. This means that after object detection is performed on each frame, a series of frames are obtained, and then the data is correlated based on motion models or visual cues. However, since most existing multi-object tracking algorithms are based on pedestrian tracking datasets, they are not suitable for unmanned vessel vision. Compared to the booming development of general multi-object tracking methods, tracking algorithms for unmanned vessel vision are relatively few. Compared to traditional pedestrian tracking scenarios, sea surface scenarios are more complex (fog, exposure, etc.). These new characteristics increase the difficulty of detection and tracking. Summary of the Invention

[0003] The purpose of this invention is to propose a new method for tracking multiple categories and multiple targets on the sea surface.

[0004] The technical solution to achieve the purpose of this invention is: a multi-target, multi-class tracking method for the sea surface based on improved D-IOU matching, comprising the following steps:

[0005] Step 1: Preprocess the input multi-category, multi-target video data of the sea surface to extract the target candidate region in each frame;

[0006] Step 2: Use the enhanced correlation coefficient algorithm to perform image registration between consecutive frames of the video to reduce the vertical displacement of targets between frames;

[0007] Step 3: Apply an improved D-IOU matching method to inter-frame targets, perform trajectory fitting and tracking for each target, and output the trajectory and related information for each target;

[0008] Step 4: Apply a detection-driven Kalman filter to each target to determine the positional changes of the target between different frames;

[0009] Step 5: Use a soft voting method to correct the trajectory category for each successfully tracked trajectory, thereby improving the stability of long trajectories.

[0010] Preferably, step 1, which involves extracting the target candidate region in each frame, comprises:

[0011] The YOLOv7 detector is used to perform frame-by-frame detection on the raw video data. The detection box of each target in the detection process is used as the candidate region of the target for subsequent tracking.

[0012] Preferably, step 2, which uses the enhanced correlation coefficient algorithm for image registration, comprises the following steps:

[0013] Using the previous frame as the reference image and the current frame as the image to be registered, both images are first converted to grayscale, and a Euclidean motion model is chosen. Then, the ECC-enhanced correlation coefficient method is used to estimate the affine transformation matrix between the two images. Finally, the calculated transformation matrix is ​​applied to the images to be registered to achieve alignment.

[0014] Preferably, the step in step 3, which uses the improved D-IOU matching method for inter-frame targets, is as follows:

[0015] Step 31: Divide the target candidate regions obtained in Step 1 into high-confidence regions x according to the detector confidence level. high and low confidence region x low ;

[0016] Step 32: Calculate the x-axis of the historical tracking trajectories before and after the current frame image. high The overlap area of ​​the IOU between them can be expressed by the formula: Where S a Let S represent the area of ​​target a. b It is represented as the area of ​​target b, used to assess the degree of overlap between the two regions of target a and target b.

[0017] Step 33: Calculate the improved D-IOU distance as the similarity between targets. The process can be expressed by the formula: Where ρ represents the center distance between the tracking bounding box and the detection bounding box, c represents the diagonal length of the maximum enclosing rectangle of the two bounding boxes, α is a coefficient, and S traj S represents the average bounding box area of ​​the current trajectory. det R represents the area of ​​the detection box. det p and q represent the detection confidence given by the detector, where p and q are equal to 0 or 1 in different cases;

[0018] Step 34: Solve the similarity matrix obtained in Step 33 using the Hungarian algorithm to obtain the tracking results between the two frames, i.e., the matching relationship between the trajectory and the detection. The result is the successfully tracked trajectory;

[0019] Step 35, x high The part that was not successfully tracked in the middle and x low Match again, following the same steps as steps 32 to 34;

[0020] Step 36: After two matches, the remaining x high Partial trajectory is matched with the currently lost trajectory using IOU to re-track the trajectory.

[0021] Preferably, the Kalman filter used for detection in step 4 is specifically:

[0022] During tracking, this invention maintains Kalman filter updates for each object state. If the current trajectory is a lost trajectory, i.e., a trajectory that has continuously failed to be tracked, the prediction result of the Kalman filter is used as the location information of the current trajectory; if the current trajectory is an active trajectory, i.e., a trajectory that has been successfully tracked continuously, the detection result of the most recently successfully tracked frame of YOLOv7 is used as the location information of the current trajectory. The location information serves as a target candidate region and is used to calculate the similarity matrix in step 3.

[0023] Preferably, the soft voting method in step 5 is as follows:

[0024] Step 51: In the similarity matrix calculation for multi-category multi-target matching, this invention relaxes the strict restrictions on categories. When historical trajectory i and detection j have the highest similarity but different categories, this invention still considers them as a possible association. For those matching pairs that are not the most similar, this invention strictly requires them to maintain consistency in categories; if the categories are inconsistent, they are considered as impossible associations.

[0025] Step 52: For all successfully tracked trajectories in Step 3, record the matched category information and frequency of occurrence; for matching results of different categories in Step 52, the voting mechanism will select the category with the highest frequency in the current category sample, record the new category and its frequency of occurrence, and correct the category.

[0026] Compared with existing technologies, the significant advantages of this invention are: this method only requires the location information of the image, does not require the image re-identification feature information, and can successfully track targets even when there is no overlap between them, with low computational complexity and real-time tracking. Furthermore, it employs detection-driven Kalman filtering and soft voting methods to improve the tracking accuracy and robustness of multiple targets across various categories on the sea surface, and can handle problems such as occlusion and motion uncertainty in complex scenes. Attached Figure Description

[0027] Figure 1 This is a flowchart of the tracking method of the present invention.

[0028] Figure 2 This includes various types of multi-target scenarios on the sea surface under the vision of unmanned vessels, including scenarios within ports, leaving ports, entering ports, and offshore areas.

[0029] Figure 3 This is a visualization of the detection-driven Kalman filtering results of this invention. The thick solid-line box represents the tracking result, and the yellow box represents the detection result. Figure 3 In the diagram, (a) represents the Kalman filter without detection-driven filtering. Figure 3 (b) in the example uses a detection-driven Kalman filter.

[0030] Figure 4 The visualization results of the improved D-IOU matching used in this invention, Figure 4 (a) in the text refers to matching using the IOU method. Figure 4 (b) in the example uses improved D-IOU matching.

[0031] Figure 5 This is a visualization of the soft voting mechanism used in this invention.

[0032] Figure 6 This invention illustrates the changes in IDF1 and MOTA indices under different area ratio thresholds.

[0033] Figure 7 This invention illustrates the changes in IDF1 and MOTA indices under different displacement thresholds. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0035] This invention presents an improved D-IOU matching-based multi-target, multi-class tracking method for the sea surface, aiming to address the challenges of multi-target tracking in the marine environment. Figure 1 This paper presents a proposed method for tracking multiple types of vessels on the sea surface. The multi-target tracking system of this invention follows a detection-then-tracking paradigm, comprising a detector, a motion model, and a tracker. This invention includes three modules: an improved D-IOU matching module, a detection-driven Kalman filter, and a soft voting mechanism, enabling it to handle the aforementioned problems and be applicable to multi-type vessel tracking scenarios on the sea surface. Based on the three modules proposed in this invention, a simple, fast, and accurate tracking framework is constructed using a detector equipped with YOLOv7. The specific steps are as follows:

[0036] Step 1: Preprocess the input multi-category, multi-target video data of the sea surface;

[0037] Figure 2 The raw video data is used. Video data preprocessing includes object detection but not depth feature extraction because existing methods struggle to extract discriminative depth features from limited data due to the complex lighting and weather conditions in marine scenes. Secondly, extracting depth features would incur high computational costs to achieve engineering goals as simply and quickly as possible, which does not guarantee tracking efficiency in practical applications. Finally, due to the rapid development of deep learning-based detectors, existing powerful object detectors and motion models are sufficient to provide a solid foundation for appearance-based matching. The detector uses the YOLOv7 detector, with YOLOv7 as the backbone network, and initialized weights using a COCO pre-trained model. The training plan is 90 epochs on the training set. The input image size is 1920×1080. Data augmentation includes mosaic data augmentation and hybrid data augmentation. The model is trained on four NVIDIA RTX 3090 GPUs with a batch size of 4. The optimizer uses a 5×10^5 weight decay optimization. -4 A SGD optimizer with momentum of 0.8. The initial learning rate is 10. -1 It features three generations of warm-up and cosine annealing scheduling. The total training time is approximately 30 hours.

[0038] Step 2: Use an enhanced correlation coefficient algorithm to perform image registration between consecutive frames of the video;

[0039] Considering the non-linear motion of ships and objects in a seascape, potentially involving vertical pitching and horizontal swaying, a motion model adaptable to these scene characteristics is needed for effective tracking. In this tracking process, the previous frame is used as the reference image, and the current frame as the image to be registered. First, both images are converted to grayscale, and a Euclidean motion model is chosen. Then, the ECC-enhanced correlation coefficient method is used to estimate the affine transformation matrix between the two images. Finally, the calculated transformation matrix is ​​applied to the image to be registered to achieve alignment. This process is performed for each new frame during real-time tracking to ensure accuracy and consistency.

[0040] Step 3, the steps for applying the improved D-IOU matching method to inter-frame targets are as follows;

[0041] Based on the detection box score, the detection boxes are divided into high-scoring regions x. high and low-scoring region x low Process them separately. First time using the high-resolution region x high Improved D-IOU matching was performed with the previous tracking trajectory. The low-resolution region x was used a second time. low The first time, no high-scoring region was matched. highImproved D-IOU matching is performed on the tracking trajectory (e.g., objects that are severely occluded in the current frame, causing a score drop). For high-scoring regions x that do not match the tracking trajectory... high This will be used to re-track the currently lost trajectory. The remaining high-resolution region x will then be determined. high This will establish a new trajectory.

[0042] The improved D-IOU matching process can be expressed by the following formula:

[0043]

[0044] in, Where S a Let S represent the area of ​​target a. b The area of ​​target b is represented as follows: Figure 3 As shown, ρ represents the center distance between the tracking bounding box and the detection bounding box, c represents the diagonal length of the maximum bounding rectangle of the two bounding boxes, α is a coefficient, and S traj S represents the average bounding box area of ​​the current trajectory. det R represents the area of ​​the detection box. det The detection confidence levels p and q represent the detection confidence given by the detector, which can be 0 or 1 in different cases. When introducing the center distance ρ, this invention uses the diagonal c of the enclosing rectangle to normalize the center distance. However, this ratio introduces some extreme cases, resulting in the same calculation for correct and incorrect matches. This invention improves the formula by adding a coefficient α, representing the ratio of the area of ​​the non-target region in the maximum envelope rectangle to the area of ​​the maximum enclosing rectangle. Furthermore, area is also important information for handling multi-class vessel tracking, as vessels vary in shape between and within different classes. This invention adds the average area of ​​existing trajectories to the ratio of the area of ​​the detection frames. This ratio rewards detection frames with areas similar to the trajectory and penalizes those with significant area differences. In particular, small targets have smaller areas, thus exhibiting greater area variation. Therefore, p = 0 when the target is very small, and p = 1 in other cases. This invention assumes that high-confidence detections are more reliable than low-confidence detections. Based on this prior, a detection confidence factor R is added. det This makes the detection cost of high confidence levels lower. The present invention uses two confidence matching levels: the first round uses high confidence matching with q=1, and the second round uses low confidence matching with q=0. Finally, thresholds are used to directly limit outliers. As shown in formula (2), the present invention sets three thresholds σ1, σ2, and σ3. Δd represents the displacement between the detection box and the tracking box. σ1 is used to filter results with excessively high computational costs, σ2 is used to filter results with excessively large area differences between the tracking box and the detection box, and σ3 is used to filter results with excessively large displacements between the tracking box and the detection box.

[0045]

[0046] Traditional IOU matching is a local matching strategy, limiting its search domain to all detections that overlap with trajectory i. Unlike traditional IOU matching, the improved D-IOU matching proposed in this invention expands the search domain to all detections, i.e., a global matching strategy. By introducing the aforementioned distance, the results for all detections with trajectory i can be calculated, regardless of whether there is IOU overlap between them. This change gives the method of this invention the nearest neighbor matching property based on IOU matching, thus leveraging the original advantages of IOU in dense scenes. Even when IOU information becomes invalid, it does not lead to matching failure; a correct match can still be obtained. Figure 4 As shown in (a), the IOU matching method lost cargo ship number 3, and an ID switch occurred during retracking. The ID changed from 3 to 5. Figure 4 As shown in (b), low IOU overlap between the two detection frames led to trajectory loss. The detector successfully detected cargo ship 3 but failed to associate it with the corresponding trajectory. It initialized as a new trajectory 5. When the detection results of the next frame arrived, trajectories 3 and 5 competed for space. An ID switch occurred when trajectory 5 was a better match. After using improved D-IOU matching, the stability of the trajectory was guaranteed. It can stably track cargo ship 3, which is helpful for long-term object tracking and is well adapted to target tracking in seascapes.

[0047] Step 4: Apply a detection-driven Kalman filter to each target;

[0048] Existing methods for motion prediction using Kalman filters primarily focus on the Kalman filter's predictions; that is, the Kalman filter's predictions are used to correct existing trajectories and then matched with detection frames to obtain the final result. This invention uses a detection-driven Kalman filter with detector results at its core, provided there is a powerful detector. During tracking, the Kalman filter is updated for each object state. However, the Kalman filter's results are only used if the trajectory is a lost trajectory. If the trajectory is an active trajectory, the detector results are used because these results are more reliable than the values ​​corrected by the Kalman filter. Figure 3The visualization results show that there were discrepancies between the tracking and detection results before using detection-driven Kalman filtering. Particularly for the sailboat with ID 18, the irregular jitter represented by the yellow and red lines persists, and the Kalman filter accumulates errors, making the offset very noticeable. Despite successful correlation, the final tracking result is inferior to the bounding box provided by the detector. In maritime scenes, due to the greater maneuverability of the vessel, jitter is more pronounced, and the bias caused by Kalman filter predictions may be more prominent and severe. Therefore, with the aid of the detector, detection-driven Kalman filtering was used, which will make the tracking results more accurate, reduce FP and FN, and thus improve tracking performance.

[0049] Step 5: Use a soft voting method to correct the trajectory category for each successfully tracked trajectory;

[0050] For detector misclassification, this invention uses soft voting to increase tolerance for categories. Compared to hard voting, this invention relaxes the strict restrictions on categories so that some objects successfully detected by the detector but misclassified can be successfully associated. Note that this invention does not completely relax the category restrictions, otherwise it would lead to many incorrect matches. Based on the improved D-IOU matching method of this invention, the category restrictions can be relaxed when the computational cost of detection box j to trajectory i is minimized, but the categories are different, as shown in Equation (3). At the same time, since the Hungarian algorithm has the property of globally optimal matching, these restrictions of this invention will not affect the matching of detection box j with other more reasonable ones.

[0051] T i.cls ≠D j.cls ∩cost(T i D j )≠max(cost(T i (3)

[0052] After obtaining matches for different categories, the categories need to be corrected. This invention uses a voting mechanism to record the frequency of trajectory categories at any given moment during trajectory association. If different categories appear, the voting mechanism selects the category with the highest frequency in the current category samples, while simultaneously recording the new category and its frequency. Based on this soft voting mechanism, if a category is incorrectly detected in several frames of a long trajectory, the correct category with a higher frequency will be selected. This is because the correct category is much more frequent than the incorrect category, thus increasing the stability of long trajectories. Figure 5 As shown, this illustrates two instances of detector misclassification within six consecutive frames, misclassifying Passenger Ship 1 as a fishing boat. Passenger Ship 1 has the best match but belongs to a different category. At this point, the soft voting mechanism relaxes the strict category restrictions, allowing a match. Since the frequency of the passenger ship category is much higher than that of the fishing boat category, the category can be corrected to passenger ship.

[0053] Referring to formula (2), a threshold is set to filter outliers when the search scope is expanded to the global search domain. Figure 6 As can be seen, small targets are sensitive to the choice of area ratio because they occupy relatively fewer pixels, and their area varies more significantly across consecutive frames than that of normal targets. Therefore, there is a difference in the area ratio range between small and normal targets. σ² = 0.3 was chosen to strike a balance between IDF1 and MOTA. Figure 7 This indicates that the threshold filtered out many correct correlations. It also reflects the peculiarity of seascapes, suggesting that many objects have relatively large horizontal displacements between consecutive frames. Setting σ3 to 2.5w yields optimal performance.

[0054] In summary, firstly, this invention introduces an improved D-IOU matching method, extending the local search domain of traditional IOU matching to the global search domain. This enables efficient target matching even under conditions of significant field-of-view fluctuations on unmanned vessels, without relying on a separate re-identification network. Secondly, to better handle the situation of drifting objects on the sea surface, this invention employs a detection-driven Kalman filter. This strategy focuses on the observation results rather than the prediction results of traditional Kalman filters, thereby improving the stability and accuracy of tracking. Finally, this invention introduces a soft voting mechanism to correct for possible erroneous class classifications by the detector. This mechanism effectively addresses the target class detection problem in complex marine environments.

Claims

1. A multi-target, multi-class tracking method for the sea surface based on improved D-IOU matching, characterized in that, Includes the following steps: Step 1: Preprocess the input multi-category, multi-target video data of the sea surface to extract the target candidate region in each frame; Step 2: Use the enhanced correlation coefficient algorithm to perform image registration between consecutive frames of the video to reduce the vertical displacement of targets between frames; Step 3: Apply an improved D-IOU matching method to inter-frame targets, perform trajectory fitting and tracking for each target, and output the trajectory and related information for each target; The steps for using the improved D-IOU matching method for inter-frame targets are as follows: Step 31: Divide the target candidate regions obtained in Step 1 into high-confidence regions according to the detector confidence level. and low confidence areas ; Step 32: Calculate the historical tracking trajectories before and after the current frame image. The overlap area of ​​the IOU between them can be expressed by the formula: ,in This represents the area of ​​target a. Represented as the area of ​​target b, it is used to assess the degree of overlap between the two regions, target a and target b. Step 33: Calculate the improved D-IOU distance as the similarity between targets. The process can be expressed by the formula: ,in, This represents the center distance between the tracking bounding box and the detection bounding box. This represents the diagonal length of the largest enclosing rectangle of the two bounding boxes. It is a coefficient. This represents the average bounding box area of ​​the current trajectory. This represents the area of ​​the detection frame. This indicates the detection confidence level given by the detector. and It can be equal to 0 or 1 in different situations; Step 34: Solve the similarity matrix obtained in Step 33 using the Hungarian algorithm to obtain the tracking results between the two frames, that is, the matching relationship between the trajectory and the detection; the result is the successfully tracked trajectory. Step 35, The parts that were not successfully tracked and Match again, following the same steps as steps 32 to 34; Step 36: After two matches, the remaining... Partial trajectory is matched with the currently lost trajectory using IOU to re-track the trajectory; Step 4: Apply a detection-driven Kalman filter to each target to determine the positional changes of the target between different frames; Step 5: For each successfully tracked trajectory, use a soft voting method to correct the trajectory category, improving the stability of long trajectories; the soft voting method is as follows: Step 51: In the similarity matrix calculation of multi-class multi-target matching, when the historical trajectory i and the detection j have the highest similarity but different categories, they are regarded as possible associations, that is, their original similarity is retained; for matching pairs that are not the most similar, they are required to be consistent in category. If the categories are inconsistent, they are regarded as impossible associations. Step 52: For all successfully tracked trajectories in Step 3, record the matched category information and frequency of occurrence; for the matching results of different categories in Step 52, the voting mechanism will select the category with the highest frequency in the current category sample, record the new category and its frequency of occurrence, and correct the category.

2. The sea surface multi-target multi-class tracking method based on improved D-IOU matching according to claim 1, characterized in that, The method for extracting the target candidate region in each frame in step 1 is as follows: The YOLOv7 detector is used to perform frame-by-frame detection on the raw video data. The detection box of each target is used as a candidate region for subsequent tracking.

3. The sea surface multi-target multi-class tracking method based on improved D-IOU matching according to claim 1, characterized in that, Step 2 uses the enhanced correlation coefficient algorithm for image registration as follows: The previous frame image is used as the reference image, and the current frame image is used as the image to be registered. First, both images are converted to grayscale and the Euclidean motion model is selected. Then, the ECC enhanced correlation coefficient method is used to estimate the affine transformation matrix between the two images. Finally, the calculated transformation matrix is ​​applied to the image to be registered to achieve their alignment.

4. The sea surface multi-target multi-class tracking method based on improved D-IOU matching according to claim 1, characterized in that, The specific Kalman filter used for detection in step 4 is as follows: During tracking, the Kalman filter is updated for each object state. If the current trajectory is a lost trajectory, i.e., a trajectory that has failed to be tracked continuously, the prediction result of the Kalman filter is used as the position information of the current trajectory. If the current trajectory is an active trajectory, i.e. a trajectory that has been successfully tracked continuously, the detection result of the latest successfully tracked frame of YOLOv7 is used as the position information of the current trajectory.

Images