Fish multi-target tracking method based on nonlinear motion modeling and category constraint

Abstract

The invention discloses a fish multi-target tracking method based on nonlinear motion modeling and category constraint, and the method comprises the following steps: constructing a basic fish target detection model which is used for generating a fish target detection frame; constructing a target tracking data set in an MOT format for target tracking and a target identification data set for appearance feature extraction, and constructing a basic fish appearance feature re-identification model for extracting appearance features of a fish target; and constructing a target tracking model associated with a nonlinear filter based on detection confidence weighting and category gating. A confidence-weighted extended Kalman filtering algorithm is provided, and the algorithm dynamically adjusts weight distribution of a detection result in a track updating process by analyzing a confidence score of a target detection frame in real time. According to the category gating mechanism, a category similarity penalty term is introduced into a cost matrix of data association, and by establishing an association barrier of category perception, it is ensured that detection frames and estimation frames of different categories are not wrongly associated to the same identity.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and artificial intelligence, particularly a multi-target underwater fish tracking method based on dynamic nonlinear motion modeling and category constraints. This method can provide important technical support for the exploration and assessment of animal resources and ecological protection in oceans, lakes, and other waters. This research was funded by the Liaoning Provincial Natural Science Foundation's "Research on Intelligent Perception of Marine Targets in Cloud-Edge Collaboration with Marine Robots." Background Art

[0002] With the rapid development of computer vision and deep learning technologies, multi-target tracking algorithms have been widely used in fields such as intelligent surveillance, autonomous driving, and motion analysis. However, in complex scenarios, especially dynamic ocean and lake environments, multi-target tracking still faces many challenges. For example, the frequent partial or complete occlusion of fish targets and the high similarity of their appearance make identification and association more difficult. Fish targets also often exhibit non-uniform or curved motion, accompanied by sudden acceleration or turns, which places higher demands on motion prediction models. Therefore, how to improve the performance of multi-target tracking in fish scenarios is an important issue that needs to be addressed.

[0003] The development of visual object tracking is relatively recent, primarily concentrated in the past decade or so. With the rapid advancement of computer vision and deep learning technologies, object tracking methods and applications have also been evolving. Early classic methods such as Meanshift and particle filters were widely used in object tracking tasks. While they achieved some success in some basic scenarios, Meanshift relies on initializing the target's color distribution. If the background changes or is noisy, the initial color model may not accurately represent the target, leading to tracking failure. Particle filters are susceptible to noise in dynamic backgrounds, affecting state estimation. Consequently, these traditional methods suffer from low overall accuracy and struggle to cope with complex underwater environments and multi-target scenarios. In recent years, research has gradually transitioned from traditional methods to deep learning approaches. Driven by the dramatic improvements in object detection performance in recent years, detection-based tracking schemes have emerged in the field of multi-object tracking and have rapidly become the mainstream framework for multi-object tracking. This shift has significantly driven the development of multi-object tracking (MOT) tasks, with representative methods including SORT, DeepSORT, and StrongSORT. These methods typically proceed as follows: first, an object detector is used to identify objects in the image and generate detection boxes. Next, a motion model is used to estimate the state of the existing trajectories to predict the object's position in the current frame. A matching method is then used to associate the detected objects with the predicted trajectories to determine the identity of each object. After matching, the system updates the motion estimation model to improve its adaptability and accuracy to target motion, thereby achieving more stable multi-object tracking. These methods have demonstrated significant advantages in the field of multi-object tracking.

[0004] However, these advances still face two problems in multi-target tracking applied to fish scenarios. The first problem is that existing multi-target tracking algorithms are mostly applied to linearly moving targets such as humans and cars, using linear filters for motion prediction. However, the underwater motion characteristics of fish exhibit frequent accelerations and sharp turns. Linear motion prediction makes the state estimation of fish targets less accurate, and the tracking accuracy of the algorithm decreases. The second problem is that existing algorithms are mostly applied to targets with large individual differences, such as portraits and license plates, and the performance requirements for the association part of the algorithm are not high. However, the feature differences between fish targets are small and fish targets occlude each other. Applying existing association algorithms to fish targets will cause the algorithm's ability to maintain correct identity over time to decrease. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a fish multi-target tracking method based on nonlinear motion modeling and category constraints in response to the deficiencies of the existing technology.

[0006] The technical solutions of the present invention are as follows:

[0007] The fish multi-target tracking method based on nonlinear motion modeling and category constraints includes steps S1-S3, which are as follows:

[0008] S1: Based on the original underwater fish image data and the corresponding manually annotated data, the fish images and annotated data are trained based on the YOLOv5 target detection model to build a basic fish target detection model for generating fish target detection frames;

[0009] S2: Collect and annotate original underwater fish video data, construct a target tracking dataset in MOT format for target tracking and a target recognition dataset for appearance feature extraction, and build a basic fish appearance feature re-identification (ReID) model to extract the appearance features of fish targets;

[0010] S3: Construct a target tracking model based on detection confidence weighted nonlinear filter and category gating association (StrongSORT with Nonlinear Filtering and Category Gating, StrongSORT-NLCG). The model structure diagram is as follows Figure 2 As shown in Figure 2. The target tracking process includes the following two key steps:

[0011] (1) Motion estimation: A nonlinear filter weighted by detection confidence is used to replace the traditional linear filter for motion modeling;

[0012] (2) Target association: Design a new association strategy that introduces a category gating mechanism to improve the accuracy of association between targets.

[0013] Finally, StrongSORT-NLCG is used to track fish targets. The specific construction process is as follows: the detection model and ReID model synchronously output the detection box and its appearance features. After the motion estimation value and the detection value are associated with the target, the trajectory status and feature information are updated based on the observation results, and finally the target tracking result is obtained.

[0014] The method flow of estimating the target frame using the confidence-weighted nonlinear filter motion model (Confidence-Weighted Extended Kalman Filter, CW-EKF) in the motion estimation part of step S3 includes steps S311-S315. Figure 3 shown.

[0015] Specifically, S311 generates the target's initial state estimate X0 and initial state covariance matrix (estimated uncertainty) P0 based on the newly detected target detection box d0 and detection confidence value C0 obtained from S1;

[0016] S312 uses a nonlinear motion model to calculate the target state X of the previous frame. k-1 Predict the target state of the next frame and construct the state transfer matrix F k , and obtain the state estimate and the state estimation covariance matrix

[0017] Specifically, in order to adapt to the characteristics of nonlinear motion of fish, the state transfer matrix F k Usually, nonlinear factors such as the acceleration of the target need to be considered. The state transfer matrix F k According to the characteristics of EKF, the Jacobian matrix can be used to construct F k It is defined as an 8-dimensional matrix with diagonal elements of 1, indicating that the current state value is directly equal to the state of the previous moment (for example, X k =X k-1 +v x dt); F k [1,5], F k [2, 6], F k [3,7] and F k The value [4, 8] is the dt time step, which indicates how the target's velocity affects the change of its position or height. According to the constructed state transfer matrix F k , the target state X of the previous frame k-1 Get state estimate and the state estimation covariance matrix The estimation formula for the next frame state is: where Q k is the process noise matrix, which reflects the uncertainty of the model's state estimation.

[0018] S313 According to d k and C k , construct the observation matrix H of the target at frame k k , we get the observation covariance matrix (detection uncertainty) R k ;

[0019] Specifically, the nonlinear Kalman filter based on detection confidence weighting is obtained by k Adjust so that the Kalman gain can be adjusted according to the detection confidence C k Dynamically adjust the observation value d k The weight in trajectory update. The formula is: Specifically, the lower the detection confidence, the larger the observation noise covariance, and the smaller the calculated Kalman gain, thereby reducing the impact of low-confidence detection values on state updates; conversely, high-confidence detection values will give the detection a higher weight.

[0020] S314 estimates the covariance matrix based on the state and the observation covariance matrix R k Calculate the Kalman gain K k ;

[0021] S315 is based on the Kalman gain K k Update the state vector X k and the state estimation covariance matrix P k .

[0022] Finally, a robust motion state estimation model for nonlinear fish motion is obtained, which can accurately estimate the state of the current frame based on the detection results and trajectory prediction information of the current frame.

[0023] In the target association part of step S3, the association strategy between the detection frame and the estimation frame is improved to address the problem that the fish target features are similar and mutually occluded. A new cost matrix is designed and an Enhanced Class-Gated Association Mechanism (ECGAM) is added to further optimize the association effect. The association process includes S321-S326. The flowchart is as follows: Figure 4 shown.

[0024] Specifically, first obtain the detection frame set D = {d1, d2, ..., d M} and the predicted trajectory set T = {t1, t2, ..., t N Then perform global linear matching to associate the trajectory set with the detection box set. The goal is to use the minimum cost matching method. to get matching results.

[0025] S321: Appearance feature f based on the detection frame det (d j ) and the appearance features of the trajectory estimation box f trk (t i ) Calculate the cosine distance and construct the cost matrix C using the minimum cost matching method cos ;

[0026] S322: Calculate the Mahalanobis distance using the motion features of the detection frame and the trajectory estimation frame, and C cos Perform weighted combination to obtain the new cost matrix C mahal ;

[0027] S323: Use category gating to control the cost matrix C mahal Filter to ensure that only trajectories and detection boxes of the same category can match, and obtain the cost matrix C classes ;

[0028] Specifically, in order to solve the problem of feature similarity between fish targets and individual target occlusion, it is necessary to improve the association algorithm and design an association strategy that is more suitable for fish target tracking. The specific method is to introduce a category gating mechanism to filter the original cost matrix and obtain the cost matrix C classes , to ensure that only trajectories and detection boxes of the same category can be matched. The construction formula is:

[0029]

[0030] Among them, class(t i ) is t i Category, class(d j ) is d j Category, gated_cost is the gated cost used to indicate infeasible matching.

[0031] S324: The cost matrix C after the final filtering using the Hungarian algorithm cost =C classes Solve the optimal allocation problem above and get the set of matched trajectories and detection boxes M (1) ={(t i ,d j )}, and the set of unmatched trajectories and unmatched detection box set The formula is:

[0032]

[0033] S325: For unmatched trajectory sets Use the IoU between the track box and the detection box to construct the cost matrix C IoU , use the Hungarian algorithm to get the final matching pair M (2) ={(t i ,d j )} and unmatched trajectory set and unmatched detection box set

[0034] S326: Unmatched detection box d j A new trajectory is generated and set to the unconfirmed state Tentative, without matching the trajectory t i Will increase the unmatched age When its unmatched age exceeds the maximum age threshold max Or in an unconfirmed state, the track will be destroyed.

[0035] Through the above steps, the detection bounding box of the current frame is matched with the trajectory bounding box, and the matching result can be used to update the motion model later. Unmatched detections and tracks are handled accordingly. Category gating verifies the categories of the track and detection bounding box, and only allows them to match when the track and detection bounding box belong to the same category. This mechanism greatly reduces mismatches caused by different categories, improving the robustness and accuracy of the entire target tracking process.

[0036] The technical effects of the present invention are as follows:

[0037] (1) Solved the problem of rapid nonlinear motion of the target and trajectory error accumulation: Traditional tracking algorithms are usually based on linear motion models and are difficult to effectively adapt to nonlinear motion characteristics such as sudden acceleration and sharp turns that occur during the swimming process of fish. To this end, the present invention proposes a confidence-weighted extended Kalman filter algorithm. The algorithm dynamically adjusts the weight distribution of detection results during the trajectory update process by analyzing the confidence score of the target detection frame in real time. At the same time, the Jacobian matrix in the extended Kalman filter is used for nonlinear state prediction, which significantly improves the system's adaptability to the irregular swimming pattern of fish and effectively reduces the trajectory prediction error.

[0038] (2) Solve the problem of mis-association of target identities caused by category confusion: To address the problem of mismatching the identities of different species of fish due to dense occlusion when underwater fish schools cross each other, the present invention innovatively designs a category gating mechanism. This mechanism introduces a category similarity penalty term into the data association cost matrix. By establishing a category-aware association barrier, it ensures that detection frames and estimation frames of different categories will not be mis-associated with the same identity. This design fundamentally solves the problem of cross-category target mistracking caused by appearance similarity, and significantly improves the accuracy and robustness of multi-target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 : StrongSORT-NLCG model structure diagram;

[0040] Figure 2 : Overall flow chart;

[0041] Figure 3 :Flowchart of the method using the detection confidence weighted nonlinear filter motion model (CW-EKF);

[0042] Figure 4 : Flowchart of the method for the association strategy using the improved enrolment category gating mechanism (ECGAM);

[0043] Figure 5 : Specific flow chart of embodiment calculation;

[0044] Figure 6:Comparison of target tracking effects between the original algorithm and StrongSORT-NLCG; DETAILED DESCRIPTION

[0045] The following describes in detail the implementation process of the present invention for tracking multiple fish targets in the public dataset underwater target tracking dataset FISHTRAC provided by Travis M et al. [6] and the self-built underwater video dataset, with reference to the accompanying drawings and examples.

[0046] The present invention is implemented by the following technical solutions, the overall process is as follows Figure 1 The process includes steps S1-S3, which are as follows:

[0047] S1: Based on the original underwater fish image data and the corresponding manually annotated data, the fish images and annotated data are trained based on the YOLOv5 target detection model to build a basic fish target detection model for generating fish target detection frames;

[0048] S2: Collect and annotate original underwater fish video data, construct a target tracking dataset in MOT format for target tracking and a target recognition dataset for appearance feature extraction, and build a basic fish appearance feature re-identification (ReID) model to extract the appearance features of fish targets;

[0049] S3: Construct a target tracking model based on detection confidence weighted nonlinear filter and category gating association (StrongSORT with Nonlinear Filtering and Category Gating, StrongSORT-NLCG). The model structure diagram is as follows Figure 2 As shown in Figure 2. The target tracking process includes the following two key steps:

[0050] (1) Motion estimation: A nonlinear filter weighted by detection confidence is used to replace the traditional linear filter for motion modeling;

[0051] (2) Target association: Design a new association strategy that introduces a category gating mechanism to improve the accuracy of association between targets.

[0052] Finally, StrongSORT-NLCG is used to track fish targets. The specific construction process is as follows: the detection model and ReID model synchronously output the detection box and its appearance features. After the motion estimation value and the detection value are associated with the target, the trajectory status and feature information are updated based on the observation results, and finally the target tracking result is obtained.

[0053] The method flow of estimating the target frame using the confidence-weighted nonlinear filter motion model (Confidence-Weighted Extended Kalman Filter, CW-EKF) in the motion estimation part of step S3 includes steps S311-S315. Figure 3 shown.

[0054] Specifically, S311 generates the target's initial state estimate X0 and initial state covariance matrix (estimated uncertainty) P0 based on the newly detected target detection box d0 and detection confidence value C0 obtained from S1;

[0055] S312 uses a nonlinear motion model to calculate the target state X of the previous frame. k-1 Predict the target state of the next frame and construct the state transfer matrix F k , and obtain the state estimate and the state estimation covariance matrix

[0056] Specifically, in order to adapt to the characteristics of nonlinear motion of fish, the state transfer matrix F k Usually, nonlinear factors such as the acceleration of the target need to be considered. The state transfer matrix F k According to the characteristics of EKF, the Jacobian matrix can be used to construct F k It is defined as an 8-dimensional matrix with diagonal elements of 1, indicating that the current state value is directly equal to the state of the previous moment (for example, X k =X k-1 +v x dt); F k [1,5], F k [2, 6], F k [3,7] and F k The value [4, 8] is the dt time step, which indicates how the target's velocity affects the change of its position or height. According to the constructed state transfer matrix F k , the target state X of the previous frame k-1 Get state estimate and the state estimation covariance matrix The estimation formula for the next frame state is: where Q k is the process noise matrix, which reflects the uncertainty of the model's state estimation.

[0057] S313 According to d k and C k , construct the observation matrix H of the target at frame k k , we get the observation covariance matrix (detection uncertainty) R k ;

[0058] Specifically, the nonlinear Kalman filter based on detection confidence weighting is obtained by k Adjust so that the Kalman gain can be adjusted according to the detection confidence C k Dynamically adjust the observation value d k The weight in trajectory update. The formula is: Specifically, the lower the detection confidence, the larger the observation noise covariance, and the smaller the calculated Kalman gain, thereby reducing the impact of low-confidence detection values on state updates; conversely, high-confidence detection values will give the detection a higher weight.

[0059] S314 estimates the covariance matrix based on the state and the observation covariance matrix R k Calculate the Kalman gain K k ;

[0060] S315 is based on the Kalman gain K k Update the state vector X k and the state estimation covariance matrix P k .

[0061] Finally, a robust motion state estimation model for nonlinear fish motion is obtained, which can accurately estimate the state of the current frame based on the detection results and trajectory prediction information of the current frame.

[0062] In the target association part of step S3, the association strategy between the detection frame and the estimation frame is improved to address the problem that the fish target features are similar and mutually occluded. A new cost matrix is designed and an Enhanced Class-Gated Association Mechanism (ECGAM) is added to further optimize the association effect. The association process includes S321-S326. The flowchart is as follows: Figure 4 shown.

[0063] Specifically, first obtain the detection frame set D = {d1, d2, ..., d M} and the predicted trajectory set T = {t1, t2, ..., t N Then perform global linear matching to associate the trajectory set with the detection box set. The goal is to use the minimum cost matching method. to get matching results.

[0064] S321: Appearance feature f based on the detection frame det (d j ) and the appearance features of the trajectory estimation box f trk (t i) Calculate the cosine distance and construct the cost matrix C using the minimum cost matching method cos ;

[0065] S322: Calculate the Mahalanobis distance using the motion features of the detection frame and the trajectory estimation frame, and C cos Perform weighted combination to obtain the new cost matrix C mahal ;

[0066] S323: Use category gating to control the cost matrix C mahal Filter to ensure that only trajectories and detection boxes of the same category can match, and obtain the cost matrix C classes ;

[0067] Specifically, in order to solve the problem of feature similarity between fish targets and individual target occlusion, it is necessary to improve the association algorithm and design an association strategy that is more suitable for fish target tracking. The specific method is to introduce a category gating mechanism to filter the original cost matrix and obtain the cost matrix C classes , to ensure that only trajectories and detection boxes of the same category can be matched. The construction formula is:

[0068]

[0069] Among them, class(t i ) is t i Category, class(d j ) is d j Category, gated_cost is the gated cost used to indicate infeasible matching.

[0070] S324: The cost matrix C after the final filtering using the Hungarian algorithm cost =C classes Solve the optimal allocation problem above and get the set of matched trajectories and detection boxes M (1) ={(t i ,d j )}, and the set of unmatched trajectories and unmatched detection box set The formula is:

[0071]

[0072] S325: For unmatched trajectory sets Use the IoU between the track box and the detection box to construct the cost matrix C IoU , use the Hungarian algorithm to get the final matching pair M (2) ={(t i ,d j )} and unmatched trajectory set and unmatched detection box set

[0073] S326: Unmatched detection box d j A new trajectory is generated and set to the unconfirmed state Tentative, without matching the trajectory t i Will increase the unmatched age When its unmatched age exceeds the maximum age threshold max Or in an unconfirmed state, the track will be destroyed.

[0074] Through the above steps, the detection bounding box of the current frame is matched with the trajectory bounding box, and the matching result can be used to update the motion model later. Unmatched detections and tracks are handled accordingly. Category gating verifies the categories of the track and detection bounding box, and only allows them to match when the track and detection bounding box belong to the same category. This mechanism greatly reduces mismatches caused by different categories, improving the robustness and accuracy of the entire target tracking process.

[0075] Example 1: Fish Multi-Target Tracking in the Underwater Target Tracking Dataset FISHTRAC

[0076] The specific process in this embodiment is as follows Figure 5 The specific steps are as follows:

[0077] The original .mat format data in the FISHTRAC dataset was converted to .txt format. Since the dataset only contains identity labels and not species labels, we manually annotated the species for each fish. The dataset was then converted to formats suitable for different tasks: YOLO format for object detection, MOT format for object tracking, and ReID format for appearance feature extraction.

[0078] First, we used YOLOv5s as the base object detection model and trained it on the converted object detection dataset to obtain the FISHTRAC fish object detection model. Then, we used the Xception model as the base feature extraction model and trained it on the object recognition dataset to obtain the ReID appearance feature extraction model.

[0079] During the target tracking process, the video is fed into the target tracking model and processed frame by frame. The steps for processing each frame are as follows:

[0080] 1) Target detection: Perform target detection on the current frame and obtain the detection frame d k =[x k ,y k ,a k ,h k ] and its confidence C k .

[0081] 2) Appearance feature extraction: Use the ReID model to extract the appearance features f of the detection target det (d j ).

[0082] 3) Trajectory creation and prediction: For the newly detected target, the detection box d k , confidence C k and appearance features f det (d j ) is passed into the model. The model uses Kalman filtering to estimate the motion state: 1) Use the detection box and confidence to initialize the trajectory state estimation vector X0 and state covariance matrix P0; 2) According to the state estimation X of the current frame k-1 and the covariance matrix P k-1 , predict the state estimation vector of the target in the next frame and the covariance matrix The calculation formula is:

[0083] X0=[xyah 0 0 0 0] T

[0084]

[0085] X k - =F k ·X k-1

[0086]

[0087] 4) Inter-frame matching: According to the detection box d of the next frame k The predicted state of the trajectory The matching process includes: 1) calculating the cosine distance between the appearance features of the detection frame and the trajectory estimation frame, and obtaining the cosine distance cost matrix C cos ; 2) Combine the cosine distance cost matrix to perform Mahalanobis distance C mahal Calculation, and then filter by category gating to get C classes , using the Hungarian algorithm to obtain the preliminary matching trajectory detection set M (1) ={(t i ,d j )} and unmatched trajectory set and unmatched detection box set 3) For unmatched items, use the IoU cost matrix C IoU Perform secondary matching and use the Hungarian algorithm again to finally obtain the successfully matched trajectory M (2) ={(t i ,dj )} and the remaining unmatched trajectories and detection box The calculation formula of each cost matrix is:

[0088] C cos (i,j)=1-cosine_similarity(f det (d j ),f trk (t i ))

[0089]

[0090] 5) Track management: for unmatched tracks Its "age" attribute increases by 1. If the maximum age is exceeded or the track is in the "unconfirmed" state, the track is deleted; for unmatched detection frames Generate a new track (i.e., process 3) and mark it as "unconfirmed". When the track successfully matches the detection box for three consecutive frames, change the track status to "confirmed".

[0091] 6) Status update: Update the successfully matched trajectory and detection box M (2) ={(t i ,d j )}, calculate the observation matrix H of the detection box k and the observation covariance matrix R k , using the calculated Kalman gain K k State estimation and the state estimation covariance matrix Update, where the calculation formula is:

[0092]

[0093] Finally, the updated trajectory state vector X is obtained k and the state estimation covariance matrix P k .

[0094] 7) Continuous iteration: Process each frame of the image in turn until the end of the video.

[0095] Finally, the record of each trajectory is saved in .txt format and compared with the gt.txt file in the MOT format dataset. Indicators such as MOTA, IDF1, HOTA, and ID Switch are calculated to evaluate the performance of the target tracking model.

[0096] The experimental results of the StrongSORT-NLCG algorithm in this embodiment are shown in Table 1.

[0097] Table 1 Experimental results of StrongSORT-NLCG on the FISHTRAC dataset

[0098]

[0099] Example 2: Tracking multiple fish species using a self-built underwater video dataset

[0100] The dataset in this embodiment was collected and generated by an underwater robot at a fixed position underwater. Compared to Example 1, the video in this embodiment contains more target types that need to be tracked, and the fish have more complex motion characteristics, and there are also many cases of mutual occlusion of targets. These characteristics increase the difficulty of target tracking, making this embodiment more prominently demonstrate the superior performance of the present invention in tracking underwater nonlinear motion and occluded targets.

[0101] The specific process in this embodiment is as follows Figure 5 The specific steps are as follows:

[0102] First, the original images and annotation results of the self-built dataset are constructed into a YOLO format dataset for target detection, and the video data is manually annotated with detection boxes, categories, and identity IDs, and converted into MOT format and target recognition format. Then, the trained target detection model, ReID model, the original video data of the self-built dataset and its MOT format annotation data are input into the StrongSORT-NLCG algorithm, and target tracking is performed according to the process in Example 1. The final experimental results are shown in Table 2, and the comparison effect is shown in Table 2. Figure 6 As shown, the superior performance of the present invention is verified in the underwater fish fast-changing and complex occlusion scenes.

[0103] Table 2 Experimental results of StrongSORT-NLCG on self-built dataset

[0104]

[0105] The source of the public dataset FISHTRAC used in Example 1: Travis M, Mark J, Emily R, et al. Detection confidence driven multi-object tracking to recover reliable tracks from unreliable detections[J]. Pattern Recognition, 2023, 135

[0106] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all such improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A fish multi-target tracking method based on nonlinear motion modeling and category constraints, characterized by: The method includes steps S1-S3: S1: Based on the original underwater fish image data and the corresponding manually annotated data, the fish images and annotated data are trained based on the YOLOv5 target detection model to build a basic fish target detection model for generating fish target detection frames; S2: Collect and annotate original underwater fish video data, construct a target tracking dataset in MOT format for target tracking and a target recognition dataset for appearance feature extraction, and build a basic fish appearance feature re-identification (ReID) model to extract the appearance features of fish targets; S3: Construct a target tracking model StrongSORT-NLCG based on detection confidence weighted nonlinear filter and category gated association. The target tracking model includes the following two steps: (1) Motion estimation: A nonlinear filter weighted by detection confidence is used to replace the traditional linear filter for motion modeling; (2) Target association: Design a new association strategy that introduces a category gating mechanism to improve the accuracy of association between targets.

2. The method according to claim 1, characterized in that The method for estimating the target frame using the nonlinear filter motion model CW-EKF with weighted detection confidence in the motion estimation part in step S3 includes steps S311-S315: S311 generates the target's initial state estimate X0 and initial state covariance matrix P0 based on the newly detected target detection box d0 and detection confidence value C0 obtained from S1; S312 uses a nonlinear motion model to calculate the target state X of the previous frame. k-1 Predict the target state of the next frame and construct the state transfer matrix F k , and obtain the state estimate and the state estimation covariance matrix S313 According to d k and C k , construct the observation matrix H of the target at frame k k , and obtain the observation covariance matrix R k ; S314 estimates the covariance matrix based on the state and the observation covariance matrix R k Calculate the Kalman gain K k ; S315 is based on the Kalman gain K k Update the state vector X k and the state estimation covariance matrix P k .

3. The method according to claim 2, characterized in that In step S312, in order to adapt to the characteristics of nonlinear movement of fish, the state transfer matrix F k Usually, nonlinear factors such as the acceleration of the target need to be considered. The state transfer matrix F k According to the characteristics of EKF, the Jacobian matrix can be used to construct F k It is defined as an 8-dimensional matrix, with the diagonal elements being 1, indicating that the state value at the current moment is directly equal to the state at the previous moment; F k [1,5],F k [2, 6], F k [3,7] and F k The value [4, 8] is the dt time step, which indicates how the target's velocity affects the change of its position or height. According to the constructed state transfer matrix F k , the target state X of the previous frame k-1 Get state estimate and the state estimation covariance matrix The estimation formula for the next frame state is: where Q k is the process noise matrix, which reflects the uncertainty of the model's state estimation.

4. The method according to claim 2, characterized in that In step S313, the nonlinear Kalman filter based on the detection confidence weighting is obtained by k Adjust so that the Kalman gain can be adjusted according to the detection confidence C k Dynamically adjust the observation value d k The weight in trajectory update; the formula is: Specifically, the lower the detection confidence, the larger the observation noise covariance, and the smaller the calculated Kalman gain, thereby reducing the impact of low-confidence detection values on state updates; conversely, high-confidence detection values will give the detection a higher weight.

5. The method according to claim 1, wherein The target association part in step S3 addresses the problem of similar features and mutual occlusion of fish targets by improving the association strategy between the detection frame and the estimation frame, designing a new cost matrix, and adding a category gating mechanism ECGAM. Specifically, it includes steps S321-S326: S321: Appearance feature f based on the detection frame det (d j ) and the appearance features of the trajectory estimation box f trk (t i ) Calculate the cosine distance and construct the cost matrix C using the minimum cost matching method cos ; S322: Calculate the Mahalanobis distance using the motion features of the detection frame and the trajectory estimation frame, and C cos Perform weighted combination to obtain the new cost matrix C mahal ; S323: Use category gating to control the cost matrix C mahal Filter to ensure that only trajectories and detection boxes of the same category can match, and obtain the cost matrix C classes ; S324: The cost matrix C after the final filtering using the Hungarian algorithm cost =C classes Solve the optimal allocation problem above and get the set of matched trajectories and detection boxes M (1) ={(t i ,d j )}, and the set of unmatched trajectories and unmatched detection box set S325: For unmatched trajectory sets Use the IoU between the track box and the detection box to construct the cost matrix C IoU , use the Hungarian algorithm to get the final matching pair M (2) ={(t i ,d j )} and unmatched trajectory set and unmatched detection box set S326: Unmatched detection box d j A new trajectory is generated and set to the unconfirmed state Tentative, without matching the trajectory t i Will increase the unmatched age When its unmatched age exceeds the maximum age threshold max Or in an unconfirmed state, the track will be destroyed.

6. The method according to claim 5, characterized in that In step S323: a category gating mechanism is introduced to filter the original cost matrix to obtain the cost matrix C classes , to ensure that only trajectories and detection boxes of the same category can be matched. The construction formula is: Among them, class(t i ) is t i Category, class(d j ) is d j Category, gated_cost is the gated cost used to indicate infeasible matching.

Images