Fish swarm tracking method based on improved YOLOX and DeepSORT

Abstract

The application discloses a fish group tracking method based on improved YOLOX and DeepSORT, and comprises the following steps: acquiring video data, and performing auxiliary labeling by using a threshold segmentation method to obtain a training data set; performing data enhancement on the data set, training a YOLOX model, and judging a two-classification model CNN occ a ReID model, and simultaneously changing an inference stage of the YOLOX model into a Soft-NMS; initializing a feature set according to a limited fish group quantity, saving appearance features and position information by using DeepSORT, sending a detection result of the YOLOX model into the DeepSORT to perform two-stage target matching, outputting a tracking result of a current frame after the two-stage target matching, and adopting an adaptive updating feature set strategy to update a target feature set in an updating stage. The application can effectively realize fish group multi-target tracking, and overcome the time-consuming and labor-consuming shortcomings of a traditional artificial monitoring mode.

Description

Technical Field

[0001] This invention relates to the technical field of target recognition and multi-target tracking, and in particular to a fish swarm tracking method based on improved YOLOX and DeepSORT. Background Technology

[0002] Fish behavior is a fundamental field of study, and its behavioral characteristics play a crucial indicative role in fish breeding, aquaculture, and environmental monitoring. Swimming behavior is a key component of fish behavior research and a vital life activity for fish in water, significantly impacting their migration, foraging, reproduction, and predator avoidance (Zhong et al., 2013). Furthermore, fish swimming habits provide important guidance for selecting aquaculture models and facility specifications. Currently, research on swimming behavior primarily relies on manual statistics. However, the rapid swimming speed of fish and the tendency for them to block each other significantly interfere with manual statistics, making it time-consuming, labor-intensive, inefficient, and inaccurate. Moreover, manual observation cannot precisely quantify behavioral parameters such as swimming speed, which can also interfere with fish behavior, leading to substantial errors in the statistical results.

[0003] Currently, among fish swarm tracking methods, patent CN106529434A uses a visual attention model to track individual fish within a swarm. Patent CN103955688A uses extreme value detection and local region ellipse fitting to locate the fish head region and estimate relevant parameters, then combines Kalman prediction and feature matching to track the swarm.

[0004] In laboratory environments, the limited space leads to fish exhibiting swarming behavior and easy occlusion between fish, resulting in frequent target ID swapping during tracking. Furthermore, existing methods rely heavily on traditional methods such as threshold segmentation, which do not provide high tracking accuracy. Therefore, to address the problems encountered in tracking fish swarms, a fish swarm tracking method based on an improved YOLOX and DeepSORT algorithm was designed, proposing a fair matching and adaptive feature update strategy to improve the tracking accuracy of fish swarms in laboratory environments. Summary of the Invention

[0005] The purpose of this invention is to reduce the workload of researchers and accurately quantify the behavioral parameters of fish schools. It proposes a fish school tracking method based on an improved YOLOX and DeepSORT model. This method considers the occlusion problem caused by the fish school's clustering behavior during overhead photography. For occlusion scenarios, the Soft-NMS method is used in the YOLOX model to improve recall in occluded scenes. Furthermore, in DeepSORT target matching, a fair matching strategy is adopted instead of cascaded matching, ignoring trajectory priority matching that depends on the duration of occlusion, ensuring fair matching of trajectories and targets with the same priority in a fixed number of scenarios. In the update phase, an adaptive feature update strategy is adopted to selectively update the trajectory feature set, ensuring the independence of features within the feature set. This improves the accuracy of IDs under occlusion conditions and reduces ID switches during tracking.

[0006] To achieve the above objectives, the technical solution provided by this invention is: a fish swarm tracking method based on improved YOLOX and DeepSORT, comprising the following steps:

[0007] S1. Obtain video data and use threshold segmentation to perform auxiliary annotation to obtain the training dataset;

[0008] S2. Perform data augmentation on the dataset and train the YOLOX model and a CNN binary classification model to determine occlusion. occ Along with the ReID model, the inference phase of the YOLOX model is changed to Soft-NMS;

[0009] S3. Initialize the feature set based on the limited number of fish, and use DeepSORT to save appearance features and location information; feed the detection results of the YOLOX model into DeepSORT for two-stage target matching. The first stage is appearance feature matching using a fair matching strategy, and the second stage is IoU matching. The fair matching strategy involves the ReID model extracting appearance features and matching them with the appearance features saved in the feature set with the same priority. After the two-stage target matching, output the tracking result of the current frame, and in the update stage, adopt an adaptive feature set update strategy to update the target feature set. This adaptive feature set update strategy utilizes CNN. occ Determine whether the target is occluded and decide whether to update the feature set.

[0010] Furthermore, the specific steps of step S1 are as follows:

[0011] S11. Use a large cardboard box that can hold the experimental box. Set up a horizontal bar above the experimental box for the camera to take pictures of the fish in the experimental box from a top-down perspective. In order to provide a stable and uniform light source, an LED white light needs to be set up above the camera, and a diffuser paper is placed between the LED white light and the camera to diffuse the light.

[0012] S12. Use a Hikvision camera to capture overhead video of the experimental chamber. The collected data includes 24-hour video data of fish activity.

[0013] S13. Randomly crop video segments from the video data to construct a dataset;

[0014] S14. For images in the dataset, the fish are separated from the background using a binarization thresholding segmentation algorithm to obtain a binarization matrix. The connected component algorithm for calculating the binarization matrix in the OpenCV open-source computer vision library is used to obtain the connected component, which is a connected region. The bounding rectangle of the connected component is used as the bounding box of the fish. Finally, the data is saved in YOLO format, and the LabelImg annotation tool is used to manually correct the difficult-to-separate data of the fish.

[0015] Furthermore, in step S2, the YOLOX model uses the anchor-free object detection algorithm YOLOX as the fish detection model. It employs a learning rate warmup strategy and Mosaic and Mixup data augmentation methods to improve model accuracy, and replaces NMS with Soft-NMS to improve the model's recall under occlusion conditions; the occlusion-determining classification model CNN... occ The ReID model uses ResNet-18 as the base network and employs copy-paste data augmentation to increase the number of occluded samples. The model uses ResNet-18 as the base network and rotation and flipping data augmentation to improve the model's accuracy.

[0016] Furthermore, in step S3, assuming there are N fish in the experimental box, DeepSORT is limited to a maximum of N trajectories. In the first stage of target matching, cascaded matching is replaced with a fair matching strategy. That is, the ReID model extracts the appearance feature vectors of B targets in the current frame, calculates the cosine distance between the feature sets of the N trajectories and the B appearance feature vectors, obtains an N*B cost matrix, and then uses the Hungarian algorithm to assign targets and corresponding trajectories. In the second stage of target matching, IoU matching is performed on the trajectories and targets that were not matched in the first stage. The IoU matrix is ​​calculated between the B targets and the position information saved in the previous frame, and then the Hungarian algorithm is used to assign targets based on the IoU matrix.

[0017] During the target update phase, a threshold parameter T is set, and the IoU between each pair of detection boxes is calculated to obtain the matrix M. IoU In M IoU Find detection boxes where IoU > T, and use CNN. occ Determine if there is occlusion within the detection box. If not, update its features to the trajectory feature set; otherwise, discard it.

[0018] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0019] 1. This invention utilizes machine learning methods to achieve multi-target tracking of fish schools, overcoming the drawbacks of traditional manual monitoring methods, which are time-consuming and labor-intensive.

[0020] 2. This invention quantifies swimming behavior by calculating the tracking trajectory of fish schools, thereby obtaining high-precision behavioral information and providing theoretical data support for researchers.

[0021] 3. This invention proposes a method for data annotation using threshold segmentation-assisted annotation, which greatly reduces the workload of annotation and saves time.

[0022] 4. This invention proposes a multi-target tracking framework that combines improved YOLOX and DeepSORT to improve tracking accuracy in scenarios with a fixed number of targets. This framework is versatile and can be used for automatic tracking of different fish groups.

[0023] 5. This invention proposes to utilize CNN in the target update stage. occ The method for determining fish occlusion maintains the independence of the target's updated appearance feature set, significantly reducing the frequency and number of ID exchanges. This provides a new approach for exploring high-precision multi-target tracking of fish swarms and its accurate quantification of behavior. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the logical flow of the method of the present invention.

[0025] Figure 2 A flowchart for creating a dataset for this invention is provided.

[0026] Figure 3 This is a diagram illustrating the algorithm design of this invention.

[0027] Figure 4 This is a comparison chart of video frames before and after the improvement. Detailed Implementation

[0028] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0029] See Figure 1As shown, this embodiment provides a fish swarm tracking method based on improved YOLOX and DeepSORT. This method can solve the difficulties in tracking fish when they gather, achieving high-precision multi-target tracking of fish swarms, reducing the workload of researchers, and accurately quantifying the behavioral parameters of fish swarms. This method is developed based on the Ubuntu 16.04 operating system and uses an Nvidia GeForce RTX 980Ti GPU on the PyTorch deep learning framework platform for model training and testing. The specific implementation steps are as follows:

[0030] S1. Set up the shooting environment and equipment, and collect footage during the fish experiment, including the following steps:

[0031] S11. Use a large cardboard box that can hold the experimental box. Set up a horizontal bar above the experimental box so that the camera can take pictures of the fish in the experimental box from a top-down perspective. In order to provide a stable and uniform light source, an LED white light needs to be set up above the camera, and a diffuser paper is placed between the LED white light and the camera to diffuse the light.

[0032] S12. Six groups of experiments were conducted simultaneously. Hikvision cameras were used to take overhead shots of the experimental boxes at the same height, and six groups of videos were obtained, each containing four experimental fish with different diets. The video frame resolution was 1920*1080 pixels and the frame rate was 25 frames per second.

[0033] S2. Use a threshold segmentation algorithm to assist in labeling fish bodies and create a dataset, as follows:

[0034] S21. Randomly crop 20 segments from the video, select 13 segments, each 15 seconds long, as the test set, and the rest as the training set for the model.

[0035] S22. After segmenting the fish population from the background using a binarization thresholding algorithm, a binarization matrix is ​​obtained. See Figure 2 As shown in Figure a, the connected component is obtained using the connected component algorithm for calculating the binarized matrix in the OpenCV open-source computer vision library. Figure 2 As shown in b, a connected component is a connected region; the bounding rectangle of the connected component is used as the bounding box of the fish, see... Figure 2 As shown in c, the data is finally saved in YOLO format, and the LabelImg annotation tool is used to manually correct data that is difficult to separate, such as fish crossings.

[0036] S23. Use Copy-Paste to augment the data. Randomly select a mask of a fish from the image segmented by thresholding in S22, and paste it onto another fish after random rotation and folding to increase the number of complex samples with cross-occlusion.

[0037] S3, Training the YOLOX model, and using a binary classification model (CNN) to determine occlusion. occ Along with the ReID model, the inference phase of the YOLOX model is also changed to Soft-NMS, as follows:

[0038] This method utilizes the PyTorch deep learning framework to train the YOLOX model and CNN on an NVIDIA GTX 980Ti GPU platform running on Ubuntu 16.04. occ ReID model;

[0039] a. CNN occ Using ResNet-18 as the base network, images with IoU are selected and cropped from the labeled bounding boxes. Then, the images within the bounding boxes are manually judged to see if they are occluded. If they are occluded, the image is marked as "1" and otherwise as "0".

[0040] b. YOLOX Model Training

[0041] To balance speed and accuracy, the anchor-free target detection algorithm YOLOX was selected as the fish detection model, and data augmentation methods such as Mixup and Mosaic were used during the training phase. The model was trained for 200 epochs.

[0042] In occlusion scenarios, detection boxes often overlap significantly. Non-maximum suppression (NMS) will suppress detection boxes with overlap exceeding an overlap threshold K, leading to missed detections of fish with high overlap under occlusion conditions. To improve recall and tracking accuracy, NMS is replaced with Soft-NMS during the YOLOX model inference stage. Soft-NMS considers the overlap between the score and the bounding boxes while using NMS, thus improving the model's recall under occlusion conditions. The formulas for bounding box overlap IoU and Soft-NMS are as follows:

[0043]

[0044]

[0045] Among them, B P With B Q s represents the area of ​​bounding boxes P and Q. i represents the confidence score of the i-th detection box, and K represents the set overlap threshold.

[0046] c. ReID model training

[0047] The model was trained using ResNet-18 as the base network, and data augmentation techniques such as rotation and flipping were used to improve the model's accuracy.

[0048] S4. Initialize the feature set based on the limited number of fish, and use DeepSORT to save appearance features and location information; such as Figure 3 As shown, the detection results of the YOLOX model are fed into DeepSORT for two-stage target matching. The first stage is appearance feature matching using a fair matching strategy, and the second stage is IoU matching. The fair matching strategy involves the ReID model extracting appearance features and matching them with the appearance features stored in the feature set with the same priority. After the two-stage target matching, the tracking result of the current frame is output. In the update stage, an adaptive feature set update strategy is adopted to update the target feature set. This adaptive feature set update strategy utilizes CNN. occ Determine whether the target is occluded and decide whether to update the feature set.

[0049] In a laboratory environment, the number of fish is a known and controllable variable, which can be used as a priori condition. Assuming there are N fish in the experimental chamber, DeepSORT is limited to a maximum of N trajectories. In the first stage of target matching, cascaded matching is replaced with a fair matching strategy. That is, the ReID model extracts the appearance feature vectors of B targets in the current frame, calculates the cosine distance between the feature sets of the N trajectories and the B appearance feature vectors, obtains an N*B cost matrix, and then uses the Hungarian algorithm to assign targets to their corresponding trajectories. In the second stage of target matching, IoU matching is performed on the trajectories and targets that failed to match in the first stage. The IoU matrix is ​​calculated between the B targets and the position information saved in the previous frame, and then the Hungarian algorithm is used to assign targets based on the IoU matrix.

[0050] Because fish schools may overlap and obstruct each other, target updates may involve the appearance features of two or more fish. Updating these appearance features can disrupt the independence of feature sets within the trajectory, causing interference in the similarity matrix calculation during target matching. To address this, an adaptive feature set update strategy is adopted: during the target update phase, a threshold parameter T is set, and the IoU between each pair of detection boxes is calculated to obtain the matrix M. IoU In M IoU Find detection boxes where IoU > T, and use CNN. occ Determine if there is occlusion within the detection box. If not, update its features to the trajectory feature set; otherwise, discard it.

[0051] The specific steps of step S4 are as follows:

[0052] S41. To perform target matching using a fair matching strategy, the first step is to manually observe and obtain the number of fish, N. This limits DeepSORT to N sets of tracks for target matching. Each frame's detection result is matched against these N tracks. This matching process uses the ReID model to extract the appearance feature vector of each detection box. The minimum cosine distance between these feature vectors and the feature sets stored in these N tracks is used to obtain the appearance feature matrix Cos_cost_matrix. Cosine distance is a similarity calculation method used to measure the degree of similarity between different individuals; a smaller cosine distance indicates greater similarity. The formula for calculating cosine distance is as follows:

[0053]

[0054] In the formula, A and B represent the appearance feature vectors between two bodies, and θ is the cosine angle.

[0055] In Cos_cost_matrix, motion information is filtered. For each track, a Kalman filter is used to predict the tracking box of the current frame based on the tracking box information of the previous frame, and the Mahalanobis distance between the tracking box and the detection box of the current frame is calculated. If the Mahalanobis distance exceeds a set threshold, the corresponding element of Cos_cost_matrix is ​​set to infinity. Finally, after filtering the motion information, Cost_matrix is ​​obtained. Cost_matrix is ​​used as input to the Hungarian algorithm to obtain the matching result between the track and the detected target.

[0056] S42. For Tracks and Detection Results that did not match successfully in the previous step, calculate the IoU matrix by calculating the position information of the predicted bounding box and the detection bounding box. If the element in the IoU matrix is ​​greater than the set threshold, set the element of the IoU matrix at the corresponding position to infinity, and then perform Hungarian algorithm matching again.

[0057] S43. Add the results of S41 and S42 to obtain the final target matching result.

[0058] S44. Based on the IoU matrix and the binary classification model CNN for occlusion detection. occ The adaptive target update strategy updates the feature set of each track as follows: Pairwise detection boxes with IoU > T are selected from the IoU matrix, and the images within these detection boxes are input into a binary classification model (CNN) for occlusion detection. occ In the process, the model output determines whether there is occlusion. If occlusion is detected, its appearance features are not updated. If no occlusion is detected, its appearance features are updated to the feature set of the corresponding Track.

[0059] S45. Repeat S41-S44 until the last frame of the video.

[0060] Improvement for example Figure 4 As shown, Figure 4 The upper left corner of 'a' and 'c' indicates the occlusion situation between two frames during the tracking process, such as... Figure 4 As shown in Figure b, an ID switch occurs after occlusion; this invention can guarantee the independence of the Track feature set under occlusion conditions, improving individual tracking accuracy, such as... Figure 4 As shown in Figure d, this invention effectively solves the ID switching problem caused by adhesion and occlusion. Therefore, this invention can ensure accurate and stable tracking of each fish ID.

[0061] The experimental results of the fish tracking method described above are explained in detail below:

[0062] This invention uses four commonly used evaluation metrics to statistically analyze the detection results of the test set, namely, precision (P), recall (R), identification F1 score (IDF1), and multiple object tracking accuracy (MOTA). Their formulas are defined as shown in formulas (1)-(4):

[0063]

[0064]

[0065]

[0066]

[0067] In the formula, TP (True Positive) is the number of positive samples in the image that the model predicts as positive, FP (False Positive) is the number of negative samples in the image that the model predicts as positive, FN (False Negative) is the number of negative samples in the image that the model predicts as positive, IDTP is the number of true IDs in the tracking box, IDFN is the number of false negative IDs in the tracking box, and IDFP is the number of false positive IDs in the tracking box.

[0068] The test set was calculated using the above formula, and the statistical results are shown in Table 1.

[0069] The comparison results between the benchmark DeepSORT method and the method proposed in this invention are shown in Table 1 below. Compared with the benchmark DeepSORT method, the method proposed in this invention has better detection effect and higher tracking accuracy.

[0070] Table 1 Comparison of Results

[0071] method Precision Recall↑ <![CDATA[IDF1↑]]> MOTA↑ Benchmark DeepSORT 99.8 99.2 93.3 98.8 This invention 99.8 99.7 99.5 99.3

[0072] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A fish swarm tracking method based on improved YOLOX and DeepSORT, characterized in that, Includes the following steps: S1. Obtain video data and use threshold segmentation to perform auxiliary annotation to obtain the training dataset. The specific steps are as follows: S11. Use a large cardboard box that can hold the experimental box. Set up a horizontal bar above the experimental box for the camera to take pictures of the fish in the experimental box from a top-down perspective. In order to provide a stable and uniform light source, an LED white light needs to be set up above the camera, and a diffuser paper is placed between the LED white light and the camera to diffuse the light. S12. Use a Hikvision camera to capture overhead video of the experimental chamber. The collected data includes 24-hour video data of fish activity. S13. Randomly crop video segments from the video data to construct a dataset; S14. For the images in the dataset, the fish group and the background are separated by a binarization thresholding segmentation algorithm to obtain a binarization matrix. The connected component algorithm for calculating the binarization matrix in the OpenCV open-source computer vision library is used to obtain the connected component, which is a connected region. The bounding rectangle of the connected component is used as the bounding box of the fish. Finally, it is saved in YOLO format, and the LabelImg annotation tool is used to manually correct the data that is difficult to separate by the intersection of the fish group. S2. Perform data augmentation on the dataset and train the YOLOX model and a CNN binary classification model to determine occlusion. occ Along with the ReID model, the inference phase of the YOLOX model is changed to Soft-NMS; The YOLOX model uses the anchor-free object detection algorithm YOLOX as the fish detection model. It employs a learning rate warmup strategy and Mosaic and Mixup data augmentation methods to improve model accuracy. Soft-NMS replaces NMS to improve recall under occlusion conditions. The occlusion-detecting classification model is a CNN. occ The ReID model uses ResNet-18 as the base network and employs copy-paste data augmentation to increase the number of occluded samples. It also uses rotation and flipping data augmentation to improve model accuracy. S3. Initialize the feature set based on the limited number of fish, and use DeepSORT to save appearance features and location information; feed the detection results of the YOLOX model into DeepSORT for two-stage target matching. The first stage is appearance feature matching using a fair matching strategy, and the second stage is IoU matching. The fair matching strategy involves the ReID model extracting appearance features and matching them with the appearance features saved in the feature set with the same priority. After the two-stage target matching, output the tracking result of the current frame, and in the update stage, adopt an adaptive feature set update strategy to update the target feature set. This adaptive feature set update strategy utilizes CNN. occ Determine whether the target is occluded and decide whether to update the feature set; Assuming there are N fish in the experimental chamber, and DeepSORT is limited to a maximum of N trajectories, in the first stage of target matching, cascaded matching is replaced with a fair matching strategy. That is, the ReID model extracts the appearance feature vectors of B targets in the current frame, calculates the cosine distance between the feature sets of the N trajectories and the B appearance feature vectors, obtains an N*B cost matrix, and then uses the Hungarian algorithm to assign targets and corresponding trajectories. In the second stage of target matching, IoU matching is performed on the trajectories and targets that did not match in the first stage. The IoU matrix is ​​calculated between the B targets and the position information saved in the previous frame, and then the Hungarian algorithm is used to assign targets based on the IoU matrix. During the target update phase, a threshold parameter T is set, and the IoU between each pair of detection boxes is calculated to obtain the matrix M. IoU In M IoU Find detection boxes where IoU > T, and use CNN. occ Determine if there is occlusion within the detection box. If not, update its features to the trajectory feature set; otherwise, discard it.

Images