A multi-target tracking method for shrimps

Abstract

The application discloses to belong to the computer vision technical field, especially relate to a kind of for shrimps Multi-target tracking method, including: constructing training dataset, building, training, using shrimps Multi-target tracking model, complete shrimps Multi-target tracking.Based on YOLOX model, increase appearance representation extraction branch, realize end-to-end simultaneously training detection model and appearance representation;Increase cross-dimension feature weighting module on detection frame regression branch, improve the prediction accuracy of target position of detector;Adopt uncertain loss function, dynamically adjust weight.When tracking prawns, adopt the strategy of cascade matching, consider the appearance features and the objects with low detection score, solve the mutual occlusion problem in dense scene.The multi-target tracking method can realize online tracking of shrimps in dense scene and obtain motion trajectory, solve the problem of small target and frequent ID switching in the process of shrimp tracking.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and specifically relates to a multi-target tracking method for shrimp. Background Technology

[0002] Multi-target tracking aims to obtain the trajectory of a set of targets. Based on the trajectory information, the behavior of the targets can be judged, and it has wide applications in video surveillance, autonomous driving, and behavioral science research. Factory farming is a developing trend in aquaculture. In factory farming workshops, intelligent monitoring technology is used to monitor the shrimp farming environment. Analyzing video data to examine abnormal states and behaviors is crucial for achieving healthy shrimp farming. Shrimp target tracking, in particular, can reflect their movement characteristics and patterns, and promptly detect abnormalities, making it an important means of achieving healthy farming. On the one hand, changes in shrimp movement are the most direct phenomenon of water quality changes, serving as a biological early warning indicator of changes in the farming environment, quickly detecting abnormalities when water quality deteriorates. On the other hand, shrimp movement patterns are also an important indicator of their health; their movement patterns change under adverse conditions such as stress, weakness, low temperature, and exposure to pollutants.

[0003] In recent years, most research on multi-object tracking in videos has focused on pedestrians and vehicles for autonomous driving. Some research has also explored tracking other targets in industrial production to aid in safe manufacturing. Detection-based tracking has become a mainstream method in recent years. This method typically involves two steps: first, locating each object in each frame using a high-precision detector; and second, performing cross-frame identity restoration for each object, which involves associating objects in the current frame with those in the previous frame, with successfully matched objects forming a continuous trajectory. Early algorithms separated object detection and appearance feature extraction, resulting in slow training and inference speeds. Later, researchers proposed training both simultaneously, a method adopted by many algorithms and known as joint detection and embedding. In factory-scale aquaculture, there has been research on tracking fish and obtaining their movement trajectories. Compared to tracking humans, fish appearance features are more difficult to identify, leading to frequent target ID switching during tracking.

[0004] In the process of shrimp tracking, the low recognition of shrimp targets under panoramic cameras and the frequent switching of target IDs in intensive farming scenarios result in poor tracking of shrimp movement trajectories, which has become a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-target tracking method for shrimp, characterized by the following steps:

[0006] S1: Collect overhead videos of shrimp to build a training dataset for a multi-target tracking model of shrimp;

[0007] S2: Construct a shrimp multi-target tracking model based on multi-task learning and cascaded strategies. The shrimp multi-target tracking model adds an appearance representation extraction branch and a cross-dimensional feature weighting module to the YOLOX model. The shrimp multi-target tracking model includes: a backbone network, a feature fusion network, and a classification prediction network. The classification prediction network is based on the YOLOX model and includes a classification branch, a confidence regression branch, and a bounding box regression branch.

[0008] S3: Use the shrimp multi-target tracking model to train the dataset, and use the transfer learning method to train the shrimp multi-target tracking model;

[0009] S4: Based on the trained shrimp multi-target tracking model, the cascaded data association method is used for online tracking to draw the shrimp's movement trajectory.

[0010] The structure of the appearance representation extraction branch in step 2 is defined as follows:

[0011]

[0012] F scale (z,avgp(f1))=F scale (σ(W1δ(W0avgp(f1))) (2)

[0013] Where * represents convolution, × represents element-wise multiplication, and o reid The output of the branch for appearance extraction, Let f represent the set of channel filters, f0 represent the feature map extracted by the backbone network, f1 represent the feature map extracted by the residual network, and F scale () represents the scaling transformation of the channel dimension, z represents a series of linear mapping combinations, avgp() represents average pooling, W0 and W1 represent the weights obtained during training, σ() represents the Sigmoid activation function, and δ() represents the ReLU activation function.

[0014] The execution steps for extracting the branch, as indicated by the appearance, are as follows:

[0015] The image features obtained from the backbone network are input into two basic combination modules for further feature extraction;

[0016] The results of feature extraction are input into a residual network to extract the appearance features of shrimp.

[0017] The appearance features are input into the SE-Net network, and the appearance features are weighted from the channel dimension to obtain the expression of the shrimp appearance features.

[0018] The structure of the cross-dimensional feature weighting module in step 2 is defined as follows:

[0019]

[0020]

[0021] in, Let f1 represent the set of channel filters, f2 represent the feature map after attention weighting of the channels, W2 and W3 represent the weights obtained during training, and o reg This represents the output of the detection box regression branch. and This represents a set of spatial filters.

[0022] The execution steps of the cross-dimensional feature weighting module are as follows:

[0023] The image features obtained from the backbone network are input into two basic combination modules for further feature extraction;

[0024] The results of feature extraction are sequentially input into one convolutional layer, one channel attention module, and one spatial attention module;

[0025] Obtain the center position, height, and width of the shrimp detection frame.

[0026] The steps for training the shrimp multi-target tracking model in step 3 are as follows:

[0027] Input labeled shrimp video frames;

[0028] Image features are extracted through a backbone network, and after passing through a feature fusion network, the image features are input into a classification and prediction network to obtain the appearance features, classification, confidence, and detection boxes of multiple objects in the video frame.

[0029] The prediction results are compared with the labels, the loss value is calculated using an uncertain loss function, and backpropagation is performed.

[0030] The parameters of the shrimp multi-target tracking model are optimized using an optimizer.

[0031] The definition of the uncertainty loss function is:

[0032]

[0033]

[0034] l det =w3×l cls +w4×l obj +w5×l reg (7)

[0035] lcls =-(y n ×log(σ(z n )))+(1-y n )×(1-log(σ(z n ))) (8)

[0036] l obj =-(y n ×log(σ(z n )))+(1-y n )×(1-log(σ(z n ))) (9)

[0037] Among them, l id The cross-entropy loss function is used for appearance representation extraction classification tasks, where N is the number of samples in the training set, K is the number of classes, and l i p(k) is the ground truth label of an individual, p(k) is the probability that the observed sample belongs to category k, and l det The loss function used for the detection task is defined by l. cls l obj l reg It consists of three tasks combined, l cls and l obj Let BCEWithLogitsLoss be the loss function, l reg The IoU loss function is defined by w3 and w4, where w3 and w4 are values ​​of 1 and w5 is value of 5, representing the Sigmoid activation function. n y represents the score for predicting the nth sample as a positive example. n Let w1 and w2 represent the label of the nth sample, respectively. id and l det The weight.

[0038] The steps of the cascading data association method in step 4 are as follows:

[0039] S41: Initialize the target detected in the first frame of the video as the tracking object, and classify the detected object into high-scoring objects and low-scoring objects based on a preset tracking confidence threshold;

[0040] S42: In subsequent video frames, the detected object and the tracked object are associated in two stages:

[0041] In the first stage, based on the target being detected at the current moment, the Kalman filter method is used to predict the position of the object being tracked.

[0042] Then, for high-scoring detection targets, the cost matrix is ​​calculated by combining the motion model and the appearance model, and the Hungarian algorithm is used to perform linear allocation based on the cost matrix;

[0043] In the second stage, low-scoring detection targets are linearly assigned based on the IoU distance of the detection boxes and using the Hungarian algorithm.

[0044] S43: Delete the tracked objects that have not been matched for a long time, and initialize the newly detected and unmatched targets as new tracked objects;

[0045] S44: Connect the trajectories of each moving object to obtain the complete movement trajectory of the individual shrimp.

[0046] The formula for calculating the cost matrix is ​​as follows:

[0047] D1=λD iou +(1-λ)D emb (10)

[0048] Among them, D iou It is the IoU distance, D emb It is the cosine distance, with λ set to 0.9.

[0049] D iou =1-IOU (11)

[0050] Wherein, IOU is the intersection-union ratio of the detection box and the predicted box;

[0051]

[0052] Where u and v are the appearance model representation matrices of the detected object and the tracked object, respectively.

[0053] The steps in step 1 for constructing the training dataset for the shrimp multi-target tracking model are as follows:

[0054] Install video capture equipment to ensure that the video captured by the equipment can cover the entire range of shrimp movement. Divide the captured video into short videos of 10 to 50 seconds, and annotate each frame of the captured video using the MOT format. Divide the annotated video into training set and test set.

[0055] The beneficial effects of this invention are as follows:

[0056] This invention discloses a multi-target tracking method for shrimp, which can reduce the number of target ID switching times in intensive aquaculture scenarios, obtain complete shrimp movement trajectories, provide support for quantitative analysis of shrimp movement patterns, and lay the foundation for analyzing abnormal shrimp behavior. It not only enables timely detection of abnormal shrimp behavior and aquaculture water quality anomalies, reducing potential economic losses, but also has significant implications for shrimp behavioral research. Specific effects are as follows:

[0057] 1. A multi-target shrimp tracking method based on multi-task learning and cascaded strategies is proposed. This method can obtain the shrimp target location and appearance features end-to-end through joint training, thereby realizing online tracking of shrimp in dense scenes, which has practical applications in actual aquaculture.

[0058] 2. This method adds an appearance representation extraction branch to the anchor-free detector YOLOX, and extracts a better shrimp appearance representation through the residual mechanism and SE attention module. It can perform matching based on appearance features during tracking, and can match individual shrimp more accurately.

[0059] 3. By adding a cross-dimensional feature weighting module to the detection box regression branch of the model, the prediction accuracy of the detector is improved, and the tracking effect is further enhanced.

[0060] 4. It adopts a cascaded data association method, which can balance the accuracy and efficiency of data association and handle scenarios with densely distributed tracking objects. Attached Figure Description

[0061] Figure 1 This invention discloses a flowchart of a multi-target tracking method for shrimp;

[0062] Figure 2 This is a diagram showing the overall structure of the shrimp multi-target tracking model disclosed in this invention.

[0063] Figure 3 The appearance representation of the extracted branch network structure diagram disclosed in this invention;

[0064] Figure 4 This is a diagram of the detection box regression branch network structure with added cross-dimensional feature weighting module disclosed in this invention;

[0065] Figure 5 This is a network structure diagram of the shrimp multi-target tracking model disclosed in this invention;

[0066] Figure 6 This is an illustration of the implementation effect of a multi-target tracking method for shrimp disclosed in this invention. Detailed Implementation

[0067] The present invention will be further described in detail below with reference to the accompanying drawings.

[0068] like Figure 1The present invention discloses a multi-target tracking method for shrimp, which is implemented in four steps. First, overhead video of shrimp is collected to construct a training dataset for a shrimp multi-target tracking model. Second, a shrimp multi-target tracking model based on multi-task learning and cascaded strategies is built. Then, the model is trained using transfer learning. Finally, online multi-target tracking of shrimp is performed based on the model, and the shrimp's movement trajectory is plotted. The specific process is as follows:

[0069] S1: Collect overhead videos of shrimp to build a training dataset for a multi-target shrimp tracking model.

[0070] We collected overhead video footage of shrimp to construct a training dataset for a multi-target tracking model. First, cameras were mounted above the shrimp ponds to capture video data from a top-down perspective. During capture, we aimed to include the entire range of shrimp movement within the lens's coverage. Then, the videos were segmented into short videos of 10-50 seconds for easier data annotation. Finally, we used the MOT format to annotate each frame of the collected videos, and then divided the annotated videos into training and testing sets.

[0071] In this embodiment, the camera is installed 1.5m directly above the aquaculture pond, capable of capturing images of all individual shrimp in the pond. The camera used for video capture is a Hikvision 3T86FWDV2-I3S (8 megapixels, 4mm focal length). The captured video resolution is 1920*2560, and the frame rate is 20fps.

[0072] S2: Construct a shrimp multi-target tracking model based on multi-task learning and cascaded strategies. The shrimp multi-target tracking model adds an appearance representation extraction branch and a cross-dimensional feature weighting module to the YOLOX model.

[0073] Figure 2 This is a diagram illustrating the overall structure of the shrimp multi-target tracking model based on multi-task learning and cascaded strategies disclosed in this invention. The following description, in conjunction with the accompanying drawings, further illustrates the multi-target tracking method for shrimp described in this invention.

[0074] like Figure 2 As shown, the shrimp multi-object tracking model includes a backbone network, a feature fusion network, and a prediction network. The classification and prediction network is based on the YOLOX model and includes a classification branch, a confidence regression branch, and a bounding box regression branch. The shrimp multi-object tracking model adds an appearance representation extraction branch and a cross-dimensional feature weighting module to the YOLOX model. After inputting video frames into the shrimp multi-object tracking model, it outputs the appearance features, classification, confidence, and bounding boxes of multiple objects in the image. The prediction results are compared with the annotations of the video frames, loss is calculated, and the shrimp multi-object tracking model is optimized through backpropagation. The specific optimization process is as follows:

[0075] First, labeled shrimp video frames are input. Image features are extracted through a backbone network. After feature fusion, the features are input into four branches, which simultaneously perform four tasks: appearance representation extraction, object classification, object confidence regression, and bounding box regression. This yields the appearance features, classification, confidence, and bounding boxes of multiple objects in the image. The prediction results are compared with the labels, and loss functions are calculated separately. An uncertain loss function is then used to combine the four results to obtain the loss value (error), which is then backpropagated. The gradient is updated through an optimizer.

[0076] The backbone network is used to extract image features. In this embodiment, the backbone network PAFPN of the YOLOX model is used. It should be noted that no specific limitation is made to the backbone network in this embodiment. For example, the backbone network DLA34 of CenterNet can also be used.

[0077] This invention improves upon the existing YOLOX model by adding an appearance representation extraction branch to obtain shrimp appearance feature representations. The designed appearance representation extraction branch combines residual network and compressed dilated network structures to extract as many shrimp appearance features as possible. The structure of this network is defined as shown in formula (1).

[0078]

[0079] F scale (z,avgp(f1))=F scale (σ(W1δ(W0avgp(f1))) (2)

[0080] Where * represents convolution, × represents element-wise multiplication, and o reid The output of the branch for appearance extraction, Let f represent the set of channel filters, f0 represent the feature map extracted by the backbone network, f1 represent the feature map extracted by the residual network, and F scale () represents the scaling transformation of the channel dimension, z represents a series of linear mapping combinations, avgp() represents average pooling, W0 and W1 represent the weights obtained during training, σ() represents the Sigmoid activation function, and δ() represents the ReLU activation function.

[0081] like Figure 3As shown, the appearance representation extraction branch of this invention includes: two basic combination modules (CBS), one residual network (ResNet), and one SE-Net network. These modules are connected sequentially. The basic combination module includes, in sequence, a convolutional layer, a batch normalization operation, and a Silu activation function. The execution steps of the appearance representation extraction branch are as follows: First, the image features obtained by the backbone network are input into the two basic combination modules for further feature extraction. The result of the feature extraction is input into the residual network to extract the appearance features of shrimp. The appearance features are then input into the SE-Net network, where they are weighted from the channel dimension to obtain an expression of the shrimp appearance features. The residual network is used to extract the appearance features of shrimp, and SE-Net is used to weight the appearance features from the channel dimension. SE-Net is a channel attention mechanism that can weight features according to their importance, thereby obtaining a better expression of the shrimp appearance features.

[0082] Considering the large number of tracked objects in dense scenes, in an optional embodiment, the representation dimension of the appearance features of the appearance representation extraction branch is set to 512 dimensions.

[0083] This invention discloses a multi-target tracking method for shrimp. It adds an appearance representation extraction branch to the anchor-free detector YOLOX, extracting a better appearance representation of aquatic products through a residual mechanism and an SE attention module. This enables matching based on appearance features during tracking, resulting in more accurate matching of individual aquatic products. The appearance representation extraction branch can extract a 512-dimensional appearance representation, encoding the shrimp's appearance features into a 512-dimensional vector. Therefore, during matching in the tracking phase, both the appearance representation and the shrimp's motion features can be considered simultaneously, resulting in more accurate trajectory and candidate object matching. By correctly matching the trajectory and the detected object, the number of ID switching during tracking can be reduced.

[0084] The YOLOX model is used to obtain the target's classification, confidence score, and location. To improve the detection accuracy of shrimp, this invention makes another improvement to the existing YOLOX model by adding a cross-dimensional feature weighting module to the detection box regression branch of the YOLOX model. For example... Figure 4 As shown, the solid-lined box represents the detection box regression branch with added cross-dimensional feature weighting module. The specific calculation formula for this branch network is as follows:

[0085]

[0086]

[0087] in, Let f0 represent the set of channel filters, f2 represent the feature map extracted by the backbone network, f3 represent the feature map after channel attention weighting, and W2 and W3 represent the training weights. reg This represents the output of the detection box regression branch. and This represents a set of spatial filters.

[0088] The cross-dimensional feature weighting module includes: two basic combination modules, one convolutional layer, one channel attention module, and one spatial attention module. The execution steps of the cross-dimensional feature weighting module are as follows: First, the image features obtained by the backbone network are input into the two basic combination modules for further feature extraction. The resulting features are then input into the confidence regression branch and the detection box regression branch, respectively. In the detection box regression branch, the features are sequentially passed through one convolutional layer, one channel attention module, and one spatial attention module to obtain the center position, height, and width of the object's detection box. The basic combination modules include: convolutional layers, batch normalization, and a Silur activation function. The channel attention module sequentially includes: dimension transformation, a fully connected layer, a ReLU activation function, a fully connected layer, and dimension transformation. The spatial attention module sequentially includes: convolutional layers, batch normalization, a ReLU activation function, a convolutional layer, batch normalization, and a Sigmoid activation function.

[0089] The described cross-dimensional feature weighting module not only reflects the spatial importance of shrimp species but also the importance of features expressed in different channels. Most importantly, this module enables cross-dimensional interaction, resolving the information loss problem inherent in sequentially implemented attention modules.

[0090] This invention discloses a multi-target tracking method for shrimp. By adding a cross-dimensional feature weighting module to the detection box regression branch of the shrimp multi-target tracking model, the method can better regress detection boxes that are closer to the target by weighting the features. This reduces the number of incorrect matches between adjacent shrimp when matching trajectories due to inaccurate detection box positioning, improves the prediction accuracy of the detector, and further enhances the tracking effect.

[0091] This completes the construction of the shrimp multi-target tracking model based on multi-task learning and cascaded strategies disclosed in this invention. The overall network structure of the shrimp multi-target tracking model is as follows: Figure 5 As shown.

[0092] S3: Use the shrimp multi-target tracking model to train the dataset, and employ transfer learning to train the shrimp multi-target tracking model:

[0093] The steps for training a shrimp multi-target tracking model are as follows:

[0094] Input labeled shrimp video frames;

[0095] Image features are extracted through a backbone network, and after passing through a feature fusion network, the image features are input into a classification and prediction network to obtain the appearance features, classification, confidence, and detection boxes of multiple objects in the video frame.

[0096] The prediction results are compared with the labels, the loss value is calculated using an uncertain loss function, and backpropagation is performed.

[0097] The parameters of the shrimp multi-target tracking model are optimized using an optimizer.

[0098] During training, a loss function with homoscedastic uncertainty is used to combine the loss functions of multiple tasks. This function can learn a balance between classification and regression loss, and the specific formula is as follows:

[0099]

[0100]

[0101] l det =w3×l cls +w4×l obj +w5×l reg (7)

[0102] l cls =-(y n ×log(σ(z n )))+(1-y n )×(1-log(σ(z n ))) (8)

[0103] l obj =-(y n ×log(σ(z n )))+(1-y n )×(1-log(σ(z n ))) (9)

[0104] Among them, l id The cross-entropy loss function is used for appearance representation extraction classification tasks, where N is the number of samples in the training set, K is the number of classes, and l i (k) is the individual's ground truth label, p(k) is the probability that the observed sample belongs to category k, and l det The loss function used for the detection task is defined by l. cls l obj l reg It consists of three tasks combined, l cls and l objLet BCEWithLogitsLoss be the loss function, l reg The IoU loss function is defined by w3 and w4, where w3 and w4 are values ​​of 1 and w5 is value of 5, representing the Sigmoid activation function. n y represents the score for predicting the nth sample as a positive example. n Let w1 and w2 represent the label of the nth sample, respectively. id and l det The weight.

[0105] In this embodiment, transfer learning is applied during model training. First, the model parameters are initialized using models pre-trained by YOLOX on the COCO dataset, which effectively accelerates the training speed. The input image size is set to 960*1280, the batch size is 8, the initial learning rate is 0.001 / 64, the momentum parameter is set to 0.9, the non-maximum suppression threshold is set to 0.5, and the learning rate decays to 10%. -5 The training process consisted of 120 epochs, with data gain enabled for the first 80 epochs and disabled for the last 40 epochs.

[0106] S4: Based on the trained shrimp multi-target tracking model, the cascaded data association method is used for online tracking to draw the shrimp's movement trajectory.

[0107] During the tracking process in this embodiment, such as Figure 6 As shown, the multi-target tracking method for shrimp disclosed in this invention is applied to track Litopenaeus vannamei. For real-time video data, the category of the detected target in the video is first obtained based on the trained multi-target tracking model for shrimp (considering the possibility of multiple types of shrimp), confidence, center point position and size of the detection box, and target appearance features.

[0108] During the tracking process, a cascading matching strategy is used for data association:

[0109] S41: Initialize the target detected in the first frame of the video as the tracking object, and classify the detected object into high-scoring objects and low-scoring objects based on the tracking confidence threshold.

[0110] S42: In subsequent video frames, the detected object and the tracked object are associated in two stages.

[0111] In the first stage, based on the detection at the current moment, the Kalman filter method is used to predict the position of the tracked object. Then, high-scoring detected targets are used as candidate objects, and a cost matrix is ​​calculated by combining the motion model and the appearance model. The Hungarian algorithm is then used for linear allocation based on the cost matrix. The formula for calculating the cost matrix by combining the motion model and the appearance model is as follows:

[0112] D1=λDiou +(1-λ)D emb (10)

[0113] Among them, D iou It is the IoU distance, D emb It is the cosine distance, with λ set to 0.9.

[0114] D iou =1-IOU (11)

[0115] Wherein, IOU is the intersection-union ratio of the detection box and the predicted box;

[0116]

[0117] Where u and v are the appearance model representation matrices of the detected object and the tracked object, respectively.

[0118] In the second stage, low-scoring detected targets are also included as trajectory candidates, and linear allocation is performed based on the IoU distance of the detection boxes and using the Hungarian algorithm.

[0119] S43: Delete the long-unmatched tracking trajectory, indicating that the trajectory has ended and no longer needs to be tracked; initialize the newly detected and unmatched target as the new tracking object.

[0120] S44: Connect the trajectories of each moving object to obtain the complete movement trajectory of the individual shrimp.

[0121] In the target tracking process of this embodiment, the non-maximum suppression threshold is also set to 0.5, the tracking threshold is set to 0.35, the matching threshold is set to 0.9, and the lost tracking object is considered to have disappeared if it does not reappear after 20 frames.

[0122] This invention discloses a multi-target tracking method for shrimp. In the online tracking phase, a cascaded data association approach is used. In the first stage, only high-scoring candidate detection boxes are matched. In the second stage, low-scoring detection boxes that should have been discarded are still matched. Considering that when shrimp are densely packed, parts of their bodies may be occluded, the cascaded data association approach ensures that even when a target shrimp's detection score is low due to occlusion, a correct trajectory match can still be obtained through a combination of motion and appearance analysis. On the one hand, matching only high-scoring candidate detection boxes in the first stage avoids matching too many objects and reducing efficiency. On the other hand, ignoring correctly matched trajectories obtained in the first stage in the second stage effectively reduces the number of trajectories that need to be matched in the second stage. Simultaneously matching high-scoring candidate detection boxes mitigates the risk of missing correct objects due to occlusion. This approach balances the accuracy and efficiency of data association and can handle scenarios with densely distributed tracked objects.

[0123] This completes the application of the multi-target tracking method for shrimp disclosed in this invention, achieving multi-target tracking of shrimp. The shrimp multi-target tracking model based on multi-task learning and cascaded strategies has the following advantages:

[0124] (1) Add an appearance representation extraction branch to the YOLOX model without anchors. This branch combines the residual network and the compression and dilation network, and uses the features extracted by the backbone network to learn the appearance feature representation. This enables simultaneous training of the detector and the appearance representation extraction network, achieving end-to-end training without the need to prepare training data in different formats, thus enabling multi-task learning.

[0125] (2) Add a cross-dimensional feature weighting module to the YOLOX model to deal with the dense distribution of farming targets in shrimp farming. In addition to capturing the importance of channels and space, this module also considers cross-dimensional interaction to improve the accuracy of the detector. Cross-dimensional interaction can consider the importance of channels and space at the same time, and no information is lost after sequential channel weighting and spatial weighting.

[0126] (3) When tracking shrimp, a cascade matching strategy is adopted, taking into account appearance features and candidate objects with low detection scores.

[0127] To demonstrate the practical effectiveness of the multi-target tracking method for shrimp disclosed in this invention, relevant experiments were conducted. The experimental results are shown in Table 1, which compares the results of ablation experiments performed using the YOLOX-s model, the appearance representation extraction module, and the cross-dimensional interaction module. In the table, the higher the value of the index indicated by the upward arrow, the better, and the lower the value of the index indicated by the downward arrow, the better.

[0128] Table 1 Ablation Experiment Results

[0129]

[0130] As shown in Table 1, adding the appearance representation extraction branch enables the use of appearance features during tracking, improving all metrics except for ID switches (IDS). Specifically, this includes: Multiple Object Tracking Accuracy (MOTA), Multiple Object Tracking Precision (MOTP), ID metric (ID+F1 Score, IDF1), and Most Tracked (MT) proportion. Furthermore, adding the cross-dimensional interaction module further enhances all metrics listed in Table 1.

[0131] Regarding the tracking strategy, the multi-target tracking method using a cascaded matching strategy disclosed in this invention is compared with MOTDT and DeepSort. The experimental results are shown in Table 2. Since the three methods use the same detection model, the values ​​of MOTA and MOTP are similar. Compared with the tracking strategies using MOTDT and DeepSort, the multi-target tracking method disclosed in this invention improves both IDF1 and MT, while reducing the number of ID switching. Compared with the tracking strategies using MOTDT and DeepSort, the multi-target tracking method disclosed in this invention significantly reduces the average forward propagation time (AFT), average track time (ATT), and average inference time (ATT).

[0132] Table 2 Comparative experimental results of different tracking strategies

[0133]

[0134] This invention discloses a multi-target tracking method for shrimp. Compared with the two-stage tracking model of DeepSORT, it does not require training a feature extraction network in addition to the detector, which effectively improves the tracking speed. On the other hand, during the training phase, it does not require building two training datasets for the target detection model and the re-identification model respectively. The multi-target tracking method for shrimp disclosed in this invention only needs to label one dataset for training once. The experimental results shown in Table 2 show that it has better tracking performance.

Claims

1. A multi-target tracking method for shrimp, characterized in that, Includes the following steps: S1: Collect overhead videos of shrimp to build a training dataset for a multi-target tracking model of shrimp; S2: Build a shrimp multi-target tracking model based on multi-task learning and cascaded strategies. The shrimp multi-target tracking model adds an appearance representation extraction branch and a cross-dimensional feature weighting module to the YOLOX model. The shrimp multi-target tracking model includes: a backbone network, a feature fusion network, and a classification prediction network. The classification prediction network is based on the YOLOX model and includes a classification branch, a confidence regression branch, and a bounding box regression branch. S3: Use the shrimp multi-target tracking model to train the dataset, and use the transfer learning method to train the shrimp multi-target tracking model; S4: Based on the trained shrimp multi-target tracking model, the cascaded data association method is used for online tracking to draw the shrimp's movement trajectory; The structure of the appearance representation extraction branch in step 2 is defined as follows: （1） （2） Where * represents convolution and × represents element-wise multiplication. The output of the branch for appearance extraction, Let f represent the set of channel filters, f0 represent the feature map extracted by the backbone network, and f1 represent the feature map extracted by the residual network. Let z represent the scale transformation of the channel dimension, and z represent a series of linear mapping combinations. This indicates average pooling, where W0 and W1 represent the weights obtained during training. This represents the Sigmoid activation function. Represents the ReLU activation function; The appearance indicates that the execution steps for extracting the branch are as follows: The image features obtained from the backbone network are input into two basic combination modules for further feature extraction; The results of feature extraction are input into a residual network to extract the appearance features of shrimp. The appearance features are input into the SE-Net network, and the appearance features are weighted from the channel dimension to obtain the expression of the shrimp appearance features; In step 3 The steps for training a multi-target tracking model for shrimp are as follows: Input labeled shrimp video frames; Image features are extracted through a backbone network, and after passing through a feature fusion network, the image features are input into a classification and prediction network to obtain the appearance features, classification, confidence, and detection boxes of multiple objects in the video frame. The prediction results are compared with the labels, the loss value is calculated using an uncertain loss function, and backpropagation is performed. The parameters of the shrimp multi-target tracking model are optimized using an optimizer.

2. The multi-target tracking method for shrimp according to claim 1, characterized in that, The structure of the cross-dimensional feature weighting module in step 2 is defined as follows: （3） （4） in, Let f1 represent the set of channel filters, f2 represent the feature map after attention-weighted processing of the channels, and W2 and W3 represent the weights obtained during training. This represents the output of the detection box regression branch. and This represents a set of spatial filters.

3. The multi-target tracking method for shrimp according to claim 2, characterized in that, The execution steps of the cross-dimensional feature weighting module are as follows: The image features obtained from the backbone network are input into two basic combination modules for further feature extraction; The results of feature extraction are sequentially input into one convolutional layer, one channel attention module, and one spatial attention module; Obtain the center position, height, and width of the shrimp detection frame.

4. The multi-target tracking method for shrimp according to claim 1, characterized in that, The definition of the uncertainty loss function is: (5) (6) (7) (8) (9) Among them, l id The cross-entropy loss function is used for appearance representation extraction classification tasks, where N is the number of samples in the training set, K is the number of classes, and l i (k) is the individual's ground truth label, p(k) is the probability that the observed sample belongs to category k, and l det The loss function used for the detection task is defined by l. cls 、 l obj 、 l reg It consists of three tasks combined, l cls and l obj Let BCEWithLogitsLoss be the loss function, l reg Let w3 and w4 be the IoU loss function, and w5 be 5. Z represents the Sigmoid activation function. n y represents the score for predicting the nth sample as a positive example. n Let w1 and w2 represent the label of the nth sample, respectively. id and l det The weight.

5. A multi-target tracking method for shrimp according to claim 1, characterized in that, The steps of the cascading data association method in step 4 are as follows: S41: Initialize the target detected in the first frame of the video as the tracking object, and classify the detected object into high-scoring objects and low-scoring objects based on a preset tracking confidence threshold; S42: In subsequent video frames, the detected object and the tracked object are associated in two stages: In the first stage, based on the detected target at the current moment, the Kalman filter method is used to predict the position of the tracked object; then, for high-scoring detected targets, the cost matrix is ​​calculated by combining the motion model and the appearance model, and the Hungarian algorithm is used to perform linear allocation based on the cost matrix; In the second stage, low-scoring detection targets are linearly assigned based on the IoU distance of the detection boxes and using the Hungarian algorithm. S43: Delete the tracked objects that have not been matched for a long time, and initialize the newly detected and unmatched targets as new tracked objects; S44: Connect the trajectories of each moving object to obtain the complete movement trajectory of the individual shrimp.

6. A multi-target tracking method for shrimp according to claim 5, characterized in that, The formula for calculating the cost matrix is ​​as follows: (10) Among them, D iou It is the IoU distance, D emb It is the cosine distance, with λ set to 0.

9. (11) Wherein, IOU is the intersection-union ratio of the detection box and the predicted box; (12) Where u and v are the appearance model representation matrices of the detected object and the tracked object, respectively.

7. A multi-target tracking method for shrimp according to claim 1, characterized in that, The steps in step 1 for constructing the training dataset for the shrimp multi-target tracking model are as follows: Install video capture equipment to ensure that the video captured by the equipment can cover the entire range of shrimp movement. Divide the captured video into short videos of 10 to 50 seconds, and annotate each frame of the captured video using the MOT format. Divide the annotated video into training set and test set.

Images