An insect multi-individual tracking method and system based on estag and sort

Abstract

The present application relates to a kind of based on EStag and SORT insect multiple individual tracking method and system, belong to the field of insect multi-target tracking.It includes: design multiple different two-dimensional plane marker EStag and respectively paste on multiple target insects;Multiple individual insect video is collected to carry out frame image processing, obtain image dataset;Wherein, the multiple individual insect video includes the video recorded to the multiple target insects and the video recorded to multiple non-target insects;Using the image dataset is trained to obtain insect target detection model, the insect target detection model is used for based on the frame image of input insect individual identification;Based on the video containing the multiple target insects, based on insect target detection model, using the SORT multi-target tracking algorithm of fusion EStag is realized to the multiple target insects of multiple individual tracking.The present application method detection efficiency is high, precision is high, can realize long-term stable visual tracking to multiple insects.

Description

Technical Field

[0001] This invention belongs to the field of multi-target tracking of insects, and particularly relates to a method and system for multi-target tracking of insects based on EStag and SORT. Background Technology

[0002] Social insects are often used as model organisms to understand various life processes, from genetics and development to social behavior. When studying multiple social insects in experiments, the most fundamental challenge is collecting spatial location information of different insect individuals over time. A popular method is to artificially label the experimental insects (tags, clipped wings) for long-term visual tracking. However, this is time-consuming, labor-intensive, and difficult, and the tracking efficiency decreases significantly as the number of experimental samples increases. Deep learning-based detection and re-identification are also very popular visual tracking methods, offering high detection accuracy and speed. However, errors can easily propagate between video frames, making them unsuitable for tracking tasks with long time spans. Summary of the Invention

[0003] Based on the above analysis, the present invention aims to provide a method and system for tracking multiple insects based on EStag and SORT. The method detects the position of individual insects through a deep convolutional network, identifies EStags attached to the insects based on the detection boxes, and combines the SORT multi-target tracking algorithm to achieve visual tracking of multiple insects.

[0004] On the one hand, the present invention provides a method for tracking multiple insects based on EStag and SORT, specifically including the following steps:

[0005] Design multiple different two-dimensional planar markers (EStags) and affix them to multiple target insects;

[0006] Multiple videos of individual insects are collected and processed into frame images to obtain an image dataset; wherein, the multiple videos of individual insects include videos recorded for the multiple target insects and videos recorded for multiple non-target insects;

[0007] An insect target detection model is trained using the image dataset, and the insect target detection model is used to identify individual insects based on the input frame images;

[0008] Based on the video containing the multiple target insects, and using the insect target detection model, the SORT multi-target tracking algorithm, which integrates EStag, is used to track multiple individuals of the multiple target insects.

[0009] Furthermore, the step of tracking multiple individuals of the multiple target insects using the SORT multi-target tracking algorithm fused with EStag, based on the video containing the multiple target insects and the insect target detection model, includes:

[0010] Based on the insect target detection model, detection boxes for multiple target insects in the current frame are determined;

[0011] Based on the previous frame of the video containing the multiple target insects, the position prediction of the multiple target insects is obtained by using a linear velocity model Kalman filter to obtain the predicted bounding boxes of the multiple target insects in the current frame.

[0012] The predicted bounding box and the detection box are matched by a metric that combines IOU and EStag similarity instead of IOU in the SORT algorithm, so as to realize the tracking of multiple bodies of the multiple target insects. Here, IOU is the similarity metric between the predicted bounding box and the detection box in the SORT tracking algorithm.

[0013] Furthermore, the calculation formula for the metric that integrates IOU and EStag similarity is as follows:

[0014] S=λ1IOU+λ2S t ;

[0015] Where S is a metric that combines IOU and EStag similarity, representing the similarity between the predicted bounding box and the detected bounding box, St represents the EStag similarity between the predicted bounding box and the detected bounding box, and λ1 and λ2 are weight coefficients, with λ2 > λ1.

[0016] Furthermore, the two-dimensional planar marker EStag is a QR code; the formula for calculating the similarity of EStags in the prediction box and the detection box is:

[0017]

[0018] Here, Hamming distance represents the Hamming distance between the QR code matrices of two EStags used to calculate similarity.

[0019] Furthermore, the design method of the two-dimensional planar marker EStag includes: each QR code center is composed of a unique 4×4 16-bit 0 and 1 encoding matrix, and the matrix is ​​surrounded by a layer of white pixels and a layer of black pixels; the 16-bit encoding consists of 10 identifier bits and 6 cyclic redundancy check codes.

[0020] Furthermore, the design method for the two-dimensional planar marker EStag also includes:

[0021] The QR code is only valid in one direction;

[0022] Each of the black and white pixels should account for no less than one-quarter of the QR code;

[0023] The Hamming distance between any two valid QR codes is no less than 3.

[0024] Furthermore, the insect target detection model is a YOLOv5s-slim model obtained by pruning the YOLOv5s model. The pruning includes removing the multi-scale feature map prediction of YOLOv5s and pruning the BottleneckCSP module of YOLOv5s.

[0025] On the other hand, the present invention also provides an insect multi-person tracking system based on EStag and SORT, comprising:

[0026] The model training module is used to train the insect target detection model using an image dataset obtained from multiple collected videos of individual insects to obtain a trained insect target detection model.

[0027] The visual tracking module is used to track multiple insect targets based on a trained insect target detection model, using the SORT multi-target tracking algorithm that integrates EStags, and obtaining the position information of multiple insects in different frames.

[0028] Furthermore, the insect target detection model is a YOLOv5s-slim model obtained by pruning the YOLOv5s model. The pruning includes removing the multi-scale feature map prediction of YOLOv5s and pruning the BottleneckCSP module of YOLOv5s.

[0029] Furthermore, the similarity metric for the SORT multi-target tracking algorithm that integrates EStags is:

[0030] S = λ1IOU + λ2St;

[0031] Where S represents the similarity between the predicted bounding box and the detected bounding box, and IOU is the similarity metric between the predicted bounding box and the detected bounding box in the SORT tracking algorithm. t λ1 represents the EStag similarity between the predicted bounding box and the detected bounding box, and λ2 represents the weight coefficients, where λ2 > λ1.

[0032] The present invention can achieve at least one of the following beneficial effects:

[0033] By incorporating EStag information as a high-confidence representation into the SORT algorithm, EStag recognition is made faster based on the detection box of the target detection model. At the same time, EStag is used to effectively maintain individual identity. When EStag cannot be correctly recognized, the fusion algorithm can degenerate into the original SORT algorithm. The fusion of the two enables long-term and stable tracking of multi-target insects.

[0034] By pruning the YOLOv5 model to obtain a lighter YOLOv5s-slim model, a target detection model for insect multi-person tracking methods based on EStag and SORT was used. This significantly improved the inference speed of target detection while maintaining accuracy.

[0035] By using the two-dimensional planar marker EStag designed in this invention, the marker can be shrunk to a sufficiently small size with a low resolution and a very small pixel side length, making it suitable for small insects and easy to implement; and it can mark 150 insects at the same time, meeting the needs of multi-target insect tracking.

[0036] By providing an insect multi-person tracking system based on EStag and SORT, users can more easily observe multiple insects and obtain individual and group characteristics.

[0037] Other features and advantages of the invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained from what is particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0038] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0039] Figure 1 This is a flowchart of the method of the present invention;

[0040] Figure 2 An example of a design for EStag, a two-dimensional planar tag;

[0041] Figure 3 A schematic diagram for designing EStag, a two-dimensional planar marker;

[0042] Figure 4 Example image of a printed QR code;

[0043] Figure 5 This is a graph showing the QR code recognition rate.

[0044] Figure 6 This is a normalized statistical representation of the width and height distribution of the bounding boxes labeled in the image dataset in Example 1.

[0045] Figure 7 For YOLOv5s models;

[0046] Figure 8 YOLOv5s is a single-scale model structure;

[0047] Figure 9 The model is YOLOv5s-slim.

[0048] Figure 10 These are screenshots of bumblebee video recordings in different scenarios from Example 2;

[0049] Figure 11 The curves showing the changes in mAP@0.5 and mAP@0.5:0.95 on the validation set of Example 2;

[0050] Figure 12 This is a comparison chart of target detection results using the YOLOv5s model, the single-scale YOLOv5s model, and the YOLOv5s-slim model. Detailed Implementation

[0051] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0052] Method Implementation Examples

[0053] Example 1

[0054] A specific embodiment of the present invention discloses a method for tracking multiple insects based on EStag and SORT, which specifically includes the following steps:

[0055] Step S01: Design multiple different two-dimensional planar markers (EStags) and paste them onto multiple target insects respectively;

[0056] Step S02: Collect multiple videos of individual insects and perform frame image processing to obtain an image dataset; wherein, the multiple videos of individual insects include videos recorded for the multiple target insects and videos recorded for the multiple non-target insects;

[0057] Step S03: Train the insect target detection model using the image dataset. The insect target detection model is used to identify individual insects based on the input frame images.

[0058] Step S04: Based on the video containing the multiple target insects, and based on the insect target detection model, the SORT multi-target tracking algorithm with EStag fusion is used to track multiple individuals of the multiple target insects.

[0059] Specifically, the multi-object tracking method of the present invention is a video-based multi-object tracking method. The video-based multi-object tracking method refers to: for multiple targets that continuously enter and exit and move in the first frame to the last frame of the video, distinguishing the multiple targets from the background, and tracking the positions of the multiple targets in different frames.

[0060] Specifically, in step S01, the two-dimensional planar marker EStag is a QR code, and its design method includes: each QR code center consists of a unique 4×4 16-bit 0 and 1 encoding matrix, and the matrix is ​​surrounded by a layer of white pixels and a layer of black pixels; the 16-bit encoding consists of 10 identifier bits and 6 cyclic redundancy check bits; wherein, the first 10 identifier bits are 1 to 2 10 The QR code is a binary representation of numbers between -1 and -1, padded with zeros on the left. A unique 6-bit cyclic redundancy check (CRC) code is generated for each identifier to detect errors. The 16-bit code is transformed into a 4×4 encoding matrix in row-major order, corresponding to black and white pixels. This QR code can be scaled up as needed and pasted onto the target for recording and observation via a camera.

[0061] Specifically, to reduce the false recognition rate, the design method of the two-dimensional planar tag EStag also includes three restrictions: the QR code must be valid only in one direction; the proportion of black and white pixels must each be no less than one-quarter of the QR code; and the Hamming distance between all valid QR codes must be no less than 3 to prevent confusion between QR codes. Based on these restrictions, 150 usable QR code samples can be obtained, which is sufficient for tagging insect populations.

[0062] Figure 2 and Figure 3 A schematic diagram of the design for the EStag two-dimensional planar marker.

[0063] Specifically, in practice, in order to reduce the impact of QR code pasting on the physiological characteristics of insects, the generated labels should be as small and lightweight as possible. In this embodiment, the visibility of the QR code in the video is quantified as the pixel side length occupied by the QR code in the video. Figure 4 The image shows an example of a printed QR code. Testing showed that at a resolution of 600 dpi and a minimum pixel side length greater than 25, almost all QR codes could be correctly recognized. Figure 5 The EStag recognition rate curve illustrates that the QR code of this invention can be scaled down to a very small size, making it suitable for use with small insects.

[0064] Specifically, the image dataset in step S02 is used to train the insect target detection model.

[0065] Specifically, step S02 includes steps S021-S023.

[0066] S021. Collect videos of multiple individual insects.

[0067] Specifically, the extracted frame image sequences from the video are used to train the insect target detection model. Optionally, multiple cameras can be used to capture multiple insect videos under different lighting and scene conditions during video recording. These multiple insect videos include videos recorded for multiple target insects with EStags affixed to them within the same scene. Optionally, to improve the generalization ability of the augmentation model, videos can also be included for multiple insects in the same scene with and without EStags, and videos for multiple insects in the same scene with different colored EStags affixed.

[0068] S022. Obtain a sequence of frame images based on multiple insect videos.

[0069] Specifically, when processing frame images from multiple collected insect videos, to enable the model to learn the characteristics of insects in different states, it is necessary to ensure that the selected images have significant differences. Optionally, frame images from the video can be extracted at equal intervals to obtain an extracted frame sequence. Then, image differencing can be performed on adjacent frames in the frame sequence, and frames with high similarity among adjacent frames can be removed based on the difference results. Optionally, the image extraction can be performed in a Python environment using the open-source image processing library OpenCV.

[0070] S023. Label the frame image sequence obtained in step S022 to obtain the labeled image sequence as the image dataset for training the insect target detection model.

[0071] Specifically, the open-source annotation tool LabelImg, which supports the YOLO format, is used to annotate the frame image sequence obtained in step S022. Specifically, the insect bodies in the frame images are annotated with rectangular boxes, ensuring the boxes are flush with the edges of the insect bodies and do not cover too much of the background area. Then, the target is assigned a category (since this invention focuses on a single species of insect, only one category needs to be defined, for example, bumblebee) and saved. The annotated frame image sequence is then used as the image dataset for training the insect target detection model.

[0072] Specifically, in step S03, the insect target detection model is a YOLOv5s-slim model obtained by pruning the YOLOv5s model. The pruning includes removing the multi-scale feature map prediction of YOLOv5s and pruning the BottleneckCSP module of YOLOv5s.

[0073] Specifically, the pruning principle is described as follows: By statistically analyzing and normalizing the width and height of all labeled bounding boxes in the image dataset, such as... Figure 6As shown, the width normalization ratio ranges from 0.03 to 0.13, and the height ranges from 0.05 to 0.22. The width and height distributions are relatively concentrated and mainly represent small objects in the image. Therefore, we discard the multi-scale feature map prediction of YOLOv5s (feature maps of different scales are used to predict targets of different sizes) and only retain the single-scale model structure YOLOv5s model, which is more effective for small object detection. The number of network layers is reduced from the original 232 layers to 184 layers (including activation layers and normalization layers, etc.), and the model size is reduced from 14.8M to 10.1M. Figure 7 and Figure 8 The diagrams show the original YOLOv5s model structure before and after this pruning step, as well as the single-scale YOLOv5s model structure. As can be seen from the diagrams... Figure 8 The single-scale model structure of YOLOv5s significantly reduces the number of layers. Furthermore, since insect target detection is a single-class target detection task, which is simpler than the detection task on the COCO dataset, the Bottleneck CSP module was removed to improve computational speed, resulting in the YOLOv5s-slim model. The number of network layers was further reduced from 184 to 124 (including activation layers and normalization layers, etc.), and the model size was further reduced to 2.5M. Figure 9 The structure of the YOLOv5s-slim model obtained after further pruning is shown. Figure 8 It is simpler and lighter.

[0074] Optionally, in step S03, the image dataset is divided into a training set, a test set, and a validation set. The training set is used to train the YOLOv5s-slim model, the test set is used to test whether the trained model achieves the expected results, and the validation set is used to evaluate the performance of the model that achieves the expected results after testing. Optionally, the training set and the validation set are used together to train the YOLOv5s-slim model.

[0075] Specifically, step S04 includes steps S041-S043.

[0076] S041. Based on the insect target detection model YOLOv5s-slim, determine the detection boxes of multiple target insects in the current frame.

[0077] Specifically, each frame of a video containing multiple target insects is input into the trained insect target detection model to obtain the detection boxes of multiple target insects in the current frame of each image.

[0078] S042. Based on the previous frame of the video containing the multiple target insects, use a linear velocity model Kalman filter to predict the positions of the multiple target insects and obtain the predicted bounding boxes of the multiple target insects in the current frame.

[0079] The linear velocity model Kalman filter is a motion model used to predict the trajectory of a target, while using tracking results with high confidence to correct the prediction.

[0080] S043. Use a metric that combines IOU and EStag similarity to replace the IOU of the SORT algorithm to perform association matching on the predicted box and the detection box, so as to realize the tracking of multiple bodies of the multiple target insects. Here, IOU is the similarity metric between the predicted box and the detection box in the SORT tracking algorithm.

[0081] Specifically, the formula for calculating the metric that combines IOU and EStag similarity is as follows:

[0082] S=λ1IOU+λ2S t (1)

[0083] Where S is a metric that combines IOU and EStag similarity, representing the similarity between the predicted bounding box and the detected bounding box. t λ1 represents the EStag similarity between the predicted bounding box and the detected bounding box, and λ2 represents the weight coefficients, where λ2 > λ1.

[0084] Specifically, the formula for calculating the EStag similarity between the predicted bounding box and the detected bounding box is as follows:

[0085]

[0086] Here, Hamming distance represents the Hamming distance between the QR code matrices of two EStags used to calculate similarity.

[0087] Specifically, the principle of equation (1) is explained as follows:

[0088] In traditional SORT target tracking algorithms, the Hungarian algorithm is used to solve for the Interchange of Units (IOU). However, due to the variable motion and frequent occlusion of targets, traditional SORT algorithms suffer from a high number of target identity changes, making it impossible to achieve clear multi-target tracking. Therefore, this invention introduces EStag similarity into the metric, which can clarify the target's identity as much as possible based on EStag similarity. Specifically, a similarity metric is calculated by fusing IOU and EStag similarity; furthermore, since EStag-based target identity confirmation carries greater weight, the weighting coefficient λ2 is set to be greater than λ1.

[0089] The principle of EStag-based target identification is as follows: each insect trajectory is assigned a trajectory ID based on the QR code; the QR code in the current detection box is identified, and the Hamming distance between the matrix of the QR code and the QR code matrix corresponding to the trajectory ID is calculated: when the Hamming distance is 0, the similarity is 1, and the two are the same QR code; when the Hamming distance is 1, it indicates that the QR code identification may have an error in one bit, and the similarity is 0.5; when the Hamming distance is 2, the QR code matching confidence is very low, and the similarity is set to 0.1; when the Hamming distance is greater than or equal to 3, the QR code information can no longer be relied upon, and the tracking algorithm of Equation (1) degenerates into the SORT algorithm.

[0090] Specifically, the method for recognizing QR codes will be explained below.

[0091] Specifically, this invention employs a strategy of searching for QR codes within the detection box based on an insect target detection model, which can effectively improve the detection speed of EStags compared to the strategy of searching the entire image in typical QR code recognition algorithms.

[0092] Specifically, methods for recognizing QR codes include image preprocessing and QR code recognition.

[0093] Specifically, image preprocessing includes: converting the frame image into a grayscale image, and using the open-source OperCV library to perform adaptive binarization of the grayscale image to obtain a binary image.

[0094] Specifically, QR code recognition involves: finding a white connected region with a square boundary within the search area (detection box) of the binary image. It should be noted that due to variations in the angle and posture of insects, the boundaries of QR codes in the image are often not strictly square. Therefore, an approximate quadrilateral outline is first found, and then an approximate square boundary is found through a series of criteria such as connected region area, parallelism, and rectangularity. The four vertices of the approximate square boundary are identified as the four vertices of the white boundary of the QR code. Based on these four vertices, the black and white pixel values ​​are read from the binary image to obtain a 4×4 encoding matrix. Since a QR code is only valid in one direction, cyclic redundancy checks are performed on the matrix in four directions to obtain the corresponding 16-bit code. Optionally, if the specific QR codes used are known before tracking, prior knowledge can be used for filtering.

[0095] This embodiment discloses a multi-target insect tracking method based on EStag and SORT. A lighter YOLOv5s-slim model, obtained by pruning the YOLOv5 model, is used as the target detection model, significantly improving the inference speed of target detection while maintaining accuracy. By designing a simple two-dimensional planar marker EStag, the marker can be shrunk to a sufficiently small size with low resolution and a small pixel side length, making it suitable for small insects and easy to implement. Furthermore, it can simultaneously mark up to 150 insects, meeting the needs of multi-target insect tracking. By incorporating EStag information as high-confidence representation information into the SORT algorithm, EStag recognition is based on the detection bounding box of the target detection model, resulting in faster EStag recognition speed. Simultaneously, EStag effectively maintains individual identity. When EStag cannot be correctly recognized, the fusion algorithm can degenerate into the original SORT algorithm, achieving long-term, stable tracking of multiple insect targets.

[0096] Example 2

[0097] Another specific embodiment of the present invention discloses a multi-entity tracking method for insects based on EStag and SORT, specifically for video-based multi-entity tracking of bumblebees.

[0098] Specifically, the steps include the following:

[0099] Step S11: Design multiple different two-dimensional planar markers (EStags) and paste them onto multiple target bumblebees.

[0100] Specifically, the EStag design method is the same as step S01 in Example 1.

[0101] Specifically, the designed EStag tags are affixed to the mesothorax of bumblebees. Testing showed that the appropriate tag size for bumblebees is 2.5mm × 2.5mm, with a weight of 2.1mg. QR codes are printed on waterproof, tear-resistant A4 adhesive paper using a laser printer with a resolution of at least 600dpi. Before affixing the QR codes, each bumblebee is anesthetized in a 4°C refrigerator and removed after complete anesthesia. Each bumblebee is assigned a unique QR code, which is then affixed to its back using biosafe adhesive. Once the bumblebees have recovered their activity, the QR codes can be used for data collection.

[0102] Step S12: Collect multiple bumblebee videos and perform frame image processing to obtain an image dataset. This includes steps S121-S123.

[0103] S121. Collect multiple individual bumblebee videos.

[0104] Specifically, to increase sample diversity, three different types of cameras were used to record videos of bumblebee behavior in various scenarios. The three cameras were a Sony FDR-AX30, a Hikvision MVL-KF1228M-12MP, and a JAIGO-5000M. To enhance the robustness of the bumblebee detection model, the Sony camera used multiple resolutions (720P, 1080P, 4K) for shooting, while the Hikvision and JAI lenses recorded at a resolution of 2560×2048. The Sony lens recorded video at 25 FPS, while the Hikvision and JAI lenses recorded video at 30 FPS.

[0105] Specifically, to enhance the model's generalization ability, videos of multiple bumblebees were captured using the three types of cameras under different lighting conditions and scenarios: Scenario 1 consisted of four observation boxes under infrared light, each containing two bumblebees, with one bumblebee having a color-coded marker affixed to its mesothorax for easy identification, filmed under low-light conditions; Scenario 2 consisted of two observation boxes under normal lighting conditions, each containing two bumblebees, with green and blue markers affixed to their mesothoraxes for easy identification; Scenario 3 and Scenario 4 were filmed under normal lighting conditions, with each bumblebee bearing an EStag. Scenario 3 had a smaller field of view containing four bumblebees, resulting in a higher pixel count in the image, while Scenario 4 had a larger field of view containing nine bumblebees, resulting in a lower pixel count in the image; Scenario 5 consisted of a small swarm observation box under normal lighting conditions, containing four bumblebees, each bearing an EStag; Scenario 6 consisted of a beehive under infrared light conditions, containing ten bumblebees, each bearing an EStag. Figure 9 The video screenshots show bumblebee recordings in different scenarios.

[0106] S122. Obtain a frame image sequence based on multiple bumblebee videos.

[0107] Specifically, frames are extracted from the video at 30-second intervals. The extraction method is the same as step S022 in Embodiment 1. In this embodiment, a total of 2313 frames were ultimately extracted.

[0108] S123. The frame image sequence obtained in step S122 is labeled to obtain the labeled image sequence as the image dataset for training the bumblebee target detection model. The labeling method is the same as in S023, and all 2313 frame images in step S122 are labeled.

[0109] Step S13: Train the bumblebee target detection model using the image dataset.

[0110] Specifically, the bumblebee target detection model adopts the YOLOv5s-slim model in step S03 of Example 1.

[0111] Specifically, the image dataset is divided into training, testing, and validation sets in a 3:1:1 ratio to train the model.

[0112] For the YOLOv5s-slim model, using the hyperparameters recommended by the official YOLOv5 model, the training batch size was set to 32, and the epoch (training all data once) was set to 300. The model was validated using a validation set. When the epoch reached 50, the model began to converge, demonstrating very good performance on the validation set. Figure 10 The changes in mAP@0.5 and mAP@0.5:0.95 on the validation set are shown. mAP@0.5 is a target detection evaluation metric in existing technologies, with a default IOU threshold of 0.5; values ​​greater than 0.5 are considered targets. mAP@0.5:0.95 is the metric obtained by calculating mAP at intervals of 0.05 from 0.5 to 0.95, and then averaging these mAP values.

[0113] Furthermore, the image dataset of this embodiment is used to verify the speed comparison of the pre-pruning YOLOv5 model, the single-scale model structure YOLOv5s model, and the YOLOv5s-slim model. Specifically, during training, the k-means clustering algorithm is used to cluster the bounding boxes on the training set, and the k cluster centers are used as the model's anchors; such as Figure 11 The detection results are shown using the YOLOv5 model, the YOLOv5s model, and the YOLOv5s-slim model. Figure 11 In FPS -1 This represents the time taken to detect each frame; Precision represents the detection rate, and Recall represents the recall rate. The calculation formulas are as follows:

[0114] precision = TP / (TP+FP);

[0115] recall = TP / (TP+FN);

[0116] In this context, TP indicates that the target region was detected correctly; FP indicates that the background region was incorrectly detected as the target region; and FN indicates that the target region was incorrectly detected as the background region (missed detection).

[0117] from Figure 11 As can be seen, the single-scale YOLOv5s model improves the detection speed compared to the YOLOv5 model, but there is no significant change with the increase of Anchors; while the YOLOv5s-slim model, which is the object detection model used in this embodiment, has almost the same accuracy (indicated by Precision and Recall) compared to the YOLOv5 model and the YOLOv5s model, and the time taken to detect each frame is greatly reduced, resulting in a significant improvement in detection speed.

[0118] Step S14: Based on the video containing the multiple target bumblebees, and based on the bumblebee target detection model, use the SORT multi-target tracking algorithm with EStag fusion to achieve multi-body tracking of the multiple target bumblebees.

[0119] Specifically, step S14 is the same as step S04 in embodiment 1. When calculating the similarity between the predicted box and the detection box based on formula (1), it is necessary to identify the EStag of the predicted box and the detection box respectively.

[0120] Specifically, the recognition effect of the EStag designed by the method of the present invention is verified through the following recognition process.

[0121] Specifically, videos of multiple target bumblebees were recorded in three scenarios: Scenario A was a brightly lit glass dome with an open top and a white background, containing 10 bumblebees with EStags; Scenario B was a four-section transparent device, with 4 bumblebees with EStags in each section, for a total of 16 bumblebees; Scenario C had a light source within the field of view, with numerous ribbon cables around the device, making the scene complex. In all three scenarios, there were virtually no obstructions, and the bumblebees appeared well in the camera's field of view, with their markings clearly visible in the video.

[0122] Specifically, three hours of video were recorded for each of the three scenarios, with a resolution of 2560×5048 and a frame rate of 30. 100 frames were captured at equal intervals within each video for QR code detection. The total number of QR codes to be detected in the three scenarios was 1000, 1600, and 1000, respectively. Table 1 shows the EStag recognition performance of this invention in the three scenarios.

[0123] Table 1. EStag recognition performance of video extracted frames in three scenarios

[0124]

[0125] As shown in Table 1, the target detection model of this invention achieves high accuracy in identifying EStags using the design method of this invention, with precision rates exceeding 99% in all three scenarios.

[0126] Specifically, in step S14, a video segment with poor QR code detection and frequent bumblebee activity is selected. The video is 5 minutes long, contains 10 bumblebees within the field of view, and has a resolution of 30 FPS. The SORT multi-target tracking algorithm fused with EStag and the detection-based EStag tracking algorithm of this embodiment are used to track multiple bumblebees. The tracking results are shown in Table 2.

[0127] Table 2 shows detection-based EStag tracking (a) and SORT tracking based on fused EStags (b).

[0128]

[0129] As shown in Table 2, the SORT multi-target tracking algorithm for EStag has significantly higher correct match count and recall than detection-based EStag tracking, with a precision of no less than 98% and a FPS of [missing information]. -1 The performance of SORT multi-target tracking algorithm for EStag is significantly lower than that of EStag tracking, indicating that its overall performance is far superior to that of detection-based EStag tracking algorithm.

[0130] System Implementation Examples

[0131] This embodiment discloses an insect multi-entity tracking system based on EStag and SORT, including a model training module, a visual tracking module, and an insect feature calculation module.

[0132] The model training module is used to train the insect target detection model using an image dataset obtained from multiple collected insect videos to obtain a trained insect target detection model.

[0133] The visual tracking module is used to track multiple individual insects based on a trained insect target detection model using the SORT multi-target tracking algorithm fused with EStag, obtaining the position information of multiple individual insects in different frames. The insect target detection model is a YOLOv5s-slim model obtained by pruning the YOLOv5 model. This pruning includes removing the multi-scale feature map predictions of YOLOv5 and pruning the Bottleneck CSP module of YOLOv5. The similarity metric of the SORT multi-target tracking algorithm fused with EStag is:

[0134] S = λ1IOU + λ2St;

[0135] Where S represents the similarity between the predicted bounding box and the detected bounding box, and IOU is the similarity metric between the predicted bounding box and the detected bounding box in the SORT tracking algorithm. t λ1 represents the EStag similarity between the predicted bounding box and the detected bounding box, and λ2 represents the weight coefficients, where λ2 > λ1.

[0136] Compared to existing technologies, this embodiment provides an insect multi-person tracking system based on EStag and SORT, which allows users to more conveniently observe and obtain individual and group characteristics of multiple bumblebees. Other beneficial effects of this embodiment are the same as those of the method embodiment, and will not be described in detail here.

[0137] It should be noted that the above embodiments are based on the same inventive concept, and any parts not described repeatedly can be referenced from each other.

[0138] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for tracking multiple insects based on EStag and SORT, characterized in that, Includes the following steps: Design multiple different two-dimensional planar markers (EStags) and affix them to multiple target insects; the two-dimensional planar markers (EStags) are QR codes. Multiple videos of individual insects are collected and processed into frame images to obtain an image dataset; wherein, the multiple videos of individual insects include videos recorded for the multiple target insects and videos recorded for multiple non-target insects; An insect target detection model is trained using the image dataset, and the insect target detection model is used to identify individual insects based on the input frame images; Based on a video containing the multiple target insects, and using an insect target detection model, the SORT multi-target tracking algorithm, which integrates EStags, is employed to track multiple individuals of the target insects. This includes: determining the detection boxes of the multiple target insects in the current frame based on the insect target detection model; using a linear velocity model Kalman filter to predict the positions of the multiple target insects in the current frame based on the previous frame of the video containing the multiple target insects, obtaining predicted boxes of the multiple target insects in the current frame; and using a metric that integrates IOU and EStag similarity to replace the IOU of the SORT algorithm for association matching between the predicted boxes and the detection boxes, thereby achieving multi-individual tracking of the multiple target insects. Here, IOU is the similarity metric between the predicted boxes and the detection boxes in the SORT tracking algorithm. The calculation formula for the metric that integrates IOU and EStag similarity is: S = λ1IOU + λ2S t Where S is a metric that combines IOU and EStag similarity, representing the similarity between the predicted bounding box and the detected bounding box. t Let λ1 and λ2 represent the EStag similarity between the predicted bounding box and the detected bounding box, respectively, where λ2 > λ1. The formula for calculating the EStag similarity between the predicted bounding box and the detected bounding box is as follows: Here, Hamming distance represents the Hamming distance between the QR code matrices of two EStags used to calculate similarity.

2. The tracking method according to claim 1, characterized in that, The design method of the two-dimensional planar marker EStag includes: each QR code center is composed of a unique 4×4 16-bit 0 and 1 encoding matrix, and the matrix is ​​surrounded by a layer of white pixels and a layer of black pixels; the 16-bit encoding consists of 10 identifier bits and 6 cyclic redundancy check bits.

3. The tracking method according to claim 2, characterized in that, The design method for the two-dimensional planar marker EStag also includes: The QR code is only valid in one direction; Each of the black and white pixels should account for no less than one-quarter of the QR code; The Hamming distance between any two valid QR codes is no less than 3.

4. The tracking method according to any one of claims 1-3, characterized in that, The insect target detection model is a YOLOv5s-slim model obtained by pruning the YOLOv5s model. The pruning includes removing the multi-scale feature map prediction of YOLOv5s and pruning the BottleneckCSP module of YOLOv5s.

5. A multi-individual insect tracking system based on EStag and SORT, characterized in that, include: The model training module is used to obtain an image dataset based on multiple collected videos of individual insects. The module includes: designing multiple different two-dimensional planar markers (EStags) and affixing them to multiple target insects; the EStags are QR codes; collecting multiple videos of individual insects and performing frame image processing to obtain an image dataset; wherein the multiple videos of individual insects include videos recorded for the multiple target insects and videos recorded for multiple non-target insects; training the insect target detection model using the image dataset to obtain a trained insect target detection model; the trained insect target detection model is used to identify individual insects based on the input frame images. The visual tracking module is used to track multiple insects in multiple volumes using the SORT multi-target tracking algorithm with EStag fusion, based on a trained insect target detection model, to obtain the position information of multiple insects in different frames. This includes: determining the detection boxes of multiple insects in the current frame based on the insect target detection model; predicting the positions of multiple insects in the current frame using a linear velocity model Kalman filter based on the previous frame containing the multiple insects; and performing association matching between the predicted boxes and detection boxes using a metric that integrates IOU and EStag similarity instead of the IOU of the SORT algorithm, thereby achieving multi-volume tracking of the multiple insects. Here, IOU is the similarity metric between the predicted boxes and detection boxes in the SORT tracking algorithm. The calculation formula for the metric integrating IOU and EStag similarity is: S = λ1IOU + λ2S t Where S is a metric that combines IOU and EStag similarity, representing the similarity between the predicted bounding box and the detected bounding box. t Let λ1 and λ2 represent the EStag similarity between the predicted bounding box and the detected bounding box, respectively, where λ2 > λ1. The formula for calculating the EStag similarity between the predicted bounding box and the detected bounding box is as follows: Here, Hamming distance represents the Hamming distance between the QR code matrices of two EStags used to calculate similarity.

6. The tracking system according to claim 5, characterized in that, The insect target detection model is the YOLOv5s-slim model obtained by pruning the YOLOv5 model. The pruning includes removing the multi-scale feature map prediction of YOLOv5s and pruning the BottleneckCSP module of YOLOv5s.

Images