Wildlife multi-object tracking method based on improved motr
By improving the MOTR model, utilizing an efficient hybrid encoder and a detection query initialization strategy based on prior knowledge, combined with a multi-scale semantic alignment module, the problems of insufficient accuracy and generalization ability of MOTR in tracking small targets in wild environments are solved, and efficient and accurate deer target tracking is achieved.
Patent Information
- Application Number
- CN202411588801.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-08
- Publication Date
- 2025-10-17
- Estimated Expiration
- 2044-11-08
AI Technical Summary
Existing MOTR models have difficulty in effectively tracking small target animals in wild environments, especially when the fur of deer is similar in color to the background, and are prone to missed detection. They also require large computational complexity and have poor generalization ability.
The improved Efficient Hybrid Encoder, feature enhancement module RFEM and SE attention are used, combined with the detection query initialization strategy based on prior knowledge and the multi-scale semantic alignment module to improve the computational efficiency and target recognition ability of the model.
The accuracy of small target tracking and the generalization ability of the model in different field scenarios are improved, the phenomenon of missed detection is reduced, and the performance of deer individual ID recognition is improved.
Smart Images
Figure CN119477986B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The technical scheme relates to an animal tracking method, in particular to a wild animal multi-target tracking method based on an improved MOTR and a multi-target tracking method of deer applied to a wild field. BACKGROUND
[0002] Animal tracking is of great significance for wild animal protection and management. Applying tracking technology to record the migration path of migratory animals and behavior pattern recognition helps to deeply understand the population activity and habitat utilization, which can provide data support for monitoring endangered species and assessing the health status of the ecosystem. Researchers can also use animal tracking technology to obtain information such as population size, distribution and activity range, so as to develop protection measures in wildlife reserves, avoid overpopulation of some species and over predation of other species. In addition, animal tracking helps to understand the living habits and health status of captive wild animals in a timely manner to improve animal welfare.
[0003] Early animal tracking technology mainly uses electronic collars to carry animals, although these methods can accurately locate and track animals, but they can cause harm to animals or cause some health problems, such as causing animals to have stress reactions, etc., in addition, the collar can also interfere with the animal's breathing and eating. Deep learning has also been applied to animal multi-target tracking, and compared with traditional methods, it has achieved better performance. Most of the current deep learning methods are based on CNN, but due to the limitations of the modeling ability of long-term dependencies of convolutional neural networks, it is difficult to perform animal target tracking tasks in long-term monitoring videos. In recent years, Transformer has gradually emerged in the field of machine vision, which can understand and process long-distance semantic relationships through the introduction of multi-head attention, and can cope with multi-target tracking tasks in complex application scenarios. MOTR is one of such methods, which is a completely end-to-end multi-target tracking framework, but the method has large computational complexity, poor ability to process small targets, and cannot cope with the situation where the fur color of the animal is similar to the background. SUMMARY
[0004] The present application proposes a wild animal multi-target tracking method based on an improved MOTR, and the design idea is introduced as follows:
[0005] Small target detection capability is the key to animal target tracking in the wild. Traditional CNN networks can only extract local features of animal targets, but ignore global context information, so it is difficult to track small animal targets in complex outdoor scenes.
[0006] MOTR (Multi-Object Tracking and Segmentation) is a Transformer-based end-to-end multi-object tracking framework. The encoder of MOTR adopts a Transformer structure, which can capture global features, but the computational complexity will multiply as the length of the feature sequence increases.
[0007] In the decoder stage of MOTR, the tracking queries generated from the detection results of the previous frame are concatenated with the detection queries of the current frame and sent to the decoder for interaction. Since the content in the detection query and its corresponding reference point coordinates are randomly initialized, there is a lack of prior knowledge, which leads to the object query blindly searching for the target in the feature map. In the wild environment, the fur of animals is often very similar in color to the environment, and the MOTR model cannot distinguish animal targets from the background well, so there will be a missed detection phenomenon. In addition, the MOTR model has poor generalization ability in different wild scenes.
[0008] The cross-attention module of the decoder can be understood as a "matching + information extraction" process, where each query first matches the corresponding region and then extracts the features for subsequent prediction. However, since the self-attention and feed-forward neural network (FFN) map the queries multiple times, the queries and features are mapped to different embedding spaces, making it difficult for the queries to match their corresponding regions and extract animal features from the corresponding regions.
[0009] To solve the above problems and further improve the accuracy of animal tracking, the present application makes the following technical improvements:
[0010] 1. Combine the Efficient Hybrid Encoder, the RFEM feature enhancement module, and the SE attention to improve the feature pyramid part of the Efficient Hybrid Encoder. Replace the original encoder of MOTR with the improved encoder to achieve low computational complexity while improving small target tracking capability.
[0011] 2. To distinguish deer targets from visually similar backgrounds, the present application establishes an initialization detection query strategy based on prior knowledge, reduces the influence of complex backgrounds, reduces the missed detection phenomenon, and improves the generalization ability of the improved MOTR model in different wild scenes.
[0012] 3. To better distinguish visually similar deer individuals of different sizes and improve the deer ID recognition performance of the model, a multi-scale semantic alignment module is proposed. The semantic alignment module projects the detection query and the feature map into the same embedding space, allowing the detection query to quickly and correctly match the corresponding position in the feature map.
[0013] The specific steps of the improved MOTR-based wildlife multi-target tracking method of the present application are as follows:
[0014] Step 1, collect deer video sequences in the field scene;
[0015] Step 2, use a convolutional neural network (Resnet50) to process the collected deer video and extract multi-scale deer features of three different resolutions;
[0016] Step 3, use an improved efficient hybrid encoder to replace the original encoder to fuse the multi-scale features, output the fused multi-scale features, and construct the improved efficient hybrid encoder, which includes the following steps:
[0017] Step 3.1, feature fusion is divided into intra-scale interaction part and cross-scale interaction part, in the intra-scale interaction part, the three different resolution feature maps obtained in step 2) are denoted as {S3, S4, S5}, wherein the smallest resolution feature map S5 is input into the Transformer for attention calculation to capture global features, and the output is denoted as F5;
[0018] Step 3.2, the Fusion module of the cross-scale interaction part will splice the two feature maps that need to be fused in the channel dimension, and the execution steps of the Fusion module include:
[0019] Step 3.2.1, use two parallel 1x1 convolutions to reduce the dimension of the two input feature maps that need to be fused;
[0020] Step 3.2.2, a 1x1 convolution of the reduced dimension feature map will pass through a parallel structure composed of a 1x1 convolution and a 3x3 convolution, the outputs of the two convolutions will be added element by element, and the feature map after addition will be subjected to a squeeze-and-excitation (SE) attention mechanism for expansion and squeezing operation, to improve the precision of model feature expression and the integration ability of global information;
[0021] Step 3.2.3, another 1x1 convolution of the reduced dimension feature will be added element by element with the feature map after element-wise addition in step 3.2.2);
[0022] Step 3.3, the cross-scale interaction part is divided into top-down and bottom-up two fusion parts, in the top-down fusion part, F5 will be processed by the RFEM feature enhancement module, specifically, the F5 feature map will be processed by three different hole rates (1, 2, 3) of the hole convolution, and then be spliced in turn to obtain the feature map F6, F6 is up-sampled into a feature map with the same resolution size as S4, and the feature map is sent into the Fusion1 module together with S4 for fusion;
[0023] Step 3.4, the output feature map of the Fusion1 module in step 3.3) is processed by the RFEM module to obtain the feature map F7, and F7 is up-sampled into a feature map with the same resolution size as S3, and the feature map is sent into the Fusion2 module together with S3 for fusion;
[0024] Step 3.5, in the bottom-up fusion part, the output of the Fusion2 module in step 3.4) is down-sampled by a 3x3 GSConv convolution into a feature map with the same resolution size as F7 in step 3.4), and the feature map is sent into the Fusion3 module together with the feature map F7 for fusion;
[0025] Step 3.6, the output of the Fusion3 module in step 3.5) is down-sampled by a 3x3 GSConv convolution into a feature map with the same resolution size as F6 in step 3.3), and the feature map is sent into the Fusion4 module together with the feature map F6 for fusion;
[0026] Step 3.7, the fused features of the Fusion2, Fusion3 and Fusion4 are output as the final multi-scale features;
[0027] Step 4, the multi-scale features fused in step 3.7) are sent into the constructed detection query initialization module based on prior knowledge, and a detection query with preliminary position information of the deer is output, and the construction of the object query initialization module based on prior knowledge includes:
[0028] Step 4.1, generating a preliminary proposal box on the fused multi-scale feature map output in step 3.7);
[0029] Step 4.2, each feature point in the generated proposal box is taken as a separate detection query, which is processed by an independent detection head, and the detection head outputs the offset of the deer position coordinates;
[0030] Step 4.3, adding the offset of the position coordinates to the initial proposal box coordinates to obtain the final position coordinates, and finally selecting multiple position coordinates with high scores according to the classification scores as the reference position coordinates of the decoder part;
[0031] Step 4.4, the reference points are processed by position embedding, followed by a fully connected layer and an LN layer to generate a new detection query;
[0032] Step 4.6, a set of learnable vectors weights are generated by a linear layer, and weights are multiplied with the generated detection query as the final detection query. For other video frames except the first frame, the tracking query does not participate in initialization, and the tracking query generated by QIM in the last frame is still used;
[0033] Step 5, the detection query generated in step 4 is input into the decoder together with the multi-scale features fused in step 3.7) to generate the prediction result of the current frame. The execution steps of the decoder are:
[0034] 5.1) The object query is first calculated by self-attention. If it is the first frame, the object query is only the detection query. If it is the subsequent frame, the object query includes the detection query and the tracking query;
[0035] 5.2) After self-attention, a semantic alignment module is added to more accurately match the corresponding positions and introduce the multi-scale features obtained in step 3.7) to better distinguish different individuals of similar visual appearance. The steps to build the multi-scale semantic alignment module include:
[0036] Step 5.2.1, the reference point coordinates generated in step 4.3) are used to extract ROI features on different size feature maps and align and fuse them;
[0037] Step 5.2.2, the fused features are again down-sampled by the interpolate operation to sample them into 300 features with a dimension of 256 and a shape of 1x1;
[0038] Step 5.2.3, the object query calculated by self-attention in step 5.1) is processed by a linear layer and normalized, and multiplied with the features sampled in step 5.2.2);
[0039] 5.3) The object query calculated by self-attention in step 5.1) is processed by the semantic alignment module in step 5.2), and the processing result is cross-attention calculated with the feature map fused in step 3.6).
[0040] Step 6, the prediction result of the current frame is processed by the QIM module to screen new-born targets, already tracked targets and extinct targets, and the new-born targets and already tracked targets are output as the tracking query of the next frame;
[0041] Step 7, repeat the above steps for each subsequent frame, except that for each subsequent frame, the detection query output by the prior knowledge-based query initialization module is spliced with the tracking query generated by the QIM module of the previous frame, and the fused multi-scale features are input into the decoder for interaction to generate the prediction result of each frame.
[0042] The beneficial effects of the present application include:
[0043] (1) A new encoder is constructed by fusing Efficient Hybrid Encoder, feature enhancement module RFEM and SE attention module, and replacing the original encoder of MOTR, so as to effectively track the deer in the video shot in the wild environment.
[0044] (2) A priori knowledge-based query initialization strategy is proposed to improve the tracking accuracy and generalization ability of the model in the wild environment similar to the appearance of the deer in the deer video, or in different environments with large background differences.
[0045] (3) A multi-scale semantic alignment module is constructed to reduce the matching difficulty of the query on the feature map, avoid the transformation of the wrong deer individual ID, and improve the ID recognition performance of the model. BRIEF DESCRIPTION OF DRAWINGS
[0046] Figure 1 The schematic diagram of the improved MOTR-based deer tracking method of the present application.
[0047] Figure 1 In the formula, CNN is the backbone network ResNet50, EHE represents the improved Efficient Hybrid Encoder, PKI refers to the prior knowledge-based detection query initialization module, the multi-scale semantic alignment module is included in the decoder Dec, Qd is the detection query used for detecting new targets, Qt is the tracking query, QIM is used to manage the appearance of new targets and the disappearance of tracked targets, and Y represents the prediction result of the network, which is Collective average loss with label G.
[0048] Figure 2 The schematic diagram of the new encoder constructed by the present application, wherein the right part of the figure is the structure of the Fusion module.
[0049] Figure 3 The schematic diagram of the prior knowledge-based detection query initialization strategy proposed by the present application.
[0050] Figure 3In the formula, weights represents a vector, (x, y, w, h) is a preset proposal box coordinate, (Delta x, Delta y, Delta w, Delta h) is a coordinate offset output by the Box head, and Top 300 represents that 300 proposal box coordinates with the highest scores are selected as reference coordinates of the decoder part according to the scores of the classification head.
[0051] Figure 4 A schematic diagram of the multiscale semantic alignment module constructed in the application. The left drawing is a decoder structure, and the right drawing is a structure diagram of the multiscale semantic alignment module.
[0052] Figure 5 A comparison chart of test results of the application of the animal tracking method of the application to deer video tracking in a small target running blurred image.
[0053] Figure 6 A comparison chart of test results of the application of the animal tracking method of the application to deer video tracking in light change.
[0054] Figure 7 A comparison chart of test results of the application of the animal tracking method of the application to deer video tracking in different scenes.
[0055] Figure 8 A result chart of the application of the animal tracking method of the application to deer motion trajectory tracking.
[0056] Figure 9 A result chart of the application of the animal tracking method of the application to deer individual counting.
[0057] Figure 10 A result chart of the application of the animal tracking method of the application to deer behavior recognition.
[0058] Figures 5 to 7 For experiments, Figures 8 to 10 For extended applications of the tracking method. DETAILED DESCRIPTION
[0059] The wild animal multi-target tracking method based on the improved MOTR of the application first collects deer videos; then uses a deer tracking algorithm for tracking, and the steps are as follows:
[0060] 1) A convolutional neural network (Resnet50) is used to process the collected videos, and multiscale deer features of three different resolutions are extracted;
[0061] 2) An improved efficient hybrid encoder is used to fuse the multiscale features, and output fused multiscale features;
[0062] 3) The fused multiscale features are input into a detection query initialization module based on prior knowledge, and a detection query with preliminary deer position information is output.
[0063] 4) The detection query is input into the decoder together with the fused multi-scale features, and in the decoder, the detection query is first subjected to self-attention calculation, then passes through the improved multi-scale semantic alignment module, and then interacts with the fused multi-scale features through cross-attention, and finally outputs the prediction result of the current video frame;
[0064] 5) The prediction result of the current frame is directly output as the final tracking result; at the same time, the prediction result of the current frame also passes through the QIM module to screen new targets, tracked targets and extinct targets, and the new targets and tracked targets are output as the tracking query of the next frame;
[0065] 6) The above operation is repeated for each subsequent frame, except that for the first frame, the detection query output by the query initialization module based on prior knowledge is spliced with the tracking query generated by the QIM module of the previous frame, and the fused multi-scale features are input into the encoder for interaction.
[0066] In the present application: the improved efficient hybrid encoder replaces the original encoder of MOTR, achieving low computational complexity while improving small target tracking capability; the initialization detection query strategy based on prior knowledge reduces the influence of complex background, reduces the missed detection phenomenon, and improves the generalization ability of the model in different environments; the multi-scale semantic alignment module better distinguishes different sizes of individuals that are similar in vision, and improves the ID recognition performance of the model.
[0067] The technical solutions of the present application are further described below with reference to the accompanying drawings.
[0068] The present application provides Figure 1 As shown in the improved MOTR-based wild animal multi-target tracking method, a new encoder that fuses the efficient hybrid encoder, the RFEM feature enhancement module, and the SE attention, the query initialization module based on prior knowledge, and the multi-scale semantic alignment module are designed, and the specific steps include:
[0069] Step 1, collect deer video sequences in the wild scene;
[0070] Step 2, use a convolutional neural network (Resnet50) to process the collected deer video to extract multi-scale deer features of three different resolutions;
[0071] Step 3, replace the original encoder with the improved Efficient Hybrid Encoder to fuse the aforementioned multi-scale features, output the fused multi-scale features, and construct the improved Efficient Hybrid Encoder, which includes the following steps:
[0072] Step 3.1, feature fusion is divided into intra-scale interaction part and cross-scale interaction part, in the intra-scale interaction part, the three different resolution feature maps obtained in step 2) are denoted as {S3, S4, S5}, where the feature map S5 with the smallest resolution is input into the Transformer for attention calculation to capture global features, and the output is denoted as F5;
[0073] Step 3.2, the Fusion module of the cross-scale interaction part will splice the two feature maps that need to be fused in the channel dimension, and the execution steps of the Fusion module include:
[0074] Step 3.2.1, use two parallel 1x1 convolutions to reduce the dimension of the two input feature maps that need to be fused;
[0075] Step 3.2.2, a 1x1 convolution is used to reduce the dimension of the feature map, which will pass through a parallel structure composed of a 1x1 convolution and a 3x3 convolution, the outputs of the two convolutions will be added element by element, and the feature map after addition will be subjected to a Squeeze-and-Excitation (SE) attention mechanism for expansion and squeezing operation, to improve the precision of feature expression and the integration ability of global information;
[0076] Step 3.2.3, the other 1x1 convolution will be added element by element with the feature map after element-wise addition in step 3.2.2);
[0077] Step 3.3, the cross-scale interaction part is divided into top-down and bottom-up fusion parts, in the top-down fusion part, F5 will be processed by the RFEM feature enhancement module, specifically, F5 feature map will be processed by three different hole rates (1, 2, 3) of the hole convolution, and will be spliced in turn to obtain the feature map F6, F6 will be up-sampled into a feature map with the same resolution size as S4, and the feature map will be input into the Fusion1 module together with S4 for fusion;
[0078] Step 3.4, the output feature map of the Fusion1 module in step 3.3) is processed by the RFEM module to obtain the feature map F7, and F7 is up-sampled into a feature map with the same resolution size as S3, and the feature map is input into the Fusion2 module together with S3 for fusion;
[0079] Step 3.5, in the self-downward fusion part, the output of the Fusion2 module in step 3.4) is down-sampled by a 3x3 GSConv convolution into a feature map with the same resolution size as F7 in step 3.4), which is fused with the feature map F7 into the Fusion3 module;
[0080] Step 3.6, the output of the Fusion3 module in step 3.5) is down-sampled by a 3x3 GSConv convolution into a feature map with the same resolution size as F6 in step 3.3), which is fused with the feature map F6 into the Fusion4 module;
[0081] Step 3.7, the features fused by the Fusion2, Fusion3 and Fusion4 are output as the final multi-scale features;
[0082] Step 4, the multi-scale features fused in step 3.7) are input into the constructed prior knowledge based detection query initialization module, and a detection query with preliminary position information of the deer is output, and the construction of the prior knowledge based object query initialization module comprises the following steps:
[0083] Step 4.1, preliminary proposal boxes are generated on the multi-scale feature map output in step 3.7);
[0084] Step 4.2, each feature point in the generated proposal box is taken as a single detection query, which is processed by an independent detection head, and the detection head outputs the offset of the position coordinates of the deer;
[0085] Step 4.3, the offset of the position coordinates is added to the coordinates of the initial proposal box to obtain the final position coordinates, and finally multiple position coordinates with high scores are selected according to the classification scores as the reference position coordinates of the decoder part;
[0086] Step 4.4, the reference point is processed by a position embedding, a fully connected layer and an LN layer to generate a new detection query;
[0087] Step 4.6, a set of learnable vectors weights are generated by a linear layer, the weights are multiplied by the generated detection query to obtain the final detection query, and for other video frames except the first frame, the tracking query does not participate in initialization, and the tracking query generated by the QIM in the last frame is still used;
[0088] Step 5, the detection query generated in step 4 is input into the decoder together with the multi-scale features fused in step 3.7) to interact, and a prediction result of the current frame is generated, and the execution steps of the decoder are as follows:
[0089] 5.1) The object query is first subjected to self-attention calculation; if it is the first frame, the object query is only the detection query; if it is the subsequent frame, the object query includes the detection query and the tracking query;
[0090] 5.2) A semantic alignment module is added after self-attention, which more accurately matches the corresponding positions and introduces the multi-scale features obtained in step 3.7) to better distinguish different deer individuals that are visually similar, and the steps for constructing the multi-scale semantic alignment module include:
[0091] Step 5.2.1, using the reference point coordinates generated in step 4.3) to extract ROI features on different size feature maps and align and fuse them;
[0092] Step 5.2.2, the fused features are again subjected to down-sampling and interpolation operation, and are sampled into 300 features with 256 dimensions and 1x1 shape;
[0093] Step 5.2.3, the object query subjected to self-attention calculation in step 5.1) is subjected to a linear layer and normalized, and is multiplied by the features sampled in step 5.2.2);
[0094] 5.3) The object query subjected to self-attention calculation in step 5.1) is subjected to the semantic alignment module in step 5.2), and the processing result is subjected to cross-attention calculation with the fused feature map in step 3.6).
[0095] Step 6, the prediction result of the current frame is subjected to the QIM module to screen new targets, tracked targets and extinct targets, and the new targets and tracked targets are output as the tracking query of the next frame;
[0096] Step 7, each subsequent frame repeats the above steps, except that each subsequent frame, except the first frame, splices the detection query output by the query initialization module based on prior knowledge in the current frame with the tracking query generated by the QIM module in the previous frame, and inputs them together with the fused multi-scale features into the decoder for interaction to generate the prediction result of each frame.
[0097] In order to verify the improvement strategy of the present application, including the encoder submodule (EHE), the query initialization submodule (PKI) and the multi-scale semantic alignment submodule (MSFSA), the present application takes deer video as an example for experiment.
[0098] The present application performs ablation experiments on a deer dataset to prove the degree of improvement of each point to the model. The experimental results are shown in Table 1, after using the improved encoder EHE, MOTA and IDF1 are improved by 1% and 1.2% respectively, the value of IDS is reduced by 20%, and the reasoning speed is 4.6 times of the original. While ensuring the accuracy, the real-time performance of the model is improved. Since PKI is only used in the detection query stage, the detection performance evaluation index MOTA is improved by 1.5%, while the IDF1 index biased towards identity recognition is only improved by 0.4%, and the IDS is also reduced. In addition, after introducing MSFSA, MOTA and IDF1 are improved by 1.9% and 2.2% respectively, and the value of IDS is reduced by 22%. The experimental results show that the improved model has a greater improvement in MOTA, IDF1 and IDS compared with the original model, which improves the accuracy of target recognition and tracking.
[0099] Table 1. Ablation experiment results (EHE represents the improved Efficient Hybrid Encoder, PKI represents the prior knowledge-based query initialization module, and MSFSA represents the multi-scale semantic alignment module)
[0100] EHE PKI MSFSA MOTA (%) IDF1 (%) IDS 79.4 84.5 30 √ 80.4 85.7 24 √ √ 81.9 86.1 22 √ √ √ 83.8 88.3 17
[0101] In order to prove the advancement of the method proposed in the present application, the following compares the proposed deer tracking method with common animal tracking networks on the constructed video dataset. The comparison includes multi-target tracking accuracy (MOTA), IDF1 score, and model complexity. The experimental results are shown in Table 2:
[0102] Table 2. Comparison of experimental results
[0103]
[0104] From the comparison of the experimental results in Table 2, it can be seen that the deer tracking method proposed in the present application has obvious advantages in tracking accuracy and parameter quantity. At the same time, although the GFLOPS of TrackFormer and TransTrack are slightly lower than the present application, the tracking accuracy of the present application is significantly improved compared with the above two methods.
[0105] In order to verify the generalization ability and robustness of the present application in complex scenes, four situations are tested: blurred images caused by shaking of the shooting device, small running animal targets, changes in light, and scenes not appearing in the training set. The results are visualized and compared with two optimal algorithms ByteTrack and TransTrack.
[0106] Figure 5The tracking results of small targets running in the blurred image are visualized. The model of the application detects all targets at frame 202, while ByteTrack misses one. In addition, at frame 256, all three methods miss the ID of a deer, but the model of the application successfully matches the ID of it again at frame 257, while the other two methods still miss it. The experimental results show that the algorithm has strong robustness for small target detection.
[0107] Figure 6 The tracking result comparison chart of the application under light changes is shown. The image contains individuals of sika deer in different postures. At frame 113, both of the two comparison algorithms have repeated tracking on the individual of sika deer with ID 12, and the individual lying in the lower right corner is not all tracked. At frame 224, the visual effect of the deer fur changes due to light changes, causing the loss of the previously tracked target. At frame 258, the light returns to the previous situation, but the missed individual is still not tracked. Thanks to the multi-scale semantic module proposed in this paper, the method of the application successfully tracks the individuals of sika deer in different postures and is not affected by light changes.
[0108] Figure 7 The generalization ability of the application in different scenes is shown. Two scenes not appearing in the training set are selected, as shown in Figure 7 Although all three algorithms have false positives and false negatives in the snow scene, overall, the method of the application is better than the other models. At frame 138 in the fence scene, TranTrack has a false detection box, and one of the three sika deer on the left side of the image is missed by ByteTrack due to occlusion. At frame 153, a small individual in the middle of the image is occluded, causing both comparison algorithms to lose the target. However, the method of the application improves the generalization ability in different scenes due to the query initialization method based on prior knowledge, and the above situations do not occur.
[0109] Figure 8 The application of the application to the tracking of the movement trajectory of deer is shown. The complete tracking path is drawn for all deer individuals using the target tracking positioning box output by the model. It can be seen that each sika deer is assigned a unique movement trajectory, and the tracking path is relatively complete without interruption. This function can help understand the activity range of deer and provide important data support for deer activity monitoring and planning and design of protected areas.
[0110] Figure 9The application of the present application in counting is demonstrated, and the number of deer tracking positioning boxes in each image frame of the monitoring video can be automatically counted, that is, the number of deer at any moment is counted, and the lost individual is found in time through the last disappearing point of each deer track.
[0111] Figure 10 The application of the present application in behavior detection is demonstrated, and a behavior monitoring classification head can be added in the proposed model, or a classification network can be added subsequently to intercept the sika deer target of the tracking positioning box, so as to realize the behavior recognition of the deer individual. This can facilitate researchers to deeply study the living habits of sika deer, and timely find the abnormal behavior and significance related health condition of the deer, for example, the animal is in a lying position for a long time, and has no feeding behavior, which is an important signal of the animal being ill, and corresponding treatment measures should be immediately developed.
Claims
1. A wildlife multi-target tracking method based on improved MOTR, characterized by the following steps include: 1) Collect video sequences of deer in the wild; 2) The collected deer videos were processed using the convolutional neural network Resnet50 to extract multi-scale features at three different resolutions. The feature maps are denoted as {S3, S4, S5} in descending order of resolution. 3) The original encoder of the multi-target tracking framework MOTR is replaced with an improved efficient hybrid encoder, and the multi-scale features extracted in step 2) are fused to output the fused multi-scale features; 4) The fused multi-scale features from step 3 are fed into the detection query initialization module, which outputs a detection query with prior knowledge of the deer's initial location; 5) The detection query obtained in step 4) and the fused multi-scale features obtained in step 3) are input into the decoder for interaction to generate the prediction result of the current frame; 6) The prediction result of the current frame passes through the QIM module to generate the tracking query for the next frame; The QIM module is used to manage the appearance of new targets and the disappearance of tracked targets, thereby filtering new targets, tracked targets, and disappeared targets, and outputting new targets and tracked targets as tracking queries for the next frame; 7) The above steps are repeated for each subsequent frame. However, for each subsequent frame, except for the first one, the detection query output by the detection query initialization module based on prior knowledge is concatenated with the tracking query generated by the previous frame through QIM. This is then fed into the decoder along with the fused multi-scale features for interaction to generate the prediction result for each frame. In step 3), the improved efficient hybrid encoder combines the Efficient Hybrid Encoder with the RFEM module and SE attention, improving the feature pyramid part of the Efficient Hybrid Encoder, including: Introducing the RFEM module into the Efficient Hybrid Encoder; Feature fusion is divided into intra-scale interaction and cross-scale interaction. The cross-scale interaction is further divided into top-down fusion and bottom-up fusion. SE attention is introduced in the fusion module of the cross-scale interaction part. The fusion module is used to fuse the two input feature maps that need to be fused in the cross-scale interaction part.
2. The method for tracking multiple targets of wild animals based on improved MOTR according to claim 1, wherein In step 3), the fusion module in the improved efficient hybrid encoder is used to fuse the two input feature maps that need to be fused in the cross-scale interaction part; the structure of the fusion module is: a) Concatenate the two input feature maps in the channel dimension and use two parallel The convolution reduces the dimension of the concatenated feature map; b) A feature map reduced by a 1x1 convolution passes through a parallel structure consisting of a 1x1 convolution and a 3x3 convolution. The outputs of the two convolutions in the parallel structure are added element-by-element to fuse the feature information from different convolutional layers. SE attention is then used to squeeze and expand the features output by the parallel structure. c) The features after another 1x1 convolution are added element-by-element to the features after element-by-element addition in step b).
3. The method for tracking multiple targets of wild animals based on improved MOTR according to claim 1 or 2, wherein In step 3), the specific steps of feature fusion are: 3.1) In the intra-scale interaction part, the feature map S5 with the minimum resolution obtained in step 2) is fed into the Transformer, and the Transformer Encoder is used to capture global features. The output feature map is denoted as F5. 3.2) In the top-down fusion part, F5 is processed by the RFEM module to obtain the feature map F6. F6 is then upsampled to a feature map with the same feature resolution as S4. This feature map is then sent to the Fusion1 module for fusion together with S4. 3.3) The output feature map of the Fusion1 module in step 3.2) is processed by the RFEM module to obtain the feature map F7. F7 is then upsampled to the same resolution as the S3 feature map. This feature map and S3 are then fed into the Fusion2 module for fusion. 3.4) In the bottom-up fusion part, the output of the Fusion2 module in step 3.3) is downsampled by a 3x3 GSConv convolution into a feature map with the same resolution as F7 in step 3.3) and this feature map is fed into the Fusion3 module together with the feature map F7 for fusion. 3.5) The output of the Fusion3 module in step 3.4) is downsampled by a 3x3 GSConv convolution into a feature map with the same resolution as F6 in step 3.2). This feature map and feature map F6 are then fed into the Fusion4 module for fusion. 3.6) The fused features of Fusion2, Fusion3 and Fusion4 are output as the final multi-scale features.
4. The method for tracking multiple targets of wild animals based on improved MOTR according to claim 1, wherein In step 4), the detection query initialization module uses the Two-Stage variant method in Deformable-DETR to initialize the query vector.
5. The method for tracking multiple targets of wild animals based on improved MOTR according to claim 4, characterized in that In step 4), the detection query initialization module is a detection query initialization module based on prior knowledge, which uses a two-stage method to initialize the detection query; The steps to build a detection query initialization module based on prior knowledge include: 4.1) Generate preliminary proposal boxes based on the fused multi-scale features output in step 3.6); 4.2) Each feature point in the generated proposal box is treated as a separate detection query and passed through an independent detection head, which outputs the offset of the deer's position coordinates; 4.3) Add the offset of the position coordinate to the initial proposal coordinate to obtain the new position coordinate. Finally, select multiple position coordinates with high scores based on the classification score as the reference position coordinates of the decoder part; 4.4) The reference position coordinates are processed through position embedding and then connected to a fully connected layer and LN layer to generate a new detection query; 4.5) Use the linear layer to generate a set of learnable vector weights, which are multiplied by the detection query generated in step 4.4) as the final detection query. For video frames other than the first frame, the tracking query does not participate in the initialization and still uses the tracking query generated by QIM in the previous frame in step 6).
6. The method for tracking multiple targets of wild animals based on improved MOTR according to claim 5, wherein In step 5), the decoder performs the following steps: 5.1) Object query first performs self-attention calculation; If it is the first frame, the object query is only a detection query; if it is a subsequent frame, the object query includes a detection query and a tracking query; 5.2) A semantic alignment module is added after the self-attention to more accurately match the corresponding positions and introduce the multi-scale features obtained in step 3) to better distinguish different visually similar deer individuals; 5.3) The object query calculated by self-attention in step 5.1) will be processed by the semantic alignment module in step 5.2), and the processing result will be cross-attention calculated with the feature map fused in step 3).
7. The method for tracking multiple targets of wild animals based on improved MOTR according to claim 6, wherein The multi-scale semantic alignment module in step 5.2) includes the following steps: 5.2.1) Using the reference position coordinates generated in step 4.3), extract ROI features on feature maps of different sizes and fuse them; 5.2.2) Perform interpolation on the fused features to reduce them to 300 feature points with a dimension of 256 and a shape of 1x1. 5.2.3) Pass the object query calculated by self-attention in step 5.1) through a linear layer and normalize it, and multiply it with the features sampled in step 5.2.2).