Wild animal multi-target tracking method based on improved MOTR

Abstract

The invention discloses a wild animal multi-target tracking method based on an improved MOTR. The method comprises the following steps: 1) extracting multi-scale deer features with different resolutions from a collected deer video by using a convolutional neural network; 2) carrying out feature fusion on the multi-scale features by using an improved efficient hybrid encoder, and outputting the fused multi-scale features; 3) inputting the fused multi-scale features into a query initialization module based on priori knowledge, and outputting a detection query with initial position information of the deer; and 4) detecting, querying and inputting the fused multi-scale features into a decoder, and finally outputting a prediction result of the current video frame. According to the invention, the improved encoder is adopted to replace the original encoder of the MOTR, so that the small target tracking capability is improved while the low calculation amount is realized; and a detection query strategy is initialized, so that the influence of a complex background is reduced, the missing detection phenomenon is reduced, and the generalization ability of the model applied to different field scenes is improved.

Description

Technical Field

[0001] The present technical scheme relates to an animal tracking method, in particular to a wild animal multi-target tracking method based on an improved MOTR, and a multi-target tracking method for deer in a wild scene. Background Art

[0002] Animal tracking is of great significance to wildlife conservation and management. The application of tracking technology to record the migration paths and behavioral pattern recognition of migratory animals can help to gain a deeper understanding of population activities and habitat utilization, which can provide data support for monitoring endangered species and assessing the health of ecosystems. Researchers can also use animal tracking technology to obtain information such as population size, distribution and range of activities in order to formulate protection measures in wildlife reserves and avoid over-breeding of certain species and over-predation of other species. In addition, animal tracking helps to timely understand the living habits and health status of captive wild animals in order to improve animal welfare.

[0003] Early animal tracking technology mainly relied on putting electronic collars on animals. Although these methods can accurately locate and track animals, they can cause harm to animals or cause some health problems, such as causing stress reactions in animals. In addition, collars can also interfere with animals' breathing and eating. Deep learning has also been applied to animal multi-target tracking, and has achieved better performance than traditional methods. Most of the current deep learning methods are based on CNN, but due to the limited modeling ability of convolutional neural networks for long-term dependencies, they are difficult to handle animal target tracking tasks in long-term surveillance videos. In recent years, Transformer has gradually emerged in the field of machine vision. By introducing multi-head attention, Transformer understands and processes long-distance semantic relationships and can handle multi-target tracking tasks in complex application scenarios. MOTR is one of the representatives of this type of method. It is a fully end-to-end multi-target tracking framework, but this method has a large amount of computation, poor ability to handle small targets, and cannot handle situations where the fur color of animals is similar to the background. Summary of the invention

[0004] The present invention proposes a wild animal multi-target tracking method based on improved MOTR. The design idea is introduced as follows:

[0005] Small target detection capability is the key to tracking animal targets in the wild. Traditional CNN networks can only extract local features of animal targets while ignoring global contextual information. Therefore, it is difficult to track small animal targets in complex scenes in the wild.

[0006] MOTR (Multi-Object Tracking and Segmentation) is an end-to-end multi-object tracking framework based on Transformer. The encoder of MOTR adopts the Transformer structure. Although it can capture global features, the amount of calculation will double as the length of the feature sequence increases.

[0007] In the decoder stage of MOTR, the tracking query generated by the detection result of the previous frame is concatenated with the detection query of the current frame and sent to the decoder for interaction. Since the content in the detection query and its corresponding reference point coordinates are randomly initialized and lack prior knowledge, the object query blindly searches for the target in the feature map. In the wild environment, the fur of animals often has a very similar color to the environment. The MOTR model cannot distinguish the animal target from the background well, so missed detections may occur. In addition, the generalization ability of the MOTR model in different wild scenes is poor.

[0008] The cross-attention module of the decoder can be understood as a "matching + information extraction" process. Each query is first matched with the corresponding area, and then features are extracted for subsequent predictions. However, since the self-attention and feedforward neural network FFN map the query multiple times, the query and features are mapped to different embedding spaces, which makes it difficult to match the query to its corresponding area and unable to extract the animal's features from the corresponding area.

[0009] In order to solve the above problems and further improve the accuracy of animal tracking, the present invention makes the following technical improvements:

[0010] 1. The efficient hybrid encoder, feature enhancement module RFEM, and SE attention are combined to improve the feature pyramid part of the efficient hybrid encoder. The improved encoder replaces the original encoder of MOTR to achieve low computational complexity while improving the ability to track small targets.

[0011] 2. In order to distinguish deer targets from visually similar backgrounds, the present invention establishes an initialization detection query strategy based on prior knowledge, which reduces the influence of complex backgrounds, reduces missed detections, and improves the generalization ability of the improved MOTR model applied to different field scenes;

[0012] 3. In order to better distinguish visually similar deer individuals of different sizes and improve the deer ID recognition performance of the model, a multi-scale semantic alignment module is proposed. The semantic alignment module projects the detection query and the feature map into the same embedding space, so that the detection query can quickly and correctly match the corresponding position in the feature map.

[0013] The specific steps of the wild animal multi-target tracking method based on improved MOTR of the present invention are as follows:

[0014] Step 1, collecting deer video sequences in wild scenes;

[0015] Step 2: Use a convolutional neural network (Resnet50) to process the collected deer videos and extract multi-scale deer features at three different resolutions;

[0016] Step 3: Use an improved efficient hybrid encoder to replace the original encoder to perform feature fusion on the aforementioned multi-scale features, and output the fused multi-scale features. The steps of constructing the improved efficient hybrid encoder include:

[0017] Step 3.1: feature fusion is divided into the intrascale interaction part and the cross-scale interaction part. In the intrascale interaction part, the three feature maps with different resolutions obtained in step 2) are denoted as {S3, S4, S5}, among which the feature map S5 with the smallest resolution is sent to the Transformer for attention calculation to capture the global features, and the output is denoted as F5.

[0018] Step 3.2: The Fusion module of the cross-scale interaction part will splice the two feature maps that need to be fused in the channel dimension. The execution steps of the Fusion module include:

[0019] Step 3.2.1, use two parallel 1x1 convolutions to reduce the dimension of the two input feature maps to be fused respectively;

[0020] Step 3.2.2, a 1x1 convolution convolution after dimensionality reduction feature map will pass through a parallel structure consisting of a 1x1 convolution and a 3x3 convolution. The outputs of the two convolutions will be added element by element. The added feature map will be expanded and squeezed through the Squeeze-and-Excitation (SE) attention mechanism to improve the refinement of the model feature expression and the ability to integrate global information;

[0021] Step 3.2.3, another 1x1 convolution. The reduced feature will be added element by element to the feature map added element by element in step 3.2.2);

[0022] Step 3.3, the cross-scale interaction part is divided into two fusion parts: top-down and bottom-up. In the top-down fusion part, F5 will be processed by the RFEM feature enhancement module. Specifically, the F5 feature map will be convolved with three different dilation rates (1, 2, 3) and concatenated in sequence to obtain the feature map F6. F6 is then upsampled to a feature map with the same feature resolution as S4. This feature map is sent to the Fusion1 module for fusion together with S4.

[0023] Step 3.4: The output feature map of the Fusion1 module in step 3.3) is processed by the RFEM module to obtain the feature map F7, which is then upsampled to a feature map with the same resolution as the S3 feature map. This feature map enters the Fusion2 module together with S3 for fusion;

[0024] Step 3.5, in the bottom-up fusion part, the output of the Fusion2 module in step 3.4) is downsampled by a 3x3 GSConv convolution into a feature map with the same resolution as F7 in step 3.4), and this feature map is fused together with the feature map F7 in the Fusion3 module;

[0025] Step 3.6: The output of the Fusion3 module in step 3.5) is downsampled by a 3x3 GSConv convolution into a feature map with the same resolution as F6 in step 3.3), and this feature map is fused with the feature map F6 in the Fusion4 module;

[0026] Step 3.7, the fused features of Fusion2, Fusion3 and Fusion4 will be output as the final multi-scale features;

[0027] Step 4, step 3.7) The fused multi-scale features will be sent to the detection query initialization module based on prior knowledge, and the detection query with the preliminary location information of the deer will be output. The steps of constructing the object query initialization module based on prior knowledge include:

[0028] Step 4.1, generating a preliminary proposal box on the fused multi-scale feature map output in step 3.7);

[0029] In step 4.2, each feature point in the generated proposal box is treated as a separate detection query and passed through an independent detection head, which outputs the offset of the deer position coordinates;

[0030] Step 4.3, add the offset of the position coordinate to the initial suggestion box coordinate as the final position coordinate, and finally select multiple position coordinates with high scores according to the classification score as the reference position coordinates of the decoder part;

[0031] Step 4.4: The reference point is processed by position embedding and then connected to a fully connected layer and LN layer to generate a new detection query;

[0032] Step 4.6, use the linear layer to generate a set of learnable vector weights, which are multiplied by the generated detection query as the final detection query. For other video frames except the first frame, the tracking query does not participate in the initialization, and the tracking query generated by QIM in the previous frame is still used;

[0033] Step 5: The detection query generated in step 4 is input into the decoder together with the multi-scale features fused in step 3.7) to interact and generate the prediction result of the current frame. The execution steps of the decoder are:

[0034] 5.1) Object query first performs self-attention calculation; if it is the first frame, the object query is only a detection query; if it is a subsequent frame, the object query includes a detection query and a tracking query;

[0035] 5.2) A semantic alignment module is added after the self-attention to more accurately match the corresponding positions and introduce the multi-scale features obtained in step 3.7) to better distinguish different deer individuals that are visually similar. The steps to construct the multi-scale semantic alignment module include:

[0036] Step 5.2.1, using the reference point coordinates generated in step 4.3), extract ROI features on feature maps of different sizes and align and fuse them;

[0037] Step 5.2.2, perform interpolate operation on the fused features again, and sample them into 300 feature points with a dimension of 256 and a shape of 1x1;

[0038] Step 5.2.3, pass the object query calculated by self-attention in step 5.1) through a linear layer and normalize it, and multiply it with the features sampled in step 5.2.2);

[0039] 5.3) The object query calculated by self-attention in step 5.1) will be processed by the semantic alignment module in step 5.2), and the processing result will be cross-attention calculated with the feature map fused in step 3.6).

[0040] Step 6: The prediction result of the current frame will pass through the QIM module to screen new targets, already tracked targets and disappeared targets, and output the new targets and already tracked targets as the tracking query output of the next frame;

[0041] Step 7: Repeat the above steps for each subsequent frame. The difference is that, except for the first frame, each subsequent frame will concatenate the detection query output by the query initialization module based on prior knowledge of the current frame with the tracking query generated by the QIM module of the previous frame, and input them into the decoder together with the fused multi-scale features for interaction to generate the prediction result of each frame.

[0042] The beneficial effects of the present invention include:

[0043] (1) A new encoder is constructed by fusing the Efficient Hybrid Encoder, the feature enhancement module RFEM and the SE attention module, and replacing the original encoder of MOTR to effectively track deer in videos shot in the wild.

[0044] (2) A query initialization strategy based on prior knowledge is proposed to improve the tracking accuracy and generalization ability of the model when applied to wild environments with visually similar appearance to the fur of deer in deer videos, or different environments with large background differences.

[0045] (3) Construct a multi-scale semantic alignment module to reduce the difficulty of query matching on the feature map, avoid erroneous deer individual ID transformations, and improve the ID recognition performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a schematic diagram of the deer tracking method based on improved MOTR of the present invention.

[0047] Figure 1 In the figure, CNN is the backbone network ResNet50, EHE stands for the improved Efficient Hybrid Encoder, PKI refers to the detection query initialization module based on prior knowledge, the multi-scale semantic alignment module is included in the decoder Dec, Qd is the detection query used to detect new targets, Qt is the tracking query, QIM is used to manage the appearance of new targets and the disappearance of tracked targets, Y represents the prediction result of the network, and the collective average loss is performed with the label G.

[0048] Figure 2 A schematic diagram of a new encoder constructed for the present invention, wherein the right part of the figure is the structure of the Fusion module.

[0049] Figure 3 This is a schematic diagram of the detection query initialization strategy based on prior knowledge proposed by the present invention.

[0050] Figure 3In the figure, weights represents a vector, (x, y, w, h) is the preset suggestion box coordinates, (Δx, Δy, Δw, Δh) is the coordinate offset output by the Box head, and Top300 represents the selection of the 300 suggestion box coordinates with the highest scores according to the scores of the classification head as the reference coordinates of the decoder part.

[0051] Figure 4 Schematic diagram of the multi-scale semantic alignment module constructed for the present invention. The left figure is the decoder structure, and the right figure is the structural diagram of the multi-scale semantic alignment module.

[0052] Figure 5 This is a comparison diagram of test results of the present invention applied to deer video tracking in a blurred image of a small running target.

[0053] Figure 6 This is a comparison chart of test results of the present invention applied to deer video tracking under light changes.

[0054] Figure 7 This is a comparison chart of test results of the present invention applied to deer video tracking in different scenarios.

[0055] Figure 8 This is a diagram showing the application result of the animal tracking method of the present invention in tracking the movement trajectory of a deer.

[0056] Fig. 9 This is a diagram showing the application results of the animal tracking method of the present invention in counting individual deer.

[0057] Fig.10 This is a diagram showing the application results of the animal tracking method of the present invention in the behavior recognition of deer.

[0058] Figures 5 to 7 For the experiment, Figures 8 to 10 It is an extended application of the tracking method. DETAILED DESCRIPTION

[0059] The improved MOTR-based wild animal multi-target tracking method of the present invention first collects a video of a deer; then uses a deer tracking algorithm to track the deer, and the steps are as follows:

[0060] 1) The collected videos were processed using a convolutional neural network (Resnet50) to extract multi-scale deer features at three different resolutions;

[0061] 2) Using the improved efficient hybrid encoder (Efficient Hybrid Encoder) to fuse the aforementioned multi-scale features and output the fused multi-scale features;

[0062] 3) Input the fused multi-scale features into the detection query initialization module based on prior knowledge, and output the detection query with the preliminary location information of the deer;

[0063] 4) The aforementioned detection query and the fused multi-scale features are input into the decoder. In the decoder, the detection query first performs self-attention calculation, then passes through the improved multi-scale semantic alignment module, and then performs cross-attention interaction with the fused multi-scale features, and finally outputs the prediction result of the current video frame;

[0064] 5) The prediction result of the current frame will be directly output as the final tracking result; at the same time, the prediction result of the current frame will also pass through the QIM module to screen new targets, already tracked targets and disappeared targets, and output the new targets and already tracked targets as the tracking query of the next frame;

[0065] 6) The above operation is repeated for each subsequent frame. The difference is that, except for the first frame, each subsequent frame will concatenate the detection query output by the query initialization module based on prior knowledge of the current frame with the tracking query generated by the QIM module of the previous frame, and input it into the encoder together with the fused multi-scale features for interaction.

[0066] In the present invention: an improved high-efficiency hybrid encoder replaces the original encoder of MOTR, achieving low computational complexity while improving small target tracking capability; an initialization detection query strategy based on prior knowledge reduces the influence of complex backgrounds, reduces missed detections, and improves the generalization capability of the model for different environments; a multi-scale semantic alignment module better distinguishes visually similar deer individuals of different sizes, and improves the deer individual ID recognition performance of the model.

[0067] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0068] The present invention provides Figure 1 The figure shows a wildlife multi-target tracking method based on improved MOTR. A new encoder integrating efficient hybrid encoder, RFEM feature enhancement module, SE attention, query initialization module based on prior knowledge and multi-scale semantic alignment module are designed. The specific steps include:

[0069] Step 1, collecting deer video sequences in wild scenes;

[0070] Step 2: Use a convolutional neural network (Resnet50) to process the collected deer videos and extract multi-scale deer features at three different resolutions;

[0071] Step 3: Use an improved efficient hybrid encoder to replace the original encoder to perform feature fusion on the aforementioned multi-scale features, and output the fused multi-scale features. The steps of constructing the improved efficient hybrid encoder include:

[0072] Step 3.1: feature fusion is divided into the intrascale interaction part and the cross-scale interaction part. In the intrascale interaction part, the three feature maps with different resolutions obtained in step 2) are denoted as {S3, S4, S5}, among which the feature map S5 with the smallest resolution is sent to the Transformer for attention calculation to capture the global features, and the output is denoted as F5.

[0073] Step 3.2: The Fusion module of the cross-scale interaction part will splice the two feature maps that need to be fused in the channel dimension. The execution steps of the Fusion module include:

[0074] Step 3.2.1, use two parallel 1x1 convolutions to reduce the dimension of the two input feature maps to be fused respectively;

[0075] Step 3.2.2, a 1x1 convolution convolution after dimensionality reduction feature map will pass through a parallel structure consisting of a 1x1 convolution and a 3x3 convolution. The outputs of the two convolutions will be added element by element. The added feature map will be expanded and squeezed through the Squeeze-and-Excitation (SE) attention mechanism to improve the refinement of the model feature expression and the ability to integrate global information;

[0076] Step 3.2.3, another 1x1 convolution. The reduced feature will be added element by element to the feature map added element by element in step 3.2.2);

[0077] Step 3.3, the cross-scale interaction part is divided into two fusion parts: top-down and bottom-up. In the top-down fusion part, F5 will be processed by the RFEM feature enhancement module. Specifically, the F5 feature map will be convolved with three different dilation rates (1, 2, 3) and concatenated in sequence to obtain the feature map F6. F6 is then upsampled to a feature map with the same feature resolution as S4. This feature map is sent to the Fusion1 module for fusion together with S4.

[0078] Step 3.4: The output feature map of the Fusion1 module in step 3.3) is processed by the RFEM module to obtain the feature map F7, which is then upsampled to a feature map with the same resolution as the S3 feature map. This feature map enters the Fusion2 module together with S3 for fusion;

[0079] Step 3.5, in the bottom-up fusion part, the output of the Fusion2 module in step 3.4) is downsampled by a 3x3 GSConv convolution into a feature map with the same resolution as F7 in step 3.4), and this feature map is fused together with the feature map F7 in the Fusion3 module;

[0080] Step 3.6: The output of the Fusion3 module in step 3.5) is downsampled by a 3x3 GSConv convolution into a feature map with the same resolution as F6 in step 3.3), and this feature map is fused with the feature map F6 in the Fusion4 module;

[0081] Step 3.7, the features fused by Fusion2, Fusion3 and Fusion4 will be output as the final multi-scale features;

[0082] Step 4, step 3.7) The fused multi-scale features will be sent to the detection query initialization module based on prior knowledge, and the detection query with the preliminary location information of the deer will be output. The steps of constructing the object query initialization module based on prior knowledge include:

[0083] Step 4.1, generating a preliminary proposal box on the fused multi-scale feature map output in step 3.7);

[0084] In step 4.2, each feature point in the generated proposal box is treated as a separate detection query and passed through an independent detection head, which outputs the offset of the deer position coordinates;

[0085] Step 4.3, add the offset of the position coordinate to the initial suggestion box coordinate as the final position coordinate, and finally select multiple position coordinates with high scores according to the classification score as the reference position coordinates of the decoder part;

[0086] Step 4.4: The reference point is processed by position embedding and then connected to a fully connected layer and LN layer to generate a new detection query;

[0087] Step 4.6, use the linear layer to generate a set of learnable vector weights, which are multiplied by the generated detection query as the final detection query. For other video frames except the first frame, the tracking query does not participate in the initialization, and the tracking query generated by QIM in the previous frame is still used;

[0088] Step 5: The detection query generated in step 4 is input into the decoder together with the multi-scale features fused in step 3.7) to interact and generate the prediction result of the current frame. The execution steps of the decoder are:

[0089] 5.1) Object query first performs self-attention calculation; if it is the first frame, the object query is only a detection query; if it is a subsequent frame, the object query includes a detection query and a tracking query;

[0090] 5.2) A semantic alignment module is added after the self-attention to more accurately match the corresponding positions and introduce the multi-scale features obtained in step 3.7) to better distinguish different deer individuals that are visually similar. The steps to construct the multi-scale semantic alignment module include:

[0091] Step 5.2.1, using the reference point coordinates generated in step 4.3), extract ROI features on feature maps of different sizes and align and fuse them;

[0092] Step 5.2.2, perform interpolate operation on the fused features again, and sample them into 300 feature points with a dimension of 256 and a shape of 1x1;

[0093] Step 5.2.3, pass the object query calculated by self-attention in step 5.1) through a linear layer and normalize it, and multiply it with the features sampled in step 5.2.2);

[0094] 5.3) The object query calculated by self-attention in step 5.1) will be processed by the semantic alignment module in step 5.2), and the processing result will be cross-attention calculated with the feature map fused in step 3.6).

[0095] Step 6: The prediction result of the current frame will pass through the QIM module to screen new targets, already tracked targets and disappeared targets, and output the new targets and already tracked targets as the tracking query output of the next frame;

[0096] Step 7: Repeat the above steps for each subsequent frame. The difference is that, except for the first frame, each subsequent frame will concatenate the detection query output by the query initialization module based on prior knowledge of the current frame with the tracking query generated by the QIM module of the previous frame, and input them into the decoder together with the fused multi-scale features for interaction to generate the prediction result of each frame.

[0097] In order to verify the improved strategy of the present invention, including the encoder submodule (EHE), the query initialization submodule (PKI) and the multi-scale semantic alignment submodule (MSFSA), the present invention takes the deer video as an example to conduct experiments.

[0098] The present invention conducts an ablation experiment on the deer data set to prove the degree of improvement of each point improvement on the model. The experimental results are shown in Table 1. After adopting the improved encoder EHE, MOTA and IDF1 are increased by 1% and 1.2% respectively, and the value of IDS is reduced by 20%. At the same time, its inference speed is 4.6 times that of the original one, which improves the real-time performance of the model while ensuring the accuracy. Since PKI is only used in the detection query stage, the detection performance evaluation index MOTA is improved by 1.5%, while the IDF1 index biased towards identity recognition is only increased by 0.4%, and IDS is also reduced. In addition, after the introduction of MSFSA, MOTA and IDF1 are increased by 1.9% and 2.2% respectively, and the value of IDS is reduced by 22%. The experimental results show that the improved model has a significant improvement in MOTA, IDF1, and IDS compared with the original model, and improves the accuracy of target recognition and tracking.

[0099] Table 1. Ablation experiment results (EHE represents the improved Efficient Hybrid Encoder, PKI represents the query initialization module based on prior knowledge, and MSFSA represents the multi-scale semantic alignment module)

[0100] EHE PKI MSFSA MOTA(%) IDF1(%) IDS 79.4 84.5 30 √ 80.4 85.7 24 √ √ 81.9 86.1 22 √ √ √ 83.8 88.3 17

[0101] In order to demonstrate the advancedness of the proposed method, the proposed deer tracking method is compared with the common animal tracking network on the constructed video dataset. The comparison includes multi-target tracking accuracy (MOTA), IDF1 score, and model complexity. The experimental results are shown in Table 2:

[0102] Table 2 Comparative experimental results

[0103]

[0104] From the comparison of the experimental results in Table 2, it can be seen that the deer tracking method proposed in the present invention has obvious advantages in tracking accuracy and parameter quantity. At the same time, although the GFLOPS of TrackFormer and TransTrack are slightly lower than that of the present invention, the tracking accuracy of the present invention is significantly improved compared with the above two methods.

[0105] In order to verify the generalization ability and robustness of the present invention in complex scenes, four situations were tested: blurred images caused by shaking of the camera, small running animals, light changes, and scenes that did not appear in the training set. The results were visualized and compared with the two best algorithms ByteTrack and TransTrack.

[0106] Figure 5The tracking results of a small target running in a blurred image are visualized. The model of the present invention detects all targets in frame 202, while ByteTrack misses one. In addition, in frame 256, all three methods miss the ID of a deer, but the model of the present invention successfully matches its ID again in frame 257, while the other two methods still miss it. The experimental results show that the algorithm has strong robustness for small target detection.

[0107] Figure 6 A comparison chart of the tracking results of the present invention under light changes is shown. The image contains sika deer individuals in different postures. At the 113th frame, both comparison algorithms repeated tracking on the sika deer individual with ID 12, and the individual lying in the lower right corner was not fully tracked. At the 224th frame, the visual effect of the deer fur changed due to the change in illumination, resulting in the loss of the previously tracked target. At the 258th frame, the illumination returned to the previous situation, but the missed individual was still not tracked. Thanks to the multi-scale semantic alignment module proposed in this article, the method of the present invention successfully tracked the sika deer individuals in different postures and was not affected by the illumination change.

[0108] Figure 7 The generalization ability of the present invention in different scenarios is demonstrated. Two scenarios that did not appear in the training set are selected, such as Figure 7 As shown in the figure, although the three algorithms all have false detections and missed detections in the snow scene, overall, the methods of the present invention are superior to other models. In the 138th frame of the fence scene, TranTrack has an incorrect detection frame, and there is an occlusion among the three sika deer on the left side of the image, so ByteTrack fails to track one of them. In the 153rd frame, a smaller individual in the middle of the image is occluded, causing both compared algorithms to lose the target. The method of the present invention benefits from the query initialization method based on prior knowledge, which improves the generalization ability in different scenarios. All of the above situations occur.

[0109] Figure 8 The present invention is demonstrated for tracking the movement trajectory of deer. By using the target tracking positioning frame output by the model, complete tracking paths are drawn for all individual deer. It can be seen that each sika deer is assigned a unique movement trajectory, and the tracking path is relatively complete without interruption. This function can help understand the activity range of deer and provide important data support for individual deer activity monitoring and the planning and design of protected areas.

[0110] Fig. 9The present invention is demonstrated to be applicable in counting, and the number of deer tracking and positioning frames in each image frame of the surveillance video can be automatically counted, that is, the number of deer at any time can be counted, abnormal numbers can be discovered in time, and the lost individuals can be found through the last vanishing point of each deer track.

[0111] Fig.10 The application of the present invention in behavior detection is demonstrated. A behavior monitoring classification head can be added to the proposed model, or a classification network can be added later to intercept the sika deer target in the tracking and positioning frame to achieve behavior recognition of individual deer. This can facilitate researchers to conduct in-depth research on the living habits of sika deer and timely discover abnormal behaviors and relevant health conditions of deer. For example, if an animal lies down for a long time and does not eat, this is an important signal that the animal is sick, and corresponding treatment measures should be formulated immediately.

Claims

1. A method for tracking multiple targets of wild animals based on improved MOTR, characterized by the following steps: include: 1) Collect video sequences of deer in wild scenes; 2) The collected deer videos were processed using the convolutional neural network Resnet50 to extract multi-scale features of three different resolutions. The feature maps are recorded as {S3, S4, S5} from large to small resolutions; 3) Using the improved efficient hybrid encoder to replace the original encoder of the multi-target tracking framework MOTR, perform feature fusion on the multi-scale features extracted in step 2), and output the fused multi-scale features; 4) The fused multi-scale features from step 3) are fed into the detection query initialization module, which outputs a detection query with prior knowledge of the initial position of the deer; 5) The detection query obtained in step 4) and the fused multi-scale features obtained in step 3) are input into the decoder for interaction to generate a prediction result for the current frame; 6) The prediction result of the current frame passes through the QIM module to generate the tracking query of the next frame; 7) The above steps are repeated for each subsequent frame. The difference is that, except for the first frame, each subsequent frame will concatenate the detection query output by the detection query initialization module based on prior knowledge of the current frame with the tracking query generated by QIM in the previous frame, and input them into the decoder together with the fused multi-scale features for interaction to generate the prediction result of each frame; In step 3), the improved efficient hybrid encoder combines the Efficient Hybrid Encoder with the RFEM module and SE attention, and improves the feature pyramid part of the Efficient Hybrid Encoder, including: Introducing the RFEM module in the Efficient Hybrid Encoder; Feature fusion is divided into the intra-scale interaction part and the cross-scale interaction part, among which the cross-scale interaction part is further divided into the top-down fusion part and the bottom-up fusion part; SE attention is introduced in the Fusion module of the cross-scale interaction part. The Fusion module is used to fuse the two input feature maps that need to be fused in the cross-scale interaction part.

2. The method for tracking multiple targets of wild animals based on improved MOTR according to claim 1, characterized in that In step 3), the Fusion module in the improved efficient hybrid encoder is used to fuse the two input feature maps that need to be fused in the cross-scale interaction part; the structure of the Fusion module is: a) Concatenate the two input feature maps in the channel dimension and use two parallel 1x1 convolutions to reduce the dimension of the concatenated feature maps; b) A feature map after dimensionality reduction by a 1x1 convolution will pass through a parallel structure consisting of a 1x1 convolution and a 3x3 convolution; the outputs of the two convolutions in the parallel structure are added element by element, the feature information of different convolutional layers is integrated, and SE attention is used to squeeze and expand the features output by the parallel structure; c) The features reduced by another 1x1 convolution will be added element-by-element to the features added element-by-element in step b).

3. The method for tracking multiple targets of wild animals based on improved MOTR according to claim 1 or 2, characterized in that In step 3), the specific steps of feature fusion are: 3.1) In the inner scale interaction part, the feature map S5 with the smallest resolution obtained in step 2) is sent to the Transformer, and the Encoder of the Transformer is used to capture the global features. The output feature map is denoted as F5; 3.2) In the top-down fusion part, F5 is processed by the RFEM module to obtain the feature map F6, which is then upsampled to a feature map with the same feature resolution as S4. This feature map is sent to the Fusion1 module for fusion together with S4; 3.3) The output feature map of the Fusion1 module in step 3.2) is processed by the RFEM module to obtain the feature map F7, which is then upsampled to a feature map with the same resolution as the S3 feature map. This feature map and S3 are then fed into the Fusion2 module for fusion; 3.4) In the bottom-up fusion part, the output of the Fusion2 module in step 3.3) is downsampled by a 3x3 GSConv convolution into a feature map with the same resolution as F7 in step 3.3), and this feature map is fused with the feature map F7 in the Fusion3 module; 3.5) The output of the Fusion3 module in step 3.4) is downsampled by a 3x3 GSConv convolution into a feature map with the same resolution as F6 in step 3.2), and this feature map is fused with the feature map F6 in the Fusion4 module; 3.6) The fused features of Fusion2, Fusion3 and Fusion4 are used as the final multi-scale feature output.

4. The method for tracking multiple targets of wild animals based on improved MOTR according to claim 1, characterized in that In step 4), the detection query initialization module uses the Two-Stage variant method in Deformable-DETR to initialize the query vector.

5. The method for tracking multiple targets of wild animals based on improved MOTR according to claim 4, characterized in that In step 4), the detection query initialization module is a detection query initialization module based on prior knowledge, and uses a two-stage method to initialize the detection query; The steps to construct a detection query initialization module based on prior knowledge include: 4.1) Generate a preliminary proposal box based on the fused multi-scale features output in step 3.6); 4.2) Each feature point in the generated proposal box is treated as a separate detection query and passed through an independent detection head, which outputs the offset of the deer position coordinates; 4.3) Add the offset of the position coordinate to the initial proposal box coordinate as the new position coordinate, and finally select multiple position coordinates with high scores according to the classification score as the reference position coordinates of the decoder part; 4.4) The reference position coordinates are processed through position embedding and then connected to a fully connected layer and LN layer to generate a new detection query; 4.5) A set of learnable vector weights is generated using the linear layer. The weights are multiplied with the detection query generated in step 4.4) as the final detection query. For other video frames except the first frame, the tracking query does not participate in the initialization, and the tracking query generated by QIM in the previous frame in step 6) is still used.

6. The method for tracking multiple targets of wild animals based on improved MOTR according to claim 1, characterized in that In step 5), the decoder performs the following steps: 5.1) Object query first performs self-attention calculation; If it is the first frame, the object query is only a detection query; if it is a subsequent frame, the object query includes a detection query and a tracking query; 5.2) A semantic alignment module is added after the self-attention to more accurately match the corresponding positions and introduce the multi-scale features obtained in step 3) to better distinguish different deer individuals that are visually similar; 5.3) The object query after the self-attention calculation in step 5.1) will be processed by the semantic alignment module in step 5.2), and the processing result will be cross-attention calculated with the feature map fused in step 3).

7. The method for tracking multiple targets of wild animals based on improved MOTR according to claim 6, characterized in that The multi-scale semantic alignment module in step 5.2) includes the following specific steps: 5.2.1) Using the reference position coordinates generated in step 4.3), extract ROI features on feature maps of different sizes and fuse them; 5.2.2) Perform interpolation operation on the fused features again and sample them into 300 feature points with a dimension of 256 and a shape of 1x1; 5.2.3) The object query calculated by self-attention in step 5.1) is passed through a linear layer and normalized, and multiplied with the features sampled in step 5.2.2).

8. The method for tracking multiple targets of wild animals based on improved MOTR according to claim 1, characterized in that In step 6), the QIM module is used to manage the appearance of new targets and the disappearance of tracked targets, thereby screening new targets, tracked targets and disappeared targets, and outputting new targets and tracked targets as tracking queries for the next frame.

Images