Animal tracking and pose estimation method and apparatus, electronic device, and storage medium

Abstract

The application discloses an animal tracking and posture estimation method and device, electronic equipment and a storage medium, the method comprising: acquiring an animal ethology video and preprocessing to obtain a plurality of original sampling images; the images are sequentially input into a pre-trained instance segmentation model and a posture estimation model respectively to obtain instance segmentation masks and posture data of each frame of image, and then the original trajectory is created based on the correlation of the instance segmentation masks of adjacent frames; the instance segmentation masks of each frame of image are superimposed into the corresponding original image, and cutting and rotating are performed to obtain instance cutting images; the original trajectory is updated based on the instance cutting images, a preset first trajectory mutual exclusion condition and a preset trajectory matching condition to obtain an animal trajectory; the animal trajectory is combined with the corresponding posture data to generate a visual animal behavior space-time graph. The application solves the problem that current multi-experimental animal tracking and posture estimation cannot be better performed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to an animal tracking and posture estimation method and device, an electronic device and a storage medium. BACKGROUND

[0002] Multi-animal behavior research is an important part of neuroscience research and medical research. The analysis of complex multi-animal social behavior under different environments, different personalities, and even different diseases is a basic step for related research. A high-reliability multi-animal tracking and posture estimation system can assist in experimental data collection, eliminate the influence of human researcher's subjective experience, and significantly reduce the workload of experimental personnel.

[0003] Existing multi-animal tracking and pose estimation systems include DeepLabCut (Lauer, J., Zhou, M., Ye, S. et al. Multi-animal pose estimation, identification and tracking with DeepLabCut. Nat Methods 19, 496-504 (2022)), SLEAP (Pereira, T. D., Tabris, N., Matsliah, A. et al. SLEAP: A deep learning system for multi-animal pose tracking. Nat Methods 19, 486-495 (2022)), SIPEC (Marks, M., Jin, Q., Sturman, O. et al. Deep-learning-based identification, tracking, pose estimation and behaviour classification of interacting primates and mice in complex environments. Nat Mach Intell 4, 331-340 (2022)), etc. The above systems require researchers to label animal IDs for neural network training, so the experimental animals need to have significant visual features for differentiation for labeling; DeepLabCut and idTracker.ai (Romero-Ferrero, F., Bergomi, M. G., Hinz, R. C. et al. idtracker.ai: tracking all individuals in small or large collectives of unmarked animals. Nat Methods 16, 179-182 (2019)) also provide deep learning-based unsupervised multi-animal tracking solutions, the former uses a triplet loss to train a Transformer-based Re-id component, and the latter designs a complex algorithm for reliable tracking of a large number of small experimental animals based on image-based experimental animal differentiation, but does not provide pose estimation functions.

[0004] In addition, some experimental animal tracking systems also design tracking methods based on graph optimization, kinematics, etc. that are irrelevant to experimental animals and not based on deep learning.

[0005] However, the prior art has many shortcomings:

[0006] 1. The tracking method independent of the appearance of experimental animals cannot handle experimental animals with highly nonlinear motion and frequent changes in shape. In intense activities, frequent occlusion, fighting and high-speed motion can seriously affect tracking accuracy. Once the experimental animals are wrongly tracked, the error propagates in space and time, making the final tracking result unpredictable.

[0007] 2. The systems such as DeepLabCut, SIPEC, SLEAP, etc. use artificial labeling of animal data for neural network training to distinguish different experimental animals. This requires experimental animals to have different appearance features for human differentiation and labeling, which is not achievable for a large number of small experimental animals (such as ants, fruit flies, etc.). In addition, experimental animals often have highly similar appearances, making it difficult for researchers to distinguish them. Introducing additional markers on experimental animals can have unpredictable effects on experimental animals and reduce the reliability of the final experimental results. In addition, artificial labeling of animal ID also introduces additional work for researchers.

[0008] 3. The DeepLabCut proposed triplet loss training based on the Re-id module of the Transformer is an unsupervised animal tracking scheme, which works well when the experimental animals interact and move regularly. However, in the case of complex interaction and complex motion, a large amount of manual verification is still required.

[0009] 4. The complex system proposed by idTracker.ai based on image classification algorithm to distinguish experimental animals is an unsupervised animal tracking scheme, but it requires a large amount of computer memory, consumes a large amount of computing resources, and takes a long time to calculate. It has high requirements for the light source and imaging equipment of the video shooting, and cannot handle highly variable experimental animals. It cannot give reliable results in complex animal interaction or complex environmental interference. In addition, this system cannot estimate the posture of experimental animals. SUMMARY

[0010] The purpose of the present application is to overcome the above technical deficiencies and provide an animal tracking and posture estimation method, device, electronic equipment and storage medium, which solves the technical problem of not being able to better track and estimate the posture of multiple experimental animals in the prior art.

[0011] To achieve the above technical purpose, the present application adopts the following technical scheme:

[0012] In a first aspect, the present application provides an animal tracking and posture estimation method, comprising the following steps:

[0013] Obtaining an animal ethology video, preprocessing the animal ethology video to obtain a plurality of original sampling images;

[0014] inputting the multiple frames of original sampling images into the pre-trained instance segmentation model and the posture estimation model respectively and sequentially to obtain instance segmentation masks and posture data of the frames of original sampling images, creating original trajectories based on a correlation relationship of the instance segmentation masks of adjacent frames;

[0015] stacking the instance segmentation masks of the frames of sampling images into the corresponding original sampling images, and performing cropping and rotating to obtain instance cropped images;

[0016] updating the original trajectories based on the instance cropped images, a preset first trajectory exclusion condition and a preset trajectory matching condition to obtain animal trajectories;

[0017] combining the animal trajectories with the corresponding posture data to generate visualized animal behavior spatiotemporal graphs.

[0018] In some embodiments, the animal behavior video is obtained, and the animal behavior video is preprocessed to obtain multiple frames of original sampling images, including:

[0019] An animal behavior video is obtained, and a region of interest containing multiple animals in the animal behavior video is obtained;

[0020] The region of interest is cropped in time and picture to obtain a video segment containing multiple animal interaction behaviors;

[0021] The video segment is processed by video frame clustering to obtain multiple frames of original sampling images.

[0022] In some embodiments, the following formula is used to obtain the correlation relationship of the instance segmentation masks of adjacent frames:

[0023]

[0024] wherein, M t is an instance segmentation mask corresponding to the t-th frame of original sampling images, M t+1 is an instance segmentation mask corresponding to the t+1-th frame of original sampling images, Thr track is a preset correlation threshold.

[0025] In some embodiments, the original trajectories are updated based on the instance cropped images, a preset first trajectory exclusion condition and a preset trajectory matching condition to obtain animal trajectories, including:

[0026] The instance cropped images are screened to obtain retained images and rejected images;

[0027] checking exclusivity of original trajectories corresponding to the reserved images based on a preset first trajectory exclusivity condition, merging determinable original trajectories, and generating an ordered list of the original trajectories sorted according to lengths of exclusive time periods;

[0028] merging the original trajectories based on the merging condition of the determinable original trajectories and the ordered list, and using a preset trajectory matching condition to obtain optimized trajectories;

[0029] updating the optimized trajectories based on trajectories corresponding to the removed images and a preset second trajectory exclusivity condition to obtain the animal trajectories.

[0030] In some embodiments, the checking exclusivity of original trajectories corresponding to the reserved images based on a preset first trajectory exclusivity condition, merging determinable original trajectories, and generating an ordered list of the original trajectories sorted according to lengths of exclusive time periods comprises:

[0031] checking exclusivity of original trajectories corresponding to the reserved images based on a preset first trajectory exclusivity condition, obtaining original trajectory exclusivity relationships, and merging determinable original trajectories in combination with a preset maximum number of tracked animals;

[0032] mapping the original trajectory exclusivity relationships to generate original trajectory sets that are mutually exclusive, and generating an ordered list of the original trajectory sets sorted according to lengths of exclusive time periods.

[0033] In some embodiments, the merging the original trajectories based on the merging condition of the determinable original trajectories and the ordered list, and using a preset trajectory matching condition to obtain optimized trajectories comprises:

[0034] determining whether there are determinable original trajectories;

[0035] when there are no determinable original trajectories and the number of elements in the original trajectory sets does not exceed a preset maximum number of tracked animals, using an instance cutout image corresponding to an original trajectory set with the longest exclusive time period as a sample, performing down-sampling processing to generate a training set, training a preset classification network using the training set, and obtaining a trained classification network;

[0036] performing classification probability calculation on the reserved images using the classification network, and performing trajectory merging operations based on the calculated classification probabilities to obtain optimized trajectories.

[0037] In some embodiments, the performing classification probability calculation on the reserved images using the classification network, and performing trajectory merging operations based on the calculated classification probabilities to obtain optimized trajectories comprises:

[0038] The classification network is used to calculate classification probability of the reserved image, and a new merged original trajectory is generated based on a preset prediction probability value.

[0039] Based on the new merged original trajectory, the number of corresponding instance segmentation images is calculated, and it is determined whether the number exceeds a preset value.

[0040] When the number exceeds the preset value, optimal matching is performed on the original trajectory set to be classified and the original trajectory set that has been classified to generate a to-be-merged list.

[0041] Based on the to-be-merged list and the new merged original trajectory, an optimized trajectory is generated.

[0042] In a second aspect, the present application further provides an animal tracking and posture estimation device, comprising:

[0043] An image acquisition module is configured to acquire an animal ethology video, and pre-process the animal ethology video to obtain a plurality of original sampling images.

[0044] An identification module is configured to input the plurality of original sampling images into a pre-trained instance segmentation model and a posture estimation model in sequence respectively, to obtain instance segmentation masks and posture data of each frame of original sampling image, and then create original trajectories based on the correlation of instance segmentation masks of adjacent frames.

[0045] A processing module is configured to superimpose the instance segmentation masks of each frame of sampling image onto the corresponding original sampling image, and perform cropping and rotation to obtain instance cut images.

[0046] An updating module is configured to update the original trajectories based on the instance cut images, a preset first trajectory mutual exclusion condition and a preset trajectory matching condition, to obtain animal trajectories.

[0047] A space-time graph generation module is configured to combine the animal trajectories and the corresponding posture data to generate a visual animal behavior space-time graph.

[0048] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory.

[0049] The memory stores a computer program that can be executed by the processor.

[0050] The processor executes the computer program to implement the steps of the animal tracking and posture estimation method described above.

[0051] In a fourth aspect, the present application also provides a computer readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps of the animal tracking and pose estimation method as described above.

[0052] Compared with the prior art, the animal tracking and pose estimation method, device, electronic equipment and storage medium provided by the present application first acquire an ethology video of an animal, pre-process the ethology video of the animal to obtain a plurality of original sampling images, then input the plurality of original sampling images into a pre-trained instance segmentation model and a pose estimation model respectively and sequentially to obtain instance segmentation masks and pose data of each original sampling image, create an original trajectory based on the correlation of the instance segmentation masks of adjacent frames, then superimpose the instance segmentation masks of each sampling image onto the corresponding original sampling image and perform cropping and rotation to obtain instance cropped images, update the original trajectory based on the instance cropped images, a preset first trajectory mutual exclusion condition and a preset trajectory matching condition to obtain an animal trajectory, and finally combine the animal trajectory with the corresponding pose data to generate a visualized animal behavior space-time graph. The animal tracking and pose estimation method provided by the present application combines a deep learning model, adopts an unsupervised learning paradigm, and tracks experimental animals under highly complex social conditions based on images, which greatly reduces the requirements for a shooting device and a data processing device, shortens the running time, and gives reliable tracking and pose estimation results of multiple experimental animals without human researcher intervention. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is an implementation platform architecture diagram of the animal tracking and pose estimation method provided by the embodiment of the present application;

[0054] Figure 2 is a flowchart of the animal tracking and pose estimation method provided by the embodiment of the present application;

[0055] Figure 3 is a flowchart of step S100 in the animal tracking and pose estimation method provided by the embodiment of the present application;

[0056] Figure 4 is an ethology video segment of an animal provided by the embodiment of the present application in the animal tracking and pose estimation method;

[0057] Figure 5 is a schematic diagram of an instance cropped image provided by the embodiment of the present application in the animal tracking and pose estimation method;

[0058] Figure 6Is the trajectory merging state diagram of the intermediate step in the animal tracking and posture estimation method provided by the embodiment of the application;

[0059] Figure 7 Is the trajectory mutual exclusion relationship diagram in the animal tracking and posture estimation method provided by the embodiment of the application;

[0060] Figure 8 Is the tracking result combined with the posture estimation visualized diagram in the animal tracking and posture estimation method provided by the embodiment of the application;

[0061] Figure 9 Is the tracking result combined with the posture estimation result spatiotemporal diagram in the animal tracking and posture estimation method provided by the embodiment of the application;

[0062] Figure 10 Is the functional module diagram of the animal tracking and posture estimation device provided by the embodiment of the application;

[0063] Figure 11 Is the hardware structure diagram of the electronic device provided by the embodiment of the application. DETAILED DESCRIPTION

[0064] In order to make the object, technical scheme and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0065] Please refer to Figure 1 and Figure 2 The animal tracking and posture estimation method provided by the present application comprises the following steps:

[0066] S100, obtaining an animal ethology video, pre-processing the animal ethology video to obtain a plurality of original sampling images;

[0067] S200, inputting the plurality of original sampling images into a pre-trained instance segmentation model and a posture estimation model in sequence respectively, obtaining instance segmentation masks and posture data of each original sampling image, and creating original trajectories based on the correlation of the instance segmentation masks of adjacent frames;

[0068] S300, superimposing the instance segmentation masks of each sampling image into the corresponding original sampling image, and performing cropping and rotating to obtain instance cut images;

[0069] S400, updating the original trajectories based on the instance cut images, a preset first trajectory mutual exclusion condition and a preset trajectory matching condition to obtain animal trajectories;

[0070] S500, combine the animal trajectory with the corresponding posture data to generate a visualized animal behavior spatiotemporal graph.

[0071] In this embodiment, a deep learning model is combined, and unsupervised learning paradigm is used to track experimental animals under highly complex social conditions based on images. Compared with other non-supervised and image-based methods, the requirements for shooting equipment and data processing equipment are greatly reduced, the running time is shortened, and reliable multi-experimental animal tracking and posture estimation results are given without human researcher intervention.

[0072] In some embodiments, referring to Figure 3 , the step S100 specifically includes:

[0073] S110, acquiring an ethology video of animals, and acquiring a region of interest containing multiple animals in the ethology video of animals;

[0074] S120, performing time and picture cutting on the region of interest to obtain a video segment containing multiple animal interactive behaviors;

[0075] S130, performing video frame clustering processing on the video segment to obtain multiple frames of original sampling images.

[0076] In this embodiment, first, an image acquisition device is used to shoot a multiple animal social behavior activity area, and the obtained results are transmitted to a computer system, as shown in Figure 1 . Then, the region of interest of the video is selected, the video is time and picture cut, which contains various complex interactive behaviors of multiple experimental animals, as shown in Figure 4 , and the input video is re-encoded into the preferred H.264 format. Then, the input video resolution is reduced to 96x96 pixels, and the video frame clustering is performed using the clustering algorithm according to the given frame extraction number. If the number of frames in the class exceeds the number of frames contained in 3 seconds, a frame is randomly selected and saved, otherwise it is skipped. In addition, for the case where the number of frames is insufficient, random sampling is used to make up. Preferably, the clustering algorithm is the K-means algorithm.

[0077] In some embodiments, in the step S200, first, the instance segmentation model and the posture estimation model are trained. Preferably, the instance segmentation model uses the RTMDet-Ins-tiny deep learning model, and the instance segmentation and experimental animal detection are performed through the model. The posture estimation model uses a deep learning model combining MobileNet V2 and a heat map (Heatmap), and the animal posture estimation is performed through the model in a top-down manner.

[0078] Of course, it needs to be explained that the instance segmentation model in the embodiment of the application is not limited to the RTMDet-Ins-tiny deep learning model, which is only one way of generating an instance mask and target detection, as long as the model, method for obtaining a mask for subsequent unsupervised tracking is within the protection scope of the application, similarly, the posture estimation model in the embodiment of the application is not limited to the deep learning model combined with MobileNet V2 and Heatmap, which is only one way of extracting the posture of an experimental animal, as long as the model, method for combining animal posture estimation with experimental animal tracking results is within the protection scope of the application.

[0079] When the model is trained, first, training data with annotations is obtained, the training data is input into the instance segmentation model and the posture estimation model respectively to obtain the trained instance segmentation model and the posture estimation model, and subsequently, instance segmentation, experimental animal detection and animal posture estimation can be directly performed through the two models respectively.

[0080] When the multi-frame original sampling images are input into the two models, the instance segmentation model can output an instance segmentation result, and then the correlation of the instance segmentation masks of adjacent frames is obtained by using the following formula:

[0081]

[0082] Wherein, M t is the instance segmentation mask corresponding to the t-th original sampling image, M t+1 is the instance segmentation mask corresponding to the t+1-th original sampling image, Thr track is a preset correlation threshold, preferably 0.7.

[0083] If the above formula is satisfied, the adjacent frames are correlated, otherwise a new track is created.

[0084] After the model is calculated, preferably, the posture coordinates, posture scores, detection boxes and original track IDs output by the model are stored using the local database system sqlite3.

[0085] It needs to be explained that the database system sqlite3 is only one way of software-generated data management, as long as the system, method for managing experimental animal tracking and posture data in the form of a database is within the protection scope of the application.

[0086] In some embodiments, the step S300 is used to superimpose the instance mask output in the step S200 onto the original sample image, and cut the original sample image using the minimum circumscribed matrix. After cutting, the posture parameters suitable for color are used to rotate and register the cut instance image, so that the instance image has a uniform direction, and then the instance image is saved into the original track ID folder, as shown in Figure 5 .

[0087] In some embodiments, the step S400 is used to perform track matching for subsequent generation of the behavior space-time graph of each animal, as shown in Figure 6 , and the step S400 specifically includes:

[0088] The instance cut image is screened to obtain a reserved image and a rejected image;

[0089] Based on a preset first track mutual exclusion condition, the original tracks corresponding to the reserved image are checked for mutual exclusion, the determinable original tracks are merged, and an ordered list sorted according to the length of the mutual exclusion period is generated;

[0090] Based on the merging of the determinable original tracks and the ordered list, the original tracks are merged using a preset track matching condition to obtain optimized tracks;

[0091] Based on the rejected image and a preset second track mutual exclusion condition, the tracks corresponding to the rejected image are classified, and the optimized tracks are updated to obtain the animal tracks.

[0092] Preferably, based on the preset first track mutual exclusion condition, the original tracks corresponding to the reserved image are checked for mutual exclusion, the determinable original tracks are merged, and an ordered list sorted according to the length of the mutual exclusion period is generated, which includes:

[0093] Based on the preset first track mutual exclusion condition, the original tracks corresponding to the reserved image are checked for mutual exclusion, the original track mutual exclusion relationship is obtained, and the determinable original tracks are merged in combination with the preset maximum number of tracked animals;

[0094] The original track mutual exclusion relationship is mapped to generate a set of original tracks mutually exclusive with each other, and an ordered list formed by the set of original tracks is generated according to the length of the mutual exclusion period.

[0095] Preferably, based on the merging of the determinable original tracks, the original tracks are merged using a preset track matching condition to obtain optimized tracks, which includes:

[0096] It is determined whether there is a determinable original track;

[0097] When there is no original trajectory that can be determined and the number of elements contained in the original trajectory set does not exceed the preset maximum number of tracked animals, the instance cut image corresponding to the original trajectory set with the longest mutually exclusive period is taken as a sample, and a training set is generated after down-sampling processing, the preset classification network is trained through the training set, and a trained classification network is obtained.

[0098] The classification network is used to calculate the classification probability of the reserved image, and a trajectory merging operation is performed based on the calculated classification probability to obtain an optimized trajectory.

[0099] Preferably, the classification network is used to calculate the classification probability of the reserved image, and a trajectory merging operation is performed based on the calculated classification probability to obtain an optimized trajectory, comprising:

[0100] The classification network is used to calculate the classification probability of the reserved image, and a new merged original trajectory is generated based on a preset prediction probability value.

[0101] Based on the new merged original trajectory, the number of instance segmentation images corresponding thereto is calculated, and it is judged whether the number exceeds a preset value.

[0102] When the number exceeds the preset value, the original trajectory set to be classified and the original trajectory set that has been classified are optimally matched to generate a to-be-merged list.

[0103] Based on the to-be-merged list and the new merged original trajectory, an optimized trajectory is generated.

[0104] In this embodiment, when generating animal trajectories, the following steps are specifically performed:

[0105] Step one: after superimposing the instance segmentation mask on the corresponding original sampling image and preferentially cutting using the minimum circumscribed matrix, the cut instance picture is rotated and registered using a preset pose parameter, so that the instance picture has a uniform direction, and then saved under the original trajectory ID folder;

[0106] Step two: the size of the instance cut image generated in step one is counted, and the average size of the cut picture set corresponding to each original trajectory is calculated, and if the cut picture area exceeds the preferred threshold of 95%, it is rejected;

[0107] Step three: reject the original trajectory containing less than the preferred length threshold of 10 frames and the original trajectory with an average detection number less than the preferred confidence threshold of 0.6;

[0108] Step four: for the cut image rejected in step three, find the corresponding frame, save it at an optimal minimum interval of 3 seconds, and prompt the user to label these data to obtain better output results;

[0109] Step five: check the exclusivity of all original trajectories, preferably, set the trajectories overlapping more than 10 frames on the time axis as exclusive, and combine the original trajectories that can be determined according to the following exclusive conditions combined with the preset maximum number of tracked animals, such as Figure 6

[0110]

[0111]

[0112] Step six: mapping of the original trajectory exclusivity, as shown in Figure 7 , preferably, using the maximum clique algorithm to generate a set of original trajectories that are exclusive to each other, and sorting according to the length of the exclusive period to generate an ordered list formed by the exclusive trajectory set;

[0113] Step seven: if there is an assigned trajectory and the cutting map of the assigned trajectory accounts for more than 98% of the total cutting map, then jump to step twelve, otherwise check the entire original trajectory set, if the number of elements in the original trajectory set does not exceed the preset maximum number of tracked animals, then enter step eight, otherwise, if it is the first time to enter step seven, then terminate the running and prompt an error, otherwise, record the shortest original trajectory that triggers the exceeding of the preset maximum number of tracked animals, if the problem trajectory triggers more than the preferred threshold of 3 times, then eliminate this trajectory, otherwise, roll back to the state before the update of the assigned trajectory;

[0114] Step eight: for the ordered list output in step six, if there is no classified trajectory, preferably, take the original trajectory set with the longest exclusive period, assign a classified trajectory to all elements in the set, otherwise, take the classified trajectory set; find the corresponding cutting map from the original trajectory corresponding to the set, divide the initial training set and the initial test set in the ratio of 9:1, then resample the initial test set to make the number of each class test set equal to generate the test set, and resample the original trajectory that has been added to the training set to generate the training set after increasing the downsampling rate according to the number of times it participates in training;

[0115] Step nine: train the classification network OSNet_x0_25 output in step eight, dynamically adjust the input size to the average size of the cutting map calculated in step two, and test using the test set. Preferably, set the label smoothing parameter to 0.1; preferably, set the first accuracy termination threshold to 99.5%, the precision improvement threshold to 0.5%, and the first training patience value to 15 rounds;

[0116] ​Step ten: input the segmentation map corresponding to all original trajectories t into the trained OSNet obtained in step nine, calculate the average probability of each category, denoted as classification probability P_t. If there is an original trajectory with a predicted probability exceeding the preferred 0.9, merge it with the classified trajectory obtained in the previous step. If there is no such trajectory, reduce the probability threshold to 0.85 and perform merging.

[0117] Step eleven: for the newly merged original trajectories in step ten, calculate the total number of segmentation maps contained. If it exceeds the preferred threshold of 50, return to step five. Otherwise, proceed to step twelve.

[0118] Step twelve: for each original trajectory set, calculate the chi-square distance between its classification probability P_t and the classification probability P_certain of the assigned trajectory set. Preferably, use the Hungarian algorithm to perform optimal matching between the original trajectory set to be analyzed and the assigned trajectory set, and calculate the average matching cost. If it is lower than the preferred average matching cost of 0.2, include it in the list of trajectories to be merged. It should be noted that the above Hungarian matching algorithm is only one way to perform data association instance matching. Any method that can perform optimal matching is within the scope of the present application. The above chi-square distance is only a way to measure the difference between the model output probability and the assigned trajectory and unassigned trajectory. Any method that can measure the distance between features and probabilities is within the scope of the present application.

[0119] Step thirteen: for the list of trajectories to be merged given in step twelve, if the list is empty, add the matching with the lowest total matching cost. Traverse the list of trajectories to be merged, and take out the trajectories to be merged with lower average matching cost one by one. Calculate the updated number of segmentation maps after merging, and if it exceeds the preferred threshold of 50, merge the trajectories and return to step five.

[0120] Step fourteen: re-add all trajectories removed in steps two and three to the original trajectory set, set the trajectory exclusion condition to 1 frame overlap on the time axis, and set the threshold for the contradiction condition to 1. Execute steps five, six, and seven to classify the remaining trajectories.

[0121] Step fifteen: use the sqlite3 local database system to update the ID of the tracked animal trajectory.

[0122] In some embodiments, after updating the ID of the tracked animal trajectory using the sqlite3 local database system, the animal trajectory ID and pose data stored in the sqlite3 local database system are combined for data visualization, and the visualized animal behavior spatiotemporal graph is obtained, as shown in Figure 8 and Figure 9 .

[0123] The technical solution provided by the present application has the following advantages compared to the prior art:

[0124] 1. The original trajectory quality is greatly improved through the instance segmentation component;

[0125] 2. The unsupervised multi-animal tracking paradigm is retained, the tracking process does not require human intervention and animal ID labeling, the tracking accuracy is greatly improved, and extremely high accuracy is achieved without manual calibration;

[0126] 3. Support processing more complex and more intense animal interaction behavior;

[0127] 4. The reliable multi-animal tracking and pose estimation module is integrated;

[0128] 5. The database component is used for data management, and the data management process is optimized;

[0129] 6. Compared with idTracker.ai and other image classification-based tracking systems, the running time and memory consumption are greatly reduced, and the tracking accuracy is improved;

[0130] 7. The robustness to complex environment and complex light is improved, and the configuration requirements of the shooting and processing equipment are reduced.

[0131] Another embodiment of the present application provides an animal tracking and pose estimation device, please refer to Figure 10 The animal tracking and pose estimation device comprises an image acquisition module 11, an identification module 12, a processing module 13, an updating module 14 and a space-time graph generation module 15.

[0132] The image acquisition module 11 is used for acquiring animal behavior video, and pre-processing the animal behavior video to obtain a plurality of original sampling images.

[0133] The identification module 12 is used for inputting the plurality of original sampling images into a pre-trained instance segmentation model and a pose estimation model in sequence respectively, obtaining instance segmentation masks and pose data of each frame of original sampling image, and creating an original trajectory based on the correlation of the instance segmentation masks of adjacent frames.

[0134] The processing module 13 is used for superimposing the instance segmentation masks of each frame of sampling image on the corresponding original sampling image, and performing cutting and rotating to obtain an instance cut image.

[0135] The updating module 14 is used for updating the original trajectory based on the instance cut image, a preset first trajectory mutual exclusion condition and a preset trajectory matching condition to obtain an animal trajectory.

[0136] The space-time graph generation module 15 is used for combining the animal trajectory with the corresponding pose data to generate a visual animal behavior space-time graph.

[0137] It should be noted that the module referred to in the present application refers to a series of computer program instruction segments capable of completing a specific function, and is more suitable for describing the execution process of animal tracking and pose estimation than programs. The specific implementation of each module is described with reference to the corresponding method embodiments described above, and will not be described here.

[0138] Another embodiment of the present application provides an electronic device, such as Figure 11 As shown in the figure, the electronic device 10 comprises:

[0139] One or more processors 110 and memories 120, Figure 11 In the embodiment, the processor 110 and the memory 120 can be connected through a bus or other means, Figure 11 In the embodiment, the connection through the bus is taken as an example.

[0140] The processor 110 is used to complete various control logics of the electronic device 10, which can be a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine), a graphics processing unit (GPU) or other programmable logic device, discrete gate or transistor logic, discrete hardware component or any combination of these components. In addition, the processor 110 can also be any conventional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, such as a combination of DSP and microprocessor, multiple microprocessors, one or more microprocessors combined with DSP and / or any other such configuration.

[0141] The memory 120 is a non-volatile computer readable storage medium, which can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions corresponding to the animal tracking and pose estimation method in the embodiment of the present application. The processor 110 executes various function applications and data processing of the electronic device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, implements the animal tracking and pose estimation method in the above method embodiment.

[0142] The memory 120 can include a program storage area and a data storage area. The program storage area can store an operating platform, applications, and / or other program instructions required by at least one function. The data storage area can store data created by the electronic device 10, etc. Additionally, the memory 120 can include a volatile memory, and also include a non-volatile memory, such as at least one magnetic storage device, a flash memory device, or other non-volatile solid state memory device. In some embodiments, the memory 120 can optionally include a memory that is remotely located from the processor 110, and can be connected to the electronic device 10 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0143] One or more units are stored in the memory 120, and when executed by the one or more processors 110, perform the animal tracking and pose estimation method in any of the method embodiments described above, for example, perform the method steps S100 to S500 in the method described above. Figure 2

[0144] Another embodiment of the present application provides a computer-readable storage medium storing computer executable instructions, which, when executed by one or more processors, perform the method steps S100 to S500 in the method described above. Figure 2

[0145] By way of example, computer-readable storage media can include a read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory can include a random access memory (RAM) as an external cache memory. By way of illustration, and not limitation, RAM can be available in many forms such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memory of the operating environments described herein are intended to include one or any other suitable type of memory.

[0146] ​​To sum up, the animal tracking and posture estimation method and device, the electronic equipment and the storage medium provided by the present application first acquire an ethology video of an animal, and pre-process the ethology video to obtain a plurality of original sampling images. Then, the plurality of original sampling images are sequentially input into a pre-trained instance segmentation model and a posture estimation model respectively to obtain instance segmentation masks and posture data of each original sampling image. Then, based on the correlation of the instance segmentation masks of adjacent frames, an original trajectory is created. Then, the instance segmentation masks of each sampling image are superimposed on the corresponding original sampling image, and are cropped and rotated to obtain instance cropped images. Then, the original trajectory is updated based on the instance cropped images, a preset first trajectory mutual exclusion condition and a preset trajectory matching condition to obtain an animal trajectory. Finally, the animal trajectory and the corresponding posture data are combined to generate a visual animal behavior space-time graph. By combining a deep learning model, the experimental animal tracking under a highly complex social condition is performed based on images using an unsupervised learning paradigm. Compared with other non-supervised image-based methods, the requirements for a shooting device and a data processing device are greatly reduced, and the running time is shortened. Without human researcher intervention, reliable multi-experimental animal tracking and posture estimation results are given.

[0147] The specific embodiments of the present application described above do not constitute a limitation on the protection scope of the present application. Any various other corresponding changes and modifications made according to the technical concept of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. An animal tracking and pose estimation method, characterized by, The method comprises the following steps: obtaining an animal behavior video, preprocessing the animal behavior video to obtain a plurality of original sampling images; inputting the plurality of original sampling images into a pre-trained instance segmentation model and a posture estimation model respectively and sequentially to obtain instance segmentation masks and posture data of each original sampling image, creating original trajectories based on the correlation of the instance segmentation masks of adjacent frames; stacking the instance segmentation masks of each sampling image onto the corresponding original sampling image, and performing cropping and rotation to obtain instance cropped images; updating the original trajectories based on the instance cropped images, a preset first trajectory mutual exclusion condition and a preset trajectory matching condition to obtain animal trajectories; combining the animal trajectories with the corresponding posture data to generate visualized animal behavior spatiotemporal graphs; the updating of the original trajectories based on the instance cropped images, the preset first trajectory mutual exclusion condition and the preset trajectory matching condition to obtain the animal trajectories comprises: screening the instance cropped images to obtain retained images and excluded images; performing mutual exclusion checking on the original trajectories corresponding to the retained images based on the preset first trajectory mutual exclusion condition, merging determinable original trajectories and generating an ordered list sorted according to the lengths of the mutual exclusion periods; merging the original trajectories based on the merging conditions of the determinable original trajectories and the ordered list to obtain optimized trajectories using the preset trajectory matching condition; updating the optimized trajectories based on the excluded images and a preset second trajectory mutual exclusion condition to obtain the animal trajectories.

2. The animal tracking and pose estimation method of claim 1, wherein, The obtaining of the animal behavior video and the preprocessing of the animal behavior video to obtain a plurality of original sampling images comprises: obtaining an animal behavior video, obtaining a region of interest containing multiple animals in the animal behavior video; performing time and picture cropping on the region of interest to obtain a video segment containing multiple animal interaction behaviors; performing video frame clustering processing on the video segment to obtain a plurality of original sampling images.

3. The animal tracking and pose estimation method of claim 2, wherein, The correlation of the instance segmentation masks of adjacent frames is obtained using the following formula: ， Wherein, M t is the instance segmentation mask corresponding to the t-th frame of original sampling image, M t+1 is the instance segmentation mask corresponding to the t+1-th frame of original sampling image, Thr track is a preset correlation threshold.

4. The animal tracking and pose estimation method of claim 3, wherein, the mutual exclusion checking of the original trajectories corresponding to the retained images based on the preset first trajectory mutual exclusion condition, the merging of determinable original trajectories and the generation of an ordered list sorted according to the lengths of the mutual exclusion periods comprise: performing mutual exclusion checking on the original trajectories corresponding to the retained images based on the preset first trajectory mutual exclusion condition to obtain original trajectory mutual exclusion relationships and combine the preset maximum number of tracked animals to merge determinable original trajectories; mapping the original trajectory mutual exclusion relationships to generate a set of mutually exclusive original trajectories, and then sorting the set to generate an ordered list formed by the set of original trajectories according to the lengths of the mutual exclusion periods.

5. The animal tracking and pose estimation method of claim 4, wherein, the merging of the original trajectories based on the merging conditions of the determinable original trajectories using the preset trajectory matching condition to obtain optimized trajectories comprises: determining whether there are determinable original trajectories; When there is no original trajectory that can be determined and the number of contained elements in the original trajectory set does not exceed the preset maximum number of tracked animals, the instance cut image corresponding to the original trajectory set with the longest mutually exclusive period is taken as a sample, and a training set is generated after down-sampling processing; the preset classification network is trained through the training set to obtain a trained classification network; The classification network is used to perform classification probability calculation on the reserved image, and a trajectory merging operation is performed based on the calculated classification probability to obtain an optimized trajectory.

6. The animal tracking and pose estimation method of claim 5, wherein, The classification network is used to perform classification probability calculation on the reserved image, and a trajectory merging operation is performed based on the calculated classification probability to obtain an optimized trajectory. The classification network is used to perform classification probability calculation on the reserved image, and a trajectory merging operation is performed based on the calculated classification probability to obtain an optimized trajectory. The classification network is used to perform classification probability calculation on the reserved image, and a trajectory merging operation is performed based on the calculated classification probability to obtain an optimized trajectory. When the number exceeds the preset value, the original trajectory set to be classified and the original trajectory set that has been classified are optimally matched to generate a list to be merged; The classification network is used to perform classification probability calculation on the reserved image, and a trajectory merging operation is performed based on the calculated classification probability to obtain an optimized trajectory.

7. An animal tracking and pose estimation apparatus, comprising: Comprise: An image acquisition module is configured to acquire an animal behavior video, and pre-process the animal behavior video to obtain a plurality of original sample images; An identification module is configured to input the plurality of original sample images into a pre-trained instance segmentation model and a pose estimation model in sequence to obtain instance segmentation masks and pose data of each original sample image, and create original trajectories based on the correlation of instance segmentation masks of adjacent frames; A processing module is configured to superimpose the instance segmentation masks of each sample image onto the corresponding original sample image, and perform cropping and rotation to obtain instance cut images; An updating module is configured to update the original trajectories based on the instance cut images, a preset first trajectory mutual exclusion condition, and a preset trajectory matching condition to obtain animal trajectories; A spatio-temporal graph generation module is configured to combine the animal trajectories and the corresponding pose data to generate a visual animal behavior spatio-temporal graph. The updating module is configured to update the original trajectories based on the instance cut images, a preset first trajectory mutual exclusion condition, and a preset trajectory matching condition to obtain animal trajectories. The updating module is configured to update the original trajectories based on the instance cut images, a preset first trajectory mutual exclusion condition, and a preset trajectory matching condition to obtain animal trajectories. The updating module is configured to update the original trajectories based on the instance cut images, a preset first trajectory mutual exclusion condition, and a preset trajectory matching condition to obtain animal trajectories. The updating module is configured to update the original trajectories based on the instance cut images, a preset first trajectory mutual exclusion condition, and a preset trajectory matching condition to obtain animal trajectories. Comprise:

8. An electronic device, comprising: A processor and a memory. ​ The memory has stored thereon a computer program executable by the processor; The processor implements the steps in the animal tracking and pose estimation method according to any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, Comprise: A processor and a memory; The memory has stored thereon a computer program executable by the processor; The processor implements the steps in the animal tracking and pose estimation method according to any one of claims 1-6 when executing the computer program.

Images