Animal three-dimensional posture estimation method, device and equipment and storage medium

Abstract

The application relates to the field of artificial intelligence and discloses an animal three-dimensional posture estimation method, device and equipment and a storage medium. The method collects experimental animal video data based on a built multi-view shooting platform, constructs a training data set and a prediction data set according to the experimental animal video data, performs frame screening processing on the training data set, constructs an animal three-dimensional posture estimation model based on key point errors, inputs the training data set and the prediction data set after frame screening into the animal three-dimensional posture estimation model to perform training, adjusts the loss during training by using a framework designed for different key point errors, and performs key point recognition and posture estimation on the experimental animal by using the trained model. Different key point training weights are set during the training stage according to the labeling error of the key points calculated in advance, so that the result error caused by the key point error is avoided, and the multi-view volume three-dimensional posture estimation performance is optimized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for estimating three-dimensional pose of animals. Background Technology

[0002] Animal 3D pose estimation is a key technique for studying animal behavior and is widely used in neuroscience, biology, and animal behavior. Its goal is to recover the animal's pose information in three-dimensional space from 2D video data.

[0003] However, animal 3D pose estimation is extremely challenging due to the morphological diversity of animals, complex background environments, and the influence of factors such as perspective and lighting. In existing technologies, most animal 3D pose estimation methods rely on deep learning techniques, especially convolutional neural networks. These methods typically require labeled training data to train the model on animal keypoints in pre-video frames. However, manually labeled training samples are prone to errors due to manual annotation, which significantly impacts the accuracy of these methods. Furthermore, most existing animal 3D pose estimation methods are fully supervised, meaning they rely entirely on labeled training data during training. This approach allows manual annotation errors to severely affect the final training results, thus impacting the final outcome. Therefore, effectively avoiding keypoint errors while improving the performance of animal 3D pose estimation is a pressing technical problem that needs to be solved.

[0004] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this invention is to provide a method, apparatus, device, and storage medium for animal three-dimensional pose estimation, aiming to solve the technical problem that manual annotation errors in the prior art seriously affect the final training results and result in poor performance of animal three-dimensional pose estimation.

[0006] To achieve the above objectives, the present invention provides a method for estimating the three-dimensional pose of an animal, the method comprising:

[0007] Video data of laboratory animals was collected based on a multi-view shooting platform that was built.

[0008] Based on the experimental animal video data, a training dataset and a prediction dataset were constructed.

[0009] Perform frame filtering on the training dataset;

[0010] A three-dimensional animal pose estimation model based on keypoint errors is constructed. The animal three-dimensional pose estimation model includes a multi-view volumetric three-dimensional pose estimation network and a framework designed for different keypoint errors.

[0011] The training dataset and the prediction dataset after frame filtering are combined into a time block and input into the animal 3D pose estimation model for training. The loss during training is adjusted by weighting the framework designed for different key point errors.

[0012] The trained model was used to identify key points and estimate poses in experimental animals.

[0013] Preferably, the multi-view shooting platform includes multiple cameras, and the multiple cameras are calibrated using a checkerboard calibration method to obtain their intrinsic and extrinsic parameters;

[0014] The multi-view shooting platform used to collect experimental animal video data includes:

[0015] Video data of experimental animals is collected synchronously by setting up the cameras of the multi-view shooting platform.

[0016] Preferably, the step of constructing a training dataset and a prediction dataset based on the experimental animal video data includes:

[0017] The experimental animal video data was labeled with key points using a 3D labeling software package to construct a training dataset.

[0018] Construct the unlabeled dataset into a prediction dataset.

[0019] Preferably, the frame filtering process on the training dataset includes:

[0020] High-dimensional features are extracted from each frame of the training dataset using a deep learning pre-trained model, and the features of all frames are represented as a feature matrix.

[0021] use The -means clustering algorithm clusters the feature matrix to obtain K clusters;

[0022] Calculate the distance from each frame of the K clusters to the corresponding cluster center, and select the frame with the smallest distance as the keyframe;

[0023] Based on the selected keyframe index, the corresponding frames are extracted from the original video and saved as representative frames of the video.

[0024] Preferably, the multi-view volumetric 3D pose estimation network is a 3D convolutional neural network, and the framework designed for different keypoint errors is constructed based on L1 loss and temporal smoothness constraints.

[0025] The construction of the animal 3D pose estimation model based on keypoint errors includes:

[0026] Projective geometry is used to construct the 3D feature space for measurement;

[0027] The 3D convolutional neural network infers landmark locations using shared features across cameras and spatial statistics of learned animal poses;

[0028] The L1 loss is used to perform standard supervised pose regression loss on the labeled frame and to establish a time constraint expression for the time smoothness constraint between keypoint coordinates.

[0029] Preferably, the inference of landmark locations using shared features across cameras and learned spatial statistics of animal poses via the 3D convolutional neural network includes:

[0030] The centroid of the animal in each view was detected using a standard 2D U-Net, and the 3D centroid of the animal was inferred through triangulation.

[0031] Based on the positional relationship of multiple cameras and the 3D centroid of the animal, a 3D volumetric frame that can accommodate the entire animal is constructed. The 3D volumetric frame is then processed by the 3D convolutional neural network to predict the location of 3D landmarks.

[0032] Preferably, the step of using the trained model to perform keypoint identification and pose estimation on the experimental animal includes:

[0033] The trained model is used to predict key points for each frame of the experimental animal, and the animal's pose is estimated by using the relative positional changes of the key points.

[0034] Furthermore, to achieve the above objectives, the present invention also proposes an animal three-dimensional pose estimation device comprising:

[0035] The collection module is used to collect video data of laboratory animals based on the constructed multi-view shooting platform;

[0036] A construction module is used to construct a training dataset and a prediction dataset based on the experimental animal video data;

[0037] A filtering module is used to perform frame filtering processing on the training dataset;

[0038] The building module is also used to build an animal 3D pose estimation model based on keypoint errors. The animal 3D pose estimation model includes a multi-view volumetric 3D pose estimation network and a framework designed for different keypoint errors.

[0039] The training module is used to input the time block composed of the training dataset after frame filtering and the prediction dataset into the animal three-dimensional pose estimation model for training, and to adjust the weight of the loss during training using a framework designed for different key point errors.

[0040] The recognition module is used to identify key points and estimate poses of experimental animals using a trained model.

[0041] In addition, to achieve the above objectives, the present invention also proposes an animal three-dimensional pose estimation device, wherein the animal three-dimensional pose estimation device stores an animal three-dimensional pose estimation program, and when the animal three-dimensional pose estimation program is executed by a processor, it implements the steps of the animal three-dimensional pose estimation method described above.

[0042] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing an animal three-dimensional pose estimation program, which, when executed by a processor, implements the steps of the animal three-dimensional pose estimation method described above.

[0043] Compared with the prior art, the beneficial effects that the present invention can achieve are at least as follows:

[0044] (1) Targeted Error Correction: The method in this embodiment can focus on key point errors. In animal 3D posture estimation, different key points have different levels of importance for the accurate description of the posture. By analyzing key point errors, those key points that are estimated inaccurately can be accurately identified. For example, in the estimation of the walking posture of quadrupeds, the errors of the limb joints and the spine key points have a significant impact on the overall posture. Through error analysis, the estimation errors of these key parts are weighted and adjusted, which can more accurately restore the true 3D posture of the animal and reduce the deviation of the overall posture estimation.

[0045] (2) Adapting to Individual Differences: Different species of animals, and even different individuals of the same species, exhibit differences in body structure and movement patterns. This method can adjust weights based on the error in keypoint estimation for each individual animal. For example, dogs and mice have different limb proportions; when estimating posture in dogs and encountering errors in leg keypoints, the required weight adjustment strategy may differ from that when estimating the same body part in mice. Error analysis can help customize more suitable posture estimation models for different individuals, thereby improving the accuracy of posture estimation for various animals.

[0046] (3) Improved anti-interference capability: In real-world scenarios, animal pose estimation may be affected by various factors, such as changes in lighting, changes in animal surface features (e.g., changes in hair length and color), and occlusion of some body parts. By adjusting the weights through keypoint error analysis, the model can focus more on keypoints that are less affected by interference and are estimated more accurately, thus reducing the impact of interference factors on the overall pose estimation. For example, when one side of the animal's body is occluded by a shadow, by assigning higher weights to keypoints on the other side that are not occluded and are estimated more accurately, the model can overcome shadow interference to a certain extent and still estimate the animal's 3D pose relatively accurately.

[0047] (4) Enhanced generalization ability: This method helps the model work better in different datasets and scenarios. If the model has been trained with weight adjustment based on keypoint error analysis, it can adapt and adjust its pose estimation strategy more quickly when faced with new animal species, new movement patterns, or new environmental conditions. For example, when converting from an animal pose estimation dataset in a laboratory environment to a dataset in a field environment, the model can readjust the weights according to the keypoint error in the new dataset, thereby improving the accuracy of animal 3D pose estimation in complex field environments and enhancing the model's generalization ability. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the structure of the animal three-dimensional pose estimation method in the hardware operating environment involved in the embodiment of the present invention;

[0049] Figure 2 This is a flowchart illustrating the first embodiment of the animal three-dimensional pose estimation method of the present invention;

[0050] Figure 3 Image (a) is a schematic diagram of the multi-view imaging platform in an embodiment of the animal three-dimensional pose estimation method of the present invention. Figure 3 (b) in the figure is a schematic diagram of the shooting angle of the multi-view shooting platform;

[0051] Figure 4 This is a schematic diagram of the marked key point structure in an embodiment of the animal three-dimensional pose estimation method of the present invention;

[0052] Figure 5 This is an image of a camera view in an embodiment of the animal three-dimensional pose estimation method of the present invention;

[0053] Figure 6 This is a structural block diagram of the first embodiment of the animal three-dimensional pose estimation device of the present invention.

[0054] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0055] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0056] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of an animal three-dimensional pose estimation device in the hardware operating environment involved in the embodiments of the present invention.

[0057] like Figure 1 As shown, the animal 3D pose estimation device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen, and optionally, it may also include a standard wired interface or a wireless interface. In this invention, the wired interface of the user interface 1003 may be a USB interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a non-volatile memory (NVM), such as a disk storage device. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0058] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the animal three-dimensional pose estimation device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0059] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an animal three-dimensional pose estimation program.

[0060] exist Figure 1 In the animal 3D pose estimation device shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the user device; the animal 3D pose estimation device calls the animal 3D pose estimation program stored in the memory 1005 through the processor 1001 and executes the animal 3D pose estimation method provided in the embodiment of the present invention.

[0061] Based on the above hardware structure, an embodiment of the animal three-dimensional pose estimation method of the present invention is proposed.

[0062] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the animal three-dimensional pose estimation method of the present invention, which presents the first embodiment of the animal three-dimensional pose estimation method of the present invention.

[0063] In the first embodiment, the animal three-dimensional pose estimation method includes the following steps:

[0064] Step S10: Collect experimental animal video data based on the constructed multi-view shooting platform.

[0065] In a specific implementation, the execution subject of this embodiment is the animal three-dimensional pose estimation device, which can be an electronic device such as a personal computer or a server, and this embodiment does not limit it.

[0066] Furthermore, in this embodiment, the multi-view shooting platform includes multiple cameras, and the multiple cameras are calibrated using the checkerboard calibration method to obtain their intrinsic and extrinsic parameters;

[0067] Step S10 includes:

[0068] Video data of experimental animals is collected synchronously by setting up the cameras of the multi-view shooting platform.

[0069] It should be noted that the multiple cameras can be six or other numbers. In this embodiment, six cameras are used as an example for explanation. The multi-view shooting platform includes six cameras, and the multi-view shooting platform is as follows: Figure 3 As shown in (a) above, the shooting angle is as follows Figure 3 As shown in (b), the checkerboard calibration method is used to calibrate the six cameras to obtain their intrinsic and extrinsic parameters. The camera extrinsic parameter matrix is ​​represented as follows:

[0070] ;

[0071] in, and These are respectively the first relative to the standard coordinate system. The 3D rotation and translation matrices of each camera, and the internal geometric expression of each camera are:

[0072] ;

[0073] in and They are the first The focal length of each camera, and These are the coordinates of the camera's principal point. This is the skew parameter. Furthermore, animal video data is collected by setting up synchronous acquisition of cameras.

[0074] Step S20: Construct a training dataset and a prediction dataset based on the experimental animal video data.

[0075] It is understood that the training dataset is labeled data, and the prediction dataset is unlabeled data, with the prediction dataset containing significantly more data than the training dataset. The labeled keypoints consist of 22 keypoints, structured as follows: Figure 4 As shown, the markings are: left ear, right ear, nose, top of spine, middle of spine, top of tail, middle of tail, tail tip, left forepaw, left forearm, left forearm, left forearm, left forearm, left forearm, left forearm, right forearm, right forearm, right forearm, right forearm, left forearm, left forearm, left hind paw, left hind hock, left hind knee, right hind paw, right hind hock, and right hind knee. The marking method utilizes the existing 3D marking software Label3D, which can simultaneously display images from all camera views, such as... Figure 5 As shown, six views can be labeled simultaneously. During the labeling process, camera calibration parameters can be used to perform triangulation on the labels to obtain three-dimensional labeled data. In this embodiment, step S20 includes: using a 3D labeling software package to label key points in the experimental animal video data to construct a training dataset; and constructing a prediction dataset from the unlabeled dataset.

[0076] Step S30: Perform frame filtering processing on the training dataset.

[0077] It should be understood that the method used in the preprocessing of the training dataset for the keypoint annotation occlusion problem is to use deep learning feature embedding to select keyframes as the training set.

[0078] Further, in this embodiment, step S30 includes:

[0079] High-dimensional features are extracted from each frame of the training dataset using a deep learning pre-trained model, and the features of all frames are represented as a feature matrix.

[0080] use The -means clustering algorithm clusters the feature matrix to obtain K clusters;

[0081] Calculate the distance from each frame of the K clusters to the corresponding cluster center, and select the frame with the smallest distance as the keyframe;

[0082] Based on the selected keyframe index, the corresponding frames are extracted from the original video and saved as representative frames of the video.

[0083] In its implementation, frame filtering based on deep learning feature embedding is a process centered on extracting deep features from video frames and performing cluster analysis. This process involves the following steps: First, a pre-trained deep learning model is used to extract high-dimensional features from each frame. Assuming the video contains... Each frame, after being extracted by the model, yields a... 3D feature vectors. Represent the features of all frames as matrices:

[0084] ;

[0085] in, It is the first The feature vector of the frame. To select representative frames, the following method is used. -means clustering algorithm on the feature matrix Perform clustering. The goal is to... Frames are divided into For each cluster, minimize the sum of squared distances from each frame to its cluster center:

[0086] ;

[0087] in, It is the first The center of each cluster. After clustering, each cluster center represents the feature mean of that cluster. To select the frame that best represents the cluster center from each cluster, the distance from each frame to its cluster center can be calculated, and the frame with the smallest distance can be selected as the keyframe. Specifically, for each cluster... Keyframe Index for:

[0088] ;

[0089] in, It is a cluster This is a set of indices for all frames in the dataset. Finally, based on the selected keyframe indices... The corresponding frames are extracted from the original video and saved as representative frames. High-level semantic features of the frames are extracted using a deep learning model and combined with... -means clustering groups and filters frames. It can effectively remove redundant frames and retain important, representative key frames.

[0090] Step S40: Construct an animal 3D pose estimation model based on keypoint errors. The animal 3D pose estimation model includes a multi-view volumetric 3D pose estimation network and a framework designed for different keypoint errors.

[0091] It should be noted that the multi-view volumetric 3D pose estimation network is a relatively advanced 3D convolutional neural network. This network uses projective geometry to construct a metric 3D feature space that is robust to perspective changes. Then, the 3D convolutional neural network uses shared features across cameras and learned spatial statistics of animal poses to infer landmark locations. Furthermore, to control the size of the 3D feature space, the volume is focused on the inferred 3D centroid of the animal. The centroid of the animal in each view is detected using a standard 2D U-Net, and the 3D centroid is inferred through triangulation. Further, a 3D volumetric framework that can accommodate the entire animal is constructed using the mathematical relationships between camera positions in S1 and the information of the 3D centroid. These volumes are then processed by the 3D convolutional neural network to directly predict the 3D landmark locations.

[0092] The training framework designed for different keypoint errors is constructed based on L1 loss and temporal smoothness constraints.

[0093] The L1 loss is a standard supervised pose regression loss that applies only to the labeled frames. Given the ground truth and predicted 3D keypoint coordinates, the supervised regression loss is defined as follows:

[0094] ;

[0095] in represents the supervised regression loss, and represents the loss function of supervised learning, used to measure the difference between the predicted value and the actual value; This indicates the number of samples in the dataset. : No. The actual 3D keypoint coordinates of each sample. : No. The predicted 3D keypoint coordinates for each sample. In this formula, the supervised regression loss... It is calculated by determining the L1 distance between the actual and predicted keypoint coordinates for each sample. This is achieved by summing the losses of all samples and dividing by the number of samples. The average loss can be obtained, which is used to measure the accuracy of the model in predicting the coordinates of 3D key points.

[0096] The temporal smoothness constraint refers to the fact that at high frame rates, the animal's movement speed per frame is very low, and their overall movement trajectory should generally be smooth, rather than sudden or discontinuous. Here, the dataset is set to contain... The data of the frame, of which a small amount of labeled data is represented as , belong One of the frames in, where This indicates a specific frame that has been marked. This represents a key point in the tagged frame. Indicate key points The three-dimensional coordinates, for , , The time smoothness constraint between key point coordinates is established by setting the time constraint expression:

[0097] ;

[0098] In the formula, This is represented as a time constraint. , , They represent time. , , Key points The three-dimensional coordinates This represents the number of 3D keypoints. This expression considers not only the first-order changes in attitude (i.e., velocity) but also the second-order changes (i.e., acceleration). This ensures that the predicted attitude sequence is smooth not only in position but also in velocity and acceleration, satisfying smoothness constraints. The goal is to minimize the second-order changes in attitude, thereby making the predicted attitude sequence as smooth as possible in terms of velocity and acceleration.

[0099] The training framework designed for different keypoint errors is achieved by adjusting... and A framework for training using the weights of two loss functions, where the two loss functions combine to form a total loss function during training: ;

[0100] in, Indicates the total loss; Indicates loss The weights; Indicates loss The weights are dynamically adjusted based on the error at each keypoint during training. and This achieves the following objectives: for keypoints with large errors, more emphasis is placed on L1 loss to reduce positional errors; for keypoints with small errors, more emphasis is placed on temporal smoothness loss to improve temporal consistency. Weights are dynamically allocated based on the L1 error of each keypoint. Position error: Dynamically adjust weights:

[0101] , ;

[0102] in, Indicate key points The total loss. It is a balancing factor used to control the minimum weight of the smoothing loss. When When it is large, Approaching 1, while As the value approaches 0, the model focuses more on the L1 loss. When smaller, As the model grows, it pays more attention to temporal smoothness. In this embodiment, the multi-view volumetric 3D pose estimation network is a 3D convolutional neural network, and the framework designed for different keypoint errors is constructed based on L1 loss and temporal smoothness constraints.

[0103] The construction of the animal 3D pose estimation model based on keypoint errors includes:

[0104] Projective geometry is used to construct the 3D feature space for measurement;

[0105] The 3D convolutional neural network infers landmark locations using shared features across cameras and spatial statistics of learned animal poses;

[0106] The L1 loss is used to perform standard supervised pose regression loss on the labeled frame and to establish a time constraint expression for the time smoothness constraint between keypoint coordinates.

[0107] Furthermore, in this embodiment, the step of inferring landmark locations using shared features across cameras and learned spatial statistics of animal poses via the 3D convolutional neural network includes:

[0108] The centroid of the animal in each view was detected using a standard 2D U-Net, and the 3D centroid of the animal was inferred through triangulation.

[0109] Based on the positional relationship of multiple cameras and the 3D centroid of the animal, a 3D volumetric frame that can accommodate the entire animal is constructed. The 3D volumetric frame is then processed by the 3D convolutional neural network to predict the location of 3D landmarks.

[0110] Step S50: The training dataset and the prediction dataset after frame filtering are combined into a time block and input into the animal three-dimensional pose estimation model for training. The loss during training is adjusted by weighting using a framework designed for different keypoint errors.

[0111] It should be understood that the combination of labeled and unlabeled data in time blocks represents that each training batch contains one labeled sample and three unlabeled samples drawn from its local neighborhood, while also introducing additional temporally continuous unlabeled segments. These segments are used solely to compute the unsupervised temporal loss, helping the model achieve better smoothness and consistency in the temporal dimension. Figure 4 It shows and How to use the two loss and time block inputs.

[0112] Step S60: Use the trained model to identify key points and estimate pose of the experimental animals.

[0113] Understandably, the optimized model is used to predict keypoints for each frame, and the relative positional changes of these keypoints are used to analyze animal behavior. The 3D keypoint effect is as follows: Figure 5 As shown in the image, the top three images are taken from three different perspectives, while the bottom image shows the predicted 3D keypoint effect. Figure 5 In the middle (1) to (6), we can see the different postures of the animals.

[0114] This embodiment provides a novel training strategy that avoids errors caused by keypoint errors by setting different keypoint training weights based on pre-calculated keypoint annotation errors during the training phase. In free-moving animal experiments, this innovative method optimizes the performance of state-of-the-art multi-view volumetric 3D pose estimation and further enhances the stability of 3D keypoint tracking. The beneficial effects that can be achieved are at least as follows:

[0115] (1) Targeted Error Correction: The method in this embodiment can focus on key point errors. In animal 3D posture estimation, different key points have different levels of importance for the accurate description of the posture. By analyzing key point errors, those key points that are estimated inaccurately can be accurately identified. For example, in the estimation of the walking posture of quadrupeds, the errors of the limb joints and the spine key points have a significant impact on the overall posture. Through error analysis, the estimation errors of these key parts are weighted and adjusted, which can more accurately restore the true 3D posture of the animal and reduce the deviation of the overall posture estimation.

[0116] (2) Adapting to Individual Differences: Different species of animals, and even different individuals of the same species, exhibit differences in body structure and movement patterns. This method can adjust weights based on the error in keypoint estimation for each individual animal. For example, dogs and mice have different limb proportions; when estimating posture in dogs and encountering errors in leg keypoints, the required weight adjustment strategy may differ from that when estimating the same body part in mice. Error analysis can help customize more suitable posture estimation models for different individuals, thereby improving the accuracy of posture estimation for various animals.

[0117] (3) Improved anti-interference capability: In real-world scenarios, animal pose estimation may be affected by various factors, such as changes in lighting, changes in animal surface features (e.g., changes in hair length and color), and occlusion of some body parts. By adjusting the weights through keypoint error analysis, the model can focus more on keypoints that are less affected by interference and are estimated more accurately, thus reducing the impact of interference factors on the overall pose estimation. For example, when one side of the animal's body is occluded by a shadow, by assigning higher weights to keypoints on the other side that are not occluded and are estimated more accurately, the model can overcome shadow interference to a certain extent and still estimate the animal's 3D pose relatively accurately.

[0118] (4) Enhanced generalization ability: This method helps the model work better in different datasets and scenarios. If the model has been trained with weight adjustment based on keypoint error analysis, it can adapt and adjust its pose estimation strategy more quickly when faced with new animal species, new movement patterns, or new environmental conditions. For example, when converting from an animal pose estimation dataset in a laboratory environment to a dataset in a field environment, the model can readjust the weights according to the keypoint error in the new dataset, thereby improving the accuracy of animal 3D pose estimation in complex field environments and enhancing the model's generalization ability.

[0119] Furthermore, this embodiment of the invention also proposes a storage medium storing an animal three-dimensional pose estimation program, which, when executed by a processor, implements the steps of the animal three-dimensional pose estimation method described above.

[0120] In addition, refer to Figure 6 This invention also proposes an animal three-dimensional pose estimation device, which includes:

[0121] Collection module 10 is used to collect video data of experimental animals based on the constructed multi-view shooting platform;

[0122] Module 20 is used to construct a training dataset and a prediction dataset based on the experimental animal video data;

[0123] The filtering module 30 is used to perform frame filtering processing on the training dataset;

[0124] The construction module 20 is also used to construct an animal three-dimensional pose estimation model based on keypoint errors. The animal three-dimensional pose estimation model includes a multi-view volumetric three-dimensional pose estimation network and a framework designed for different keypoint errors.

[0125] Training module 40 is used to input the training dataset after frame filtering and the prediction dataset into the animal three-dimensional pose estimation model to train it, and to adjust the weight of the loss during training using a framework designed for different key point errors.

[0126] The recognition module 50 is used to identify key points and estimate poses of experimental animals using a trained model.

[0127] Other embodiments or specific implementations of the animal three-dimensional pose estimation device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0128] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0129] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the unit claims listing several devices, several of these devices may be embodied by the same hardware item. The use of the terms first, second, and third, etc., does not indicate any order and can be interpreted as identifiers.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as a magnetic disk or optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0131] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. An animal three-dimensional pose estimation method, characterized by, The animal three-dimensional posture estimation method comprises the following steps: Collecting experimental animal video data based on a built multi-view shooting platform; According to the experimental animal video data, a training data set and a prediction data set are constructed; Frame screening processing is performed on the training data set; An animal three-dimensional posture estimation model based on key point error is constructed, which comprises a multi-view volumetric three-dimensional posture estimation network and a framework designed for different key point errors; The frame-screened training data set and the prediction data set are input into the animal three-dimensional posture estimation model to perform training, and the framework designed for different key point errors is used to adjust the weight of the loss during training; The trained model is used to identify key points and estimate the posture of the experimental animal; Wherein, the experimental animal video data collected based on the built multi-view shooting platform comprises: Synchronously collecting experimental animal video data by setting up the camera of the multi-view shooting platform; The multi-view volumetric three-dimensional posture estimation network is a 3D convolutional neural network, and the framework designed for different key point errors is constructed according to L1 loss and time smoothness constraint; The construction of the animal three-dimensional posture estimation model based on key point error comprises: A 3D feature space is constructed by using projection geometry; The 3D convolutional neural network uses shared features across cameras and learned spatial statistics of animal poses to infer landmark positions; The L1 loss is used for standard supervised pose regression loss on labeled frames, and a time constraint expression is established for the time smoothness constraint between key point coordinates; Wherein, the L1 loss is used for standard supervised pose regression loss on labeled frames, and a time constraint expression is established for the time smoothness constraint between key point coordinates, comprising: The L1 loss is used for standard supervised pose regression loss on labeled frames, and the supervised regression loss is defined as follows: ； In the formula, This represents the loss function in supervised learning, used to measure the difference between predicted and actual values. Indicates the number of samples. For the first The actual 3D key point coordinates of each sample For the first Predicted 3D keypoint coordinates for each sample; Aiming at , , The time smoothness constraint between the key point coordinates of , , establishes a time constraint expression: ； wherein, denotes a time constraint, , , denote the three-dimensional coordinates of the key points at time , , , is the number of 3D key points;​ By adjusting the weights of the two losses and The two losses are trained, and the two loss functions form a total loss function during the training process: ; wherein, represents the total loss; represents the loss of weight; represents the loss of weight; For the position error of the key point , the weight is dynamically adjusted as follows: ​ ， ； wherein, represents the total loss of key points β is a balancing factor, β >

0.

2. The animal 3D pose estimation method of claim 1, wherein, The multi-view shooting platform comprises a plurality of cameras, and the plurality of cameras are calibrated by using a checkerboard calibration method to obtain intrinsic and extrinsic parameters thereof.

3. The animal 3D pose estimation method of claim 1, wherein, According to the experimental animal video data, a training data set and a prediction data set are constructed, comprising: Key points of the experimental animal video data are labeled by using a 3D labeling software package to construct a training data set; An unlabeled data set is constructed as a prediction data set.

4. The animal 3D pose estimation method of claim 1, wherein, The frame screening processing of the training data set comprises: A deep learning pre-training model is used to extract high-dimensional features from each frame of the training data set, and the features of all frames are represented as a feature matrix; using means clustering algorithm to cluster the feature matrix to obtain K clusters; The distance of each frame of the K clusters to the corresponding cluster center is calculated, and the frame with the smallest distance is selected as the key frame; According to the selected key frame index, the corresponding frame is extracted from the original video and saved as a representative frame of the video.

5. The animal 3D pose estimation method of claim 1, wherein, The 3D convolutional neural network uses shared features across cameras and learned spatial statistics of animal poses to infer landmark positions, comprising: The standard 2D U-Net detects the center of mass of the animal in each view, and the 3D center of mass of the animal is inferred by triangulation; According to the relationship between the multiple camera positions and the 3D centroid of the animal, a 3D volume framework capable of accommodating the whole animal is constructed, and the 3D volume framework is processed by the 3D convolutional neural network to predict the 3D landmark position.

6. The animal three-dimensional pose estimation method according to any one of claims 1 to 5, wherein The key point recognition and posture estimation of the experimental animal by using the trained model comprises: The key points of each frame of the experimental animal are predicted by using the trained model, and the posture of the animal is estimated by using the relative position change relationship of the key points.

7. An animal three-dimensional pose estimation device, characterized by, The animal three-dimensional posture estimation device comprises: The collection module is configured to collect experimental animal video data based on the built multi-view shooting platform; The construction module is configured to construct a training data set and a prediction data set according to the experimental animal video data; The screening module is configured to perform frame screening processing on the training data set; The construction module is further configured to construct an animal three-dimensional posture estimation model based on key point error, wherein the animal three-dimensional posture estimation model comprises a multi-view volume three-dimensional posture estimation network and a framework designed for different key point errors; The training module is configured to input the frame-screened training data set and the prediction data set into the animal three-dimensional posture estimation model for training in a time block, and to adjust the loss during training by using the framework designed for different key point errors; The recognition module is configured to perform key point recognition and posture estimation of the experimental animal by using the trained model; The collection of experimental animal video data based on the built multi-view shooting platform comprises: The camera of the multi-view shooting platform is used to synchronously collect experimental animal video data; The multi-view volume three-dimensional posture estimation network is a 3D convolutional neural network, and the framework designed for different key point errors is constructed according to L1 loss and time smoothness constraint; The construction of the animal three-dimensional posture estimation model based on key point error comprises: A 3D feature space is constructed by using projection geometry; The 3D convolutional neural network uses shared features across cameras and learned spatial statistics of animal poses to infer landmark positions; The L1 loss is used for standard supervised pose regression loss on labeled frames, and a time constraint expression is established for the time smoothness constraint between key point coordinates; The L1 loss is used for standard supervised pose regression loss on labeled frames, and a time constraint expression is established for the time smoothness constraint between key point coordinates, which comprises: The L1 loss is used for standard supervised pose regression loss on labeled frames, and the supervised regression loss is defined as follows: ； In the formula, This represents the loss function in supervised learning, used to measure the difference between predicted and actual values. Indicates the number of samples. For the first The actual 3D key point coordinates of each sample For the first Predicted 3D keypoint coordinates for each sample; Aiming at , , The time smoothness constraint between the key point coordinates of , , establishes a time constraint expression: ； wherein, denotes a time constraint, , , denote three-dimensional coordinates of key points at time , , , is the number of 3D key points;​ By adjusting and the weights of the two losses, the two loss functions form a total loss function during the training process: ; wherein, represents the total loss; represents the loss of weight; represents the loss of weight; For the position error of the key point , the weight is dynamically adjusted as follows: ​ ， ； wherein, represents the total loss of key points β is a balancing factor, β >

0.

8. An animal three-dimensional pose estimation device, comprising: The animal three-dimensional posture estimation device stores an animal three-dimensional posture estimation program, and the animal three-dimensional posture estimation program, when executed by a processor, implements the steps of the animal three-dimensional posture estimation method according to any one of claims 1 to 6.

9. A storage medium, characterized by The storage medium stores an animal three-dimensional posture estimation program, and the animal three-dimensional posture estimation program, when executed by a processor, implements the steps of the animal three-dimensional posture estimation method according to any one of claims 1 to 6.

Images