An animal behavior analysis method and system based on multi-view three-dimensional reconstruction and hidden space representation

Abstract

The application discloses an animal behavior analysis method and system based on multi-view three-dimensional reconstruction and hidden space representation, which comprises the following steps: acquiring multi-view synchronous video streams of experimental animals; introducing physiological constraints and geometric constraints in a preset image feature extraction network to construct an improved multi-view reconstruction model, and using the model to generate a high-fidelity three-dimensional grid with a continuous physical surface according to the multi-view synchronous video streams; after voxelizing the high-fidelity three-dimensional grid model, compressing the high-dimensional space model into a low-dimensional continuous hidden vector by using a three-dimensional variational autoencoder; and generating a high-dimensional behavior feature vector by time sequence feature aggregation to realize behavior analysis. The observation method of animal behavior is changed from the displacement of skeleton points to the time sequence evolution of hidden space vectors, the full biological information such as volume change and surface deformation is effectively captured, and the technical problems of serious information loss and high-frequency jitter of labeled data existing in the prior animal behavior analysis are solved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and biological behavior analysis technology, and in particular to an animal behavior analysis method and system based on multi-view three-dimensional reconstruction and latent space representation. Background Technology

[0002] Currently, the mainstream technical approach for automated animal behavior analysis is keypoint-based pose estimation. The workflow of this type of method (typical examples include DeepLabCut and SLEAP) typically involves: first, predefining a set of sparse body keypoints (such as the tip of the nose, auricle, limb joints, and tail root) according to experimental requirements; then, using deep neural networks to identify and track the two-dimensional or three-dimensional coordinates of these keypoints in video frames; finally, by analyzing the spatiotemporal trajectories of these discrete coordinate points, calculating kinematic parameters such as velocity and angle, thereby achieving the identification and classification of specific animal behaviors.

[0003] However, these methods have obvious technical drawbacks: First, the extremely simplified discrete sampling completely loses the volume and surface deformation information of the organism, and the errors introduced by manual annotation will lead to high-frequency jitter noise; Second, the model architecture is strongly bound to the human-defined key points, and once the experimental requirements change, the model input structure will change, making it unusable and with extremely poor scalability; Third, dense annotation is costly, and the end-to-end classification model lacks interpretability and cannot discover undefined new behaviors.

[0004] Therefore, it is necessary to propose an animal behavior analysis method and system based on multi-view 3D reconstruction and latent space representation to solve the problems of information loss and high-frequency jitter caused by existing technologies relying on discrete key point displacement analysis, rigid model architecture that is difficult to reuse, and the lack of physical interpretability and ability to discover unknown behaviors in deep learning models. Summary of the Invention

[0005] In view of this, the present invention provides an animal behavior analysis method and system based on multi-view 3D reconstruction and latent space representation, in order to solve the technical problems in existing animal behavior analysis methods, such as the loss of full biological information due to reliance on discrete key points, poor model reusability due to strong architecture binding, and lack of physical interpretability of deep classification models.

[0006] In a first aspect, the present invention provides a method for analyzing animal behavior based on multi-view three-dimensional reconstruction and latent space representation, including: Acquire multi-view synchronized video streams of laboratory animals; Physiological and geometric constraints are introduced into the preset image feature extraction network to construct an improved multi-view reconstruction model. The model is then used to analyze the multi-view synchronous video stream and output a high-fidelity 3D mesh model with continuous physical surfaces and accurate pose. After the high-fidelity 3D mesh model is voxelized, it is input into a 3D variational autoencoder, which converts the high-fidelity 3D mesh model into a continuous low-dimensional latent space vector. The contextual dynamic information of the low-dimensional latent space vector sequence is extracted using the temporal feature aggregation analysis method to generate a high-dimensional behavioral feature vector, and animal behavior analysis is performed based on the high-dimensional behavioral feature vector.

[0007] Furthermore, physiological constraints are introduced into the pre-defined image feature extraction network, including: A physiological constraint term is introduced into the loss function of the preset image feature extraction network; the physiological constraint term includes joint limit loss and self-collision penalty loss; The joint limit loss is used to penalize unnatural bending that violates the anatomical structure by using Hinge Loss, based on a preset Euler angle threshold for biological skeletons. The self-collision penalty loss is used to prevent interleaving of experimental animal limb meshes based on capsule collision detection, thus ensuring the rationality of the topology.

[0008] Furthermore, the method for calculating the joint limit loss includes: ; in, Indicates the ultimate loss of the joint. This represents the set of all joints in the skeletal topology. For joint angle, This represents the upper limit of the maximum allowable angle of bending of a joint, set based on prior biological knowledge. This represents the lower limit of the maximum allowable angle of bending of a joint, based on prior biological knowledge.

[0009] Furthermore, the method for calculating the self-collision penalty loss is as follows: in, Indicates the penalty loss for collision. These represent the geometric primitives bound to the skeletal model of the organism; This indicates the calculation of the shortest Euclidean distance between the axes of two geometric primitives; This is the preset collision threshold.

[0010] Furthermore, geometric constraints are introduced into the pre-defined image feature extraction network, including: A differentiable volume loss term is introduced into the loss function of the preset image feature extraction network. The differentiable volume loss term is used to calculate the volume of the closed surface at the vertices of the triangular mesh based on the divergence theorem. The volume consistency constraint is used to ensure that the reconstruction model maintains a constant experimental biological volume between different viewpoints and different video frames.

[0011] Furthermore, the encoder of the three-dimensional variational autoencoder includes a cascaded three-dimensional convolutional module and a dual-path parallel fully connected layer; The three-dimensional convolution module includes multiple feature extraction units, each of which includes a cascaded three-dimensional convolutional layer, a batch normalization layer, and a nonlinear activation layer. The dual-path parallel fully connected layer includes a mean mapping branch and a variance mapping branch, which are used to generate mean vectors and log-variance vectors that characterize the animal posture distribution features in parallel.

[0012] Furthermore, the low-dimensional latent space vector is obtained by reparameterizing and sampling the mean vector and the log-variance vector; The reparameterized sampling includes: randomly sampling auxiliary noise variables from a standard normal distribution; and obtaining a low-dimensional latent space vector based on the auxiliary noise variables, the mean vector, and the log-variance vector.

[0013] Furthermore, the step of extracting the contextual dynamic information of the low-dimensional latent space vector sequence using a temporal feature aggregation algorithm to generate a high-dimensional behavioral feature vector includes: Use a sliding window of preset length to extract a continuous-time low-dimensional latent space vector sequence; The low-dimensional latent space vector sequence is input into a temporal encoder, and the motion trend and posture evolution features within the sequence are aggregated through temporal convolution operations or attention mechanisms. The aggregated temporal features are mapped to a fixed-dimensional high-dimensional behavioral feature vector, which contains dynamic semantic information about the current time and its context.

[0014] Furthermore, animal behavior analysis is performed based on the high-dimensional behavioral feature vector, including: known behavior classification and unsupervised action clustering and phenotypic discovery; The known behavior classification is used to input the high-dimensional behavior feature vector into a pre-trained classifier, output the confidence probability of the current time period corresponding to the predefined standard behavior category, and generate a real-time behavior labeling sequence. The unsupervised action clustering and phenotypic discovery method is used to map the high-dimensional behavioral feature vectors to a low-dimensional manifold space using a manifold dimensionality reduction algorithm, construct an animal behavior map, and perform density clustering in the behavior map to define high-density clustered regions as independent behavioral modalities. Feature vectors that are far from known distribution clusters are marked as potential novel behaviors or abnormal phenotypes.

[0015] Secondly, this invention provides an animal behavior analysis system based on multi-view three-dimensional reconstruction and latent space representation, comprising: A multi-view video acquisition module is used to acquire synchronized video streams from multiple perspectives of laboratory animals. The 3D reconstruction module is used to introduce physiological and geometric constraints into a preset image feature extraction network, construct an improved multi-view reconstruction model, and use the model to analyze the multi-view synchronous video stream, outputting a high-fidelity 3D mesh model with continuous physical surfaces and accurate pose. The generative representation learning module is used to voxelize the high-fidelity 3D mesh model and then input it into the 3D variational autoencoder, which transforms the high-fidelity 3D mesh model into continuous low-dimensional latent space vectors. The behavioral analysis module is used to extract the contextual dynamic information of the low-dimensional latent space vector sequence using a temporal feature aggregation algorithm, generate a high-dimensional behavioral feature vector, and perform animal behavior analysis based on the high-dimensional behavioral feature vector.

[0016] Compared with existing technologies, the animal behavior analysis method and system based on multi-view three-dimensional reconstruction and latent space representation proposed in this invention have the following advantages: (1) This invention captures all biological information, including volume changes and muscle tremors, through a high-fidelity three-dimensional grid. By compressing the high-dimensional physical space into the latent space, the animal behavior is transformed into a continuous vector signal that evolves smoothly over time. This not only completely eliminates the high-frequency jitter noise caused by key point annotation, but also allows the use of continuous signal processing methods such as filtering to separate behavioral features of different frequencies, thereby achieving a more refined and interference-resistant quantitative evaluation than discrete point analysis.

[0017] (2) This invention constructs a general analysis architecture with high reusability and anti-occlusion capability, realizing high reusability and flexible decomposition of the model architecture. Since the input of physical modeling is always a normalized three-dimensional spatial field, whether the number of cameras increases or decreases, it only affects the accuracy of reconstruction without changing the input structure of the core model. This makes the encoder model and behavior vector library highly reusable, and with the accumulation of experimental data, a "behavior map" containing rich actions can be constructed. At the same time, this architecture supports decomposable analysis, and researchers can directly extract the three-dimensional mesh features of specific parts (such as the tail) and feed them into the encoder, so as to achieve independent research on local behavior without retraining.

[0018] (3) This invention solves the problems of labeling difficulties and black boxes in existing technologies. By adopting an end-to-end self-supervised or weakly supervised learning paradigm, only basic segmentation masks are needed for training, completely eliminating the dependence on expensive manual labeling of dense key points and significantly reducing experimental preparation costs. In addition, this invention utilizes the generative characteristics of the VAE architecture to endow the model with strong interpretability: the difference between two behavior vectors in the latent space can be reversed by the decoder to restore the morphological difference in the three-dimensional physical space. This allows researchers to intuitively see which part of the body caused the change based on their knowledge of the behavior change, thereby achieving a true combination of quantitative and qualitative analysis. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the animal behavior analysis method based on multi-view 3D reconstruction and latent space representation provided by the present invention. Figure 2 This is a schematic diagram of the algorithm execution flow provided by the present invention; Figure 3 This is a schematic diagram of the structure of the animal behavior analysis system based on multi-view three-dimensional reconstruction and latent space representation provided by the present invention. Detailed Implementation

[0020] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0021] Example 1

[0022] Please see Figure 1 This embodiment provides a method for analyzing animal behavior based on multi-view 3D reconstruction and latent space representation, including: Step S101: Acquire multi-view synchronized video streams of experimental animals; Step S102: Introduce physiological and geometric constraints into the preset image feature extraction network to construct an improved multi-view reconstruction model, and use the model to analyze the multi-view synchronous video stream to output a high-fidelity three-dimensional mesh model with continuous physical surfaces and accurate pose. Step S103: After voxelizing the high-fidelity 3D mesh model, input it into the 3D variational autoencoder, and convert the high-fidelity 3D mesh model into a continuous low-dimensional latent space vector through the 3D variational autoencoder. Step S104: Use the temporal feature aggregation algorithm to extract the contextual dynamic information of the low-dimensional latent space vector sequence, generate a high-dimensional behavioral feature vector, and perform animal behavior analysis based on the high-dimensional behavioral feature vector.

[0023] The method in this embodiment first acquires multi-view video streams of experimental animals and reconstructs a high-fidelity 3D mesh. Second, the high-dimensional mesh data is input into a 3D variational autoencoder (3D VAE) and mapped into continuous low-dimensional latent space vectors. Finally, by calculating the difference between two latent space vectors, it is inversely decoded to restore a 3D deformation field in physical space, thereby intuitively locating the specific body parts causing behavioral differences. This method transforms the definition of animal behavior from the traditional "displacement of skeletal points" to "the temporal evolution of latent space vectors," thus capturing comprehensive biological information, including volume changes and surface deformations. This effectively solves the problems of severe information loss and high-frequency jitter in labeled data in existing technologies.

[0024] As a specific embodiment, in step S101, this method uses multiple rigorously calibrated cameras (acquiring internal parameters K and external parameters R, T) to surround the experimental observation area and acquire multi-view synchronous video streams of experimental animals (in this embodiment, mice are selected as experimental animals).

[0025] As a preferred embodiment, in step S102, unlike traditional single-view reconstruction schemes, this scheme designs an improved MoReMouse reconstruction model. This model not only utilizes an image feature extraction network to extract semantic information from multi-view (non-traditional single-view) images, but also introduces physiological constraints and volume conservation mechanisms to ensure that the generated mesh topology conforms to the biological anatomical skeleton of mice during the mapping process from 2D images to 3D meshes. This guarantees that the total volume of the mouse does not undergo unnatural mutations between consecutive frames. Through this process, a high-fidelity 3D mesh model with background interference removed, continuous physical surfaces, and accurate pose is output, effectively solving the problems of occlusion loss and point jitter in traditional keypoint detection.

[0026] Specifically, in this embodiment, the preset image feature extraction network is the DinoV2 network. Besides using the features extracted by DinoV2 to calculate the semantic consistency loss... ) and photometric reconstruction loss ( In addition, this method also specifically introduces physiological and geometric constraint terms into the loss function of the model: The physiological constraints include joint limit loss and self-collision penalty loss; the geometric constraints include differentiable volume loss. The joint limit loss is used to penalize unnatural bending that violates the anatomical structure by using Hinge Loss, based on a preset Euler angle threshold for biological skeletons. The self-collision penalty loss is used to prevent limb mesh interpenetration based on capsule collision detection, ensuring the rationality of the topology; The differentiable volume loss term is used to calculate the volume of a closed surface at the vertices of a triangular mesh based on the divergence theorem. The volume consistency constraint ensures that the reconstructed model maintains a constant experimental biological volume across different viewpoints and different video frames.

[0027] Furthermore, the joint limit loss The calculation methods include: in, This represents the set of all joints in the skeletal topology. For joint angle, This represents the upper limit of the maximum allowable angle of bending of a joint, set based on prior biological knowledge. This represents the lower limit of the maximum allowable angle of bending of a joint, based on prior biological knowledge.

[0028] The self-collision loss function The calculation methods include: in, These represent geometric primitives bound to the mouse skeletal model (capsule bodies are used in this embodiment). This indicates the calculation of the shortest Euclidean distance between the axes of two geometric primitives; The preset collision threshold (i.e., the sum of the radii of the two capsules) By applying a penalty-based repulsive force field in the formula, when the distance between non-adjacent geometric primitives is detected to be less than a threshold... At that time, a backpropagation signal of the quadratic gradient is generated, which forces the skeleton pose parameters. The solution space is updated to a collision-free solution space, thus effectively avoiding the phenomenon of limbs intersecting.

[0029] The differentiable volume loss ( Based on the divergence theorem, the volume of a closed surface is calculated directly at the vertices of the triangular mesh. The formula for calculating the surface volume is as follows: in, This represents the overall physical volume of the closed triangular mesh surface of the 3D reconstructed model. This represents the three-dimensional coordinate vectors of the three vertices of any discrete triangular face on the surface of a three-dimensional mesh in the spatial coordinate system. This represents the global summation of the directed volume of the tetrahedron formed by all discrete triangular facets on the mesh surface, based on the divergence theorem.

[0030] By setting volume consistency constraints, the reconstructed model maintains a constant biological mass across different viewpoints and frames, thus resolving the "flattening" problem caused by depth blur in single-view scenarios. The differentiable volume loss ( The formula for calculating ) is: in, and These represent the first and second digits in the time series, respectively. Frame (currently calculated frame) and the The overall volume of the 3D reconstructed mesh output from the previous calculation frame. The overall optimization objective (global total loss function) of the reconstruction model is expressed as: in, Hyperparameters are used to balance the weights of each task.

[0031] In a preferred embodiment, in step S103, after obtaining the high-fidelity three-dimensional mesh, in order to extract semantic features from the high-dimensional geometric data and solve the problem of quantization analysis, the present invention performs voxelization processing on the high-fidelity three-dimensional mesh model and inputs it into a three-dimensional variational autoencoder. The three-dimensional variational autoencoder transforms the high-fidelity three-dimensional mesh model into a continuous low-dimensional latent space vector, compressing and mapping the complex, high-dimensional biophysical morphological features into a smooth and continuous low-dimensional latent space vector, thereby eliminating high-frequency noise caused by discrete annotation and realizing full quantization and refined characterization and analysis of animal behavioral features.

[0032] As a specific embodiment, the workflow of the 3D Variational Autoencoder (3D VAE) is as follows: Step 1, Data Preprocessing and Voxelization Input: First, the 3D mesh generated in step S102 is voxelized to transform it into a regular shape. (For example The three-dimensional binary occupied grid or distance field matrix X is used as the standard input of the encoder.

[0033] The second step involves using an encoder to compress the high-dimensional voxel data into low-dimensional distributed features. The encoder structure consists of multiple layers of 3D convolutional blocks. Each 3D convolutional block comprises cascaded 3D convolutional layers, batch normalization layers, and nonlinear activation layers. 3D convolutional layers: using The convolutional kernel extracts spatial geometric features, and downsampling is performed using a stride of 2 to reduce dimensionality; Batch normalization layer: used to accelerate convergence and prevent overfitting.

[0034] Nonlinear activation layer: used to introduce nonlinear feature maps.

[0035] After L layers (e.g., 4-5 layers) of convolution processing, the original volume data is compressed into a flattened high-dimensional feature vector.

[0036] The third step involves latent space mapping and reparameterization. To construct a continuous and sampleable latent space, this invention does not directly map a single vector, but instead learns the probability distribution of the latent variables. The encoder output is mapped to the mean vector of the latent space distribution through two parallel fully connected layers. Sum of logarithmic variance vector .

[0037] Furthermore, to address the issue of non-differentiability in the sampling process, this invention introduces a reparameterization technique: sampling an auxiliary noise variable from the standard normal distribution N(0, I). The latent space vector is calculated. The vector z (e.g., 128-dimensional or 256-dimensional) is a compact representation that encapsulates the mouse's overall posture, orientation, and deformation characteristics at the current moment.

[0038] In one specific embodiment, the 3D variational autoencoder further includes a decoder for performing the inverse process of the encoder to verify the completeness of the features. The decoder's structure consists of multiple layers of 3D deconvolution modules. Its working principle is as follows: The decoder receives the latent vector z, maps it back to the initial volume dimension through a fully connected layer, and then gradually restores the spatial resolution through layer-by-layer upsampling and convolution operations, finally outputting a reconstructed 3D voxel or mesh model. .

[0039] During the training phase, the model performs end-to-end optimization by minimizing the weighted sum of reconstruction error and KL divergence, thereby forcing the latent space vector z to exhibit continuous and smooth normal distribution characteristics.

[0040] To address the pain points of end-to-end models being "black boxes" that are uninterpretable and have poor reusability, this invention calculates the difference between two latent space vectors and decodes them inversely to restore them to a three-dimensional deformation field in physical space, thereby intuitively locating the specific body parts that cause behavioral differences (such as showing "the difference between action A and B lies in the amplitude of tail swing" through a heat map).

[0041] It should be noted that in some embodiments, the latent space can also be obtained through a GANInversion scheme based on generative adversarial networks (GANs). Specifically, a GAN that generates a high-quality 3D mouse model is trained, and during analysis, the input video frames are mapped back to the latent space of the GAN using inversion technology.

[0042] In a preferred embodiment, after obtaining a continuous sequence of latent space vectors in step S104, the behavioral analysis stage begins.

[0043] Since latent vectors at a single time point only represent instantaneous pose and are insufficient to describe complete action semantics, this invention introduces a temporal feature aggregation analysis method. First, a sliding window technique is used to extract a latent vector sequence of length T. And then, a high-dimensional behavioral feature vector containing contextual dynamic information is extracted through a temporal encoder (such as Bi-LSTM or Transformer Encoder). Based on this high-dimensional behavioral feature, the following two types of downstream tasks are executed in parallel: The first category is known behavior classification tasks, mainly targeting defined standard actions (such as grooming, standing upright, and walking). Behavioral feature vectors are input into a fully connected classification layer, which outputs a confidence probability distribution for each behavior. This distribution is used to generate real-time behavior-labeled sequences.

[0044] The second type of task is action clustering and phenotypic discovery: to uncover undefined or subtle subtype behaviors, a coupled "dimensionality reduction-clustering" strategy is used to generate behavioral maps. Specifically: Using t-SNE or UMAP algorithms, high-dimensional behavioral feature vectors are mapped to a two-dimensional plane to form a visualized behavioral map. Points with similar spatial distances in the map represent similar action patterns. In the reduced-dimensional space or the original high-dimensional space, density clustering algorithms (such as HDBSCAN or Leiden algorithm) are applied to automatically segment data points, and high-density clustered regions in the map are defined as independent behavioral modes. The classification results are mapped onto a behavior map. Clusters overlapping with known behaviors are labeled as standard behaviors; independent clusters deviating from known distributions are labeled as novel behaviors or anomalous phenotypes. Combining classification statistics and cluster topology, a detailed phenotypic analysis report is finally generated, including behavior frequency, transition probability, and features of the novel phenotype.

[0045] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the method described in this example. This method, through an analysis path of physical modeling, latent space compression, and temporal analysis, achieves a new generation of animal behavior analysis that requires no intensive manual annotation, possesses physical interpretability, and exhibits highly reusable models.

[0046] Example 2

[0047] like Figure 3 As shown, this embodiment of the invention also provides an animal behavior analysis system 300 based on multi-view three-dimensional reconstruction and latent space representation, comprising: The multi-view video acquisition module 301 is used to acquire multi-view synchronous video streams of laboratory animals. The 3D reconstruction module 302 is used to introduce physiological and geometric constraints into a preset image feature extraction network, construct an improved multi-view reconstruction model, and use the model to analyze the multi-view synchronous video stream, outputting a high-fidelity 3D mesh model with continuous physical surfaces and accurate pose. Generative representation learning module 303 is used to process the high-fidelity three-dimensional mesh model into voxels and then input it into a three-dimensional variational autoencoder. The three-dimensional variational autoencoder transforms the high-fidelity three-dimensional mesh model into a continuous low-dimensional latent space vector. The behavioral analysis module 304 is used to extract the contextual dynamic information of the low-dimensional latent space vector sequence using a temporal feature aggregation algorithm, generate a high-dimensional behavioral feature vector, and perform animal behavior analysis based on the high-dimensional behavioral feature vector.

[0048] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for analyzing animal behavior based on multi-view 3D reconstruction and latent space representation, characterized in that, include: Acquire multi-view synchronized video streams of laboratory animals; Physiological and geometric constraints are introduced into the preset image feature extraction network to construct an improved multi-view reconstruction model. The model is then used to analyze the multi-view synchronous video stream and output a high-fidelity 3D mesh model with continuous physical surfaces and accurate pose. After the high-fidelity 3D mesh model is voxelized, it is input into a 3D variational autoencoder, which converts the high-fidelity 3D mesh model into a continuous low-dimensional latent space vector. The contextual dynamic information of the low-dimensional latent space vector sequence is extracted using the temporal feature aggregation analysis method to generate a high-dimensional behavioral feature vector, and animal behavior analysis is performed based on the high-dimensional behavioral feature vector.

2. The method according to claim 1, characterized in that, Physiological constraints are introduced into the pre-defined image feature extraction network, including: A physiological constraint term is introduced into the loss function of the preset image feature extraction network; the physiological constraint term includes joint limit loss and self-collision penalty loss; The joint limit loss is used to penalize unnatural bending that violates the anatomical structure by using Hinge Loss, based on a preset Euler angle threshold for biological skeletons. The self-collision penalty loss is used to prevent interleaving of experimental animal limb meshes based on capsule collision detection, thus ensuring the rationality of the topology.

3. The method according to claim 2, characterized in that, The method for calculating the joint limit loss includes: ； in, Indicates the ultimate loss of the joint. This represents the set of all joints in the skeletal topology. For joint angle, This represents the upper limit of the maximum allowable angle of bending of a joint, set based on prior biological knowledge. This represents the lower limit of the maximum allowable angle of bending of a joint, based on prior biological knowledge.

4. The method according to claim 2, characterized in that, The method for calculating the self-collision penalty loss is as follows: in, Indicates the penalty loss for collision. These represent the geometric primitives bound to the skeletal model of the organism; This indicates the calculation of the shortest Euclidean distance between the axes of two geometric primitives; This is the preset collision threshold.

5. The method according to any one of claims 1, characterized in that, Geometric constraints are introduced into the pre-defined image feature extraction network, including: A differentiable volume loss term is introduced into the loss function of the preset image feature extraction network. The differentiable volume loss term is used to calculate the volume of the closed surface at the vertices of the triangular mesh based on the divergence theorem. The volume consistency constraint is used to ensure that the reconstruction model maintains a constant experimental biological volume between different viewpoints and different video frames.

6. The method according to claim 1, characterized in that, The encoder of the three-dimensional variational autoencoder includes a cascaded three-dimensional convolutional module and a dual-path parallel fully connected layer. The three-dimensional convolution module includes multiple feature extraction units, each of which includes a cascaded three-dimensional convolutional layer, a batch normalization layer, and a nonlinear activation layer. The dual-path parallel fully connected layer includes a mean mapping branch and a variance mapping branch, which are used to generate mean vectors and log-variance vectors that characterize the animal posture distribution features in parallel.

7. The method according to claim 6, characterized in that, The low-dimensional latent space vector is obtained by reparameterizing the mean vector and the log-variance vector; The reparameterized sampling includes: randomly sampling auxiliary noise variables from a standard normal distribution; and obtaining a low-dimensional latent space vector based on the auxiliary noise variables, the mean vector, and the log-variance vector.

8. The method according to claim 1, characterized in that, The step of extracting contextual dynamic information from the low-dimensional latent space vector sequence using a temporal feature aggregation algorithm to generate a high-dimensional behavioral feature vector includes: Use a sliding window of preset length to extract a continuous-time low-dimensional latent space vector sequence; The low-dimensional latent space vector sequence is input into a temporal encoder, and the motion trend and posture evolution features within the sequence are aggregated through temporal convolution operations or attention mechanisms. The aggregated temporal features are mapped to a fixed-dimensional high-dimensional behavioral feature vector, which contains dynamic semantic information about the current time and its context.

9. The method according to claim 1, characterized in that, Animal behavior analysis is performed based on the high-dimensional behavioral feature vector, including: known behavior classification and unsupervised action clustering and phenotypic discovery; The known behavior classification is used to input the high-dimensional behavior feature vector into a pre-trained classifier, output the confidence probability of the current time period corresponding to the predefined standard behavior category, and generate a real-time behavior labeling sequence. The unsupervised action clustering and phenotypic discovery method is used to map the high-dimensional behavioral feature vectors to a low-dimensional manifold space using a manifold dimensionality reduction algorithm, construct an animal behavior map, and perform density clustering in the behavior map to define high-density clustered regions as independent behavioral modalities. Feature vectors that are far from known distribution clusters are marked as potential novel behaviors or abnormal phenotypes.

10. An animal behavior analysis system based on multi-view three-dimensional reconstruction and latent space representation, characterized in that, include: A multi-view video acquisition module is used to acquire synchronized video streams from multiple perspectives of laboratory animals. The 3D reconstruction module is used to introduce physiological and geometric constraints into a preset image feature extraction network, construct an improved multi-view reconstruction model, and use the model to analyze the multi-view synchronous video stream, outputting a high-fidelity 3D mesh model with continuous physical surfaces and accurate pose. The generative representation learning module is used to process the high-fidelity 3D mesh model into voxels and then input it into the 3D variational autoencoder. The 3D variational autoencoder converts the high-fidelity 3D mesh model into continuous low-dimensional latent space vectors. The behavioral analysis module is used to extract the contextual dynamic information of the low-dimensional latent space vector sequence using a temporal feature aggregation algorithm, generate a high-dimensional behavioral feature vector, and perform animal behavior analysis based on the high-dimensional behavioral feature vector.

Images