Animal surface motion capturing method based on micro rendering

Abstract

The invention discloses an animal surface motion capture method based on micro-rendering, which comprises the steps of high-precision animal grid template construction based on CT scanning, multi-view synchronous animal motion image acquisition, supervision information extraction based on deep learning and grid registration based on micro-rendering. And calculating a grid in an animal motion state to realize animal surface motion capture. According to the method, the three-dimensional grid in the animal motion state can be obtained, the motion state, the surface shape and the color texture of the animal can be reflected, and compared with a traditional animal behavior capture algorithm, the behavior information of the animal is described more accurately and detailedly, and powerful support is provided for animal behavior analysis.

Description

Technical Field

[0001] The invention belongs to the field of computational biology, and in particular relates to an animal surface motion capture method based on microscopic rendering. Background Art

[0002] Accurate perception of animal behavior is of great significance in basic science and engineering applications. In basic science, accurate, intelligent and efficient methods of animal motion perception can help researchers easily obtain individual and social behaviors of animals, provide new research ideas for solving important scientific problems in disciplines such as neuroscience, animal behavior and bionics, and help reveal the neural mechanisms of animal communication, mate selection and other behaviors. In engineering applications, in-depth research on animal motion behavior will help to achieve complex animal motion behavior regulation and promote the construction of a new generation of animal robots with simple system structure, superior motion ability, optimized system energy consumption and excellent concealment performance. It has broad application prospects in many fields such as scientific research, search and rescue in dangerous environments, meteorological observation, and industrial production.

[0003] With the development of computer vision and machine learning technology, animal behavior perception technology has shown a trend of development from artificial to intelligent, from single modality to multi-modality, from plane to three-dimensional, from low precision to high precision, from sparse representation to dense representation, and from single individual to multi-individual interaction. The current mainstream methods focus on the estimation of two-dimensional and three-dimensional key points in the motion state of animals. The representative method for two-dimensional key point estimation is DeepLabCut, which is based on deep neural networks and realizes the accurate prediction of the position of animal body key points in the image. The representative method for three-dimensional key point estimation is Anipose, which uses multi-camera triangulation and spatiotemporal filtering on the basis of DeepLabCut to effectively obtain the position of animal body key points in three-dimensional space.

[0004] However, key points are a sparse representation of the animal's body. Simplifying the animal's body into a number of key points loses important information such as the animal's body surface shape, color texture, and motion state. As a dense representation of the animal's body, the three-dimensional grid effectively makes up for the defects of the key point representation and can reflect the animal's information more finely. Because the three-dimensional grid has a more complex data form than the key points, and the animal has rapid surface deformation in motion, and there are complex situations such as occlusion and motion blur, it is difficult for existing methods to capture the surface of animals in motion. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to propose an animal surface motion capture method based on micro-rendering, including: (1) high-precision animal mesh template construction based on CT scanning, (2) followed by multi-view synchronous animal motion image acquisition, (3) followed by supervised information extraction based on deep learning, and (4) finally mesh registration based on micro-rendering. This method can reconstruct the surface mesh of animals in motion, capture the surface shape, color texture and motion state of animals, and describe the behavior information of animals more accurately and in detail compared with traditional animal behavior capture methods, providing strong support for animal behavior analysis.

[0006] The method for constructing a high-precision animal mesh template based on CT scanning in step (1) uses a high-precision CT scanner to scan animal specimens and construct the scanning results into a three-dimensional mesh template. The mesh template finely depicts the shape of the animal's body, provides a powerful prior for capturing the surface of animals in motion, and improves the authenticity of the surface capture effect. A bumblebee mesh template obtained based on this method is as follows Figure 2 As shown; a built-in skeleton that conforms to the animal's anatomical structure is implanted in the mesh template to obtain a vertex-bone binding matrix, and three-dimensional key points are implanted in the template to establish a vertex-key point mapping matrix to drive mesh deformation.

[0007] The multi-perspective synchronized animal motion image acquisition method of step (2) is based on an animal image acquisition system composed of multiple high-speed industrial cameras, a synchronization signal generator, and an animal experimental platform. In the experiment, the animal is placed on the experimental platform, and multiple high-speed cameras are placed around the experimental platform to capture images of the animal's movement process from different perspectives, providing animal motion information from different perspectives. The synchronized shooting of multiple cameras is controlled by a synchronization signal generator to ensure frame synchronization at each perspective. The multi-perspective images obtained by the acquisition record the animal's behavior information from multiple different angles, and effective information complementarity can be formed between the various perspectives, providing a full and comprehensive description of the animal's motion state. An example of the system is as follows Figure 4 shown.

[0008] The supervised information extraction method based on deep learning in step (3) includes: 2D key point extraction based on DeepLabCut, contour mask extraction based on SimpleClick, and optical flow extraction based on Raft. The 2D posture tool DeepLabCut based on deep learning is used to perform key point detection on the acquired image to obtain the 2D pixel coordinates of the key points under each viewing angle, providing supervision for the position of the key points of the animal body. Figure 4An example of key point supervision obtained by this method is given; the SimpleClick segmentation tool is used to segment the acquired images. This method inputs the key points obtained by DeepLabCut into the SAM model as prompts. The SAM model implements image segmentation based on the prompts and obtains the pixel area where the animal is located in each image, that is, the contour mask of the animal. Figure 5 An example of the contour mask obtained by the above method is given; the Raft optical flow estimation model is used to extract the optical flow map between two adjacent frames under each viewing angle, indicating the dense pixel correspondence between the two frames of images, providing supervision for the direction of animal movement, Figure 6 An example of an optical flow map obtained by the above method is given. The extracted key points, contour mask map, and optical flow will be used as supervision information for the mesh registration algorithm based on differentiable rendering.

[0009] The mesh registration algorithm based on differentiable rendering in step (4) includes the following steps: first, a series of learnable mesh deformation parameters are introduced to characterize the difference between the real animal and the template animal; then, a non-rigid mesh deformation method is used to transform the template mesh through the effect of parameters to obtain a predicted mesh; finally, a loss function combining data loss, regularization loss and timing loss is used to calculate the difference between the predicted mesh and the real motion state, and the parameters are updated using the gradient descent method, thereby realizing the capture of the animal surface in motion. The flowchart of the algorithm is as follows: Figure 7 shown.

[0010] The learnable mesh deformation parameters include {β, s, γ, φ, f, θ}. β represents the local scaling factor, which allows each bone to control the three-degree-of-freedom scaling of the surface mesh around it through the vertex-bone binding matrix, thereby depicting the proportional differences between different parts of the body of the real animal and the template animal. s represents the global scaling factor, which is used to control the overall size scaling of the template. γ and φ represent displacement parameters, which are respectively used to represent the translation and rotation of the entire model in the world coordinate system, thereby capturing the overall movement of the animal. f represents the color texture parameter, which is represented by the RGB color value corresponding to each mesh vertex and is used to reflect the color texture of the animal's body surface. θ represents the posture parameter, where each element θ b Represents the rotation angle of bone b relative to its parent bone in the kinematic tree, where the rotation angle of the root bone represents the global rotation.

[0011] The non-rigid mesh deformation method introduces a local scaling factor β into the traditional linear blend skinning algorithm, so that the built-in skeleton can not only control the mesh to perform rigid posture transformation, but also control the mesh to perform non-rigid local scaling, thereby capturing the shape difference between the real animal and the template mesh. The mesh deformation function is denoted as D. Similar to the linear blend skinning algorithm, the calculation process of this function involves two transformation matrices: the world-local transformation matrix A corresponding to each bone b in the initial state b and the world-local transformation matrix T that controls the mesh deformation b To calculate A b , starting from bone b, traverse its ancestors in the kinematic tree from bottom to top until the root bone. During the traversal process, merge the transformation matrices of each bone relative to its father bone to obtain A b , the process is formulated as: in and represents the rotation angle and starting position of bone b in its father's coordinate system in the initial state, R(.) represents the function of converting a certain rotation angle into a rotation matrix, and Ψ(b) represents all ancestors of bone b. b In the same traversal process, we need to consider the influence of local scaling, a non-rigid transformation. When calculating, we first consider the translation effect caused by the local scaling of the father ψ(b) of bone b, and get T b ′ Then consider the scaling effect of the local scaling of bone b itself (given by the scaling matrix Indicates), we get T b :

[0012] The coordinates of vertex i on the mesh in the world coordinate system in the initial state are First use (A b ) -1 Transform it to the local coordinate system of bone b, and then use T b Get its coordinates in the world coordinate system after being transformed by bone b. If the total number of bones is B, the above operation can get B different coordinates for each vertex. Use the vertex-bone binding weight to perform weighted summation of these coordinates to get the coordinate V of the vertex in the world coordinate system. i ‘ Then, the global scaling, translation, and rotation represented by s,γ,φ are applied to the network V i ‘ , we get the final coordinate V of the vertex in the world coordinate system i , the mathematical form of this process is: V i =sR φ V' i +γ

[0013] The loss function contains three different losses, namely: data loss E data , regularization loss E reg And the timing loss E temp . These three different losses use actual observation data, prior regularization constraints, and temporal supervision as learning targets, respectively, to achieve alignment between the model and the real behavior of the animal and complete the surface capture of the animal in motion. The data items use the key points, contour masks, and original color images obtained by the supervised information extraction method based on deep learning as supervision, and the expression is (in the following formula, T represents the total number of video frames recorded, and t represents each frame in the video):

[0014] It contains three items: E kp , E sil , E rgb And their corresponding weights λ kp ,λ sil ,λ rgb . These three items represent the key point loss term, the contour mask loss term, and the image color loss term respectively. The key point loss term encourages the matching between the model key points and the actual key points, thereby promoting the model to capture the true movement state of the animal. The specific calculation method of the key point loss term is to map the model vertices after the mesh deformation function D into three-dimensional key point coordinates through the vertex-key point mapping matrix, and project the three-dimensional key points to each perspective through the camera projection function to obtain the two-dimensional key point coordinates generated by the model. And calculate its two-dimensional key point supervision y under each perspective (t) The contour mask loss term encourages the overlap between the model contour and the actually observed animal contour, thereby facilitating the model to capture the surface shape of the animal. The specific calculation method of the contour mask loss term is to use a differentiable renderer to render the model after the mesh deformation function D into a binary contour mask at each viewing angle. And calculate its correlation with the supervised contour mask map S (t) The image color loss term encourages the model’s color to match the color in the actual observed image, thereby facilitating the model to capture the surface color texture of the animal. The specific calculation method of the image color loss term is to use a differentiable renderer to render the model and its surface texture f after the mesh deformation function D into color images at different viewing angles. And calculate its difference with the real color image I (t) The pixel color error between .

[0015] Regularization loss introduces a priori constraints to restrict posture and shape parameters, encouraging the model to learn more realistic and reasonable parameters. Its expression is:

[0016] E reg =λ β1 E β1 +λ β2 E β2 +λ θ E θ It contains three items, namely, shape symmetry constraint E β1 , shape specification constraint E β2 and the attitude specification constraint E θ And its corresponding weight λ β1 ,λ β2 ,λ θ The shape symmetry constraint encourages anatomically symmetrical body parts to have the same local scaling coefficients by penalizing differences in the local scaling coefficients of symmetric parts. The shape norm constraint restricts the shape parameters to vary within an allowed range by penalizing shape parameters that are outside the allowed range. Similar to the shape norm constraint, the pose norm constraint encourages poses to vary within an allowed range and penalizes pose parameters that are outside the allowed range.

[0017] The timing loss introduces optical flow, an important timing information, to enhance the robustness of the method when key point supervision fails. In actual experiments, due to factors such as the rapid movement of animals and environmental occlusion, there are often a large number of errors or omissions in key point supervision, which seriously affects the surface motion capture effect. Optical flow supervision can provide timing information, indicating the dense pixel correspondence between two frames of images, effectively reflecting the direction of animal movement between the two frames, and compensating for the impact of missing key points. The expression of timing loss is:

[0018] Where E mor is the multi-view optical flow loss, λ mor The specific calculation method is to first calculate the displacement vector between the vertex projection of the t+1 frame and the vertex projection of the t frame at each viewing angle to form the optical flow feature of each vertex, and then input the mesh and optical flow features into the differentiable renderer to render the optical flow map at each viewing angle. Combine it with the supervised optical flow map O (t) A pixel-wise loss is computed to encourage the model to account for surface motion between frames consistent with the optical flow supervision.

[0019] By utilizing the gradients generated during the loss function calculation process, the values ​​of the learnable parameters can be updated using gradient descent and continuously iterated to achieve alignment from the template grid to the real animal, thereby achieving surface capture of animals in motion.

[0020] The present invention has the beneficial effects: (1) The present invention can reconstruct the three-dimensional grid of the animal in motion, reflect the animal's motion state, surface shape and color texture, accurately and in detail describe the animal's behavior information, and provide strong support for fine animal behavior analysis; (2) The present invention is robust and can effectively deal with environmental occlusion, key point detection errors, and other situations that occur in animal experiments, and is helpful for realizing animal behavior perception in complex experimental scenarios; (3) The present invention is versatile and can be applied to a variety of animal experiments, including bumblebees, fruit fly larvae, and mice. There is no need to design specific algorithms for different animals, and it can adapt to a variety of experimental needs in the field of biomedical research. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the implementation steps of this method.

[0022] Figure 2 High-precision animal mesh templates constructed for CT scans.

[0023] Figure 3 It is a multi-view synchronous animal image acquisition system for bumblebees.

[0024] Figure 4 Keypoint supervision obtained for deep learning based supervised information extraction methods.

[0025] Figure 5 Contour mask supervision obtained for deep learning based supervised information extraction methods.

[0026] Figure 6 Optical flow supervision obtained for deep learning based supervised information extraction methods.

[0027] Figure 7 Flowchart of the mesh registration method based on differentiable rendering.

[0028] Figure 8 Definition of Bumblebee key points used in the implementation case.

[0029] Fig. 9 The bumblebee surface motion capture effect in the implementation case. DETAILED DESCRIPTION

[0030] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0031] Example 1: Using the method of the present invention to capture the surface motion of a bumblebee in motion

[0032] According to step (1) shown in the accompanying figure of the abstract, a high-precision bumblebee mesh template is first constructed based on CT scanning. The three-dimensional mesh model that finely depicts the shape of the animal's body is as follows: Figure 2 As shown in the figure, the final mesh template is obtained. According to the anatomical structure of the bumblebee body, a built-in skeleton is implanted in the mesh template. According to the connection relationship between the bones, a skeleton kinematic tree is constructed, and a vertex-bone binding matrix is ​​obtained to realize skeleton-driven mesh deformation; three-dimensional key points are implanted in the template, and according to the relative position relationship between each key point and the mesh surface vertices, a vertex-key point mapping matrix is ​​established to map the mesh vertices to the three-dimensional coordinates of the key points.

[0033] According to step (2) shown in the abstract figure, the following is constructed: Figure 3 The multi-view bumblebee motion image acquisition system shown in the figure is equipped with 6 high-speed industrial cameras (Hikrobot MV-CS016-10UC array camera, Hikrobot MVL-HF2524M-10MP lens). In the experiment, a synchronization signal generator is used to control all cameras to shoot synchronously by external triggering. The shooting frame rate is set to 200 frames and the image resolution is 800*800. Before placing the experimental bumblebee, multiple cameras are calibrated first, and the parameters of each camera are obtained using the multi-view calibration method based on Anipose. After the calibration is completed, the head of the experimental bumblebee is fixed on the air pump suspension device, and an LED display is placed in front of it. The pattern changes on the LED screen stimulate it to move forward, backward, turn, etc. At the same time, its movement process is synchronously recorded from 6 different perspectives, and the recorded video is saved in the host computer.

[0034] According to step (3) shown in the attached figure of the abstract, a supervised information extraction method based on deep learning is used to extract information from the multi-view images. 46 key points on the body of the bumblebee are defined in advance, and the key points are defined as follows: Figure 8As shown in the figure, there are 6 key points on each leg: coxa-trochanter, trochanter-femur, femur-tibia, tibia-basal tarsus, basal tibia-distal tarsus, and tarsal claw; 3 key points on each tentacle: head-antenna, tentacle node, and tentacle end; and 4 key points on the trunk: mouth, head-thorax, thorax-abdomen, and tail. A total of 18,000 frames of experimental data were collected, from which a continuous 2,000-frame segment was selected as the dataset for mesh reconstruction. The supervised information extraction method based on deep learning described above is used to extract supervisory information from the video. In order to obtain the 2D key point supervision corresponding to each image, the pre-trained DeepLabCut model is used to estimate the posture of the collected images. A total of 46 key points are detected on each image. The results are shown in Figure 2. Figure 4 To obtain the contour mask data corresponding to each image, the key points are input into SimpleClick, and the image is automatically segmented using the key points as prompts to obtain the mask corresponding to the bumblebee in the image. The result is shown in Figure 5 To obtain optical flow supervision, the Raft algorithm is used to calculate the optical flow map between two adjacent frames. The result is as follows Figure 6 shown.

[0035] According to step (4) shown in the abstract figure, the surface motion of the bumblebee is captured using a grid registration algorithm based on differentiable rendering. The algorithm flow is as follows: Figure 7As shown in the figure. The algorithm is implemented based on PyTorch and PyTorch3D frameworks. The objective function introduced above is used to learn parameters in combination with data loss, regularization loss and temporal loss. The data loss uses the extracted key points, contour masks and color images in the bumblebee motion image to reconstruct the motion state, surface shape and color texture of the bumblebee. The regularization loss constrains the shape parameters and posture parameters of the bumblebee to improve the rationality of parameter learning. The temporal loss introduces multi-view optical flow supervision to enhance the stability of the algorithm when key point detection errors occur. In order to balance the impact of each item in the objective function on the final learning result, the corresponding weight coefficients of each item need to be adjusted. In this experiment, the weight values ​​of each item determined after actual debugging are shown in Table 1. When optimizing parameters, the Adam optimizer in PyTorch is used. The parameters are updated by gradient descent. The initial learning rate of the optimizer is set to 0.001. 200 iterations are performed per frame. The cosine annealing learning rate decay strategy is used to gradually reduce the learning rate during the iteration process. The final learning rate is 0.00001. After the parameter learning process is completed, the reconstruction effect is evaluated using the 3D key point Euclidean distance indicator and the contour mask overlap IOU indicator. 50 frames of the total 2000 frames were manually annotated as a test set. The average value of the 3D key point Euclidean distance error obtained by this method on the test set is less than 5% of the bee's body length, and the contour overlap IOU is 0.801. This result proves that the surface motion capture result of this method has extremely high accuracy. The mesh reconstruction result on one of the frames is shown in Figure 1. Fig. 9 shown.

[0036] Table 1 <![CDATA[λ kp ]]> <![CDATA[λ sil ]]> <![CDATA[λ rgb ]]> <![CDATA[λ β1 ]]> <![CDATA[λ β2 ]]> <![CDATA[λ θ ]]> <![CDATA[λ mor ]]> 10.0 5.0 1.0 0.3 0.5 0.5 1.0

Claims

1. A method for capturing animal surface motion based on differentiable rendering, characterized in that: This is achieved through the following steps: (1) construction of a high-precision animal mesh template based on CT scans, (2) followed by multi-view synchronous animal motion image acquisition, (3) followed by supervised information extraction based on deep learning, and (4) finally mesh alignment based on differentiable rendering.

2. The method for capturing animal surface motion based on differentiable rendering according to claim 1, characterized in that: The step (1) is a high-precision animal mesh template construction method based on CT scanning. It uses CT scanning of animal specimens to reconstruct a mesh template as a shape prior for capturing the surface of the animal in motion, and implants bones and key points into the template to obtain a vertex-bone binding matrix and a key point-vertex mapping matrix as a drive for template deformation.

3. The method for capturing animal surface motion based on micro-rendering according to claim 1, characterized in that: The step (2) uses a multi-view synchronous image acquisition system to capture images of animals in motion from multiple camera positions, providing animal motion information from different viewpoints.

4. The method for capturing animal surface motion based on differentiable rendering according to claim 1, characterized in that: The step (3) is based on a supervised information extraction method based on deep learning. The deep learning method is used to extract two-dimensional key points, contour mask maps and optical flows from the collected multi-view images as supervised information used in the subsequent grid registration method based on differentiable rendering.

5. The method for capturing animal surface motion based on differentiable rendering according to claim 1, characterized in that: The step (4) is based on a mesh registration method for differentiable rendering, firstly introducing a series of learnable mesh deformation parameters to depict the difference between the real animal and the template animal; Then, a non-rigid mesh deformation method is used to transform the template mesh into a predicted mesh through the action of parameters; finally, a loss function combining data loss, regularization loss and timing loss is used to calculate the difference between the predicted mesh and the actual animal behavior, and the parameters are updated using the gradient descent method to achieve surface motion capture.

6. The method for capturing animal surface motion based on differentiable rendering according to claim 5, characterized in that: The learnable mesh deformation parameters include {β, s, γ, φ, f, θ}, where β represents a local scaling factor, which is used to describe the difference in proportions of different body parts between the real animal and the template; s represents a global scaling factor, which is used to control the overall size scaling of the template; γ and φ represent displacement parameters, which are used to represent the translation and rotation of the template in the world coordinate system; f represents the color texture parameter, which is used to reflect the color texture of the animal's body surface; θ represents the posture parameter, which is used to control the posture of the model.

7. The method for capturing animal surface motion based on differentiable rendering according to claim 5, characterized in that: The non-rigid mesh deformation method introduces a local scaling factor β in the linear blend skinning algorithm, and controls the local scaling of the surrounding mesh vertices through bones, so as to capture the difference in shape between the real animal and the template mesh.

8. The method for capturing animal surface motion based on differentiable rendering according to claim 5, characterized in that: The loss function includes data loss, regularization loss and timing loss. The data loss uses key points, contour masks and color image data as supervision to achieve alignment between the model and actual observation data; the regularization loss introduces shape symmetry constraints, shape normalization constraints and posture normalization constraints to limit parameters within a realistic and reasonable range; the timing loss introduces multi-view optical flow information to improve the robustness of the method in the case of key point errors or omissions.

Images