Method for three-dimensional reconstruction of animals based on body shape priors

Abstract

The application discloses an animal three-dimensional reconstruction method based on body shape prior, relates to the technical field of computer vision, and screens optimal video frames through pose self-consistency scoring; independently trims the screened frames and extracts foreground masks; unconditional multi-view geometric estimation and parameterized body shape estimation are performed in parallel to obtain camera parameters, a rough point cloud and an SMAL grid; the mask centroid is used to correct the weak perspective camera parameters, SMAL vertexes are back-projected to the camera space and multi-view depth prior diagrams are rendered; the image, the depth prior and the camera parameters are input into a conditional reconstruction model, and the SMAL parameters are encoded into multiple tokens of semantic decomposition, an original Gaussian point cloud is generated through token preposition and feature linear modulation double-path injection guidance, foreground point clouds are obtained through four-stage cascade filtering, and the final three-dimensional Gaussian reconstruction result is obtained through white bottom synthesis photometric loss and mask loss optimization after initialization, thereby realizing full-automatic reconstruction from a single image to a high-quality animal three-dimensional model.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a method for three-dimensional reconstruction of animals based on body shape priors. Background Technology

[0002] 3D reconstruction from a single image is an important topic in the field of computer vision, with broad application prospects in augmented reality, digital entertainment, e-commerce displays, and other areas. However, for non-rigid objects such as animals, 3D reconstruction from a single image still faces significant challenges due to their varied poses, complex textures, and the lack of accurate template models.

[0003] In existing technologies, video generation and multi-view geometry estimation methods based on diffusion models have made progress. For example, video diffusion models such as Wan2.2 can generate orbital videos around objects from a single image, while models such as HunyuanWorld-Mirror can estimate camera poses from multiple images and output 3D Gaussian point clouds. However, these methods have the following shortcomings: First, the geometric quality of the generated orbital videos is inconsistent, lacking an automatic filtering mechanism; poor-quality videos can lead to complete reconstruction failure. Second, the parametric animal body shape model uses weak perspective projection, while the multi-view geometry estimation model uses a perspective camera system; the two coordinate systems are incompatible and cannot work directly together. Third, existing conditional reconstruction models only support the injection of low-level geometric priors such as depth and pose, and do not support parametric body shape priors. Even with token pre-processing, its guiding effect is relatively weak, a problem pointed out in a public GitHub issue. Fourth, the generated Gaussian point clouds contain a large amount of background noise, making it difficult for traditional post-processing to cleanly separate the foreground, affecting the final rendering quality. Summary of the Invention

[0004] The purpose of this invention is to provide a method for three-dimensional reconstruction of animals based on body shape priors, in order to solve the above-mentioned problems.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a method for three-dimensional reconstruction of animals based on body shape priors, comprising the following steps:

[0006] S1: Obtain a single animal reference image, extract structured animal descriptions, and combine multiple random seeds and prompt word configurations to generate multiple candidate track videos;

[0007] S2: For each candidate track video, run the multi-view geometric estimation model by sampling multiple frame subsets to obtain the relative pose of the first and last frames of each subset, calculate the self-consistency score between the relative poses, select the candidate video with the best score and output its corresponding frames.

[0008] S3: Perform animal detection, square cropping, and foreground mask extraction independently on each frame output from S2 to obtain a multi-view preprocessed image;

[0009] S4: Perform unconditional multi-view geometric estimation on the multi-view preprocessed image to obtain the camera pose, camera intrinsic parameters and coarse point cloud for each view; at the same time, run the parameterized animal body size estimation model on the first frame image to obtain the SMAL mesh and weak perspective camera parameters.

[0010] S5: Correct the weak perspective camera parameters using the foreground mask centroid, backproject the corrected SMAL vertices from the weak perspective projection space to the first frame camera space, then use the first frame camera pose transformation to the world coordinate system, and render the SMAL depth prior map for each viewpoint.

[0011] S6: The multi-view preprocessed image, SMAL depth prior map, camera pose and camera intrinsic parameters are input conditions to form a multi-view geometric estimation model. At the same time, the SMAL parameters are encoded into multiple prior tokens of semantic decomposition. The model is guided to generate the original three-dimensional Gaussian point cloud through a dual-path injection method of token pre-positioning and feature linear modulation.

[0012] S7: Perform opacity threshold filtering, multi-view mask projection filtering, statistical outlier removal and SMAL mesh proximity filtering on the original 3D Gaussian point cloud in sequence to obtain the foreground Gaussian point cloud.

[0013] S8: Initialize the Gaussian center using the foreground Gaussian point cloud, and obtain the final 3D Gaussian reconstruction result by jointly optimizing the photometric loss and mask loss of the white background synthesis.

[0014] Furthermore, the calculation of the self-consistency score in S2 specifically includes:

[0015] For each candidate track video, construct M random frame subsets and 1 uniformly anchored frame subset. Each subset contains the first frame and K-1 frames sampled from the remaining frames.

[0016] Run the multi-view geometric estimation model for each subset to obtain the camera pose of each frame, and calculate the relative pose between the first and last frames.

[0017] Calculate the pose distance between all pairs of relative poses, where the pose distance is the sum of the rotation geodesic distance and the translational angular distance;

[0018] The subset with the smallest sum of distances to other subsets is selected as the medoid subset. The average distance between the medoid subset and all random subsets is calculated as the medoid distance. The relative pose distance between the medoid subset and the uniformly anchored subset is calculated as the anchor distance. The weighted sum of the medoid distance and the anchor distance is used as the final score.

[0019] Furthermore, the step S5, which involves correcting the weak perspective camera parameters using the foreground mask centroid, specifically includes:

[0020] Scale the foreground mask of the first frame to the input size of the parametric animal size estimation model and calculate the centroid of the foreground pixels;

[0021] Calculate the bounding box center of the SMAL mesh projection based on the weak perspective camera parameters output by the parametric animal body size estimation model.

[0022] Transform the mask centroid to normalized coordinates, calculate the deviation between the mask centroid and the center of the projected bounding box, and then add this deviation to the weak perspective translation parameters.

[0023] Furthermore, in step S5, when backprojecting the corrected SMAL vertex from the weak perspective projection space to the first frame camera space, the z-coordinate of the SMAL vertex in the weak perspective model is directly used as the depth value. The camera intrinsic parameters of the first frame and the corrected weak perspective projection coordinates are used for backprojection to obtain the three-dimensional point in the camera coordinate system.

[0024] Furthermore, in S6, encoding the SMAL parameters into multiple prior tokens for semantic decomposition specifically includes:

[0025] Encode the SMAL shape parameters as shape tokens, the SMAL trunk joint rotation parameters as trunk tokens, and the SMAL limb joint rotation parameters as limb tokens;

[0026] Each token is mapped to a unified embedding dimension through an independent multilayer perceptron network;

[0027] The term "Token fronting" refers to inserting three prior tokens into the token sequence of the visual geometric transformer of the conditional multi-view geometric estimation model, before the image patch token.

[0028] The aforementioned feature linear modulation refers to the process of average pooling three prior tokens, generating scaling and translation parameters through linear projection, and then performing channel-level modulation on the sum of the image patch token and depth token.

[0029] Furthermore, S6 also includes a two-stage fine-tuning of the prior injection module:

[0030] The first stage freezes the parameterized volume prior encoder and trains only the feature linear modulation layer. The loss function includes depth loss and RGB loss.

[0031] The second stage jointly trains the encoder, the feature linear modulation layer and a fidelity prediction head. The loss function includes RGB loss, depth loss, mask loss and fidelity loss. The fidelity loss is the L2 distance between the SMAL parameters output by the fidelity prediction head and the real SMAL parameters.

[0032] In the second stage, the injected prior token and the feature linear modulation parameter are set to zero with a predetermined probability, but the fidelity loss is always calculated using the real SMAL parameters.

[0033] Furthermore, the SMAL mesh proximity filtering in S7 specifically includes:

[0034] Using SMAL depth prior maps from various viewpoints, 3D points corresponding to non-zero depth pixels are extracted from the coarse point cloud output by unconditional multi-view geometric estimation to form a proxy point cloud.

[0035] Calculate the diagonal length of the axis-aligned bounding box of the proxy point cloud, and use the diagonal length multiplied by a predetermined ratio as the distance threshold;

[0036] For each Gaussian point, calculate its distance to the nearest neighbor in the proxy point cloud. If the distance exceeds the distance threshold, discard the Gaussian point.

[0037] Furthermore, the initialization of the Gaussian center using the foreground Gaussian point cloud in S8 specifically includes:

[0038] From the coarse point cloud output by the unconditional multi-view geometric estimation, select 3D points that simultaneously satisfy the conditions of non-zero SMAL depth prior map and true foreground mask in each view to form the initial point set;

[0039] If the number of initial points exceeds the preset upper limit, random downsampling will be performed.

[0040] Calculate the average k-nearest neighbor distance for each initial point, and take half of the average distance of all points as the initial scale.

[0041] Furthermore, the photometric loss in the white background synthesis in S8 is specifically as follows:

[0042] The composite rendered image is obtained by multiplying the rendered image pixel by pixel with the rendered alpha image and adding a white background.

[0043] The supervised image is multiplied pixel by pixel with the foreground mask and then a white background is added to obtain the synthesized supervised image;

[0044] The weighted sum of L1 loss and single-scale SSIM loss between the synthetic rendered image and the synthetic supervised image is calculated as the photometric loss.

[0045] Furthermore, after selecting the candidate video with the best score in S2, the frames corresponding to the medoid subset of that video are output as input for the subsequent S3.

[0046] Compared with existing technologies, the animal three-dimensional reconstruction method based on body shape prior provided by this invention has the following beneficial effects:

[0047] (I) Regarding video selection, a method for selecting the geometric quality of generated videos based on pose self-consistency is proposed. By constructing a random frame subset and a uniformly anchored frame subset, the geodesic rotation distance and translational angular distance between the relative poses of the first and last frames of each subset are calculated. The weighted sum of the medoid distance and the anchoring distance is used as the scoring index to automatically select the video with the most stable geometric structure. The self-consistency concept of InterPose is transferred from two-image interpolated videos to single-image track videos. The introduction of a uniformly anchored subset can identify the failure mode of "consistent random subsets but global deviation". High-quality input can be selected from multiple candidate videos without manual intervention, significantly improving the success rate and robustness of subsequent reconstruction.

[0048] (II) Regarding cross-model collaboration, a weak perspective-perspective camera bridging and SMAL depth prior generation method are employed. The foreground mask centroid is used as a shared invariant between the two coordinate systems to correct the AniMer weak perspective translation parameters, aligning the SMAL projection with the foreground. Based on this, the inherent z-coordinates of the SMAL vertices are directly used as shape depth priors, participating in C2W matrix backprojection within the first frame camera and rendering multi-view depth maps. This avoids explicitly solving the transformation between the two coordinate systems and eliminates the need to re-estimate depth from the point cloud, fundamentally resolving the scale and coordinate inconsistency problem between the weak perspective model and the perspective model. This enables the previously incompatible parametric shape model and multi-view geometric estimation model to work seamlessly together, providing a high-value depth prior for subsequent conditional reconstruction.

[0049] (III) Regarding prior injection, a dual-path injection mechanism and a phased fine-tuning strategy for parameterized body shape priors are proposed. SMAL parameters are decomposed into three sets of tokens based on anatomical semantics: shape, torso, and limbs. After independent MLP encoding, these are injected into the visual geometric transformer simultaneously through two paths: token pre-positioning and feature linear modulation. For the injection module, a two-stage fine-tuning using FiLM-first is employed: the first stage trains only the FiLM layer to avoid noise adaptation when the encoder has not converged; the second stage jointly trains the encoder, FiLM, and fidelity prediction head, and introduces a PreserveLoss and PriorDropout collaborative mechanism to ensure effective gradient propagation and robustness of the model to the presence or absence of priors. This significantly enhances the guiding ability of parameterized priors on the reconstruction model, solves the problem of weak influence of existing token pre-positioning methods, and generates Gaussian point clouds that better match the true morphology of animals.

[0050] (iv) In terms of point cloud post-processing, a four-stage cascaded foreground filtering and SMAL prior-guided initialization method is adopted. Building upon the first three stages, a fourth stage, SMAL mesh proximity filtering, is added: surrogate point clouds are extracted from the coarse point cloud using the SMAL depth prior map, and Gaussian points that deviate too far from the animal surface are discarded using the proportion of the AABB diagonal length as an adaptive distance threshold. During the initialization stage, initial points are collected using the intersection of non-zero SMAL depth and a true foreground mask, and the initial scale is estimated based on the k-nearest neighbor average distance. Compared to traditional filtering methods that only use opacity and SOR, this method effectively removes isolated noise and floating points far from the subject, ensuring that all initial points fall on the animal surface with appropriate density, significantly improving the fidelity and visual effect of the final reconstruction.

[0051] (v) Regarding loss optimization, a photometric loss function with a white background is proposed. The rendered image and alpha image are composited onto a white background, and the supervised image and foreground mask are also composited onto a white background. Then, the weighted sum of L1 and SSIM is calculated as the photometric loss, which is combined with the mask loss for joint optimization. This unifies the background processing methods for rendering and supervision, allowing the photometric loss to penalize both color and opacity errors, avoiding interference from inconsistent backgrounds on training, resulting in more stable optimization convergence and sharper reconstructed edges. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0053] Figure 1 This is the overall flowchart of the present invention. Detailed Implementation

[0054] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0055] Please see Figure 1 A method for 3D animal reconstruction based on body size priors includes the following steps:

[0056] S1: Obtain a single animal reference image, extract structured animal descriptions, and combine multiple random seeds and prompt words to generate multiple candidate track videos;

[0057] S2: For each candidate track video, run the multi-view geometric estimation model by sampling multiple frame subsets to obtain the relative pose of the first and last frames of each subset, calculate the self-consistency score between the relative poses, select the candidate video with the best score and output its corresponding frames.

[0058] S3: Perform animal detection, square cropping, and foreground mask extraction independently on each frame output from S2 to obtain a multi-view preprocessed image;

[0059] S4: Perform unconditional multi-view geometric estimation on the multi-view preprocessed images to obtain the camera pose, camera intrinsic parameters and coarse point cloud for each view; at the same time, run the parameterized animal body size estimation model on the first frame image to obtain the SMAL mesh and weak perspective camera parameters.

[0060] S5: Use the foreground mask centroid to correct the weak perspective camera parameters, backproject the corrected SMAL vertices from the weak perspective projection space to the first frame camera space, then use the first frame camera pose transformation to the world coordinate system, and render the SMAL depth prior map of each view.

[0061] S6: The multi-view geometric estimation model is formed by inputting multi-view preprocessed images, SMAL depth prior maps, camera pose and camera intrinsic parameters. At the same time, the SMAL parameters are encoded into multiple prior tokens of semantic decomposition. The model is guided to generate the original 3D Gaussian point cloud through a dual-path injection method of token pre-positioning and feature linear modulation.

[0062] S7: Perform opacity threshold filtering, multi-view mask projection filtering, statistical outlier removal and SMAL mesh proximity filtering on the original 3D Gaussian point cloud to obtain the foreground Gaussian point cloud.

[0063] S8: Initialize the Gaussian center using the foreground Gaussian point cloud, and obtain the final 3D Gaussian reconstruction result by jointly optimizing the photometric loss and mask loss of the white background synthesis.

[0064] First, to obtain a sufficient number of candidate viewpoints, a large model is used to understand the content of the photographs, and then 12 short videos surrounding the animal are generated using different random seeds, each containing 81 frames. However, the geometric quality of these videos varies. S2 estimates the consistency of camera motion by repeatedly extracting several frames from the videos, selecting the videos with the most stable motion to ensure the basic quality of subsequent reconstruction. Next, each selected frame is independently cropped to ensure the animal remains centered. Subsequently, two different models work in parallel: one estimates the camera position and a coarse 3D point cloud, while the other infers the parameterized 3D mesh of the animal from the first frame. Since the coordinate systems used by the two models are incompatible, step S5 innovatively aligns the SMAL mesh to the camera coordinate system and renders a "depth prior map" for each viewpoint—this tells the computer the relative distances of different parts of the animal, like a rough 3D template. S6 inputs the image, depth prior map, and SMAL parameters into the reconstruction model. The SMAL parameters are encoded as three semantic tokens, and the model is forcibly guided to generate a more accurate 3D Gaussian point cloud through two paths: "pre-token" and "feature modulation." This point cloud contains a large amount of background noise. S7 filters it layer by layer through four layers to retain only the points on the animal's surface. Finally, S8 initializes the Gaussian sphere with these clean points, and iteratively optimizes by comparing the differences between the rendered image and the real photo to obtain the final 3D animal model that can be viewed from any angle. This method achieves fully automatic reconstruction from a single photo to a high-quality 3D animal, and can be applied to fields such as games, VR, and e-commerce displays.

[0065] The calculation of the self-consistency score in S2 specifically includes:

[0066] For each candidate track video, construct M random frame subsets and 1 uniformly anchored frame subset. Each subset contains the first frame and K-1 frames sampled from the remaining frames.

[0067] Run the multi-view geometric estimation model for each subset to obtain the camera pose of each frame, and calculate the relative pose between the first and last frames.

[0068] Calculate the pose distance between all pairs of relative poses. The pose distance is the sum of the rotation geodesic distance and the translation angular distance.

[0069] The subset with the smallest sum of distances to other subsets is selected as the medoid subset. The average distance between the medoid subset and all random subsets is calculated as the medoid distance. The relative pose distance between the medoid subset and the uniformly anchored subset is calculated as the anchor distance. The weighted sum of the medoid distance and the anchor distance is used as the final score.

[0070] If several frames extracted from a video clip at different times can yield the same camera motion trajectory, it indicates that the video's 3D geometry is stable. In practice, for each video, 10 random samples are taken, selecting the first frame and four other frames each time; additionally, a subset with uniform intervals is selected. For each subset, the camera pose of each frame is estimated using an existing model, and the relative motion matrix between the first and last frames within the subset is calculated. The distance between the two relative motion matrices consists of two parts: the rotation angle difference and the angle between the translation directions. For example, the formula for the rotation angle difference is...

[0071]

[0072] The distances between all pairs of subsets are arranged in a table. The "most central" subset, with the smallest average distance to other random subsets, is denoted as the medoid distance. Simultaneously, its distance to the uniform subset is denoted as the anchoring distance. The two distances are added together to obtain the final score; a lower score indicates better geometric consistency of the video. Compared to the original InterPose method, this approach innovatively incorporates uniform anchoring subsets, which can identify false positives where "random subsets are consistent with each other but deviate from the true trajectory as a whole." It automatically selects the video with the highest geometric quality from 12 candidate videos, avoiding manual selection and significantly improving the reconstruction success rate.

[0073] The S5 method for correcting weak perspective camera parameters using the foreground mask centroid specifically includes:

[0074] Scale the foreground mask of the first frame to the input size of the parametric animal size estimation model and calculate the centroid of the foreground pixels;

[0075] Calculate the bounding box center of the SMAL mesh projection based on the weak perspective camera parameters output by the parametric animal body size estimation model.

[0076] Transform the mask centroid to normalized coordinates, calculate the deviation between the mask centroid and the center of the projected bounding box, and then add this deviation to the weak perspective translation parameters.

[0077] The weak perspective camera model output by the parametric animal size estimation model is only applicable to the image size used during its internal training, and its projection center often has a systematic offset from the foreground center of real animal photos. Without correction, the subsequently rendered SMAL mesh will deviate from the animal's actual position. The "foreground mask centroid" is used as a common reference point between the two coordinate systems. First, the foreground mask of the first frame is scaled to 224×224, and the average coordinates of all white pixels are calculated to obtain the mask centroid. Then, using the weak perspective formula, all vertices of the SMAL image are projected onto a 224×224 image, and the bounding box center of the projected points is calculated.

[0078] Next, the mask centroid is transformed from pixel coordinates to the normalized coordinate range [-1, 1]:

[0079]

[0080] Similarly, finally, calculate the deviation.

[0081] ,

[0082] in, , The target's normalized coordinates are obtained by converting the centroid of the foreground mask. It is the current center of the SMAL projection bezel.

[0083] And directly add it to the weak perspective translation parameter Above. After this correction, the projected contour of the SMAL mesh will roughly coincide with the real foreground, laying the foundation for subsequent depth prior generation. No additional registration network needs to be trained; cross-model alignment is achieved solely through geometric invariants, making it simple and efficient.

[0084] In S5, when backprojecting the corrected SMAL vertices from the weak perspective projection space to the camera space of the first frame, the z-coordinate of the SMAL vertex in the weak perspective model is directly used as the depth value. The camera intrinsic parameters of the first frame and the corrected weak perspective projection coordinates are used for backprojection to obtain the three-dimensional points in the camera coordinate system.

[0085] The weak perspective model only has a 2D projection and a relative "shape depth," while a perspective camera requires real 3D points. Instead of attempting to convert the weak perspective depth into real physical depth, we directly use the z-coordinates of the SMAL vertices themselves as the "shape prior depth." First, based on the corrected weak perspective parameters, we project the SMAL vertices onto the image to obtain their pixel coordinates. At the same time, take the z-coordinate of the vertex after translation. Then, using the camera intrinsic parameters estimated in S4 for the first frame, the three-dimensional coordinates of the point in the first frame's camera coordinate system are calculated using the inverse formula for perspective projection:

[0086] , ,

[0087] in, , , The three-dimensional coordinates in the camera coordinate system obtained by back projection. This is a multi-view SMAL depth prior map.

[0088] While the calculated 3D point's depth value is not the actual physical distance, it reflects the relative unevenness of the animal's surface. This avoids the scale inconsistency problem caused by estimating depth from point clouds, ensuring that the SMAL depth prior and the subsequent geometric model are aligned in a relative sense.

[0089] In S6, the SMAL parameters are encoded into multiple prior tokens for semantic decomposition, specifically including:

[0090] Encode the SMAL shape parameters as shape tokens, the SMAL trunk joint rotation parameters as trunk tokens, and the SMAL limb joint rotation parameters as limb tokens;

[0091] Each token is mapped to a unified embedding dimension through an independent multilayer perceptron network;

[0092] Token pre-positioning refers to inserting three prior tokens into the token sequence of the visual geometric transformer of the conditional multi-view geometric estimation model, before the image patch token;

[0093] Feature linear modulation refers to the process of performing mean pooling on three prior tokens, generating scaling and translation parameters through linear projection, and then performing channel-level modulation on the sum of the image patch token and depth token.

[0094] First, the hundreds of parameters of the SMAL model are divided into three groups according to animal anatomy: shape parameters, trunk joints, and limb joints. Each group is fed into an independent small neural network and encoded into three semantically rich vectors, with each token having the same dimension. Then, two paths are used for injection: (1) Token preposition: The three tokens are inserted at the beginning of the model input sequence, alongside tokens such as camera and pose, so that the model's attention mechanism can interact with these global conditions. (2) Feature linear modulation: The three tokens are first averaged to obtain a global vector, and then a scaling vector is generated by two fully connected layers. and offset vector Then, the features of each small patch of the image are analyzed. with depth features Modulation of the sum:

[0095]

[0096] in, Control channel-level scaling, Control channel-level offset, This indicates element-wise multiplication.

[0097] This is equivalent to "dynamically adjusting" the strength and bias of each channel of the image features based on body shape priors. The two paths are complementary: token pre-processing provides position-related conditional interactions, while FiLM provides strong global channel modulation. Compared to using only token pre-processing, dual-path injection significantly improves the guidance capability of the priors, making the generated Gaussian point cloud more consistent with the true morphology of the animal.

[0098] S6 also includes a two-stage fine-tuning of the prior injection module:

[0099] The first stage freezes the parameterized volume prior encoder and trains only the feature linear modulation layer. The loss function includes depth loss and RGB loss.

[0100] The second stage jointly trains the encoder, the feature linear modulation layer and a fidelity prediction head. The loss function includes RGB loss, depth loss, mask loss and fidelity loss. The fidelity loss is the L2 distance between the SMAL parameters output by the fidelity prediction head and the real SMAL parameters.

[0101] In the second stage, the injected prior token and the feature linear modulation parameter are set to zero with a predetermined probability, but the fidelity loss is always calculated using the real SMAL parameters.

[0102] Phase 1: Train only the FiLM layer and freeze the SMAL encoder. The loss function only includes RGB image reconstruction error and depth map error, without adding fidelity loss. The FiLM learning rate is set to 2x, and random dropout is not used because if FiLM is dropped, gradients cannot be passed to any trainable parameters. Phase 2: Jointly train the encoder, FiLM layer, and a newly added "fidelity prediction head." The loss function includes mask loss and fidelity loss. The fidelity loss allows the prediction head to reconstruct the original SMAL parameters from the encoded tokens, requiring very small reconstruction errors, which forces the encoder to retain complete parameter information. Furthermore, in the second phase, the injected tokens and FiLM parameters are randomly set to zero with a 50% probability, but the fidelity loss is always calculated using the true parameters, keeping the encoder continuously learning. This design solves the long-path gradient decay problem and enhances the model's robustness to the presence or absence of priors. After two phases of fine-tuning, the reconstruction model can stably and effectively utilize SMAL priors, generating a more complete point cloud that better fits the animal surface.

[0103] The SMAL grid proximity filtering in S7 specifically includes:

[0104] Using SMAL depth prior maps from various viewpoints, 3D points corresponding to non-zero depth pixels are extracted from the coarse point cloud output by unconditional multi-view geometric estimation to form a proxy point cloud.

[0105] Calculate the diagonal length of the axis-aligned bounding box of the proxy point cloud, and use the diagonal length multiplied by a predetermined ratio as the distance threshold;

[0106] For each Gaussian point, calculate its distance to the nearest neighbor in the proxy point cloud. If the distance exceeds the distance threshold, discard the Gaussian point.

[0107] After the first three layers of filtering, "floating" noise points far from the animal surface may still remain in the point cloud. This step uses the SMAL depth prior map to construct an "animal surface proxy point cloud" as a geometric constraint. In S5, we have already obtained the SMAL depth map for each viewpoint, where non-zero pixels correspond exactly to the projection area of ​​the SMAL mesh on the image. From the coarse point cloud output in S4, we extract the 3D points at these projection locations. The points from all viewpoints combined constitute a sparse but accurate proxy point cloud covering the animal surface. Then, calculate the diagonal length of the circumscribed cuboid of this proxy point cloud. Set distance threshold as For candidate Gaussian points, the Euclidean distance to the nearest point in the proxy point cloud is calculated. If the distance is greater than a threshold, it indicates that the Gaussian point is too far from the animal surface and is discarded. Compared to removing outliers using only statistical outliers, utilizing the global shape prior of a parametric model can effectively remove isolated noise far from the subject while preserving surface details.

[0108] Initializing Gaussian centers using foreground Gaussian point clouds in S8 specifically includes:

[0109] From the coarse point cloud output by the unconditional multi-view geometric estimation, select 3D points that simultaneously satisfy the conditions of non-zero SMAL depth prior map and true foreground mask in each view to form the initial point set;

[0110] If the number of initial points exceeds the preset upper limit, random downsampling will be performed.

[0111] Calculate the average k-nearest neighbor distance for each initial point, and take half of the average distance of all points as the initial scale.

[0112] A set of initial particles is required before optimization can begin with a 3D Gaussian point cloud. The quality of the initialization directly affects the final reconstruction accuracy. This method does not use random sampling or the entire point cloud, but instead utilizes the intersection of the SMAL depth prior and the foreground mask to accurately locate points on the animal's surface. It iterates through each viewpoint, locating pixel positions that simultaneously satisfy the condition of a non-zero SMAL depth map, and extracts the corresponding 3D points from the coarse point cloud. These points from all views are then merged to obtain the initial point set. To avoid slow computation due to an excessive number of points, if the number exceeds 100,000, it is randomly downsampled to 100,000 points. Then, for each initial point, a KD-tree is used to search for its three nearest neighbors, the average distance to these neighbors is calculated, and the average of all average distances is multiplied by 0.5 to serve as the initial scale for each Gaussian particle. This scale can roughly reflect the point cloud density of the local area, which is beneficial for rapid convergence in subsequent optimizations. Ensuring that all initial points fall on the animal surface and have a moderate density is more accurate than purely random or purely masked methods, reducing noise in the early stages of optimization.

[0113] The specific optical loss in S8 white background synthesis is as follows:

[0114] The composite rendered image is obtained by multiplying the rendered image pixel by pixel with the rendered alpha image and adding a white background.

[0115] The supervised image is multiplied pixel by pixel with the foreground mask and then a white background is added to obtain the synthesized supervised image;

[0116] The weighted sum of L1 loss and single-scale SSIM loss between the synthetic rendered image and the synthetic supervised image is calculated as the photometric loss.

[0117] When optimizing 3D Gaussian point clouds, it's typically necessary to compare the differences between the rendered image and the actual input image, and then backpropagate to adjust parameters. However, the input image in this method has already had its background removed and set to white, while the rendered image has a black transparent background by default. Direct comparison would lead to optimization errors due to this inconsistency. Therefore, this step unifies both images onto a white background before comparison. Specific formula: Let the rendered RGB image be... Transparency is The synthesized rendered image is

[0118]

[0119] in Indicates pixel-by-pixel multiplication. This is due to the contribution of the white background. Similarly, in real-world supervised images...

[0120]

[0121] in, For the target supervised image, To expand to 3-channel alpha / mask.

[0122] Then, the photometric loss is the sum of two terms: L1 loss and single-scale SSIM loss, with weights of 0.8 and 0.2, respectively. SSIM better preserves image texture and contrast. White background synthesis allows the photometric loss to penalize both color and opacity errors, avoiding interference from background inconsistencies, resulting in more stable training and sharper reconstructed edges.

[0123] After selecting the candidate video with the best score in S2, output the frames corresponding to the medoid subset of that video as input for the subsequent S3.

[0124] This step clarifies the output format of the screening results. In S2, we calculated the self-consistency score for each candidate video and selected the video with the best score. However, this video contains 81 frames, and using all of them for subsequent reconstruction would be computationally intensive and result in highly redundant information between adjacent frames. Therefore, this step does not output the entire video, but instead outputs the frames contained in the "medoid subset" of the video. The medoid subset is the 5-frame subset selected in S2 that has the best overall consistency with other random subsets. These 5 frames are roughly evenly distributed in time, covering all 360° viewpoints of the animal, and have the most reliable geometric poses. Complete multi-view reconstruction is achieved with a very small number of frames, greatly reducing the computational cost of subsequent geometric estimation and optimization, while ensuring the quality of the input viewpoints. This also achieves a seamless transition from "video generation" to "multi-view reconstruction".

[0125] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for three-dimensional reconstruction of animals based on body size priors, characterized in that, Includes the following steps: S1: Obtain a single animal reference image, extract structured animal descriptions, and combine multiple random seeds and prompt word configurations to generate multiple candidate track videos; S2: For each candidate track video, run the multi-view geometric estimation model by sampling multiple frame subsets to obtain the relative pose of the first and last frames of each subset, calculate the self-consistency score between the relative poses, select the candidate video with the best score and output its corresponding frames. S3: Perform animal detection, square cropping, and foreground mask extraction independently on each frame output from S2 to obtain a multi-view preprocessed image; S4: Perform unconditional multi-view geometric estimation on the multi-view preprocessed image to obtain the camera pose, camera intrinsic parameters and coarse point cloud for each view; at the same time, run the parameterized animal body size estimation model on the first frame image to obtain the SMAL mesh and weak perspective camera parameters. S5: Correct the weak perspective camera parameters using the foreground mask centroid, backproject the corrected SMAL vertices from the weak perspective projection space to the first frame camera space, then use the first frame camera pose transformation to the world coordinate system, and render the SMAL depth prior map for each viewpoint. S6: The multi-view preprocessed image, SMAL depth prior map, camera pose and camera intrinsic parameters are input conditions to form a multi-view geometric estimation model. At the same time, the SMAL parameters are encoded into multiple prior tokens of semantic decomposition. The model is guided to generate the original three-dimensional Gaussian point cloud through a dual-path injection method of token pre-positioning and feature linear modulation. S7: Perform opacity threshold filtering, multi-view mask projection filtering, statistical outlier removal and SMAL mesh proximity filtering on the original 3D Gaussian point cloud in sequence to obtain the foreground Gaussian point cloud. S8: Initialize the Gaussian center using the foreground Gaussian point cloud, and obtain the final 3D Gaussian reconstruction result by jointly optimizing the photometric loss and mask loss of the white background synthesis.

2. The animal three-dimensional reconstruction method based on body shape prior as described in claim 1, characterized in that, The calculation of the self-consistency score in S2 specifically includes: For each candidate track video, construct M random frame subsets and 1 uniformly anchored frame subset. Each subset contains the first frame and K-1 frames sampled from the remaining frames. Run the multi-view geometric estimation model for each subset to obtain the camera pose of each frame, and calculate the relative pose between the first and last frames. Calculate the pose distance between all pairs of relative poses, where the pose distance is the sum of the rotation geodesic distance and the translational angular distance; The subset with the smallest sum of distances to other subsets is selected as the medoid subset. The average distance between the medoid subset and all random subsets is calculated as the medoid distance. The relative pose distance between the medoid subset and the uniformly anchored subset is calculated as the anchor distance. The weighted sum of the medoid distance and the anchor distance is used as the final score.

3. The animal three-dimensional reconstruction method based on body shape prior as described in claim 1, characterized in that, The modification of weak perspective camera parameters using the foreground mask centroid in S5 specifically includes: Scale the foreground mask of the first frame to the input size of the parametric animal size estimation model and calculate the centroid of the foreground pixels; Calculate the bounding box center of the SMAL mesh projection based on the weak perspective camera parameters output by the parametric animal body size estimation model. Transform the mask centroid to normalized coordinates, calculate the deviation between the mask centroid and the center of the projected bounding box, and then add this deviation to the weak perspective translation parameters.

4. The method for animal three-dimensional reconstruction based on body shape prior as described in claim 1, characterized in that, In step S5, when backprojecting the corrected SMAL vertex from the weak perspective projection space to the first frame camera space, the z-coordinate of the SMAL vertex in the weak perspective model is directly used as the depth value. The camera intrinsic parameters of the first frame and the corrected weak perspective projection coordinates are used for backprojection to obtain the three-dimensional point in the camera coordinate system.

5. The animal three-dimensional reconstruction method based on body shape prior as described in claim 1, characterized in that, The encoding of SMAL parameters into multiple prior tokens for semantic decomposition in S6 specifically includes: Encode the SMAL shape parameters as shape tokens, the SMAL trunk joint rotation parameters as trunk tokens, and the SMAL limb joint rotation parameters as limb tokens; Each token is mapped to a unified embedding dimension through an independent multilayer perceptron network; The term "Token fronting" refers to inserting three prior tokens into the token sequence of the visual geometric transformer of the conditional multi-view geometric estimation model, before the image patch token. The aforementioned feature linear modulation refers to the process of performing mean pooling on three prior tokens, generating scaling and translation parameters through linear projection, and then performing channel-level modulation on the sum of the image patch token and depth token.

6. The method for animal three-dimensional reconstruction based on body shape prior according to claim 1, characterized in that, S6 also includes a two-stage fine-tuning of the prior injection module: The first stage freezes the parameterized volume prior encoder and trains only the feature linear modulation layer. The loss function includes depth loss and RGB loss. The second stage jointly trains the encoder, the feature linear modulation layer and a fidelity prediction head. The loss function includes RGB loss, depth loss, mask loss and fidelity loss. The fidelity loss is the L2 distance between the SMAL parameters output by the fidelity prediction head and the real SMAL parameters. In the second stage, the injected prior token and the feature linear modulation parameter are set to zero with a predetermined probability, but the fidelity loss is always calculated using the real SMAL parameters.

7. The animal three-dimensional reconstruction method based on body shape prior as described in claim 1, characterized in that, The SMAL mesh proximity filtering in S7 specifically includes: Using SMAL depth prior maps from various viewpoints, 3D points corresponding to non-zero depth pixels are extracted from the coarse point cloud output by unconditional multi-view geometric estimation to form a proxy point cloud. Calculate the diagonal length of the axis-aligned bounding box of the proxy point cloud, and use the diagonal length multiplied by a predetermined ratio as the distance threshold; For each Gaussian point, calculate its distance to the nearest neighbor in the proxy point cloud. If the distance exceeds the distance threshold, discard the Gaussian point.

8. The method for animal three-dimensional reconstruction based on body shape prior as described in claim 1, characterized in that, The initialization of Gaussian centers using the foreground Gaussian point cloud in S8 specifically includes: From the coarse point cloud output by the unconditional multi-view geometric estimation, select 3D points that simultaneously satisfy the conditions of non-zero SMAL depth prior map and true foreground mask in each view to form the initial point set; If the number of initial points exceeds the preset upper limit, random downsampling will be performed. Calculate the average k-nearest neighbor distance for each initial point, and take half of the average distance of all points as the initial scale.

9. The method for animal three-dimensional reconstruction based on body shape prior as described in claim 1, characterized in that, The specific optical loss in the white background synthesis in S8 is as follows: The composite rendered image is obtained by multiplying the rendered image pixel by pixel with the rendered alpha image and adding a white background. The supervised image is multiplied pixel by pixel with the foreground mask and then a white background is added to obtain the synthesized supervised image; The weighted sum of L1 loss and single-scale SSIM loss between the synthetic rendered image and the synthetic supervised image is calculated as the photometric loss.

10. The method for animal three-dimensional reconstruction based on body shape prior according to claim 1, characterized in that, After selecting the candidate video with the best score in S2, the frames corresponding to the medoid subset of that video are output as input for the subsequent S3.

Images