Method, device, electronic device and storage medium for equine 4d reconstruction

Abstract

This application provides a method, apparatus, electronic device, and storage medium for 4D reconstruction of equines, relating to the field of image processing technology. The method includes: acquiring a monocular video sequence containing equine activity; performing motion estimation on each frame of the monocular video sequence to obtain motion parameters for each frame; wherein the motion parameters include model parameters and camera parameters of a parametric model designed based on equines, the model parameters including pose parameters, shape parameters, and global displacement parameters; selecting a frame from the monocular video sequence as a reference image, performing appearance reconstruction on the reference image to obtain 3D Gaussian attributes of 3D Gaussian units; and rendering based on the motion parameters and the 3D Gaussian attributes to obtain a target 4D video. This application can efficiently and faithfully reconstruct the 4D geometry and appearance of equines from monocular videos, and exhibits high robustness to incomplete viewpoint coverage.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, electronic device and storage medium for 4D reconstruction of equines. Background Technology

[0002] Equines (such as horses, donkeys, and zebras) have played a vital role in human civilization, widely used in transportation, agricultural production, warfare, sports, cultural symbolism, and animal-assisted therapy. With the development of computer vision and graphics technologies, 4D animal reconstruction technology based on monocular video has gradually become a research hotspot. This technology aims to realistically recover the three-dimensional shape, motion, and appearance of animals from single-view video input, thereby generating a digital copy of real-world animal activities. This has significant implications for animal welfare monitoring, virtual reality, filmmaking, and digital asset creation.

[0003] Traditional monocular 4D reconstruction methods typically target general scenes, attempting to reconstruct dynamic objects and static backgrounds simultaneously from videos. However, reconstruction of specific non-rigid objects, such as animals, presents significant challenges. Especially when observational data is incomplete (e.g., the animal is occluded in the video or only has a partial viewpoint), general-purpose methods often struggle to recover the complete geometry. To address this issue, researchers have begun exploring specialized reconstruction algorithms for animal categories, utilizing parametric models or neural networks to construct animable 4D representations from video or image sets.

[0004] Despite advancements in existing 4D animal reconstruction techniques, numerous challenges remain when dealing with equines. Existing methods can be broadly categorized into model-free and model-based methods. Model-free methods typically estimate deformable radiance fields or point clouds; however, due to the lack of explicit structural priors, the generated geometric details are often poor and prone to artifacts. Model-based methods usually extract textures directly from the input image. This approach makes the final appearance quality highly dependent on the alignment accuracy between the mesh and the image; any misalignment will result in misaligned or blurred textures. Summary of the Invention

[0005] The main objective of this application is to propose a 4D reconstruction method, a 4D reconstruction device, an electronic device, and a computer-readable storage medium for equines, which can efficiently and faithfully reconstruct the 4D geometry and appearance of equines from monocular video, and has high robustness to incomplete viewpoint coverage.

[0006] To achieve the above objectives, a first aspect of this application proposes a 4D reconstruction method for equines, the method comprising:

[0007] Obtain monocular video sequences containing equine activity; Motion estimation is performed on each frame of the monocular video sequence to obtain motion parameters for each frame; wherein, the motion parameters include model parameters of a parameterized model designed based on equines and camera parameters, and the model parameters include pose parameters, shape parameters, and global displacement parameters; A frame is selected from the monocular video sequence as a reference image, and the appearance of the reference image is reconstructed to obtain the 3D Gaussian properties of the 3D Gaussian primitives. The target 4D video is obtained by rendering based on the motion parameters and the 3D Gaussian properties.

[0008] Optionally, the motion estimation for each frame of the monocular video sequence to obtain motion parameters for each frame includes: The monocular video sequence is segmented to obtain multiple video segments; Spatial features are extracted from each frame of each video segment to obtain a segment spatial feature sequence; Temporal feature modeling is performed on each of the segment spatial feature sequences to obtain the segment spatiotemporal feature sequences; The spatiotemporal feature sequence of each segment is decoded to obtain the motion parameters of each frame of the video segment.

[0009] Optionally, after performing motion estimation on each frame of the monocular video sequence to obtain the motion parameters for each frame, the method further includes: The motion parameters are projected back onto a 2D plane using a differentiable renderer to obtain a 2D image; The pseudo-real values ​​are extracted from the 2D image using a 2D pose estimation tool, and the error between the 2D key points and mask of the 2D image and the pseudo-real values ​​is calculated. The attitude parameters and the shape parameters are adjusted based on minimizing the error.

[0010] Optionally, the appearance reconstruction of the reference image to obtain the 3D Gaussian properties of the 3D Gaussian elements includes: The standard spatial grid vertices are subdivided to obtain an increased number of vertices, and the initial center position of the 3D Gaussian unit is determined as the vertex; Multi-scale feature extraction is performed on the reference image to obtain multiple initial image features at different scales, and all initial image features are fused to obtain the reference image features; The 3D coordinates of the vertices are encoded to obtain the position features of the 3D Gaussian units, and the position features of all the vertices are perceived by a multilayer perceptron to obtain the point cloud geometric features. Based on the reference image features and the point cloud geometric features, the 3D Gaussian properties of the 3D Gaussian units in standard space are predicted.

[0011] Optionally, predicting the 3D Gaussian properties of 3D Gaussian units in standard space based on the reference image features and the point cloud geometric features includes: The reference image features are subjected to average pooling to obtain image context features; The reference image features, the point cloud geometric features, and the image context features are fused to obtain 3D appearance features; Gaussian attribute prediction is performed on the 3D appearance features to obtain the 3D Gaussian attributes of the 3D Gaussian primitives in standard space; wherein, the 3D Gaussian attributes include position offset, rotation, scaling, color, and opacity.

[0012] Optionally, the feature fusion of the reference image features, the point cloud geometric features, and the image context features to obtain 3D appearance features includes: A dual-stream Transformer Gaussian decoding block is obtained, wherein the dual-stream Transformer Gaussian decoding block includes a first normalization layer, a first modulation layer, a first linear layer, a second normalization layer, a self-attention layer, a first scaling layer, a first summing layer, a third normalization layer, a second modulation layer, a first multilayer perceptron, a second scaling layer, a second summing layer, a fourth normalization layer, a third modulation layer, a second linear layer, a fifth normalization layer, a third scaling layer, a third summing layer, a sixth normalization layer, a fourth modulation layer, a second multilayer perceptron, a fourth scaling layer, and a fourth summing layer; The reference image features are normalized by the first normalization layer to obtain the first image features, and the point cloud geometric features are normalized by the fourth normalization layer to obtain the first vertex features. The first image feature is modulated by the first modulation layer based on the image context feature to obtain the second image feature, and the first vertex feature is modulated by the third modulation layer based on the image context feature to obtain the second vertex feature; The first linear layer performs feature transformation on the second image features to obtain initial image query features, initial image key features, and image value features. The second linear layer also performs feature transformation on the second vertex features to obtain initial vertex query features, initial vertex key features, and vertex value features. The initial image query features and initial image key features are normalized by the second normalization layer to obtain the target image query features and target image key features. The initial vertex query features and initial vertex key features are normalized by the fifth normalization layer to obtain the target vertex query features and target vertex key features. Through the self-attention layer, self-attention calculation is performed on the target image query features, target image key features, and image value features to obtain image attention features, and self-attention calculation is performed on the target vertex query features, target vertex key features, and vertex value features to obtain vertex attention features; The image context features and the image attention features are scaled by the first scaling layer to obtain image scaling features, and the image context features and the vertex attention features are scaled by the third scaling layer to obtain vertex scaling features; The first summation layer sums the reference image features and the image scaling features to obtain image summation features, and the third summation layer sums the point cloud geometric features and the vertex scaling features to obtain vertex summation features. The image summation features are normalized by the third normalization layer to obtain the third image features, and the vertex summation features are normalized by the sixth normalization layer to obtain the third vertex features; The third image feature is modulated based on the image context feature by the third modulation layer to obtain the fourth image feature, and the third vertex feature is modulated based on the image context feature by the fourth modulation layer to obtain the fourth vertex feature; The first multilayer perceptron perceives the fourth image feature to obtain the fifth image feature, and the second multilayer perceptron perceives the fourth vertex feature to obtain the fifth vertex feature. The image context feature and the fifth image feature are scaled by the second scaling layer to obtain the sixth image feature, and the image context feature and the fifth vertex feature are scaled by the fourth scaling layer to obtain the sixth vertex feature; The second summation layer sums the sixth image features and the reference image features to obtain the first appearance feature, and the fourth summation layer sums the sixth vertex features and the point cloud geometric features to obtain the second appearance feature. The first appearance feature and the second appearance feature are combined to obtain the 3D appearance feature.

[0013] Optionally, the step of rendering the target 4D video based on the motion parameters and the 3D Gaussian properties includes: By using linear blending skinning technology, the motion parameters of each frame are used to transform the 3D Gaussian properties to the observation space of each frame, thus obtaining the observation parameters of each frame. The observed parameters for each frame are rendered using a Gaussian splash renderer to obtain the target 4D video.

[0014] To achieve the above objectives, a second aspect of this application provides a 4D reconstruction device for equines, the device comprising: The video acquisition module is used to acquire monocular video sequences containing equine animal activities; The motion estimation module is used to perform motion estimation on each frame of the monocular video sequence to obtain motion parameters for each frame; wherein, the motion parameters include model parameters of a parameterized model designed based on equines and camera parameters, and the model parameters include pose parameters, shape parameters, and global displacement parameters; The appearance reconstruction module is used to select a frame from the monocular video sequence as a reference image, perform appearance reconstruction on the reference image, and obtain the 3D Gaussian properties of the 3D Gaussian elements. The video rendering module is used to render the target 4D video based on the motion parameters and the 3D Gaussian properties.

[0015] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0016] To achieve the above objectives, a fourth aspect of the present application provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0017] This application proposes a 4D reconstruction method, a 4D reconstruction device for equines, an electronic device, and a computer-readable storage medium. This application utilizes a parametric model specifically designed for equines as a geometric bridge, combined with 3D Gaussian Splatting technology to represent appearance. This combination leverages strong geometric priors to ensure shape accuracy while utilizing the powerful rendering capabilities of Gaussian Splatting to achieve photorealistic appearance details, surpassing the reconstruction quality of traditional texture mapping and generic models. Furthermore, this application decomposes the complex 4D reconstruction task into dynamic motion reconstruction and static appearance reconstruction, eliminating the need for time-consuming per-instance optimization for each video. Particularly for appearance reconstruction, this application can directly predict the complete 3D Gaussian avatar from a single image, significantly reducing computational costs and time overhead. Therefore, this application can efficiently and faithfully reconstruct the 4D geometry and appearance of equines from monocular videos, exhibiting high robustness to incomplete viewpoint coverage.

[0018] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0019] Figure 1 This is an optional flowchart of the 4D reconstruction method for equines provided in the embodiments of this application; Figure 2 yes Figure 1 Flowchart for step 102; Figure 3 yes Figure 1 Flowchart for step 103; Figure 4 yes Figure 3 Flowchart for step 304; Figure 5 yes Figure 1 Flowchart for step 105; Figure 6 This is an optional flowchart of the 4D reconstruction method for equines provided in the examples of this application; Figure 7 This is a block diagram of the modular structure of the equine 4D reconstruction device provided in the embodiments of this application; Figure 8 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and applications to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert devices. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0024] Backpropagation: The general principle of backpropagation is as follows: The training data is input into the input layer of the neural network, passes through the hidden layer, and finally reaches the output layer of the neural network to output the result; Since there is an error between the output result of the neural network and the actual result, the error between the estimated value and the actual value is calculated, and this error is backpropagated from the output layer to the hidden layer until it reaches the input layer; During the backpropagation process, the values ​​of various parameters are adjusted according to the error; The above process is iterated continuously until convergence.

[0025] In recent years, more advanced technologies have attempted to combine implicit fields or 3D Gaussian Splatting (3DGS) to represent animal appearance. For example, higher-quality rendering has been achieved by binding Gaussian primitives to a mesh. However, most of these methods are optimization-based. Optimization-based methods suffer from two main bottlenecks: high computational cost: requiring lengthy iterative optimization for each video instance, making real-time processing impossible; and demanding data requirements: obtaining a complete appearance typically requires a clear, 360-degree surround ("turntable") video sequence. However, real-world video recordings often only capture a limited perspective of the animal (incomplete observation), leading to severe artifacts or "floating objects" in unseen areas for optimization-based methods. Furthermore, while existing feed-forward methods avoid the time-consuming problem of instance-by-instance optimization, they often sacrifice the shape realism and detail accuracy of specific species (such as horses) in order to accommodate multiple animal categories. Below, we introduce some model-based methods, optimization-based reconstruction methods, and general feed-forward reconstruction methods: Option 1, SMALR / SMALST (Traditional Geometry and Texture Mapping Reconstruction Methods): Technical Description: These methods are based on the SMAL statistical model. SMALR reconstructs quadrupeds from image sets through shape deformation and texture stitching; SMALST focuses on zebra reconstruction. They utilize the geometric priors of the SMAL model to constrain the optimization process, estimating pose and shape parameters from images. Disadvantages: These methods directly extract texture maps from the input images. Due to the lack of deep networks supporting appearance generation, their final rendering quality is extremely sensitive to the alignment accuracy of the "mesh-image". If the geometric estimation is slightly off, the texture will show obvious misalignment, resulting in insufficient realism.

[0026] Option 2, GART: Technical Description: GART is an optimization-based 4D reconstruction method. It binds 3D Gaussians to a standard-space SMAL template mesh. For the input video, it fits the appearance in the video by optimizing the properties of the Gaussians (position, color, opacity, etc.) and renders using Gaussian splashing. It can deform the Gaussian volume with the animal's movement through skeletal skinning animation (LBS). Disadvantages: GART is based on a "per-instance optimization" method. This means that for each new video, a lengthy retraining process is required, resulting in low computational efficiency. More importantly, it heavily relies on multi-view observation data. If the input video only contains a limited number of views (not 360 degrees), GART will produce severe artifacts and blurring in unobserved views, making it impossible to infer the complete appearance from a single frame image.

[0027] Option 3, 3D-Fauna / 4D Fauna (General Class Reconstruction): Technical Description: 3D-Fauna and its extension 4D-Fauna aim to learn a general class model capable of handling hundreds of quadrupedal species. They learn from internet images or videos, directly predicting the shape and appearance of animals. Disadvantages: To achieve generality across a large number of different species, these methods sacrifice modeling accuracy for specific species (such as horses). They often lack strong specific geometric priors (such as high-quality horse models), resulting in potentially distorted horse shapes and less realistic appearance details compared to dedicated models.

[0028] Although current monocular 4D reconstruction technology has made some progress in general scenes or specific categories, it still has the following significant drawbacks in high-fidelity reconstruction tasks for equines: 1. Low computational efficiency and high training time cost: Most mainstream 4D reconstruction methods (such as GART, DogRecon, etc.) are "optimization-based" methods. These methods require lengthy per-instance optimization for each input video instance to fit a high-fidelity appearance. This approach has huge computational overhead and cannot meet the needs of fast processing or real-time applications.

[0029] 2. Strict requirements for the completeness of observation data and poor robustness: Optimization-based methods typically rely on 360-degree "turntable-style" video input to reconstruct a complete 3D model. However, real-world videos often have limited viewpoints, with occlusions or only a single-sided view. In such cases, existing methods cannot effectively infer information about unseen areas, leading to severe rendering artifacts, missing textures, or "floating objects" in unobserved viewpoints, making it impossible to generate a complete 3D image.

[0030] 3. Difficulty in Balancing Geometric Accuracy and Realistic Appearance: Insufficient Accuracy in Generalization Methods: Some feed-forward methods (such as 3D-Fauna) attempt to build general tetrapod models, sacrificing geometric details and shape realism specific to particular species (such as horses) in order to accommodate diversity. Texture Misalignment in Traditional Methods: Traditional model-based methods (such as SMALR) typically sample textures directly from the input image. This approach is extremely sensitive to the accuracy of shape matching; even a slight misalignment between the mesh and the image will result in misaligned and blurred textures, lacking photorealistic quality. Structural Deficiencies in Model-Free Methods: Model-free methods lack explicit skeletal or mesh priors, leading to reconstructed geometries that often do not conform to anatomical principles and have poor detail.

[0031] 4. Lack of temporal consistency in motion estimation (jitter problem): Many existing reconstruction methods perform pose estimation based on single-frame images, ignoring the temporal context between video frames. This results in discontinuous motion sequences in the output video, exhibiting obvious inter-frame jitter and lacking natural and smooth dynamic effects.

[0032] Therefore, there is currently a lack of a technical solution that can efficiently and faithfully reconstruct the 4D geometry and appearance of equines from monocular videos, and is robust to incomplete viewpoint coverage.

[0033] To address the shortcomings of the existing technologies, the purpose of this application is to provide a monocular video 4D reconstruction method for equines based on motion and appearance decoupling and 3D Gaussian splashing. Specifically, the objectives of this application include: 1. Significantly Improved Reconstruction Efficiency and Enabled Fast Inference: By proposing a decoupled architecture, the complex 4D reconstruction task is decomposed into dynamic motion reconstruction and static appearance reconstruction. This application designs a feed-forward network structure, eliminating the need for time-consuming instance-by-instance optimization for each video. In particular, for appearance reconstruction, this application can directly predict the complete 3D Gaussian avatar from a single image, greatly reducing computational cost and time overhead.

[0034] 2. Enhanced robustness to incomplete observation data: Utilizing an appearance network (EquineGS) pre-trained on a large-scale synthetic dataset, this application possesses powerful completion capabilities. Even with limited viewpoints or only one image in the input video, this application can infer a reasonable appearance for unseen viewpoints based on learned prior knowledge, generating a 360-degree complete, artifact-free, high-fidelity 3D Gaussian model, overcoming the limitations of existing technologies that rely on full-view input.

[0035] 3. Achieving High-Fidelity Geometry and Appearance Reconstruction: This application utilizes a high-quality VAREN parametric model (a parametric model specifically designed for equines) as a geometric bridge, combined with 3D Gaussian Splatting technology to represent the appearance. This combination leverages strong geometric priors to ensure shape accuracy while utilizing the powerful rendering capabilities of Gaussian Splatting to achieve photorealistic appearance details, surpassing the reconstruction quality of traditional texture mapping and generic models.

[0036] 4. Generating smooth and precisely aligned motion sequences: By introducing the Spatio-temporal Transformer network (AniMoFormer), this application can utilize the temporal information in video clips to regress smooth motion parameters, mitigating inter-frame jitter. Simultaneously, by combining a post-optimization strategy, it further ensures pixel-level precise alignment between the reconstructed 3D model and the 2D input image.

[0037] The equine 4D reconstruction method of this application can be executed by a server alone, by a terminal alone, or by both a terminal and a server. Furthermore, the equine 4D reconstruction method provided in this application can also be software running on a server. The server can be configured as a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed device. It can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The software can be an application that implements the equine 4D reconstruction method, but is not limited to the above forms.

[0038] This application provides a 4D reconstruction method for equines, a 4D reconstruction device for equines, an electronic device, and a computer-readable storage medium. The specific embodiments are described below. First, the 4D reconstruction method for equines in this application is described.

[0039] See Figure 1 A 4D reconstruction method for equines according to an embodiment of this application includes: Step 101: Obtain a monocular video sequence containing equine animal activities; Step 102: Perform motion estimation on each frame of the monocular video sequence to obtain the motion parameters of each frame; Step 103: Select a frame from the monocular video sequence as a reference image, perform appearance reconstruction on the reference image, and obtain the 3D Gaussian properties of the 3D Gaussian primitives. Step 104: Render the target 4D video based on motion parameters and 3D Gaussian properties.

[0040] The advantage of the embodiments of steps 101 to 104 described above is that it is possible to efficiently and faithfully reconstruct the 4D geometry and appearance of equines from monocular video, and it has high robustness to incomplete viewpoint coverage.

[0041] In step 101, the monocular video sequence containing equine activities can be obtained in the following ways: (1) Obtain the monocular video sequence from a public dataset, such as directly downloading monocular videos (containing horse activity scenes) from a public dataset, such as datasets related to behavior recognition or animal monitoring (e.g., visual datasets containing horses, wild animal monitoring datasets, horse farm training scenes, etc.). (2) Self-collect (shoot) monocular video sequences, such as placing a monocular camera (e.g., a regular USB camera, sports camera, or a stabilizer-mounted camera) at an appropriate height (about 2 to 4 meters) to cover a wide field of view to include the activities of multiple horses, and then using the monocular camera to collect videos of equines to obtain a monocular video sequence.

[0042] In step 102, the motion parameters include model parameters (also called VAREN model parameters) of the parametric model designed based on equines (which can be called the horse model or VAREN model) and camera parameters. The VAREN model parameters include pose parameters, shape parameters, and global displacement parameters.

[0043] In one example, the AniMoFormer network can be used to process video frames in a monocular video sequence, outputting the VAREN model parameters (including pose parameters θ, shape parameters β, and global translation parameters γ) and camera parameters for each frame. AniMoFormer employs a two-stage design: a regression network in the first stage and a post-optimization network in the second. The regression network includes a spatial Transformer layer, a temporal Transformer layer, and a parametric regression decoder.

[0044] In one embodiment, reference is made to Figure 2 Step 102 may include: Step 201: Segment the monocular video sequence to obtain multiple video segments; Step 202: Extract spatial features from each frame of each video segment to obtain a segment spatial feature sequence; Step 203: Perform temporal feature modeling on each segment's spatial feature sequence to obtain the segment's spatiotemporal feature sequence; Step 204: Decode the spatiotemporal feature sequence of each segment to obtain the motion parameters of each frame of the video segment.

[0045] In step 201, the monocular video can be divided into multiple video segments of length N, each video segment containing N frames, where N can be 16.

[0046] In step 202, the AniMoFormer network includes a SpatialTransformer layer. Spatial features are extracted from N frames of each video segment using the SpatialTransformer layer to obtain N frames of image spatial features. These N frames of image spatial features are then concatenated to obtain a segment spatial feature sequence. Specifically, the SpatialTransformer layer processes each frame independently, extracting local image texture and structural information to obtain the image spatial features of each frame.

[0047] In step 203, the AniMoFormer network also includes a temporal Transformer layer, which can input a sequence of segment spatial features containing image spatial features from N frames. The temporal Transformer layer performs self-attention calculations over the N frames to capture the temporal correlations and motion context between frames, thus obtaining a segment spatiotemporal feature sequence. In this way, by utilizing information from adjacent frames to smooth the current prediction, the jitter caused by single-frame prediction is effectively eliminated.

[0048] In step 204, the AniMoFormer network also includes a parametric regression decoder (VAREN TransformerDecoder). The spatiotemporal feature sequence of the segment is input into the parametric regression decoder to regress the VAREN model parameters for each frame in the current segment: shape parameters, pose parameters (axis-angle rotation of 38 joints), global displacement parameters, and camera parameters.

[0049] The benefit of the embodiments of steps 201 to 204 described above is that they improve the smoothness of motion estimation, thereby reducing inter-frame jitter and thus improving the quality of 4D reconstruction.

[0050] In one embodiment, after step 102, the equine 4D reconstruction method may further include: projecting motion parameters back to a 2D plane using a differentiable renderer to obtain a 2D image; extracting pseudo-real values ​​from the 2D image using a 2D pose estimation tool, and calculating the error between the 2D keypoints and the mask and the pseudo-real values ​​of the 2D image; and adjusting the pose parameters and shape parameters based on minimizing the error.

[0051] Specifically, considering that while the spatiotemporal Transformer can guarantee smooth motion, there may be misalignment with image pixels, this embodiment introduces a second-stage post-optimization network of the AniMoFormer network: 1) Projecting the motion parameters (equivalent to a 3D mesh) back to the 2D plane using a differentiable renderer to obtain a 2D image. 2) Calculating the error between the 2D keypoints and mask of the projected 2D image and the pseudo ground truth (Pseudo GT, obtained using existing 2D pose estimation tools such as ViTPose++) extracted from the 2D image. 3) Fine-tuning the pose and shape parameters by minimizing this error. This achieves pixel-level alignment between the geometric model and image evidence, further improving the quality of 4D reconstruction.

[0052] In step 103, a frame is selected from the monocular video sequence as a reference image, and its appearance is reconstructed to obtain the 3D Gaussian properties of the 3D Gaussians. For example, a frame (usually the first frame) from the video is selected as the reference image and input into the EquineGS network. This network predicts the 3D Gaussian properties (position, rotation, scaling, opacity, spherical harmonics) of the 3D Gaussians attached to the grid surface based on the standard spatial grid of the VAREN model. The EquineGS network is a feedforward network that can generate animable 3D Gaussian models from a single image without training for each video.

[0053] In one embodiment, reference is made to Figure 3 Step 103 may include: Step 301: Subdivide the standard spatial mesh vertices to obtain an increased number of vertices, and determine the initial center position of the 3D Gaussian unit as the vertex; Step 302: Perform multi-scale feature extraction on the reference image to obtain multiple initial image features at different scales, and fuse all initial image features to obtain the reference image features; Step 303: Encode the 3D coordinates of the vertices to obtain the position features of the 3D Gaussian units, and use a multilayer perceptron to perform feature perception on the position features of all vertices to obtain the point cloud geometric features; Step 304: Based on the features of the reference image and the geometric features of the point cloud, predict the 3D Gaussian properties of the 3D Gaussian elements in the standard space.

[0054] In step 301, the standard spatial mesh is the template mesh of the VAREN model. To capture sufficient appearance detail, using only the original vertices (approximately 13,873) of the VAREN model as Gaussian centers is insufficient. This embodiment performs a subdivision operation on the template mesh of the VAREN model: adding vertices at the midpoint of each edge, splitting each face into four. This subdivision yields approximately 55,486 vertices, which serve as the initial center positions for the 3D Gaussian primitives, thereby constructing a high-density canonical point cloud.

[0055] In steps 302 to 303, reference image features and point cloud geometric features can be extracted through two parallel feature extraction streams contained in the Dual-Stream Feature Extraction network: 1) Image stream: Multi-scale features of the input image are extracted using a pre-trained large visual model (such as DINOv3), and reference image features are obtained by fusing them through convolutional layers. 2) Point cloud stream: Positional encoding is performed on the 3D coordinates of the vertices, and point cloud geometric features are extracted through a multilayer perceptron (MLP).

[0056] In step 304, the reference image features and point cloud geometric features can be predicted using a pre-trained predictor to obtain the 3D Gaussian properties of the 3D Gaussian units in standard space. The 3D Gaussian properties include position offset, rotation, scaling, color, and opacity.

[0057] The advantage of the embodiments of steps 301 to 304 described above is that they improve the accuracy of obtaining 3D Gaussian properties.

[0058] In one embodiment, reference is made to Figure 4 Step 304 may include: Step 401: Perform average pooling on the reference image features to obtain image context features; Step 402: Perform feature fusion on the reference image features, point cloud geometric features, and image context features to obtain 3D appearance features; Step 403: Perform Gaussian attribute prediction on the 3D appearance features to obtain the 3D Gaussian attributes of the 3D Gaussian primitives in the standard space.

[0059] In step 401, the reference image features are averaged and pooled to extract the image context features, which are used to guide the overall lighting and texture style.

[0060] In step 402, the reference image features, point cloud geometric features, and image context features can be fused using a dual-stream Transformer Gaussian Decoder (DSTG Decoder) to obtain 3D appearance features. The DSTG Decoder simultaneously receives the reference image features, point cloud geometric features, and image context features. The DSTG Decoder utilizes a cross-attention mechanism and adaptive layer normalization (norm) to modulate the point cloud geometric features based on the reference image features. This effectively maps the appearance information in the 2D image precisely onto the 3D geometric structure.

[0061] In step 403, the fused 3D appearance features are directly predicted by the final MLP layer to predict five attributes for each Gaussian point: 1) Position offset: fine-tunes the position of the canonical point cloud to match the specific horse's body details. 2) Rotation: represented by quaternions. 3) Scaling: controls the shape and size of the Gaussian point. 4) Color / Appearance: spherical harmonic coefficients (SH) are used to represent view-dependent color. 5) Opacity: controls the transparency of the point.

[0062] The advantage of the embodiments of steps 401 to 403 described above is that it enables the accurate mapping of appearance information in 2D images to 3D geometry, thereby improving the accuracy of 4D reconstruction of equines.

[0063] In one embodiment, step 402 may include: Obtain a dual-stream Transformer Gaussian decoding block, which includes a first normalization layer, a first modulation layer, a first linear layer, a second normalization layer, a self-attention layer, a first scaling layer, a first summing layer, a third normalization layer, a second modulation layer, a first multilayer perceptron, a second scaling layer, a second summing layer, a fourth normalization layer, a third modulation layer, a second linear layer, a fifth normalization layer, a third scaling layer, a third summing layer, a sixth normalization layer, a fourth modulation layer, a second multilayer perceptron, a fourth scaling layer, and a fourth summing layer. The reference image features are normalized by the first normalization layer to obtain the first image features, and the point cloud geometric features are normalized by the fourth normalization layer to obtain the first vertex features. The first image features are modulated based on the image context features by the first modulation layer to obtain the second image features, and the first vertex features are modulated based on the image context features by the third modulation layer to obtain the second vertex features. The first linear layer performs feature transformation on the second image features to obtain initial image query features, initial image key features, and image value features. The second linear layer performs feature transformation on the second vertex features to obtain initial vertex query features, initial vertex key features, and vertex value features. The initial image query features and initial image key features are normalized by the second normalization layer to obtain the target image query features and target image key features. The initial vertex query features and initial vertex key features are normalized by the fifth normalization layer to obtain the target vertex query features and target vertex key features. Through a self-attention layer, self-attention calculation is performed on the target image query features, target image key features, and image value features to obtain image attention features. Self-attention calculation is also performed on the target vertex query features, target vertex key features, and vertex value features to obtain vertex attention features. The image context features and image attention features are scaled by the first scaling layer to obtain the image scaling features, and the image context features and vertex attention features are scaled by the third scaling layer to obtain the vertex scaling features. The first summation layer sums the reference image features and the image scaling features to obtain the image summation features, and the third summation layer sums the point cloud geometric features and the vertex scaling features to obtain the vertex summation features. The image summation features are normalized by the third normalization layer to obtain the third image features, and the vertex summation features are normalized by the sixth normalization layer to obtain the third vertex features. The third image feature is modulated based on the image context features by the third modulation layer to obtain the fourth image feature, and the third vertex feature is modulated based on the image context features by the fourth modulation layer to obtain the fourth vertex feature. The fourth image feature is perceived by the first multilayer perceptron to obtain the fifth image feature, and the fourth vertex feature is perceived by the second multilayer perceptron to obtain the fifth vertex feature; The image context features and the fifth image features are scaled by the second scaling layer to obtain the sixth image features, and the image context features and the fifth vertex features are scaled by the fourth scaling layer to obtain the sixth vertex features. The first appearance feature is obtained by summing the features of the sixth image and the reference image through the second summing layer, and the second appearance feature is obtained by summing the features of the sixth vertex and the geometric features of the point cloud through the fourth summing layer. By combining the first appearance feature and the second appearance feature, a 3D appearance feature is obtained.

[0064] The advantage of the above embodiments is that they enable the effective "texturing" of 2D image features into 3D point clouds, thereby improving the accuracy of 4D reconstruction.

[0065] In step 104, the target 4D video is obtained by rendering based on motion parameters and 3D Gaussian properties.

[0066] In one embodiment, reference is made to Figure 5 Step 105 may include: Step 501: Using linear blending skinning technology, the 3D Gaussian properties are transformed to the observation space of each frame using the motion parameters of each frame, so as to obtain the observation parameters of each frame. Step 502: Use the Gaussian splash renderer to render the observation parameters of each frame to obtain the target 4D video.

[0067] Specifically, after obtaining the 3D Gaussian attributes in standard space, this application uses the skeleton binding weights defined by the VAREN model to transform the Gaussian center and rotation attributes with the pose parameters of each frame through the LBS formula, thereby generating a dynamic horse sequence, which is the target 4D video.

[0068] In one embodiment, to train the network described above, this application constructed two large-scale synthetic datasets: 1) VarenPoser dataset: Used to train the AniMoFormer network. By fitting the VAREN model to real motion capture data (such as the PFERD dataset) and using a texture generation model (MV-Adapter) to generate diverse textures, combined with video data rendered from simulated camera trajectories (push-pull, wrap-around, fixed), it provides accurate 3D pose and shape ground truth.

[0069] 2) VarenTex dataset: used to train the EquineGS network. Utilizing a multi-view diffusion model (such as UniTex), high-quality horse images with consistent appearance across multiple views are generated conditioned on the normal and coordinate maps of the VAREN mesh. This addresses the problem of the lack of ground truth values ​​for multi-view appearance in real-world data.

[0070] In one example, refer to Figure 6 The 4D reconstruction method for equines may include the following steps: (1) Obtain monocular video sequences containing equine animal activities.

[0071] (2) Segment the monocular video sequence to obtain multiple video segments. Divide the video frames of each video segment into blocks to obtain image blocks. Extract features from the image blocks to obtain image block features. Merge the image block features of each frame to obtain image feature tokens.

[0072] (3) Spatial features are extracted from the image feature tokens of each frame of each video segment through the spatial Transformer layer to obtain the segment spatial feature sequence.

[0073] (4) Temporal feature modeling is performed on the spatial feature sequence of each segment and the image feature tokens of each frame of the video segment through the temporal Transformer layer to obtain the spatiotemporal feature sequence of the segment.

[0074] (5) The spatiotemporal feature sequence of each segment is decoded by a parametric regression decoder to obtain the motion parameters of each frame of the video segment.

[0075] (6) Adjusting pose and shape parameters through post-optimization layer: Specifically, the motion parameters are projected back to the 2D plane using a differentiable renderer to obtain a 2D image; pseudo-real values ​​are extracted from the 2D image using a 2D pose estimation tool, and the error between the 2D key points and the mask and the pseudo-real values ​​of the 2D image is calculated; based on minimizing the error, the pose and shape parameters are adjusted to obtain the reconstructed mesh.

[0076] (7) Select a frame from the monocular video sequence as a reference image and obtain the standard spatial grid vertices.

[0077] (8) Use a pre-trained large visual model (such as DINOv3) to extract multi-scale features of the reference image and fuse them through convolutional layers to obtain the reference image features.

[0078] (9) Positional encoding is performed on the 3D coordinates of the vertices of the standard spatial grid, and point cloud geometric features are extracted by multilayer perceptron (MLP).

[0079] (10) Average pooling is performed on the reference image features to obtain image context features; and the reference image features, point cloud geometric features and image context features are fused through a dual-stream Transformer Gaussian decoding block to obtain 3D appearance features; (11) Gaussian attribute prediction of 3D appearance features is performed by multilayer perceptron to obtain 3D Gaussian attributes of 3D Gaussian elements in standard space; among which, 3D Gaussian attributes include position offset (μ), rotation (T), scaling (s), color (c), and opacity (0).

[0080] (12) Using motion parameters, the 3D Gaussian properties of the 3D Gaussian primitives are transformed into the observation space of each frame through the Linear Blend Skinning (LBS) technique, and the final 4D video is generated using the Gaussian splash renderer.

[0081] This application achieves the following effects through the above implementation methods: 1) Decoupled design: Appearance reconstruction is no longer limited by video length and motion complexity; a high-quality model can be generated with just a single frame. 2) Spatiotemporal smoothing: AniMoFormer effectively solves the jitter problem in video reconstruction. 3) High-fidelity details: Combining mesh subdivision and 3D Gaussian splashing, the reconstructed horse has realistic hair texture and muscle details, with clear edges and no blur artifacts found in traditional methods.

[0082] This application offers significant advantages in computational efficiency, observational robustness, and reconstruction quality: 1) Extremely high inference efficiency and robustness to incomplete observations: Existing optimization-based methods require iterative optimization for each video, lasting tens of minutes, and rely on full-view video coverage to avoid artifacts. In contrast, this application employs a decoupled feedforward network design, eliminating the need for test-time optimization and improving inference speed by several orders of magnitude (e.g., processing one frame on an A100 GPU takes only about 11 seconds, while GART requires a fixed 15 minutes). Furthermore, the EquineGS network in this application possesses powerful completion capabilities, inferring a complete 360-degree 3D Gaussian avatar from a single image, completely resolving the "texture floating" or geometric loss issues inherent in existing technologies under limited or occluded viewing angles. 2) High-fidelity geometric details and temporal smoothness: Compared to generalized methods that sacrifice species-specific details for generality, this application utilizes a high-quality VAREN model combined with 3D Gaussian splashing technology to capture finer geometry, outperforming existing technologies across multiple evaluation metrics. Furthermore, to address the jitter problem commonly encountered in single-frame prediction, this application introduces a spatiotemporal Transformer (AniMoFormer) and a post-optimization strategy that effectively utilizes the temporal context, ensuring that the reconstructed motion sequence is smooth and natural in time and accurately aligned with the input video at the pixel level, significantly reducing acceleration errors.

[0083] Please see Figure 7 This application also provides a 4D reconstruction apparatus for equines, which can realize the above-described 4D reconstruction method for equines. Figure 7 This is a block diagram of the modular structure of the equine 4D reconstruction device provided in the embodiments of this application. The device includes: The video acquisition module 701 is used to acquire monocular video sequences containing equine animal activities; The motion estimation module 702 is used to perform motion estimation on each frame of the monocular video sequence to obtain the motion parameters of each frame. The motion parameters include model parameters of the parameterized model designed based on equines and camera parameters. The model parameters include pose parameters, shape parameters, and global displacement parameters. The appearance reconstruction module 703 is used to select a frame from the monocular video sequence as a reference image, perform appearance reconstruction on the reference image, and obtain the 3D Gaussian properties of the 3D Gaussian primitives. The video rendering module 704 is used to render the target 4D video based on motion parameters and 3D Gaussian properties.

[0084] It should be noted that the specific implementation of the equine 4D reconstruction device is basically the same as the specific implementation of the equine 4D reconstruction method described above, and will not be repeated here.

[0085] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned 4D reconstruction method for equines. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0086] Please see Figure 8 , Figure 8 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 801 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 802 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 802 can store operating devices and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 using the equine 4D reconstruction method of the embodiments of this application. The 803 input / output interface is used to implement information input and output. The communication interface 804 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 805 transmits information between various components of the device (e.g., processor 801, memory 802, input / output interface 803, and communication interface 804); The processor 801, memory 802, input / output interface 803, and communication interface 804 are connected to each other within the device via bus 805.

[0087] This application embodiment also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described 4D reconstruction method for equines.

[0088] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0089] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0090] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0091] Those skilled in the art will understand that all or some of the steps, apparatuses, or functional modules / units in the methods disclosed above can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0092] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0093] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0094] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0095] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0097] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for 4D reconstruction of equines, characterized in that, The method includes: Obtain monocular video sequences containing equine activity; Motion estimation is performed on each frame of the monocular video sequence to obtain motion parameters for each frame; wherein, the motion parameters include model parameters of a parameterized model designed based on equines and camera parameters, and the model parameters include pose parameters, shape parameters, and global displacement parameters; A frame is selected from the monocular video sequence as a reference image, and the appearance of the reference image is reconstructed to obtain the 3D Gaussian properties of the 3D Gaussian primitives. The target 4D video is obtained by rendering based on the motion parameters and the 3D Gaussian properties.

2. The method according to claim 1, characterized in that, The motion estimation for each frame of the monocular video sequence to obtain motion parameters for each frame includes: The monocular video sequence is segmented to obtain multiple video segments; Spatial features are extracted from each frame of each video segment to obtain a segment spatial feature sequence; Temporal feature modeling is performed on each of the segment spatial feature sequences to obtain the segment spatiotemporal feature sequences; The spatiotemporal feature sequence of each segment is decoded to obtain the motion parameters of each frame of the video segment.

3. The method according to claim 1, characterized in that, After performing motion estimation on each frame of the monocular video sequence to obtain the motion parameters for each frame, the method further includes: The motion parameters are projected back onto a 2D plane using a differentiable renderer to obtain a 2D image; The pseudo-real values ​​are extracted from the 2D image using a 2D pose estimation tool, and the error between the 2D key points and mask of the 2D image and the pseudo-real values ​​is calculated. The attitude parameters and the shape parameters are adjusted based on minimizing the error.

4. The method according to any one of claims 1 to 3, characterized in that, The process of reconstructing the appearance of the reference image to obtain the 3D Gaussian properties of the 3D Gaussian primitives includes: The standard spatial grid vertices are subdivided to obtain an increased number of vertices, and the initial center position of the 3D Gaussian unit is determined as the vertex; Multi-scale feature extraction is performed on the reference image to obtain multiple initial image features at different scales, and all initial image features are fused to obtain the reference image features; The 3D coordinates of the vertices are encoded to obtain the position features of the 3D Gaussian units, and the position features of all the vertices are perceived by a multilayer perceptron to obtain the point cloud geometric features. Based on the reference image features and the point cloud geometric features, the 3D Gaussian properties of the 3D Gaussian units in standard space are predicted.

5. The method according to claim 4, characterized in that, The step of predicting the 3D Gaussian properties of 3D Gaussian units in standard space based on the reference image features and the point cloud geometric features includes: The reference image features are subjected to average pooling to obtain image context features; The reference image features, the point cloud geometric features, and the image context features are fused to obtain 3D appearance features; Gaussian attribute prediction is performed on the 3D appearance features to obtain the 3D Gaussian attributes of the 3D Gaussian primitives in standard space; wherein, the 3D Gaussian attributes include position offset, rotation, scaling, color, and opacity.

6. The method according to claim 5, characterized in that, The feature fusion of the reference image features, the point cloud geometric features, and the image context features to obtain 3D appearance features includes: A dual-stream Transformer Gaussian decoding block is obtained, wherein the dual-stream Transformer Gaussian decoding block includes a first normalization layer, a first modulation layer, a first linear layer, a second normalization layer, a self-attention layer, a first scaling layer, a first summing layer, a third normalization layer, a second modulation layer, a first multilayer perceptron, a second scaling layer, a second summing layer, a fourth normalization layer, a third modulation layer, a second linear layer, a fifth normalization layer, a third scaling layer, a third summing layer, a sixth normalization layer, a fourth modulation layer, a second multilayer perceptron, a fourth scaling layer, and a fourth summing layer; The reference image features are normalized by the first normalization layer to obtain the first image features, and the point cloud geometric features are normalized by the fourth normalization layer to obtain the first vertex features. The first image feature is modulated by the first modulation layer based on the image context feature to obtain the second image feature, and the first vertex feature is modulated by the third modulation layer based on the image context feature to obtain the second vertex feature; The first linear layer performs feature transformation on the second image features to obtain initial image query features, initial image key features, and image value features. The second linear layer also performs feature transformation on the second vertex features to obtain initial vertex query features, initial vertex key features, and vertex value features. The initial image query features and initial image key features are normalized by the second normalization layer to obtain the target image query features and target image key features. The initial vertex query features and initial vertex key features are normalized by the fifth normalization layer to obtain the target vertex query features and target vertex key features. Through the self-attention layer, self-attention calculation is performed on the target image query features, target image key features, and image value features to obtain image attention features, and self-attention calculation is performed on the target vertex query features, target vertex key features, and vertex value features to obtain vertex attention features; The image context features and the image attention features are scaled by the first scaling layer to obtain image scaling features, and the image context features and the vertex attention features are scaled by the third scaling layer to obtain vertex scaling features; The first summation layer sums the reference image features and the image scaling features to obtain image summation features, and the third summation layer sums the point cloud geometric features and the vertex scaling features to obtain vertex summation features. The image summation features are normalized by the third normalization layer to obtain the third image features, and the vertex summation features are normalized by the sixth normalization layer to obtain the third vertex features; The third image feature is modulated based on the image context feature by the third modulation layer to obtain the fourth image feature, and the third vertex feature is modulated based on the image context feature by the fourth modulation layer to obtain the fourth vertex feature; The first multilayer perceptron perceives the fourth image feature to obtain the fifth image feature, and the second multilayer perceptron perceives the fourth vertex feature to obtain the fifth vertex feature. The image context feature and the fifth image feature are scaled by the second scaling layer to obtain the sixth image feature, and the image context feature and the fifth vertex feature are scaled by the fourth scaling layer to obtain the sixth vertex feature; The second summation layer sums the sixth image features and the reference image features to obtain the first appearance feature, and the fourth summation layer sums the sixth vertex features and the point cloud geometric features to obtain the second appearance feature. The first appearance feature and the second appearance feature are combined to obtain the 3D appearance feature.

7. The method according to any one of claims 1 to 3, characterized in that, The process of rendering the target 4D video based on the motion parameters and the 3D Gaussian properties includes: By using linear blending skinning technology, the motion parameters of each frame are used to transform the 3D Gaussian properties to the observation space of each frame, thus obtaining the observation parameters of each frame. The observed parameters for each frame are rendered using a Gaussian splash renderer to obtain the target 4D video.

8. A 4D reconstruction device for equines, characterized in that, The device includes: The video acquisition module is used to acquire monocular video sequences containing equine animal activities; The motion estimation module is used to perform motion estimation on each frame of the monocular video sequence to obtain motion parameters for each frame; wherein, the motion parameters include model parameters of a parameterized model designed based on equines and camera parameters, and the model parameters include pose parameters, shape parameters, and global displacement parameters; The appearance reconstruction module is used to select a frame from the monocular video sequence as a reference image, perform appearance reconstruction on the reference image, and obtain the 3D Gaussian properties of the 3D Gaussian elements. The video rendering module is used to render the target 4D video based on the motion parameters and the 3D Gaussian properties.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images