Three-dimensional model sequence generation method and related equipment

Abstract

The embodiment of the invention discloses a three-dimensional model sequence generation method and related equipment. The related equipment can comprise a three-dimensional model sequence generation device, electronic equipment, a computer program product and a computer readable storage medium. According to the embodiment of the invention, feature extraction is carried out on video frames in a monocular video to obtain image features, an initial noise sequence corresponding to a three-dimensional model of a target object is generated, denoising is carried out on the initial noise sequence according to the image features to obtain a feature sequence set, and based on the frame positions of the video frames and the time distance between the video frames, the target object is obtained. Screening at least one reference feature block associated with the feature block from the feature sequence set, denoising the feature block according to the reference feature block to obtain a target feature sequence of the video frame, and generating a three-dimensional model sequence of the target object based on the target feature sequence; according to the scheme, the reference feature blocks can be screened to perform block-level cross-frame information interaction, so that the generation quality of the three-dimensional model sequence can be improved.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically to a method for generating a three-dimensional model sequence and related equipment. The related equipment may include a three-dimensional model sequence generation device, electronic equipment, computer program products, and computer-readable storage media. Background Technology

[0002] With the rapid development of generative artificial intelligence technology, the generation technology of four-dimensional (4D) dynamic assets (i.e., three-dimensional model sequences with a time dimension) is becoming increasingly important for fields such as film and television production, virtual reality, digital twins and interactive media.

[0003] In the process of researching and practicing current technologies, the inventors of this application discovered that the direct fine-tuning method based on a 3D diffusion transformer generates a sequence of 3D models by independently generating each frame or introducing a simple cross-frame attention mechanism. However, simple frame-by-frame generation or weakly coupled generation can cause uncontrollable drift and abrupt changes in object identity attributes (such as shape, texture, and topology) over time, disrupting the coherence of dynamic content. Furthermore, traditional all-temporal attention mechanisms require calculating the relationships between all time frames and all spatial points, resulting in high computational complexity. Therefore, the accuracy of the generated 3D model sequence is insufficient and the generation efficiency is low. Summary of the Invention

[0004] This application provides a method for generating three-dimensional model sequences and related equipment. The related equipment may include a three-dimensional model sequence generation device, electronic equipment, computer program products, and computer-readable storage media, which can improve the generation quality and efficiency of three-dimensional model sequences.

[0005] A method for generating a three-dimensional model sequence, comprising: Obtain the monocular video corresponding to the target object, wherein the monocular video includes at least one video frame; Feature extraction is performed on the video frame to obtain the image features of the video frame, and an initial noise sequence corresponding to the three-dimensional model of the target object is generated; Based on the image features, the initial noise sequence is denoised to obtain a feature sequence set, which includes the feature sequence corresponding to the video frame, and the feature sequence includes at least one feature block; Based on the frame position of the video frame and the time distance between the video frames, at least one reference feature block associated with the feature block is selected from the feature sequence set; Based on the reference feature block, the feature block is denoised to obtain the target feature sequence of the video frame, and based on the target feature sequence, a three-dimensional model sequence of the target object is generated.

[0006] Accordingly, embodiments of this application provide a three-dimensional model sequence generation apparatus, including: An acquisition unit is used to acquire a monocular video corresponding to a target object, wherein the monocular video includes at least one video frame. The extraction unit is used to extract features from the video frame, obtain the image features of the video frame, and generate an initial noise sequence corresponding to the three-dimensional model of the target object; A denoising unit is configured to denoise the initial noise sequence based on the image features to obtain a feature sequence set, the feature sequence set including the feature sequence corresponding to the video frame, and the feature sequence including at least one feature block; A filtering unit is configured to filter at least one reference feature block associated with the feature block from the feature sequence set based on the frame position of the video frame and the time distance between the video frames; The generation unit is configured to denoise the reference feature block to obtain the target feature sequence of the video frame, and generate a three-dimensional model sequence of the target object based on the target feature sequence.

[0007] In some embodiments, the filtering unit may be specifically configured to filter out the current video frame to which the feature block belongs from the video frames, and use the video frames in the monocular video as reference video frames for the current video frame, the reference video frames including the current video frame; calculate the time distance between the current video frame and the reference video frame based on the frame position of the video frame, and determine the sampling step size corresponding to the reference video frame according to the time distance; and sample at least one reference feature block associated with the feature block from the feature sequence corresponding to the reference video frame based on the sampling step size.

[0008] In some embodiments, the filtering unit may be specifically used to obtain the number of preset sampling steps in a preset sampling step set, wherein the preset sampling step corresponds to a step index identifier; determine the current step index identifier corresponding to the reference video frame based on the time distance and the number of steps; and filter out the sampling step of the feature sequence corresponding to the reference video frame from the preset sampling step set according to the current step index identifier.

[0009] In some embodiments, the filtering unit may be specifically used to identify the block position of the feature block in the current video frame, and filter out the feature block at the block position in the feature sequence corresponding to the reference video frame to obtain a reference feature block; using the reference feature block as the sampling center, sample at least one candidate reference feature block associated with the feature block in the feature sequence corresponding to the reference video frame based on the sampling step size; update the candidate reference feature block according to the frame position of the current video frame to obtain the reference feature block.

[0010] In some embodiments, the filtering unit may be specifically used to filter out the first video frame in the monocular video based on the frame position of the video frame; when the current video frame is a video frame other than the first video frame, add the unsampled feature blocks in the feature sequence of the first video frame to the candidate reference feature block to obtain the reference feature block; when the current video frame is the first video frame, use the candidate reference feature block as the reference feature block.

[0011] In some embodiments, the generation unit may be specifically used to denoise the feature block according to the reference feature block to obtain an updated feature sequence of the video frame; use the updated feature sequence as the initial noise sequence of the video frame, and return to execute the step of denoising the initial noise sequence according to the image features until the number of denoising times reaches a preset number, so as to obtain the target feature sequence of the video frame.

[0012] In some embodiments, the generation unit may be specifically configured to identify the current position in the feature sequence to which the reference feature block belongs, and based on the current position, generate a mask sequence of a reference video frame corresponding to the feature sequence; fuse the mask sequence with the feature sequence to obtain a reference feature sequence of the reference video frame; and update the feature block based on the reference feature sequence to obtain the updated feature sequence of the video frame.

[0013] In some embodiments, the generation unit may be specifically configured to: determine the attention weight of the reference video frame based on the reference feature sequence and the feature block, wherein the attention weight indicates the degree of correlation between the feature sequence and the feature block of the reference video frame; perform attention weighting on the reference feature block according to the attention weight to obtain a weighted feature block; replace the feature block with the weighted feature block to update the feature sequence corresponding to the feature block, thereby obtaining the updated feature sequence of the video frame.

[0014] In some embodiments, the denoising unit may be specifically used to employ a denoising network in a 3D model sequence generation model to perform attention weighting on the initial noise sequence and the image features to obtain a first intermediate feature sequence of the video frame corresponding to the image features, wherein the first intermediate feature sequence includes at least one noise point; to perform attention weighting on the noise points in the same first intermediate feature sequence to obtain a second intermediate feature sequence of the video frame; and to divide the noise points in the second intermediate feature sequence into blocks to obtain the feature sequence set.

[0015] In some embodiments, a training unit is further included. Specifically, the training unit can be used to acquire sample monocular videos and sample 3D model sequences corresponding to sample objects. The sample 3D model sequence includes a sample 3D model corresponding to each sample video frame in the sample monocular video. The training unit then samples the sample 3D models using a preset 3D model sequence generation model to obtain a set of sampling points, and generates an original feature distribution sequence of the sample 3D model corresponding to the sample video frame based on the set of sampling points. Based on a preset noise sequence, the original feature distribution sequence is denoised to obtain a sample feature sequence of the sample 3D model corresponding to the sample video frame. The sample feature sequence is denoised using a denoising network of the preset 3D model sequence generation model to obtain a predicted feature distribution sequence. Based on the predicted feature distribution sequence and the original feature distribution sequence, the generation loss of the sample object is determined, and the denoising network is converged based on the generation loss to obtain a 3D model sequence generation model.

[0016] In some embodiments, the training unit may be specifically used to select a benchmark sample 3D model from the sample 3D model sequence, and use the sample 3D models other than the benchmark sample 3D model as deformed sample 3D models; to sample in the benchmark sample 3D model to obtain at least one benchmark sampling point, and to identify the sampling position corresponding to the benchmark sampling point in the deformed sample 3D model; based on the sampling position, to sample the consistency sampling point corresponding to the benchmark sampling point in the deformed sample 3D model to obtain a set of sampling points.

[0017] In some embodiments, the training unit may be specifically used to extract the index identifier and centroid coordinates of the triangular facet to which the benchmark sampling point belongs in the benchmark sample 3D model; identify the deformation vertex coordinates of the triangular facet corresponding to the index identifier in the deformed sample 3D model; and determine the coordinates of the consistency sampling point corresponding to the basic sampling point in the deformed sample 3D model based on the deformation vertex coordinates and the centroid coordinates, thereby obtaining the sampling position.

[0018] In some embodiments, the training unit may specifically be used to project the sampling points in the sampling point set onto the watertight mesh surface corresponding to the sample 3D model to which the sampling points belong, so as to obtain a guide point set corresponding to the sample 3D model; to select at least one query point from the guide point set of the benchmark sample 3D model; to select the guide point corresponding to the query point from the guide point set of the deformed sample 3D model, so as to obtain a query point set; and to determine the original feature distribution sequence of the sample 3D model corresponding to the sample video frame based on the query point set and the guide point set.

[0019] In some embodiments, the training unit may specifically be used to project the sampling points in the sampling point set onto the watertight mesh surface corresponding to the sample 3D model to which the sampling points belong, to obtain a projection point set; determine the global bounding space of the sample 3D model based on the position of the projection points in the projection point set, wherein any projection point is within the global bounding space; and normalize the projection points according to the size and center position of the global bounding space to obtain a guide point set.

[0020] In some embodiments, the training unit may be specifically used to obtain the center point of at least one triangular facet in the watertight mesh corresponding to the sample 3D model, and determine the spatial distance between the center point and the sampling point; based on the spatial distance, at least one candidate point is selected from the center points, and the distance weight of the candidate point is determined according to the distance between the candidate point and the sampling point; based on the distance weight, the coordinates of the candidate point are weighted to obtain the projected coordinates of the sampling point on the watertight mesh, and the sampling point is projected onto the projected coordinates to obtain a set of projected points.

[0021] In some embodiments, the training unit may be specifically used to construct a sample bounding space of the sample 3D model based on the model attribute information of the sample 3D model; uniformly divide the sample bounding space into at least one voxel grid, and calculate the distance from the center point of the voxel grid to the surface of the sample 3D model to obtain at least one grid distance; based on the grid distance, generate at least one target triangular facet corresponding to the voxel grid, and stitch the target triangular facets together to obtain the watertight grid surface corresponding to the sample 3D model.

[0022] Furthermore, embodiments of this application also provide an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the three-dimensional model sequence generation method provided in embodiments of this application.

[0023] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the three-dimensional model sequence generation methods provided in embodiments of this application.

[0024] Furthermore, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the three-dimensional model sequence generation method provided in embodiments of this application.

[0025] In this embodiment, after acquiring the monocular video corresponding to the target object, feature extraction is performed on the video frames in the monocular video to obtain the image features of the video frames, and an initial noise sequence corresponding to the 3D model of the target object is generated. Then, based on the image features, the initial noise sequence is denoised to obtain a feature sequence set. Then, based on the frame position of the video frame and the time distance between video frames, at least one reference feature block associated with the feature block is selected from the feature sequence set. Then, based on the reference feature block, the feature block is denoised to obtain the target feature sequence of the video frame, and based on the target feature sequence, a 3D model sequence of the target object is generated. Since this scheme can select reference feature blocks associated with the current feature block from the feature sequence of each frame to perform cross-frame information interaction, it can effectively improve the generation quality of the 3D model sequence. At the same time, denoising and updating at the feature block level can effectively reduce computational complexity. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of a scenario for the three-dimensional model sequence generation method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the three-dimensional model sequence generation method provided in the embodiments of this application; Figure 3 This is a schematic diagram of the denoising network in the three-dimensional model sequence generation model provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure for training a three-dimensional model sequence generation model provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the three-dimensional model sequence generation model application provided in the embodiments of this application; Figure 6This is a schematic diagram of the generated three-dimensional model sequence provided in the embodiments of this application; Figure 7 This is a flowchart illustrating the training method for the three-dimensional model sequence generation model provided in this application embodiment; Figure 8 This is another flowchart illustrating the three-dimensional model sequence generation method provided in this application embodiment; Figure 9 This is a schematic diagram of the structure of the three-dimensional model sequence generation device provided in the embodiments of this application; Figure 10 This is another schematic diagram of the three-dimensional model sequence generation device provided in the embodiments of this application; Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] This application provides a method for generating a three-dimensional model sequence and related equipment. The related equipment may include a three-dimensional model sequence generation device, an electronic device, a computer program product, and a computer-readable storage medium. The three-dimensional model sequence generation device can be integrated into an electronic device, which may be a server or a terminal, etc.

[0030] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0031] For example, see Figure 1Taking the integration of a 3D model sequence generation device into an electronic device as an example, after acquiring the monocular video corresponding to the target object, the electronic device can extract features from the video frames in the monocular video to obtain the image features of the video frames and generate an initial noise sequence corresponding to the 3D model of the target object. Then, based on the image features, the initial noise sequence is denoised to obtain a set of feature sequences. Then, based on the frame position of the video frame and the time distance between video frames, at least one reference feature block associated with the feature block is selected from the set of feature sequences. Then, based on the reference feature block, the feature block is denoised to obtain the target feature sequence of the video frame. Based on the target feature sequence, the 3D model sequence of the target object is generated, thereby improving the generation quality and efficiency of the 3D model sequence.

[0032] It is understood that, in the specific embodiments of this application, data such as monocular video corresponding to the target object is involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0033] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0034] This embodiment will be described from the perspective of a three-dimensional model sequence generation device, which can be integrated into an electronic device, such as a server or a terminal. The terminal can include tablet computers, laptops, personal computers (PCs), wearable devices, virtual reality devices, or other smart devices capable of generating three-dimensional model sequences.

[0035] A method for generating a three-dimensional model sequence, comprising: The process involves: acquiring a monocular video corresponding to the target object, the monocular video comprising at least one video frame; extracting features from the video frame to obtain image features and generating an initial noise sequence corresponding to the 3D model of the target object; denoising the initial noise sequence based on the image features to obtain a feature sequence set, the feature sequence set including feature sequences corresponding to the video frame, and each feature sequence including at least one feature block; selecting at least one reference feature block associated with the feature block from the feature sequence set based on the frame position of the video frame and the time distance between video frames; denoising the feature block based on the reference feature block to obtain the target feature sequence of the video frame, and generating a 3D model sequence of the target object based on the target feature sequence.

[0036] like Figure 2 As shown, Figure 2This is a schematic diagram of an optional flow of a three-dimensional model sequence generation method provided in this embodiment. The three-dimensional model sequence generation method can be executed by a server or a terminal alone, or it can be executed by a terminal in cooperation with a server. The specific flow of the three-dimensional model sequence generation method can be as follows: 101. Obtain the monocular video corresponding to the target object.

[0037] Monocular video refers to a two-dimensional video sequence captured by a single camera, containing continuous changes of the target object over time. Monocular video provides only a single viewpoint, meaning the viewing angle of the target object is the same across all video frames. For example, a 30-frame video of a person turning their head, filmed from the front with a mobile phone. Monocular video includes at least one video frame, which is a single still image, a discrete sample of the video at a specific point in time. The video frames are arranged chronologically to create a continuous visual change.

[0038] The target object refers to the entity from which a sequence of 3D models needs to be generated, such as a person, animal, or object; that is, the continuously changing visual subject in a monocular video. Examples include a talking face, a running cat, or a spinning teacup. The 3D model sequence refers to the dynamic 3D asset ultimately generated from the monocular video, i.e., the sequence of 3D models of the target object changing over time. A 3D model is the geometric representation of the target object in 3D space, and can be represented using meshes, point clouds, voxels, etc.

[0039] There are several ways to obtain the monocular video corresponding to the target object, such as searching and downloading from a public database, or actively acquiring a continuous image sequence of an entity from a certain perspective using a monocular camera to obtain the monocular video corresponding to that entity.

[0040] 102. Extract features from video frames to obtain image features of the video frames, and generate the initial noise sequence corresponding to the three-dimensional model of the target object.

[0041] Image features are abstract semantic representations extracted from video frames by an image encoder. They can be represented in the form of high-dimensional vectors or feature maps, encoding information such as the visual content, texture, and pose of the video frame. For example, CNNs (Convolutional Neural Networks), ViTs (Vision Transformers), or dedicated feature extractors (such as Distillation with No Labels version 2, i.e., the DINOv2 visual encoder) can be used to extract features from video frames. For instance, using DINOv2 to extract features from a face video frame yields a 2048-dimensional feature vector encoding information such as facial expressions and head pose, i.e., image features.

[0042] After obtaining the image features corresponding to each video frame in a monocular video, a three-dimensional model corresponding to each video frame can be generated based on the image features by generating a model through a preset three-dimensional model sequence, thereby obtaining a three-dimensional model sequence (Mesh Sequence) with a time dimension, such as a 4D mesh sequence.

[0043] In some embodiments, the preset 3D model sequence generation model can be a diffusion model (e.g., DDPM or DDIM): by utilizing the rich visual information encoded by image features, a diffusion model is guided and controlled to "de-noise" from random noise to produce a 3D model sequence that is semantically identical to the image and temporally coherent. The diffusion model is pre-trained to learn the inverse reconstruction process from noise to the original data (the training process is described below).

[0044] In this embodiment, an initial noise sequence needs to be generated for each video frame. This initial noise sequence is the starting point of the diffusion model's reverse denoising process and consists of completely random Gaussian noise sampled for the entire 3D model sequence. For example, 4096 noise points (each with a dimension of 1024) can be randomly sampled from a standard normal distribution to obtain the initial noise sequence corresponding to each video frame. Initially, the noise points in the initial noise sequence are randomly sampled and have no practical meaning. However, through the reverse diffusion model process (denoising), the dimensional features of the noise points are updated based on image features, gradually transforming them into structured latent variables. That is, the information carried by the noise points gradually evolves from pure random noise into 3D geometric features with clear semantics. After denoising, each noise point corresponds to an abstract representation of a certain spatiotemporal location in the 3D model.

[0045] Considering that in the process of generating 3D model sequences based on diffusion models, it is often desirable for the generated 3D models to be smooth in time, without flickering or jumps. If the initial noise sequence of each frame is sampled independently, even if the image features are continuous, the randomness of the initial noise may cause the denoising process of each frame to go in different directions, thus causing discontinuities between frames. Therefore, this embodiment can directly generate an initial noise sequence for the target object and then apply it to all frames to achieve noise sharing. That is, the initial noise sequence of each video frame is the same, and by integrating the initial noise sequences of all video frames, a set of initial noise sequences for the monocular video of the target object can be formed. Thus, by concentrating randomness into shared initial noise, the inconsistency caused by independent randomness in each frame is avoided, ensuring that the objects in the generated 3D model sequence are the same entity, without identity drift, and that the changes in objects are continuous and smooth.

[0046] 103. Based on the image features, the initial noise sequence is denoised to obtain a set of feature sequences.

[0047] The feature sequence set includes feature sequences corresponding to video frames, and each feature sequence includes at least one feature block. The feature sequence represents the intermediate state of the 3D model during any denoising step of the diffusion model. As the denoising steps progress, noise gradually decreases, and geometric information gradually emerges. Feature blocks are local sub-units obtained by spatially segmenting the feature sequence of each frame to reduce computational complexity. For example, the 512-dimensional feature sequence of a video frame is divided into 32 blocks, each with 16 dimensions.

[0048] In some embodiments, the process of denoising an initial noise sequence based on image features to obtain a feature sequence set includes: using a denoising network in a three-dimensional model sequence generation model to perform attention weighting on the initial noise sequence and image features to obtain a first intermediate feature sequence of a video frame corresponding to the image features, wherein the first intermediate feature sequence includes at least one noise point; performing attention weighting on the noise points in the same first intermediate feature sequence to obtain a second intermediate feature sequence of the video frame; and dividing the noise points in the second intermediate feature sequence into blocks to obtain a feature sequence set.

[0049] In this context, noise points are the basic units constituting the feature sequence. They represent the incomplete and noisy state information of the 3D model at a specific spatial location and time frame during the diffusion denoising process. Essentially, a noise point is a floating-point vector, and each noise point encodes a "probability distribution" of the local geometry and appearance of its corresponding spatiotemporal location. For example, when generating a 3D sequence of a rotating cube, a noise point can be understood as follows: in the early stages of denoising, its value is almost random; in the later stages of denoising, its value becomes stable, explicitly indicating "this is a vertex of the cube, and its color is red."

[0050] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the denoising network in the 3D model sequence generation model provided in this application embodiment. For example... Figure 3 As shown, the denoising network includes a cross-attention layer, a self-attention layer, and a block sparse attention layer connected in sequence.

[0051] After obtaining the image features and initial noise sequence for each frame through step 102 above, both are input into the denoising network in the 3D model sequence generation model. First, the initial noise sequence is made to "pay attention" to the image feature sequence through the cross-attention layer of the denoising network, thereby injecting image conditions into the noise points and guiding the denoising process. In the cross-attention layer, the noise points (Query) in the initial noise sequence of each frame "query" the image features (Key / Value) of that video frame: for each noise point, it is used as the Query, and the similarity with the image feature Key is calculated. Then, the image feature Value is weighted and summed according to the similarity weight to obtain the updated noise point representation, thus obtaining the first intermediate feature sequence of that video frame. In this way, the image information of that frame is fused into the noise points, ensuring that the generated content is semantically aligned with the image of that frame.

[0052] For example, in the scenario of generating a sequence of 3D models of a "smiling profile," assuming the input is a monocular video showing a face turning from front to side and smiling: First, for each frame, extract image features. For example, frame t is a profile smile. Semantics such as "profile" and "smile" are encoded. Simultaneously, an initial noise sequence for the frame is generated, where each noise point represents a location in 3D space (e.g., left eye, right corner of mouth, nose, etc.). During the cross-attention process, for frame t, all noise points are used as queries, along with their image features. Use cross-attention. For example, a noise point representing the "corner of the mouth" will receive high-weighted attention. The "smiling" feature in the noise is then shifted upwards during denoising to form a smiling shape. A noise point representing the "cheek" is then focused on. In the "Side Profile" section, adjust its position to create a side profile curve.

[0053] After processing the initial noise sequence and image features using a cross-attention layer to obtain the first intermediate feature sequence of the video frame corresponding to the image features, a self-attention layer performs attention updates within the first intermediate feature sequence: allowing all noise points in the first intermediate feature sequence to exchange information. That is, each noise point (Query) in the first intermediate feature sequence evaluates its relevance to all other noise points (Keys) in the sequence, and then updates itself by weighting the information (Values) of all other noise points according to the relevance weights, thus obtaining the second intermediate feature sequence of the video frame. In this way, different noise points coordinate with each other within the same frame. For example, a noise point representing the "corner of the mouth" needs to communicate with noise points representing the "cheek" and "nose" to ensure that the local geometry they form can be seamlessly stitched together to form a complete and reasonable face.

[0054] The first intermediate feature sequence of each frame is processed by a self-attention layer to obtain the second intermediate feature sequence for each frame. Then, the noise points in the second intermediate feature sequence are divided into blocks to obtain the feature sequence for each frame, thus obtaining the feature sequence set for the entire monocular video. For example, assuming the second intermediate feature sequence of a video frame contains 4096 noise points, and each block contains 128 points, it can be divided into 32 blocks. If there are 10 frames, the length of the feature sequence set for the entire monocular video is 10 × 32.

[0055] The segmented feature sequences undergo temporal attention processing through subsequent block sparse attention layers, enabling the attention mechanism to operate at the block level rather than through global computation, thereby reducing computational complexity. The block sparse attention process is detailed in step 105 below.

[0056] In some embodiments, the training process of the 3D model sequence generation model is as follows: First, please refer to... Figure 4 , Figure 4 This is a schematic diagram of the structure for training a 3D model sequence generation model provided in an embodiment of this application. For example... Figure 4 As shown, the 3D model sequence generation model includes a coding network ( Figure 4 The part before the denoising network), the denoising network and the decoding network ( Figure 4 (The part after the denoising network). The encoding and decoding networks can use the Vector Set Autoencoder (VAE) and its corresponding decoder from the Hunyuan3D model (used to generate 3D mesh models). During training, their network parameters do not need to be updated; only the parameters of the denoising network need to be updated. Specifically, the training and optimization process of the denoising network includes: S1. Obtain the sample monocular video and sample 3D model sequence corresponding to the sample object.

[0057] The sample 3D model sequence includes the sample 3D model corresponding to each sample video frame in the sample monocular video.

[0058] A sample object refers to an entity with a clear spatiotemporal change as a learning target during the training process. It is a dynamic entity in the real world or virtual environment whose appearance can be captured by monocular video and whose complete geometric shape can be reconstructed using 3D modeling technology. Examples include a gymnast performing gymnastics, a leaf swaying in the wind, or a piece of clay being molded.

[0059] A sample monocular video refers to a sequence of two-dimensional images continuously captured from the perspective of a fixed or moving single camera, showing the motion and changes of a sample object within a time period [0, T]. It serves as the conditional input signal for generating the entire three-dimensional model sequence. A sample monocular video consists of multiple temporally ordered sample video frames. Each frame represents a two-dimensional projection observation of the sample object from a specific perspective; that is, each frame is a mapping of the three-dimensional state of the sample object at time t onto the camera's imaging plane (affected by lighting, material, etc.). During training, each frame of the sample monocular video is encoded as an image feature, serving as a condition for the denoising network, guiding it to restore the noise to a three-dimensional shape corresponding to the appearance of the current frame.

[0060] The sample 3D model sequence is a ground truth sequence that is strictly aligned in time with the sample monocular video and consists of the 3D geometric models corresponding to the sample objects at each moment. Specifically, the t-th 3D model in the sample 3D model sequence corresponds to the t-th video frame in the sample monocular video. The sample 3D model sequence serves as the "standard answer" during training. In the training process, the "original feature distribution sequence" can be encoded based on the sample 3D model sequence and used to calculate the loss function, directly supervising the 3D model sequence generation model to learn correct 4D geometric generation.

[0061] In essence, the goal of training a 3D model sequence generation model—that is, the goal of the denoising network—is to learn the mapping function from monocular videos to 3D model sequences. Through training with massive amounts of such paired data (sample monocular videos and sample 3D model sequences), the model can eventually generate corresponding, high-quality 3D model sequences based on a new, never-before-seen monocular video (such as a dancer's performance).

[0062] In this embodiment, the sample monocular video corresponding to the sample object can be obtained as follows: A 3D model sequence of the animation object is downloaded from a public dataset, and multi-view images are rendered for each 3D model sequence. That is, a set of 2D images containing multiple perspectives is generated for each time frame, and these images are paired with the 3D model (geometric data) of that time frame. It is easy to understand that the 3D model corresponding to the 2D images from multiple perspectives within the same time frame is the same. Then, by extracting the 2D images from the same perspective in chronological order, the 2D image sequence of the animation object at each perspective, and the corresponding 3D model sequence, can be obtained.

[0063] For example, a sequence of 3D models of 13,000 animated objects was obtained from the Objaverse dataset. Then, using the Hammersley sequence sampling method, 24 points were selected on a virtual sphere surrounding the 3D models to simulate viewing the models from different angles, resulting in 24 different 2D images at that time frame. The Hammersley sequence sampling method ensures comprehensive viewpoint coverage, capturing the most complete object appearance information with the fewest possible viewpoints, thus providing high-quality 2D visual cues for the generative model. Next, based on the 24 sampled fixed viewpoints, a small random rotation offset was added to each animated object as a whole to prevent the generative model from "overfitting" to these 24 fixed viewpoints. In this way, by introducing randomness, the generalization ability of the generative model can be enhanced, enabling it to learn to understand the omnidirectional 3D structure of objects, rather than just the 2D projections from those 24 specific angles. Then, for each time frame of the animated object's 3D model, a virtual camera is placed at one of the 24 (randomly offset) viewpoints, and a rendering engine (such as Blender or Unity) is used to generate a 512×512 resolution image for each. For example, for a 100-frame sequence of 3D models, 100 frames × 24 viewpoints = 2400 images will be obtained. Finally, from the 24 fixed viewpoints, one viewpoint is arbitrarily selected (for example, always selecting viewpoint number 7), and the images rendered from this specific viewpoint across all time frames (from frame 1 to frame 100) are extracted in chronological order and combined into a 2D image sequence. For example, 100 consecutive images from viewpoint number 7 constitute a "sample monocular video." It simulates a "video" obtained by a fixed-position camera capturing the 3D model sequence from a specific angle.

[0064] In addition, to ensure the consistency of image constraints, the following sequence-level preprocessing can be performed on a 2D image sequence from a single viewpoint: First, image segmentation techniques (such as segmentation models or chroma keying) are used to separate the foreground object (i.e., the object to be generated into a 3D model) from the background in each frame of the rendered image, generating a binary mask to eliminate background interference, allowing subsequent calculations to focus only on the object itself. Next, for each frame with the background removed, the axially aligned bounding box of its foreground object pixels is calculated, and the bounding boxes of all frames are traversed to calculate a "total bounding box" that can cover all positions and poses of the object in the entire image sequence. Then, based on the maximum size of the bounding boxes in the image sequence, the object in each frame is scaled proportionally to a fixed size, with the center of the object's own bounding box as the origin during scaling. Then, each scaled frame image is translated to the absolute center position of the image (the average of the center points of the bounding boxes of the object in all frames), so that no matter how the object moves in the original frame, it is located in the same position in the image after preprocessing. Finally, the empty areas (i.e., non-foreground areas) of the transformed image are uniformly filled with pure white. Finally, the image, after being centered and scaled as described above, is adjusted to the fixed input resolution of 518×518 pixels required by the DINOv2 encoder to meet its input specifications.

[0065] During the training of the 3D model sequence generation model, if the original 2D image sequence is directly input, drastic changes in the object's position and size will be treated as strong interference signals by the encoder (DINOv2). The generation model may incorrectly couple "motion" with "appearance changes" or "spatial displacement," making it difficult to learn stable 3D shape priors. This embodiment, through the aforementioned sequence-level preprocessing, removes the object's translation and scale changes within the image plane, forcing the generation model to learn its 3D structure and temporal motion from the object's own deformation and texture changes, rather than relying on simple 2D displacement. Simultaneously, it provides standardized input to the DINOv2 encoder, facilitating the extraction of more stable and effective image visual features.

[0066] S2. The sample 3D model is sampled using a preset 3D model sequence to obtain a set of sampling points, and based on the set of sampling points, the original feature distribution sequence of the sample 3D model corresponding to the sample video frame is generated.

[0067] In this context, the sampling point set refers to a representative set of discrete points obtained from the surface of a 3D model using a specific sampling strategy (such as farthest-point sampling, uniform sampling, etc.). Each point contains its position coordinates (x, y, z) in 3D space and its geometric features (such as normal vectors). This set provides a lightweight and information-rich numerical representation of the original 3D model.

[0068] In some embodiments, the process of sampling a sample 3D model using a preset 3D model sequence to obtain a set of sampling points includes: (1) Select the benchmark sample three-dimensional model from the sample three-dimensional model sequence, and use the sample three-dimensional models other than the benchmark sample three-dimensional model as the deformed sample three-dimensional model.

[0069] The reference sample 3D model is a specific sample 3D model in the sequence that is selected as the spatial sampling reference frame when processing a dynamic 3D model sequence with temporal consistency. The reference sample 3D model is the "template" or "anchor point" in the entire 4D data processing flow, and its core function is to establish the semantic consistency of the sampling points throughout the entire time dimension.

[0070] In this embodiment, firstly, a 3D model is selected from the entire 3D model sequence (e.g., 10 frames) as a reference model and labeled as the reference sample 3D model. Then, all other sample 3D models in the sequence besides the reference sample 3D model are labeled as deformable sample 3D models. Thus, a fixed reference system is established, and all sampling operations are first defined on the reference sample 3D model and then "propagated" to other frames.

[0071] When selecting a baseline 3D model, you can choose the first model in the sequence of 3D model samples as the baseline 3D model. For example, ... Figure 4 As shown, in the sample monocular video of "opening the door", the sample 3D model corresponding to the 0th frame (t=0) is selected as the reference sample 3D model. The sample 3D models from the 1st frame (t=1) to the Tth frame (t=T) are all regarded as deformed sample 3D models because their "opening state" has been deformed relative to the reference sample 3D model.

[0072] (2) Sample in the three-dimensional model of the reference sample to obtain at least one reference sampling point, and identify the sampling position corresponding to the reference sampling point in the three-dimensional model of the deformed sample.

[0073] The reference sampling points can be sharp edge points or random surface points. Sharp edge points refer to feature points on the surface of the 3D model where the curvature changes drastically, forming edges or contours. Examples include the edges of a cube, the screen border of a mobile phone model, and the bridge of the nose in a human model. These points are crucial for defining the shape, category, and detail of an object. If only uniform random sampling is performed, these key feature areas may be undersampled due to their small area, leading to loss of detail and blurred contours in the 3D model. Random surface points refer to points that are uniformly or randomly distributed across the entire surface of the 3D model to provide a coherent representation of the overall shape and volume of the 3D model. This ensures that the sampling points cover all areas of the 3D model, providing a basic and continuous geometric prior for the 3D model.

[0074] In this embodiment, the two methods can be combined to sample 124,928 sharp edge points and 124,928 uniform points (i.e., random surface points) from the reference sample 3D model, thereby obtaining the sampling point set corresponding to the reference sample 3D model.

[0075] After obtaining the set of sampling points corresponding to the benchmark sample 3D model, the consistent sampling points corresponding to the benchmark sampling points can be identified and extracted in the deformed sample 3D model to obtain the complete set of sampling points corresponding to the entire sample 3D model sequence.

[0076] The process of identifying the sampling position corresponding to the reference sampling point in the deformable sample 3D model includes: extracting the index identifier and centroid coordinates of the triangular facet to which the reference sampling point belongs in the reference sample 3D model; identifying the deformation vertex coordinates of the triangular facet corresponding to the index identifier in the deformable sample 3D model; and determining the coordinates of the consistency sampling point corresponding to the basic sampling point in the deformable sample 3D model based on the deformation vertex coordinates and centroid coordinates to obtain the sampling position.

[0077] The 3D model can be viewed as a 3D mesh composed of many triangular facets. Each triangular facet consists of three vertices and is the basic geometric unit of the 3D mesh model. Multiple triangular facets combine to form a continuous surface by sharing vertices and edges, collectively defining the shape of a 3D mesh model.

[0078] An index identifier is a unique integer value assigned to each triangle in a 3D mesh, used for quickly locating and accessing a specific triangle in the program. The index identifier is the storage location index of the triangle in the mesh facet list. For example, a 3D mesh with 1024 triangles would have index identifiers that are integers from 0 to 1023, with each triangle uniquely identified by its index.

[0079] The barycentric coordinates are a coordinate system used to describe the position of any sampling point inside a triangular facet. They consist of three weighted values ​​(u, v, w) satisfying u + v + w = ​​1, indicating that the point can be represented as a weighted sum of the coordinates of the three vertices of the triangle. It's easy to understand that the barycentric coordinates of a triangular facet are independent of the size, position, and orientation of the triangle; they only describe the relative position of the sampling point within the triangle. Even if the triangle deforms (vertices move), as long as the barycentric coordinates remain unchanged, the relative position of the sampling point within the deformed triangle remains the same.

[0080] In this embodiment, firstly, for each reference sampling point The system determines which triangular facet it is located on through spatial query (or finds the nearest facet through projection), and obtains and records the index of that triangular facet in the reference sample 3D model. Next, within this triangular facet, the centroid coordinates (u, v, w) of the reference sampling point relative to the three vertices are calculated and recorded as follows. Therefore, the topological mapping of the reference sampling point in the 3D model of the reference sample can be obtained. , ).

[0081] Since the deformed sample 3D model and the baseline sample 3D model have the exact same mesh topology (i.e., the same vertex connections), but the vertex positions have changed, the basic sampling points are determined in the deformed sample 3D model. Corresponding consistency sampling points At that time, firstly, based on the index identifier of the basic sampling point recorded above. In the 3D model of the deformed sample, the i-th triangular facet is directly located, and the 3D coordinates (A', B', C') of its three vertices are read, representing the new positions of the deformed facet. Finally, using the same centroid coordinates (u, v, w) and the coordinates of the deformed vertices (A', B', C'), the base sampling points are calculated through linear interpolation. Corresponding consistency sampling points The coordinates (sampling positions) are obtained, that is, the points on the 3D model of the deformed sample that are semantically consistent with the reference sampling points, the basic sampling points. and its corresponding consistency sampling points It represents the same "part" on a 3D model.

[0082] (3) Based on the sampling location, sample the consistent sampling points corresponding to the reference sampling points in the three-dimensional model of the deformed sample to obtain the sampling point set.

[0083] Obtain basic sampling points Consistency sampling points in the 3D model of the deformed sample After obtaining the coordinates, consistent sampling can be performed in the 3D model of the deformed sample (e.g., Figure 4 As shown in the figure, the set of sampling points corresponding to the three-dimensional model of the deformed sample is obtained.

[0084] For example, suppose the baseline sample 3D model is a 3D mesh of a face with a neutral expression, and a deformed sample 3D model is a 3D mesh of a face with a smiling expression, and the two face 3D models have the same triangular mesh topology. First, select a point near the corner of the mouth on the 3D mesh of the face with the neutral expression as the baseline sampling point. Next, determine that the point lies on the triangular facet. Within a range of 3045, its centroid coordinates are (0.3, 0.6, 0.1). Then, within the 3D mesh of the smiling face, the corresponding index identifier is found. A triangle with a coordinate system of 3045 is obtained, and the coordinates of its three deformed vertices are acquired: vertex A' = (2.1, 5.3, 0.9), vertex B' = (2.3, 5.1, 0.8), and vertex C' = (2.0, 5.0, 1.0). Then, using the same centroid coordinates (0.3, 0.6, 0.1) and deformed vertex coordinates (A', B', C'), the base sampling points are calculated through linear interpolation. Corresponding consistency sampling points Coordinates: =0.3×(2.1,5.3,0.9)+0.6×(2.3,5.1,0.8)+0.1×(2.0,5.0,1.0)=(2.21,5.15, 0.84) therefore, It refers to the corners of the mouth in the 3D mesh of a smiling face and the 3D mesh of a neutral face. Corresponding consistent sampling points. In this way, the point representing the corner of the mouth in the baseline sample 3D model always corresponds to the same anatomical position in each deformed sample 3D model, ensuring temporal consistency.

[0085] In this embodiment, by using the same topological index and barycentric coordinates, the sampling point sets generated from the 3D models of samples in different frames have an accurate correspondence. Regardless of how the 3D model is deformed (rotated, translated, scaled, or bent), as long as the mesh topology remains unchanged, the correspondence remains unchanged, avoiding point matching errors. At the same time, based on the barycentric coordinates, only simple linear interpolation is needed to calculate the position of the consistent sampling point corresponding to the baseline sampling point, reducing computational complexity and making it suitable for scenarios that process large-scale sequence data.

[0086] After obtaining the set of sampling points corresponding to the 3D model of each frame through the above sampling method, the original feature distribution sequence of the 3D model corresponding to the sample video frame can be generated based on the set of sampling points. This process may include: (1) Project the sampling points in the sampling point set onto the watertight mesh surface corresponding to the sample three-dimensional model to which the sampling points belong, so as to obtain the guide point set corresponding to the sample three-dimensional model.

[0087] The watertight mesh surface is a completely closed, hole-free, and boundaryless triangular mesh model that clearly distinguishes the internal and external spaces of the 3D model. Its surface normals are continuous and consistent, and each edge is shared by exactly two triangular faces. In other words, the watertight mesh surface has no missing faces, cracks, or holes, forming a "sealed" shell.

[0088] Since the 3D models of sample objects obtained from public datasets are usually non-watertight, and robust training of the VAE encoder requires a watertight mesh, it is first necessary to construct a corresponding watertight mesh surface for each sample 3D model in the sample 3D model sequence. This process includes: Based on the model attribute information of the sample 3D model, a sample bounding space of the sample 3D model is constructed; at least one voxel mesh is uniformly divided in the sample bounding space, and the distance from the center point of the voxel mesh to the surface of the sample 3D model is calculated to obtain at least one mesh distance; based on the mesh distance, at least one target triangular facet corresponding to the voxel mesh is generated, and the target triangular facets are spliced ​​to obtain the watertight mesh surface corresponding to the sample 3D model.

[0089] The model attribute information is a set of metadata that describes the geometric characteristics of the sample 3D model, including but not limited to: bounding box: the minimum to maximum coordinate range of the 3D model in 3D space, number of vertices: the total number of vertices contained in the 3D model, number of faces: the total number of triangular faces contained in the 3D model, geometric center: the centroid or bounding box center of the 3D model, and scale information: the dimensions of the 3D model in each coordinate axis direction.

[0090] The sample bounding space is a regular cubic spatial region that completely contains the 3D model of the sample. A voxel mesh is a regular 3D mesh unit formed by uniformly dividing the sample bounding space along the three coordinate axes; that is, each voxel is a small cube.

[0091] For example, based on the bounding box range information in the model attribute information of the sample 3D model, the extreme coordinate values ​​of all vertices of the original sample 3D model can be determined to obtain the initial bounding box. Then, the initial bounding box is appropriately expanded (e.g., by adding a 5%-10% margin) to ensure that the sample 3D model is completely contained and that there is sufficient distance between the surface and the boundary, thus obtaining the sample bounding space of the sample 3D model. Then, the sample bounding space is uniformly divided 128 times along each coordinate axis direction to obtain a 128×128×128 voxel mesh.

[0092] After dividing the sample bounding space of the 3D model into a voxel mesh, the distance from the center point of each voxel mesh to the surface of the 3D model is calculated, yielding at least one mesh distance. The mesh distance refers to the shortest Euclidean distance from the center point of the voxel mesh to the original surface of the 3D model. The mesh distance can be calculated using an unsigned distance field function (Universal Data File, UDF) and then the sign determined by point testing, or directly calculated using a signed distance field (SDF).

[0093] After obtaining the mesh distance, the moving cubes algorithm is applied to generate the watertight mesh surface corresponding to the sample 3D model. In this process, firstly, for each voxel mesh cell, the intersection of the isosurface (SDF=0) with the voxel mesh cell is determined based on the SDF values ​​of its eight corner points. Then, the precise intersection coordinates are calculated using linear interpolation at the locations where the isosurface intersects with the voxel edges of the voxel mesh cell. Next, according to the standard moving cubes lookup table, the intersection points are connected to form triangular facets, generating at least one target triangular facet corresponding to the voxel mesh cell. These target triangular facets approximate the isosurface of the sample 3D model (i.e., the surface with an SDF distance of 0); that is, each target triangular facet is a part of the isosurface. Finally, all voxel mesh cells are traversed, and the generated target triangular facets are combined into a complete mesh; that is, the target triangular facets are stitched together to obtain the watertight mesh surface corresponding to the sample 3D model. Thus, various potentially defective sample 3D models (point clouds, non-manifold meshes, and porous models) are converted into standard watertight mesh representations, automatically filling holes and correcting topological errors, ensuring the geometric integrity of the sample 3D models input to the denoising network.

[0094] While the above process yields a high-quality watertight mesh, it disrupts the original topology of the sample 3D model. The number of vertices, connection methods, and triangular face divisions of the original 3D mesh model and the completed watertight mesh model are independent of each other, losing the one-to-one correspondence between vertices between frames. Therefore, this embodiment also needs to project the sampling points obtained from the original sample 3D model sequence through consistent sampling onto the watertight mesh surface corresponding to the sample 3D model to which the sampling point belongs, in order to obtain the final set of guide points for the input encoder. In this way, these guide points maintain temporal consistency (because they originate from the same topological mapping) and are located on a high-quality watertight mesh surface (after projection).

[0095] The process of projecting the sampling points in the sampling point set onto the watertight mesh surface corresponding to the sample 3D model to which the sampling points belong, in order to obtain the guide point set, may include: projecting the sampling points in the sampling point set onto the watertight mesh surface corresponding to the sample 3D model to which the sampling points belong, to obtain the projection point set; determining the global bounding space of the sample 3D model based on the position of the projection points in the projection point set, wherein any projection point is within the global bounding space; and normalizing the projection points according to the size and center position of the global bounding space to obtain the guide point set.

[0096] The projection point set is a new set of points obtained by projecting each sampling point from the original sample 3D model's sampling point set onto the surface of the corresponding watertight mesh of the original sample 3D model using a specific algorithm. These points are strictly located on the triangular facets of the watertight mesh and have precise surface normals and geometric properties.

[0097] The process of projecting sampling points from the sampling point set onto the surface of the watertight mesh corresponding to the 3D model of the sample to obtain the projection point set may include: obtaining the center point of at least one triangular facet in the watertight mesh corresponding to the 3D model of the sample, and determining the spatial distance between the center point and the sampling point; based on the spatial distance, selecting at least one candidate point from the center points, and determining the distance weight of the candidate point according to the distance between the candidate point and the sampling point; based on the distance weight, weighting the coordinates of the candidate point to obtain the projection coordinates of the sampling point on the watertight mesh, and projecting the sampling point onto the projection coordinates to obtain the projection point set.

[0098] Spatial distance refers to the Euclidean distance between the sampling point and the center point of the triangular facet of the watertight mesh, i.e., the straight-line distance in three-dimensional space. Candidate points are the k closest triangular facet center points to the current sampling point, selected from all triangular facet center points of the watertight mesh during the projection process based on spatial distance. These points represent the potential surface regions that the sampling point may project onto. For example, k can be chosen in the range of 3-10 to balance computational accuracy and efficiency.

[0099] For example, we can first calculate the coordinates of the center point of each triangular facet in the watertight mesh: (A+B+C) / 3, where A, B, and C are the coordinates of the three vertices of the triangular facet in the watertight mesh. Next, for each sampling point, we calculate its spatial distance to the center points of all triangular faces. Then, for each sampling point, we find the k nearest (e.g., 5) center points in the set of triangular facet center points, obtaining a set of k candidate points corresponding to each sampling point.

[0100] After selecting multiple candidate points corresponding to the sampling points from the center points of the triangular facets of the watertight mesh, the k-nearest neighbor algorithm is used to determine the distance weight of each candidate point based on the distance between the candidate point and the sampling point. The distance weight is a weighting coefficient calculated based on the distance between the candidate point and the sampling point, used for weighted averaging among multiple candidate points. Candidate points that are closer to each other have a greater weight, reflecting the principle that "nearer points have a greater influence."

[0101] For example, for each candidate point of the sampling point, its inverse distance weight (the reciprocal of the square of its spatial distance) is calculated and normalized to the interval [0,1] to obtain the distance weight of the candidate point. It is easy to understand that the sum of the distance weights of all candidate points corresponding to a sampling point is 1. Furthermore, if the spatial distance between a candidate point and the sampling point is 0, that is, the sampling point coincides with the candidate point, then the distance weight of the candidate point is directly assigned to 1, and the distance weights of other candidate points are assigned to 0.

[0102] Finally, for a sampling point, the coordinates of the candidate point are weighted based on the distance weight of each candidate point to obtain the projected coordinates of the sampling point on the watertight grid. The sampling point is then projected from the original sample 3D model to the calculated projected coordinate position on the watertight grid to obtain the set of projected points corresponding to the sampling point set of the sample 3D model.

[0103] For example, suppose the sampling point In the original 3D face model, the point located near the lips has coordinates (10.2, 5.1, 3.3). The watertight mesh corresponding to this original 3D face model contains 50,000 triangular faces. The task is to determine this sampling point within this watertight mesh. Projection point: First, by calculating the sampling point The spatial distance to the center point of each triangular facet in the watertight mesh of the face is obtained from the sampling point. The 5 closest candidate points, for example, candidate point C1: distance from the sampling point 0.1mm, the triangular facet is located on the upper lip, candidate point C2: distance from this sampling point 0.15mm, the triangular facet is located at the lip crease, candidate point C3: distance from this sampling point 0.2mm, the triangular facet is located on the lower lip, candidate point C4: distance from this sampling point 0.25mm, the triangular facet is located around the lip, candidate point C5: distance from this sampling point The distance is 0.3mm, and the triangular facet it occupies is located on the cheek. Based on this, the distance weights of these 5 candidate points are calculated as follows: =0.51, =0.23, =0.13, =0.08, =0.05. Finally, the weighted sum is calculated to obtain the value of this sampling point. projection point The coordinates in the watertight grid are: =0.51×C1 + 0.23×C2 + 0.13×C3 + 0.08×C4 + 0.05×C5.

[0104] This k-nearest neighbor weighted projection method strikes a good balance between computational efficiency and projection quality. Through local weighted averaging, it maintains semantic consistency of sampling points while ensuring spatial smoothness and surface consistency of the projection results, providing high-quality geometric input for subsequent 3D model processing.

[0105] Thus, after obtaining the set of projection points corresponding to the set of sampling points of the original sample 3D model in the watertight mesh, the coordinates of the projection points can be further normalized to obtain the set of guide points: based on the position of the projection points in the set of projection points, the global bounding space of the sample 3D model is determined, and any projection point is within the global bounding space; according to the size and center position of the global bounding space, the projection points are normalized to obtain the set of guide points.

[0106] The global bounding space is a minimal, axis-aligned cuboid (AABB) that completely encloses all projected points in the set of projected points. It is a regular three-dimensional spatial region defined by minimum and maximum coordinate values. The dimensions of the global bounding space refer to its length in the X, Y, and Z dimensions, i.e., the length, width, and height of the cuboid. The center of the global bounding space refers to the coordinates of its geometric center point, i.e., the midpoint of the cuboid's diagonal. The guide point set is the normalized set of projected points, preserving all the topological and geometric relationships of the original point set, but with its coordinate values ​​standardized.

[0107] In the process of normalizing the projection points to obtain the guide point set, firstly, all projection points in the entire projection point set are traversed, and the maximum values ​​of these projection points on the X, Y, and Z coordinate axes are found respectively. and minimum value These six values ​​uniquely determine a minimum axis-aligned bounding box that contains all projected points, i.e., the global bounding space. Then, the center of the global bounding space is translated to the origin (0, 0, 0). This means subtracting the center position from the coordinates of each projected point in the set of projected points to obtain the translated coordinates. Then, the coordinates of each translated point are divided by the maximum size of the global bounding space. , , The maximum value among the three dimensions is used to normalize the maximum scale of the sample 3D model to a preset range, such as [-1, 1], thereby ensuring that the entire sample 3D model sequence is constrained within this range. Within a unit of space.

[0108] In this way, by normalizing the projection points, a standardized, unitless "shape description space" is established for the sample 3D model sequence, eliminating scale and position priors that are irrelevant to the essence (shape and structure) of the 3D model. This allows the denoising network to focus on learning the features of the shape itself, thereby improving the stability, convergence speed and generalization ability of the denoising network training.

[0109] (2) Select at least one query point from the set of guide points of the benchmark sample 3D model.

[0110] Here, the query point is a representative guide point selected from the guide point set of the benchmark 3D model. For example, at least one query point can be selected from the guide point set of the benchmark 3D model using a farthest point sampling algorithm, resulting in the query point set of the benchmark 3D model. That is, the query point set is a subset of the guide point set.

[0111] For example, first, a first guide point is randomly selected from the guide point set of the baseline sample 3D model. Then, from the remaining guide points, the point farthest from the selected guide point is selected and added to the query point set. This process is repeated until a predetermined number of query points (e.g., 512) are selected, resulting in a complete query point set corresponding to the baseline sample 3D model. Because these query points are spatially farthest from each other, they can represent the shape information and geometric features of the entire sample 3D model to the greatest extent possible with the fewest possible points.

[0112] (3) Select the guide points corresponding to the query points from the guide point set of the deformed sample three-dimensional model to obtain the query point set.

[0113] Since each sample 3D model in the sample 3D model sequence has a consistent topological structure (as does the corresponding watertight mesh model), the guide points in the guide point sets of each sample 3D model obtained after consistency sampling and watertight mesh projection in step S1 above are corresponding. That is, the i-th guide point in the guide point set of the baseline sample 3D model and the i-th guide point in the guide point set of the remaining deformed sample 3D models represent the same anatomical location of the sample 3D model, for example, both represent the "tip of the nose".

[0114] Therefore, after selecting at least one query point from the set of guide points in the baseline 3D model, guide points corresponding to these query points can be selected from the set of guide points in the deformed 3D models, thus obtaining the query point set corresponding to each deformed 3D model. In subsequent processing, the query points act as "anchor points" for encoder feature extraction and are used to generate the original feature distribution sequence of the sample 3D models.

[0115] (4) Based on the query point set and the guide point set, determine the original feature distribution sequence of the sample three-dimensional model corresponding to the sample video frame.

[0116] The original feature distribution sequence of the sample 3D model refers to the probability distribution parameters of the sample 3D model in space obtained after encoding the sample 3D model by an encoder. The original feature distribution sequence characterizes the statistical representation of the geometry of the corresponding sample 3D model in the latent space.

[0117] like Figure 4 As shown in the encoding network in the front part, after obtaining the set of guide points and query points corresponding to each sample 3D model through consistent sampling, watertight grid projection and farthest point screening, the two are input into the variational autoencoder (VAE) for compression encoding to achieve information compression while retaining the key geometric features of the sample 3D model, and obtain the potential distribution parameters of the sample 3D model shape, i.e. the original feature distribution sequence.

[0118] For example, the Variational Autoencoder (VAE) internally aggregates guide point features through cross-attention to update query point features, causing query points to "attention" to the guide points. This aggregates global shape information from the dense guide points to each query point. The updated query point features contain contextual information extracted from the dense point cloud. Then, self-attention optimization is performed within the query point set, allowing information exchange between query points to further optimize the features and capture the geometric relationships between points. Finally, these updated query point features are aggregated into a global feature vector (e.g., through max pooling or average pooling), and this global feature vector is mapped to the parameters of the latent variable distribution—the mean—through a fully connected layer. and variance (use (Represented by) the original feature distribution sequence of the sample 3D model.

[0119] for example, (256-dimensional mean vector). (256-dimensional standard deviation vector), then a 256-dimensional Gaussian distribution is defined through this original feature distribution sequence, which compactly encodes the complete 3D shape information of the sample's 3D model.

[0120] S3. Based on the preset noise sequence, noise is added to the original feature distribution sequence to obtain the sample feature sequence of the sample three-dimensional model corresponding to the sample video frame.

[0121] The preset noise sequence is a single, globally shared noise vector; that is, this global noise vector is used when adding noise to the original feature distribution sequence of each sample 3D model. It can be obtained by sampling from a standard normal distribution: .

[0122] In the process of adding noise to the original feature distribution sequence based on a preset noise sequence, instead of sampling an independent noise vector for each frame, a global noise vector is obtained by sampling only once. This is the preset noise sequence. Then, this single global noise vector is... The broadcast is given to all frames in the sample monocular video. This means that from frame 0 to frame T, all frames use the exact same noise vector. Finally, for each frame t, the latent distribution parameters of that frame are used ( and global noise vector To calculate the latent variables of this frame The sample feature sequence corresponding to the sample video frame is obtained as follows: + Ultimately, the latent variables of all frames The set of sample feature sequences that together constitute the sample 3D model sequence of the entire monocular video.

[0123] Existing variational autoencoders (VAEs) obtain noise vectors by randomly sampling each frame individually. And based on this, calculate the latent variables of the frame: + That is, the sample feature sequence, which would disrupt the temporal continuity of the entire sample 3D model sequence. Therefore, this embodiment uses a global noise vector. Eliminating randomness in the time dimension and making latent variables The dynamic changes are only caused by and The deterministic changes drive the process, thereby ensuring that the final generated set of sample feature sequences can faithfully and smoothly reflect the dynamic changes of the sample 3D model shape sequence.

[0124] S4. A denoising network for generating a model using a preset 3D model sequence is used to denoise the sample feature sequence in order to obtain a predicted feature distribution sequence.

[0125] like Figure 3 As shown, the denoising network of this application adds a new temporal self-attention module, namely a block sparse attention layer, to the Hunyuan3D model (used to generate 3D mesh models). Specifically, in the denoising network, each frame is first operated independently through the original cross-attention layer and self-attention layer (step 103 above) to capture complex spatial relationships. Subsequently, the block sparse attention layer models the motion and temporal dependencies of the target object across frames (steps 104 and 105). Thus, the static 3D generation capability of the Hunyuan3D model is extended to the dynamic 4D sequence generation capability.

[0126] One key challenge facing this new temporal self-attention module is computational complexity. A naive implementation of all-time-space attention has a complexity of O(N). 2 )=O((T×P) 2 ), where T is the number of frames, and P is the number of query points in the query point set of each frame (in the application of denoising networks, P can be understood as the number of noise points in the feature sequence of each frame). That is, any two query points (or noise points) are mutually attenuating. However, this computational complexity is too high. Therefore, this application designs an efficient block sparse attention mechanism specifically for this temporal self-attention module (see steps 104 and 105 below), which performs attention interaction at the feature block level, thus forming a block sparse attention layer. Furthermore, to ensure training stability, the output layer of the newly added temporal self-attention module is zero-initialized, enabling the denoising network to smoothly learn temporal relationships without destroying the powerful pre-trained weights.

[0127] Wherein, it is assumed that the total length of the input sequence of the temporal self-attention module is T is the number of frames, and P is the number of query points in the query point set of each frame. This is achieved by defining a fixed block size. (For example, 128 query points), divide the P query points in each frame into spatial dimensions. A contiguous block ( Then, inter-frame attention interactions are performed at the block level, at which point the computational complexity is O(N). 2 )=O((T× ) 2 This approach effectively reduces the computational complexity of the denoising network by dividing the network into blocks.

[0128] like Figure 4 As shown, the image features and sample feature sequences are input into the denoising network for denoising. The denoising network sequentially predicts the global noise vector added to the original feature distribution sequence through a cross-attention layer, a self-attention layer, and a block sparse attention layer, and then denoises to recover the original data, thus obtaining the predicted feature distribution sequence. The specific denoising process is similar to step 103 above, and steps 104 and 105 below; please refer to the context, and it will not be elaborated here.

[0129] As is easily understood, there can be multiple denoising networks (e.g., 21), which are sequentially stacked in series to form a complete denoising network. These denoising networks can integrate hybrid expert layers and use RMSnorm for normalization to output the intermediate denoising results corresponding to that network. Furthermore, the intermediate denoising results output by the previous denoising network serve as the input to the output of the next denoising network, while image features are input into each denoising network separately. Guided by the image features, denoising is gradually achieved, resulting in the final predicted feature distribution sequence for each frame.

[0130] S5. Based on the predicted feature distribution sequence and the original feature distribution sequence, determine the generation loss of the sample object, and based on the generation loss, converge the denoising network to obtain the three-dimensional model sequence generation model.

[0131] Specifically, by comparing the predicted feature distribution sequence of the denoising network with the original feature distribution sequence (i.e., the "standard answer") obtained from the real sample 3D model, the difference between the two is calculated as the generation loss. The gradient of the generation loss with respect to the parameters of the denoising network is also calculated. Finally, an optimizer (e.g., Adam) is used to update the weights of the denoising network according to the gradient to train and optimize the denoising network so that its predicted feature distribution sequence is closer to the original feature distribution sequence, until the denoising network converges, and a trained 3D model sequence generation model is obtained.

[0132] In addition, such as Figure 4 As shown, after obtaining the predicted feature distribution sequence, the predicted feature distribution sequence can be decoded by the decoding network to obtain the predicted three-dimensional model sequence. Then, based on the difference between the watertight grid model sequence corresponding to the original sample three-dimensional model sequence and the predicted three-dimensional model sequence, the generation loss is calculated to update the weights of the denoising network.

[0133] For example, the training process consisted of 24,000 iterations with a global batch size of 32, and each training sample contained 16 frames. During training, 4,096 query points were selected for loss calculation using farthest-point sampling. The entire training process was performed on eight GPUs with 96GB of VRAM each, taking approximately three days.

[0134] Thus, by iterating this process repeatedly on massive samples (adding noise → denoising → calculating loss → updating weights), the denoising network learns how to recover clean data consistent with the real 3D sequence based on the noise-damaged data and guided by image conditions. Ultimately, the denoising network converges into a powerful 3D model sequence generation model. This converged generation model has internalized the generation rules from monocular video to its corresponding 4D geometry (i.e., 3D model sequence), and can therefore be used to generate the corresponding 3D model sequence from monocular video.

[0135] It should be noted that in application, only the denoising network and decoding network of the 3D model sequence generation model are needed, and the encoding network is no longer required (because in application, there is no longer a 3D model corresponding to the video frame as input, and the 3D model is the generation target).

[0136] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of the three-dimensional model sequence generation model application provided in the embodiments of this application. For example... Figure 5 As shown, after obtaining the image features of the video frames through step 102 and generating the initial noise sequence corresponding to the 3D model of the target object (the initial noise sequence is the same for each video frame), the image features and the initial noise sequence are directly input into the denoising network of the 3D model sequence generation model. Through the trained denoising network with generative capabilities, the initial noise sequence is progressively denoised according to the image features to generate the feature distribution sequence corresponding to each frame (including step 103 above, and steps 104 and 105 below). Then, the feature distribution sequence is decoded by the decoding network to generate the 3D model corresponding to each frame, thereby obtaining the 3D model sequence corresponding to the entire monocular video of the target object.

[0137] 104. Based on the frame position of the video frame and the time distance between video frames, select at least one reference feature block associated with the feature block from the feature sequence set.

[0138] Here, frame position refers to the position index of a video frame in the time dimension, used to determine the temporal sequence of video frames. For example, the frame position of a video frame can be represented as... Temporal distance refers to the time interval between two frames, which can be calculated based on frame position; that is, temporal distance is measured in frames. For example, the [frame number missing]... Video frames and the first The temporal distance between video frames is 17. The reference feature block is a relevant feature block selected from the feature sequences of other frames to provide spatiotemporal consistency when denoising the current feature block.

[0139] like Figure 3As shown, after inputting the image features and the initial noise sequence into the denoising network, the initial noise sequence is first denoised based on the image features through a cross-attention layer and a self-attention layer, resulting in a feature sequence for each frame. The noise points in the feature sequence are then divided into blocks (for example, if the feature sequence contains 4096 noise points, each block contains 128 points, resulting in 32 blocks), thus obtaining the feature sequence set corresponding to the entire monocular video (i.e., step 103 above). Then, a block sparse attention layer filters at least one reference feature block associated with the current feature block from the feature sequence set to update the features of the current feature block, gradually achieving denoising.

[0140] Specifically, based on the frame position of the video frame and the temporal distance between video frames, at least one reference feature block associated with the feature block is selected from the feature sequence set, including: S1. Filter out the current video frame to which the feature block belongs in the video frame, and use the video frame in the monocular video as the reference video frame of the current video frame. The reference video frame includes the current video frame.

[0141] Specifically, for any given feature block, during the process of filtering its associated reference feature blocks, the video frame corresponding to the feature sequence to which the feature block belongs is taken as the current video frame. Simultaneously, all video frames in the monocular video are taken as reference video frames for the current video frame. That is, it is necessary to perform correlation analysis between the current video frame to which the current feature block belongs and each video frame (including the current video frame itself) to filter out the reference feature blocks associated with the current feature block from the feature sequence corresponding to each video frame.

[0142] S2. Based on the frame position of the video frame, calculate the time distance between the current video frame and the reference video frame, and determine the sampling step size corresponding to the reference video frame according to the time distance.

[0143] The difference between the frame positions of the current video frame and the reference video frame represents the time distance between them. For example, suppose the frame position of the current video frame is... The frame position of a certain reference video frame is The time distance between the two for: .

[0144] After obtaining the temporal distance between the current video frame and the reference video frame based on their frame positions, the sampling step size corresponding to the reference video frame can be determined based on this temporal distance. This allows for the sampling of a reference feature block associated with the current feature block in the current video frame. The calculation process for the sampling step size includes: Obtain the number of preset sampling steps in the preset sampling step set, and the step index identifier corresponding to the preset sampling step; determine the current step index identifier corresponding to the reference video frame based on the time distance and the number of steps; and select the sampling step of the feature sequence corresponding to the reference video frame from the preset sampling step set according to the current step index identifier.

[0145] The preset sampling step size set includes multiple preset sampling steps and a step size index identifier corresponding to each preset sampling step size. The step size index identifier is used to retrieve the corresponding preset sampling step size from the preset sampling step size set. The number of steps refers to the length of the preset sampling step size set, that is, the number of preset sampling steps included in the preset sampling step size set.

[0146] For example, the preset sampling step size set can be set as follows: The step size index identifier corresponding to each preset sampling step size is: Furthermore, the number of step sizes is 6, meaning that the preset sampling step size set contains 6 preset sampling step sizes. The preset sampling step size is represented by 4, that is, the step size index in the preset sampling step size set is 4. The preset sampling step size is 4.

[0147] After obtaining the number of preset sampling steps in the preset sampling step size set, for the current feature block, the current step size index identifier corresponding to the reference video frame can be calculated using the step size function based on the time distance between the current video frame to which the current feature block belongs and the reference video frame, as well as the number of steps. Then, in the preset sampling step size set, the preset sampling step size corresponding to the current step size index identifier is found and used as the sampling step size of the feature sequence corresponding to the reference video frame.

[0148] Wherein, step size function It can be as follows: ,in, This indicates the time distance between the current video frame and the reference video frame. Indicates the step size, and... and The smaller one is used as the current step size index identifier corresponding to the reference video frame.

[0149] For example, suppose the preset sampling step size set is: The frame position of the current video frame to which the current feature block belongs is =99, now determine the position of this feature block relative to the frame. When determining the reference relationship between reference video frames, firstly, the time distance between the two frames is calculated. The value is 49, and the preset sampling step size set has a step size of 6. Since 5 is less than 49, therefore Then, the step size index is found in the preset sampling step size set. The preset sampling step size is 8, therefore, the reference video frame at this time The sampling step size is 8, and then in the reference video frame... By sampling the feature sequence at intervals with a step size of 8, the reference feature block associated with the current feature block in the reference video frame can be obtained.

[0150] If the feature block is determined to be related to the frame position... When determining the reference relationship between reference video frames, firstly, the time distance between the two frames is calculated. It is 1 at this time. The step size index is found in the preset sampling step size set. The preset sampling step size is 1, therefore, the reference video frame at this time The sampling step size is 1, and then in the reference video frame... By sampling the feature sequence at intervals with a step size of 1, the reference feature block associated with the current feature block in the reference video frame can be obtained. That is, all feature blocks in the feature sequence of the reference video frame are reference feature blocks of the current feature block.

[0151] In addition, when determining the position of this feature block relative to the frame... When determining the reference relationship between a feature block and other feature blocks in its frame, the temporal distance is calculated. It is 0 at this time. The step size index is found in the preset sampling step size set. The preset sampling step size is 1, therefore, the reference video frame at this time The sampling step size is 1, that is, all feature blocks in the feature sequence of the reference video frame (i.e. the current video frame) are reference feature blocks associated with the current feature block. In other words, other feature blocks in the feature sequence of the current video frame are reference feature blocks of the current feature block.

[0152] Thus, for the current feature block, the closer the video frame is to it, the smaller the sampling step size, and the more reference feature blocks are selected from it to capture the global motion of the target object; the farther the video frame is to it, the larger the sampling step size, and the fewer reference feature blocks are selected from it to capture the fine-grained local motion of the target object.

[0153] S3. Based on the sampling step size, sample at least one reference feature block associated with the feature block from the feature sequence corresponding to the reference video frame.

[0154] After obtaining the sampling step size, interval sampling can be performed in the feature sequence of the reference video frame according to this sampling step size to obtain the reference feature block associated with the current feature block in the reference video frame. The specific sampling process includes: The block position of the feature block is identified in the current video frame, and the feature block at the block position is selected in the feature sequence corresponding to the reference video frame to obtain the reference feature block. Using the reference feature block as the sampling center, at least one candidate reference feature block associated with the feature block is sampled in the feature sequence corresponding to the reference video frame based on the sampling step size. The candidate reference feature block is updated according to the frame position of the current video frame to obtain the reference feature block.

[0155] The reference feature block is a feature block in the feature sequence of a reference video frame that has the exact same spatial location as the current feature block. The reference feature block represents the noise state of the same spatial region at another time point. Candidate reference feature blocks are a group of spatially adjacent feature blocks selected from the feature sequence of the reference video frame, centered on the reference feature block and arranged according to the sampling step size. These blocks are potential alternatives that the current feature block can query.

[0156] When sampling at intervals according to the sampling step size in the feature sequence of the reference video frame, firstly, the position of the current feature block in the feature sequence of the current video frame is determined, and a feature block at the same position is found in the feature sequence of the reference video frame as the reference feature block. For example, if the current feature block is the 15th feature block in frame 20, then the corresponding 15th feature block is found in the feature sequence of the reference video frame in frame 10 and used as the reference feature block. This establishes a spatial correspondence across time, ensuring that the comparison is performed between the "same part" at different times, which is the basis for effective temporal modeling. This is similar to comparing photos of a person's right arm at age 20 and age 10, ensuring comparability.

[0157] Then, in the feature sequence of the reference video frame, sampling is performed at intervals forward and backward, centered on the reference feature block and according to the sampling step size corresponding to that reference video frame (including the reference feature block itself). For example, in the feature sequence of the 10th frame of the reference video frame, assuming that the feature sequence contains 40 feature blocks, where the 15th feature block is the reference feature block and the sampling step size is 4, then in the feature sequence of the 10th frame, centered on the 15th feature block, the 11th, 7th, and 3rd feature blocks are sampled forward as candidate reference feature blocks, and simultaneously, the 19th, 23rd, 27th, 31st, 35th, and 39th feature blocks are sampled backward as candidate reference feature blocks. Afterward, this reference feature block, together with the candidate feature blocks, serves as the candidate reference feature block associated with the current feature block in the feature sequence of that reference video frame. This ensures that the current feature block can always focus on the corresponding feature block at the same position and other relatively aligned and related feature blocks in the far frame. This not only maintains the consistency and coherence of the target object's identity, but also efficiently skips a large number of irrelevant feature blocks, effectively captures the relative local motion patterns of the target object, and reduces computational complexity.

[0158] Finally, based on the frame position of the current video frame, it is determined whether the candidate reference feature block needs to be updated to obtain the reference feature block associated with the current feature block in the final feature sequence of the reference video frame. This determination process includes: Based on the frame position of the video frame, the first video frame is selected in the monocular video. When the current video frame is a video frame other than the first video frame, the unsampled feature blocks in the feature sequence of the first video frame are added to the candidate reference feature block to obtain the reference feature block. When the current video frame is the first video frame, the candidate reference feature block is used as the reference feature block.

[0159] Specifically, when the current video frame to which the current feature block belongs is a video frame other than the first video frame, that is, when the current video frame is not the first video frame, when referencing the first video frame, it is necessary to add the feature blocks that were not sampled after being sampled in the feature sequence of the first video frame in the above manner to the candidate reference feature blocks when referencing the first video frame. Thus, all feature blocks in the feature sequence of the first video frame are used as reference feature blocks for the current feature block.

[0160] For example, suppose the current feature block is the first... The 5th feature block of the video frame, then this feature block is referenced to the 5th feature block. When processing the first video frame, in addition to referencing the candidate feature blocks obtained by sampling at intervals as described above, it is also necessary to refer to all remaining feature blocks (unsampled feature blocks) in the feature sequence of that first video frame. Therefore, the unsampled feature blocks in the feature sequence of the first video frame are added to the candidate reference feature blocks to obtain the reference feature block for the current feature block in the first frame. Finally, all reference feature blocks from all video frames together constitute the reference feature block set corresponding to the current feature block, which is then used to update the current feature block for denoising.

[0161] When the current video frame is the first video frame, there is no need to update the candidate reference feature block; the candidate reference feature block sampled in the above manner is directly used as the reference feature block for the current feature block. This is because when sampling in the above manner, the feature block of this frame will refer to all other feature blocks in this frame. That is, the current feature block is the first... When sampling any feature block of the first video frame, the above sampling method will sample all feature blocks in the feature sequence of the first video frame as candidate reference feature blocks for the current feature block.

[0162] Thus, regardless of whether the current frame is the first or a non-first video frame, all feature blocks in the feature sequence of the first video frame need to be referenced. In other words, the feature blocks of all video frames must reference all information from the first video frame. Specifically, all feature blocks in the feature sequence of the first video frame are designated as global anchors, and all frame feature blocks retain the ability to reference all feature blocks from the first frame. This first-frame anchoring mechanism ensures that the denoising network always has access to the initial state, providing a stable reference for the target identity and appearance of the entire 3D model sequence and preventing semantic drift.

[0163] It's easy to understand that when determining the reference feature block corresponding to the current feature block through the first frame anchoring mechanism and step-size sampling mechanism described above, only feature blocks that meet the following conditions are considered reference feature blocks for the current feature block, and these blocks will be considered when the current feature block is updated:

[0164] in, This represents the spatial block index of the current feature block (query block) within the feature sequence of its corresponding video frame. For example, It may represent the "right elbow" area. This represents the spatial block index of the feature block (key block) to be referenced within the feature sequence of its respective video frame. Indicates time distance, i.e. and The frame rate difference between the corresponding video frames, It is a distance-step function, which can adjust the distance based on time. Dynamically output a sampling step size (modulus). Wherein, The larger (the further apart in time). The larger the value, the sparser the reference connections become. This represents the modulo (remainder) operation, therefore, it only applies if the current feature block... and feature blocks Divide by step size When the remainders obtained are equal, the feature block Only then is it the current feature block Only by referencing feature blocks can they be associated and attention connections established.

[0165] In this way, a feature block can always focus on feature blocks in other frames that are spatially identical to itself. This is the cornerstone of maintaining identity consistency and associating motion trajectories. For example, the "right elbow" block can be associated with the states of all "right elbow" blocks in historical frames. Simultaneously, it allows the denoising network to capture spatial relationships within a local range. For example, the "right elbow" block (u=17) can also reference information from the "right upper arm" block (v=1), which is crucial for understanding joint linkage. Furthermore, when s(d)=1 or very small, the condition u mod 1 = v mod 1 holds for any v (any number modulo 1 is 0), meaning that there is almost full attention between neighboring frames, allowing for fine-grained capture of rapid changes and details. When s(d) becomes large (e.g., 16), the condition becomes very strict, allowing only feature blocks of 1 / s(d) to establish connections. This significantly reduces computation while preserving the most structural and regular long-term connections, preventing the denoising network from forgetting important early information.

[0166] For example, during the denoising process, the feature sequence of each video frame is divided into P=64 feature blocks (e.g., blocks 0-15 represent the head, blocks 16-31 represent the torso, blocks 32-47 represent the left arm, and blocks 48-63 represent the right arm). The current feature block is a feature block of the right forearm in frame t=100, and its spatial block index is... When it is necessary to find the reference feature block of the current feature block in the feature sequence at frame t=95: first, determine the time distance between the two frames. And based on the scheduling table, such as S=[1,1,2,4,8,16], and the distance-step function You can find out when hour, =8. At this point, the filtering criteria are: ,Right now This means that at frame t=95, The current feature block can only focus on those spatial block indices. satisfy The feature block. At frame t=95, it satisfies... The feature blocks are: These feature blocks contain: identical locations: (i.e., the right forearm block at frame t=95) – used to correlate its own motion. Adjacent parts: (May correspond to the upper right arm or right shoulder) (Possibly corresponding to the right hand) — Used to capture limb coordination. More distant parts: such as... (Possibly located on the left shoulder or upper torso) – This could be used to capture coordinated movements of the opposite side of the body. Thus, through this rule, the attention required to calculate 64 feature blocks is now reduced to 8 reference feature blocks (reducing computation by 87.5%), while the most critical information channels (self, local neighbors, and global regularity points) are preserved.

[0167] 105. Based on the reference feature block, denoise the feature block to obtain the target feature sequence of the video frame, and generate a three-dimensional model sequence of the target object based on the target feature sequence.

[0168] The target feature sequence refers to the predicted feature distribution sequence corresponding to the video frame obtained after all denoising steps have been completed. Its noise has been basically eliminated and it contains clear three-dimensional geometric information.

[0169] In the process of denoising the feature block based on the reference feature block, the feature block is first denoised to obtain the updated feature sequence of the video frame. The current position is identified in the feature sequence to which the reference feature block belongs, and a mask sequence of the reference video frame corresponding to the feature sequence is generated based on the current position. The mask sequence and the feature sequence are fused to obtain the reference feature sequence of the reference video frame. Based on the reference feature sequence, the feature block is updated to obtain the updated feature sequence of the video frame.

[0170] The mask sequence is a weight vector with the same length as the feature sequence of the reference video frame. Each element quantifies whether the feature block at the corresponding position in the reference video frame is used to update the current feature block. For example, the mask sequence of a reference video frame containing 4 feature blocks can be [0,1,1,0]. That is, the 2nd and 3rd feature blocks in the feature sequence of the reference video frame are the reference feature blocks of the current feature block. When the current feature block is updated, for this reference video frame, only these two feature blocks are considered for updating, and the other two are ignored.

[0171] During the denoising and updating of feature blocks, firstly, based on the current position of the reference feature block in its respective feature sequence, a mask sequence of the reference video frame corresponding to the feature sequence is generated. For example, by creating a weight vector with the same length as the feature sequence of the reference video frame, setting the position corresponding to the reference feature block to 1, and setting the rest to 0, the mask sequence of the reference video frame corresponding to this feature sequence is obtained. For the current feature block, the mask sequences of the feature sequences of all video frames together form the attention mask matrix corresponding to the current feature block.

[0172] In this embodiment, the attention mask matrix corresponding to the current feature block can be:

[0173] in, This represents the spatial block index of the current feature block (query block) within the feature sequence of its corresponding video frame. The index represents the spatial block index of the feature block (key block) to be referenced within the feature sequence of its video frame, j represents the frame position of the video frame, and the j=0 frame represents the first video frame of the monocular video. Indicates if feature block If this condition is met, then it is the current feature block. The reference feature blocks. Thus, for all feature blocks, a diagonal attention pattern is defined.

[0174] This can be understood as follows: in the feature sequence of the first video frame, the mask for all feature blocks is 1, meaning that updates to all current feature blocks must focus on the first frame. In the feature sequence of non-first video frames, the mask for reference feature blocks associated with the current feature block is 1, and the mask for the remaining feature blocks is 0, meaning that updates to the current feature block only need to focus on a subset of feature blocks within that frame.

[0175] Then, the vector dot product of the mask sequence and the feature sequence is calculated, i.e., the feature sequence of the reference video frame is weighted and averaged using the mask sequence to obtain the reference feature sequence of the reference video frame. After obtaining the reference feature sequence of the reference video frame, the feature blocks are updated accordingly to obtain the updated feature sequence of the video frame: based on the reference feature sequence and the feature blocks, the attention weights of the reference video frame are determined, and the attention weights indicate the degree of correlation between the feature sequence and the feature blocks of the reference video frame; according to the attention weights, the reference feature blocks are attention-weighted to obtain weighted feature blocks; the feature blocks are replaced with the weighted feature blocks to update the feature sequences corresponding to the feature blocks, thus obtaining the updated feature sequence of the video frame.

[0176] The process involves using the current feature block as the query and the reference feature sequence of a reference video frame as the key to calculate an attention score. This score quantifies the correlation between the "state of the current feature block" and the "various information provided by the reference frame." Then, the attention score is normalized using the softmax function to the [0, 1] interval, yielding the attention weight for that reference video frame. Next, based on the attention weight of each reference video frame, the reference feature sequences of all reference video frames are weighted and summed to obtain a weighted feature block, which is then used to update the current feature block. This process is repeated for each feature block of each video frame to obtain the updated feature sequence for each video frame.

[0177] Finally, the updated feature sequence is used as the initial noise sequence of the video frame, and the process of denoising the initial noise sequence based on the image features is returned until the number of denoising operations reaches a preset number, thus obtaining the target feature sequence of the video frame.

[0178] like Figure 5 As shown, the target feature sequence output by the denoising network has reached a stable state after multiple iterations of denoising. It contains temporally coherent 4D shape information that is semantically aligned with the input video frame. Therefore, after obtaining the target feature sequence of each video frame, it can be decoded by the VAE decoder and finally rendered to generate the 3D model sequence corresponding to the monocular video of the target object.

[0179] Please refer to Figure 6 , Figure 6 This is a schematic diagram of the generated three-dimensional model sequence provided in the embodiments of this application. For example... Figure 6 As shown, for a 6-frame monocular video, a 3D model corresponding to each time frame was generated, resulting in a 3D model sequence with temporal order. Specifically, in each time frame, the main image on the right displays the 3D mesh from viewpoint 1 (front view), the image in the upper left corner displays the 3D mesh from viewpoint 2 (back view), and the image in the lower left corner displays the input video frame.

[0180] In this scheme, during the training process of the 3D model sequence generation model (including the denoising network), benchmark sampling points are obtained by sampling in the benchmark sample 3D model, and the topological mapping of the benchmark sampling points in the benchmark sample 3D model is recorded. , To obtain a set of sampling points, consistent sampling points corresponding to the baseline sampling points are sampled from the remaining sample 3D models. These sampling points are then projected onto the watertight mesh surface corresponding to the sample 3D model to obtain a set of guiding points. This ensures that subsequent input points to the encoder possess both temporal continuity and are located on a high-quality watertight mesh surface, improving the training effect of the denoising network. Simultaneously, a globally shared noise vector is used for noise addition during encoding to eliminate temporal randomness, further enhancing the training effect of the denoising network. Furthermore, a block sparse attention layer is introduced into the denoising network to model motion and temporal dependencies across frames. By selecting reference feature blocks from the feature sequence of each frame for attention association, cross-frame attention interaction is performed at the feature block level, effectively improving the generation quality of the 3D model sequence and reducing computational complexity.

[0181] As shown in Table 1, the 3D model sequence generation model described above in this application (last row in Table 1) is compared with L4GM (Latent Space 4D Gaussian Model), V2M4 (Video to 4D Model), and GVFD (Generative Volumetric Feature Distillation Model) models on a test set of 50 4D models from the Objaverse dataset. Two reference baselines, Hunyuan3D and Hunyuan3D*, are established, representing frame-by-frame 3D generation methods without and with globally shared noise, respectively. The model is trained based on 16 frames of input. We follow the ShapeGen4D evaluation protocol and use three main metrics to evaluate the geometric quality of the generated 4D sequences: 1. Chamfer↓, which measures the geometric difference between two point clouds; a lower value is better. 2. IoU↑, which measures the degree of overlap between two volumes; a higher value is better. 3. F-Score↑, which is a measure of the combined precision and recall; a higher value is better. Due to the difficulty in converting their Gaussian outputs to a watertight mesh, the IoU metrics for L4GM and GVFD were omitted. As shown in Table 1, the 3D model sequence generation method of this application significantly outperforms all compared methods in all metrics, demonstrating its superiority.

[0182] Table 1 Quantitative Comparison

[0183] Secondly, as shown in Table 2, ablation studies were conducted to analyze the impact of key design choices in this application. We evaluated the following variants of the 3D model sequence generation model: (1) without consistent sampling, which uses random sampling of the watertight mesh surface in the shape VAE encoding of each frame instead of our consistent sampling method; (2) without shared noise, where the VAE encoder uses different random noise for each frame instead of our global shared noise; (3) without sharp edge sampling, which is trained using only points sampled from the random surface, omitting sharp edge point samples; (4) temporal attention, which replaces our proposed temporal self-attention block with a pure temporal attention mechanism; and (5) full attention, which uses a complete, non-sparse spatiotemporal attention mechanism instead of our block sparse attention design. As shown in Table 2, we evaluated the computational complexity of the 3D model sequence generation model in PFLOPs (petaflops) based on a 16-frame image input sequence. The results clearly demonstrate that the proposed method significantly reduces the overall computational load of the network. Further layer-by-layer analysis reveals that our block sparse attention mechanism requires only 35% of the computational cost (PFLOPs) required by the standard full attention mechanism. This efficiency advantage becomes even more pronounced as the length of the image input sequence increases.

[0184] Table 2 Ablation Studies

[0185] As described above, in this embodiment, after obtaining the monocular video corresponding to the target object, feature extraction is performed on the video frames in the monocular video to obtain the image features of the video frames, and an initial noise sequence corresponding to the 3D model of the target object is generated. Then, based on the image features, the initial noise sequence is denoised to obtain a feature sequence set. Then, based on the frame position of the video frame and the time distance between video frames, at least one reference feature block associated with the feature block is selected from the feature sequence set. Then, based on the reference feature block, the feature block is denoised to obtain the target feature sequence of the video frame, and based on the target feature sequence, a 3D model sequence of the target object is generated. Since this scheme can select reference feature blocks associated with the current feature block from the feature sequence of each frame to perform cross-frame information interaction, it can effectively improve the generation quality of the 3D model sequence. At the same time, denoising and updating at the feature block level can effectively reduce computational complexity.

[0186] Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

[0187] In this embodiment, the three-dimensional model sequence generation device will be specifically integrated into an electronic device, with the electronic device being a server, as an example for explanation. like Figure 7 As shown, a training method for a 3D model sequence generation model is provided during the training process. The specific process is as follows: 201. The server obtains the sample monocular video and sample 3D model sequence corresponding to the sample object.

[0188] For example, the server downloads a sequence of 3D models of an animated object from a public dataset and renders multi-view images for each 3D model sequence. That is, it generates a set of 2D images containing multiple viewpoints for each time frame and pairs them with the 3D model (geometric data) of that time frame. Then, by extracting the 2D images from the same viewpoint in chronological order, the server obtains the sequence of 2D images of the animated object from each viewpoint, as well as the corresponding 3D model sequence, i.e., the sample monocular video and the sample 3D model sequence.

[0189] 202. The server samples in the reference sample 3D model to obtain at least one reference sampling point, and samples the corresponding consistency sampling points in the deformed sample 3D model to obtain a set of sampling points.

[0190] For example, the server selects a 3D model from the entire sequence of 3D models (e.g., 10 frames) as a reference and labels it as the reference sample 3D model. Then, all other sample 3D models in the sequence besides the reference sample 3D model are labeled as deformable sample 3D models. In this way, a fixed reference frame is established, and all sampling operations are first defined on the reference sample 3D model and then "propagated" to other frames.

[0191] The server samples several sharp edge points and uniform points (i.e., random surface points) from the benchmark 3D model, thus obtaining the set of sampling points corresponding to the benchmark 3D model. Then, for each benchmark sampling point... The system determines which triangular facet it is located on through spatial query (or finds the nearest facet through projection), and obtains and records the index of that triangular facet in the reference sample 3D model. Next, within this triangular facet, the centroid coordinates (u, v, w) of the reference sampling point relative to the three vertices are calculated and recorded as follows. This allows us to obtain the topological mapping of the reference sampling point in the three-dimensional model of the reference sample. , Then, based on the index identifier of that basic sampling point... In the 3D model of the deformed sample, the i-th triangular facet is directly located, and the 3D coordinates (A', B', C') of its three vertices are read, representing the new positions of the deformed facet. Finally, using the same centroid coordinates (u, v, w) and the deformed vertex coordinates (A', B', C'), the base sampling points are calculated through linear interpolation. Corresponding consistency sampling points The coordinates (sampling positions) are determined, and consistent sampling is performed in the 3D model of the deformed sample to obtain the set of sampling points corresponding to the 3D model of the deformed sample.

[0192] 203. The server projects the sampling points in the sampling point set onto the watertight mesh surface corresponding to the sample 3D model to which the sampling points belong, so as to obtain the guide point set corresponding to the sample 3D model.

[0193] For example, based on the model attribute information of the sample 3D model, the server constructs the sample bounding space of the sample 3D model, uniformly divides the sample bounding space into at least one voxel mesh, calculates the distance from the center point of the voxel mesh to the surface of the sample 3D model, obtains at least one mesh distance, generates at least one target triangular facet corresponding to the voxel mesh based on the mesh distance, and splices the target triangular facets to obtain the watertight mesh surface corresponding to the sample 3D model.

[0194] The server obtains the center point of at least one triangular facet in the watertight mesh corresponding to the sample 3D model and determines the spatial distance between the center point and the sampling point. Based on the spatial distance, at least one candidate point is selected from the center points, and the distance weight of the candidate point is determined according to the distance between the candidate point and the sampling point. Based on the distance weight, the coordinates of the candidate point are weighted to obtain the projected coordinates of the sampling point on the watertight mesh, and the sampling point is projected to the projected coordinates to obtain the set of projected points. Then, based on the position of the projected points in the set of projected points, the global bounding space of the sample 3D model is determined, and the projected points are normalized according to the size and center position of the global bounding space to obtain the set of guide points.

[0195] 204. The server selects at least one query point from the set of guide points in the baseline sample 3D model, and selects the guide point corresponding to the query point from the set of guide points in the deformed sample 3D model, thus obtaining the query point set.

[0196] For example, the server first randomly selects the first guide point from the guide point set of the baseline sample 3D model. Then, from the remaining guide points, it selects the point farthest from the selected guide point and adds it to the query point set. This process is repeated until a predetermined number of query points (e.g., 512) are selected, resulting in the complete query point set corresponding to the baseline sample 3D model. Then, the server filters out the guide points corresponding to the query points from the guide point set of the deformed sample 3D model, resulting in the query point set of the deformed sample 3D model.

[0197] 205. Based on the query point set and the guide point set, the server determines the original feature distribution sequence of the sample 3D model corresponding to the sample video frame.

[0198] For example, the server inputs the query point set and the guide point set into a variational autoencoder (VAE) for compressed encoding. It then aggregates guide point features through cross-attention to update query point features, ensuring that query points "attention" to guide points. This aggregates global shape information from dense guide points onto each query point. Next, self-attention optimization is applied within the query point set, allowing information exchange between query points to further optimize features and capture geometric relationships between them. Finally, these updated query point features are aggregated into a global feature vector (e.g., through max pooling or average pooling), and this global feature vector is mapped to the parameters of the latent variable distribution—the mean—through a fully connected layer. and variance (use (Represented by) the original feature distribution sequence of the sample 3D model.

[0199] 206. The server adds noise to the original feature distribution sequence based on a preset noise sequence to obtain the sample feature sequence of the sample three-dimensional model corresponding to the sample video frame.

[0200] For example, the server obtains a global noise vector through random sampling. Then, this single global noise vector... Broadcast to all frames in the sample monocular video. Finally, for each frame t, use the latent distribution parameters of that frame itself ( and global noise vector To calculate the latent variables of this frame The sample feature sequence corresponding to the sample video frame is obtained as follows: + Ultimately, the latent variables of all frames The set of sample feature sequences that together constitute the sample 3D model sequence of the entire monocular video.

[0201] 207. Based on image features, the server uses a denoising network of a preset 3D model sequence to denoise the sample feature sequence in order to obtain a predicted feature distribution sequence.

[0202] For example, the server inputs image features and sample feature sequences into a denoising network for denoising: the denoising network sequentially predicts the global noise vector added to the original feature distribution sequence through a cross attention layer, a self attention layer, and a block sparse attention layer, and then denoises to recover the original data, thus obtaining the predicted feature distribution sequence.

[0203] 208. The server determines the generation loss of the sample objects based on the predicted feature distribution sequence and the original feature distribution sequence, and converges the denoising network based on the generation loss to obtain the three-dimensional model sequence generation model.

[0204] For example, the server compares the predicted feature distribution sequence of the denoising network with the original feature distribution sequence obtained from the real sample 3D model (i.e., the "standard answer"), calculates the difference between the two as the generation loss, and calculates the gradient of the generation loss with respect to the parameters of the denoising network. Finally, an optimizer (e.g., Adam) is used to update the weights of the denoising network based on the gradient to train and optimize the denoising network, making its predicted feature distribution sequence closer to the original feature distribution sequence, until the denoising network converges, resulting in a trained 3D model sequence generation model. Alternatively, after obtaining the predicted feature distribution sequence, the server decodes the predicted feature distribution sequence using a decoding network to obtain the predicted 3D model sequence. Then, based on the difference between the watertight mesh model sequence corresponding to the original sample 3D model sequence and the predicted 3D model sequence, the server calculates the generation loss to update the weights of the denoising network, resulting in a trained 3D model sequence generation model.

[0205] In the training process of the 3D model sequence generation model (including the denoising network), this scheme obtains benchmark sampling points by sampling in the benchmark sample 3D model and records the topological mapping of the benchmark sampling points in the benchmark sample 3D model. , Based on this, consistent sampling points corresponding to the baseline sampling points are sampled from the remaining sample 3D models, resulting in a sampling point set. These sampling points are then projected onto the watertight mesh surface corresponding to the sample 3D model to obtain a guide point set. This ensures that subsequent input points to the encoder possess both temporal continuity and are located on a high-quality watertight mesh surface, thereby improving the training effect of the denoising network. Simultaneously, a globally shared noise vector is used for noise addition during encoding to eliminate temporal randomness, further enhancing the training effect of the denoising network.

[0206] like Figure 8As shown, a method for generating 3D model sequences is provided in the application of 3D model sequence generation. The specific process is as follows: 301. The server retrieves the monocular video corresponding to the target object.

[0207] For example, the server searches and downloads from a public database, or actively acquires a continuous image sequence of an entity from a certain perspective using a monocular camera, thus obtaining the monocular video corresponding to that entity.

[0208] 302. The server extracts features from the video frames to obtain the image features of the video frames and generates the initial noise sequence corresponding to the three-dimensional model of the target object.

[0209] For example, the server extracts features from face video frames using DINOv2 to obtain the image features of the video frames. At the same time, the server randomly samples multiple noise points from a standard normal distribution to construct an initial noise sequence for each video frame.

[0210] 303. The server denoises the initial noise sequence based on the image features to obtain a set of feature sequences.

[0211] For example, after obtaining the image features and initial noise sequence for each frame, the server inputs both into the 3D model sequence to generate the denoising network in the model. First, the initial noise sequence is made to "pay attention" to the image feature sequence through the cross-attention layer of the denoising network, thereby injecting image conditions into the noise points and guiding the denoising process. In the cross-attention layer, each noise point (Query) in the initial noise sequence of each frame "queries" the image features (Key / Value) of that video frame: for each noise point, it is used as a Query, and its similarity to the image feature Key is calculated. Then, the image feature Value is weighted and summed according to the similarity weight to obtain the updated noise point representation, thus obtaining the first intermediate feature sequence of the video frame. Then, the server allows all noise points in the first intermediate feature sequence to exchange information. That is, each noise point (Query) in the first intermediate feature sequence evaluates its own relevance to all other noise points (Key) in the sequence, and then, according to the relevance weight, the information (Value) of all other noise points is weighted and summed to update itself, thus obtaining the second intermediate feature sequence of the video frame. Then, the server divides the noise points in the second intermediate feature sequence into blocks to obtain the feature sequence corresponding to each frame, and then obtains the feature sequence set corresponding to the entire monocular video.

[0212] 304. The server calculates the time distance between the current video frame and the reference video frame based on the frame position of the video frame, and determines the sampling step size corresponding to the reference video frame based on the time distance.

[0213] For example, the server calculates the temporal distance between the current video frame and the reference video frame based on the difference in their frame positions. Then, it obtains the number of preset sampling steps from the preset sampling step size set. Based on the temporal distance between the current video frame and the reference video frame to which the current feature block belongs, and this number of steps, it calculates the current step size index identifier corresponding to the reference video frame using a step size function. Finally, it finds the preset sampling step size corresponding to the current step size index identifier in the preset sampling step size set and uses it as the sampling step size for the feature sequence corresponding to the reference video frame.

[0214] 305. Based on the sampling step size, the server samples at least one candidate reference feature block associated with the feature block from the feature sequence corresponding to the reference video frame.

[0215] For example, the server identifies the block position of the feature block in the current video frame, and filters out the feature block at the block position in the feature sequence corresponding to the reference video frame to obtain the reference feature block. Then, with the reference feature block as the center, it performs interval sampling forward and backward according to the sampling step size corresponding to the reference video frame, and the reference feature block together with the candidate feature blocks are used as candidate reference feature blocks associated with the current feature block in the feature sequence of the reference video frame.

[0216] 306. The server updates the candidate reference feature blocks based on the frame position of the current video frame to obtain the reference feature blocks associated with the current feature block in the feature sequence of the reference video frame.

[0217] For example, when the current video frame to which the current feature block belongs is a video frame other than the first video frame (i.e., the current video frame is not the first video frame), the server adds the feature blocks that were not sampled after being sampled in the feature sequence of the first video frame in the above manner to the candidate reference feature blocks when referencing the first video frame. Thus, all feature blocks in the feature sequence of the first video frame serve as reference feature blocks for the current feature block. When the current video frame is the first video frame, there is no need to update the candidate reference feature blocks; the candidate reference feature blocks sampled in the above manner are directly used as the reference feature blocks for the current feature block.

[0218] 307. The server identifies the current position in the feature sequence to which the reference feature block belongs, and generates a mask sequence of the reference video frame corresponding to the feature sequence based on the current position.

[0219] For example, the server creates a weight vector with the same length as the feature sequence of the reference video frame, sets the position corresponding to the reference feature block to 1, and sets the rest to 0, thus obtaining the mask sequence of the reference video frame corresponding to the feature sequence. For the current feature block, the mask sequences of the feature sequences of all video frames together form the attention mask matrix corresponding to the current feature block.

[0220] 308. The server denoises the feature blocks based on the mask sequence and reference feature blocks to obtain the target feature sequence of the video frame, and generates a three-dimensional model sequence of the target object based on the target feature sequence.

[0221] For example, the server calculates the vector dot product of the mask sequence and the feature sequence, i.e., it uses the mask sequence to perform a weighted average of the feature sequences of the reference video frame to obtain the reference feature sequence of the reference video frame. Then, using the current feature block as the query and the reference feature sequence of the reference video frame as the key, an attention score is calculated, and the attention score is normalized using the softmax function to normalize it to the [0, 1] interval, thus obtaining the attention weight of the reference video frame. Then, based on the attention weight of each reference video frame, the reference feature sequences of all reference video frames are weighted and summed to obtain a weighted feature block, which is then used to update the current feature block. This process is performed on each feature block of each video frame to obtain the updated feature sequence of each video frame. Then, the updated feature sequence is used as the initial noise sequence of the video frame, and the server returns to perform a step of denoising the initial noise sequence according to the image features until the denoising count reaches a preset number, thus obtaining the target feature sequence of the video frame. Then, the target feature sequence is decoded by the VAE decoder, and finally the 3D model sequence corresponding to the monocular video of the target object is rendered.

[0222] As described above, in this embodiment, after obtaining the monocular video corresponding to the target object, feature extraction is performed on the video frames in the monocular video to obtain the image features of the video frames, and an initial noise sequence corresponding to the 3D model of the target object is generated. Then, based on the image features, the initial noise sequence is denoised to obtain a feature sequence set. Then, based on the frame position of the video frame and the time distance between video frames, at least one reference feature block associated with the feature block is selected from the feature sequence set. Then, based on the reference feature block, the feature block is denoised to obtain the target feature sequence of the video frame, and based on the target feature sequence, a 3D model sequence of the target object is generated. Since this scheme can select reference feature blocks associated with the current feature block from the feature sequence of each frame to perform cross-frame information interaction, it can effectively improve the generation quality of the 3D model sequence. At the same time, denoising and updating at the feature block level can effectively reduce computational complexity.

[0223] To better implement the above methods, this application also provides a three-dimensional model sequence generation device, which can be integrated into electronic devices, such as servers or terminals. The terminal may include tablet computers, laptop computers, and / or personal computers.

[0224] For example, such as Figure 9 As shown, the 3D model sequence generation device may include an acquisition unit 301, an extraction unit 302, a denoising unit 303, a filtering unit 304, and a generation unit 305, as follows: (1) Obtain unit 301; Acquisition unit 301 is used to acquire the monocular video corresponding to the target object.

[0225] (2) Extraction unit 302; The extraction unit 302 is used to extract features from video frames, obtain image features of video frames, and generate an initial noise sequence corresponding to the three-dimensional model of the target object.

[0226] For example, the extraction unit 302 can be used to extract features from face video frames using DINOv2 to obtain image features, and randomly sample multiple noise points from a standard normal distribution to obtain the initial noise sequence corresponding to the video frame.

[0227] (3) Noise reduction unit 303; The denoising unit 303 is used to denoise the initial noise sequence according to the image features to obtain a feature sequence set.

[0228] For example, the denoising unit 303 can be used to use the denoising network in the three-dimensional model sequence generation model to perform attention weighting on the initial noise sequence and image features to obtain the first intermediate feature sequence of the video frame corresponding to the image features. The first intermediate feature sequence includes at least one noise point. Attention weighting is performed on the noise points in the same first intermediate feature sequence to obtain the second intermediate feature sequence of the video frame. The noise points are divided into blocks in the second intermediate feature sequence to obtain a feature sequence set.

[0229] (4) Filtering unit 304; The filtering unit 304 is used to filter at least one reference feature block associated with a feature block from the feature sequence set based on the frame position of the video frame and the time distance between video frames.

[0230] For example, the filtering unit 304 can be specifically used to filter out the current video frame to which the feature block belongs in the video frame, and use the video frame in the monocular video as the reference video frame of the current video frame. Based on the frame position of the video frame, the time distance between the current video frame and the reference video frame is calculated, and the sampling step size corresponding to the reference video frame is determined according to the time distance. Based on the sampling step size, at least one reference feature block associated with the feature block is sampled in the feature sequence corresponding to the reference video frame. Then, based on the frame position of the video frame, the first video frame is filtered out in the monocular video. When the current video frame is a video frame other than the first video frame, the unsampled feature blocks in the feature sequence of the first video frame are added to the candidate reference feature blocks to obtain the reference feature blocks. When the current video frame is the first video frame, the candidate reference feature blocks are used as the reference feature blocks.

[0231] (5) Generation unit 305; The generation unit 305 is used to denoise the feature block according to the reference feature block to obtain the target feature sequence of the video frame, and generate a three-dimensional model sequence of the target object based on the target feature sequence.

[0232] For example, the generation unit 305 can be used to identify the current position in the feature sequence to which the reference feature block belongs, and based on the current position, generate a mask sequence of the reference video frame corresponding to the feature sequence. The mask sequence is fused with the feature sequence to obtain a reference feature sequence of the reference video frame. Based on the reference feature sequence, the feature block is updated to obtain the updated feature sequence of the video frame. The updated feature sequence is used as the initial noise sequence of the video frame, and the step of denoising the initial noise sequence according to the image features is returned to be executed until the number of denoising times reaches a preset number to obtain the target feature sequence of the video frame. Then, the target feature sequence is decoded by the decoder, and finally the 3D model sequence corresponding to the monocular video of the target object is rendered and generated.

[0233] like Figure 10 As shown, the 3D model sequence generation device may further include a training unit 306: Training unit 306 trains the preset 3D model sequence generation model to obtain the 3D model sequence generation model.

[0234] For example, training unit 306 can be used to obtain sample monocular videos and sample 3D model sequences corresponding to sample objects. The sample 3D model sequence includes the sample 3D model corresponding to each sample video frame in the sample monocular video. A preset 3D model sequence generation model is used to sample the sample 3D model to obtain a set of sampling points. Based on the set of sampling points, the original feature distribution sequence of the sample 3D model corresponding to the sample video frame is generated. Based on a preset noise sequence, noise is added to the original feature distribution sequence to obtain the sample feature sequence of the sample 3D model corresponding to the sample video frame. The denoising network of the preset 3D model sequence generation model is used to denoise the sample feature sequence to obtain the predicted feature distribution sequence. Based on the predicted feature distribution sequence and the original feature distribution sequence, the generation loss of the sample object is determined. Based on the generation loss, the denoising network is converged to obtain the 3D model sequence generation model.

[0235] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0236] As can be seen from the above, in this embodiment, after the acquisition unit 301 acquires the monocular video corresponding to the target object, the extraction unit 302 extracts features from the video frames in the monocular video to obtain the image features of the video frames and generates an initial noise sequence corresponding to the three-dimensional model of the target object. Then, the denoising unit 303 denoises the initial noise sequence according to the image features to obtain a feature sequence set. The filtering unit 304 filters at least one reference feature block associated with the feature block from the feature sequence set based on the frame position of the video frame and the time distance between video frames. Then, the generation unit 305 denoises the feature block according to the reference feature block to obtain the target feature sequence of the video frame, and generates a three-dimensional model sequence of the target object based on the target feature sequence. Since this scheme can filter the reference feature block associated with the current feature block from the feature sequence of each frame to perform cross-frame information interaction, it can effectively improve the generation quality of the three-dimensional model sequence. At the same time, denoising and updating at the feature block level can effectively reduce computational complexity.

[0237] This application also provides an electronic device, such as... Figure 11 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically: The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 11The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 402, and calls data stored in the memory 402, to perform various functions and process data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0238] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0239] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0240] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0241] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows: The process involves: acquiring a monocular video corresponding to the target object, the monocular video comprising at least one video frame; extracting features from the video frame to obtain image features and generating an initial noise sequence corresponding to the 3D model of the target object; denoising the initial noise sequence based on the image features to obtain a feature sequence set, the feature sequence set including feature sequences corresponding to the video frame, and each feature sequence including at least one feature block; selecting at least one reference feature block associated with the feature block from the feature sequence set based on the frame position of the video frame and the time distance between video frames; denoising the feature block based on the reference feature block to obtain the target feature sequence of the video frame, and generating a 3D model sequence of the target object based on the target feature sequence.

[0242] For example, feature extraction is performed on face video frames using DINOv2 to obtain image features. Multiple noise points are randomly sampled from a standard normal distribution to obtain an initial noise sequence corresponding to the video frame. Then, a denoising network in a 3D model sequence generation model performs attention weighting on the initial noise sequence and image features to obtain a first intermediate feature sequence for the video frame corresponding to the image features. The first intermediate feature sequence includes at least one noise point. Attention weighting is then applied to the noise points in the same first intermediate feature sequence to obtain a second intermediate feature sequence for the video frame. Noise points in the second intermediate feature sequence are then divided into blocks to obtain a feature sequence set. Finally, the current video frame to which the feature block belongs is selected from the video frames, and the video from the monocular video is processed. The current video frame serves as a reference video frame. Based on the frame position of the current video frame, the temporal distance between the current video frame and the reference video frame is calculated, and the number of preset sampling step sizes in the preset sampling step size set is obtained. Based on the temporal distance and the number of step sizes, the current step size index identifier corresponding to the reference video frame is determined. According to the current step size index identifier, the sampling step size of the feature sequence corresponding to the reference video frame is selected from the preset sampling step size set. Then, the block position of the feature block is identified in the current video frame, and the feature block at the block position is selected from the feature sequence corresponding to the reference video frame to obtain the reference feature block. Using the reference feature block as the sampling center, at least one candidate parameter associated with the feature block is sampled from the feature sequence corresponding to the reference video frame based on the sampling step size. The method involves considering feature blocks. Based on the frame position of a video frame, the first video frame is selected from the monocular video. When the current video frame is not the first frame, unsampled feature blocks from the feature sequence of the first frame are added to the candidate reference feature block, resulting in a reference feature block. When the current video frame is the first frame, the candidate reference feature block is used as the reference feature block. At least one reference feature block associated with the feature block is sampled from the feature sequence corresponding to the reference video frame. Then, the current position is identified in the feature sequence to which the reference feature block belongs, and a mask sequence of the reference video frame corresponding to the feature sequence is generated based on the current position. The mask sequence is fused with the feature sequence to obtain the reference feature sequence of the reference video frame. Finally, based on the reference features... The sequence and feature blocks are used to determine the attention weights of the reference video frame. The attention weights indicate the degree of correlation between the feature sequence and the feature blocks of the reference video frame. Based on the attention weights, the reference feature blocks are attentionally weighted to obtain weighted feature blocks. The feature blocks are then replaced with the weighted feature blocks to update the feature sequences corresponding to the feature blocks, resulting in the updated feature sequences of the video frame. Then, the updated feature sequences are used as the initial noise sequences of the video frame, and the process of denoising the initial noise sequences based on image features is repeated until the denoising count reaches a preset number, resulting in the target feature sequences of the video frame. Finally, the target feature sequences are decoded by a decoder, and the 3D model sequence corresponding to the monocular video of the target object is rendered.

[0243] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0244] As described above, in this embodiment, after obtaining the monocular video corresponding to the target object, feature extraction is performed on the video frames in the monocular video to obtain the image features of the video frames, and an initial noise sequence corresponding to the 3D model of the target object is generated. Then, based on the image features, the initial noise sequence is denoised to obtain a feature sequence set. Then, based on the frame position of the video frame and the time distance between video frames, at least one reference feature block associated with the feature block is selected from the feature sequence set. Then, based on the reference feature block, the feature block is denoised to obtain the target feature sequence of the video frame, and based on the target feature sequence, a 3D model sequence of the target object is generated. Since this scheme can select reference feature blocks associated with the current feature block from the feature sequence of each frame to perform cross-frame information interaction, it can effectively improve the generation quality of the 3D model sequence. At the same time, denoising and updating at the feature block level can effectively reduce computational complexity.

[0245] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0246] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the three-dimensional model sequence generation methods provided in embodiments of this application. For example, the instructions can execute the following steps: The process involves: acquiring a monocular video corresponding to the target object, the monocular video comprising at least one video frame; extracting features from the video frame to obtain image features and generating an initial noise sequence corresponding to the 3D model of the target object; denoising the initial noise sequence based on the image features to obtain a feature sequence set, the feature sequence set including feature sequences corresponding to the video frame, and each feature sequence including at least one feature block; selecting at least one reference feature block associated with the feature block from the feature sequence set based on the frame position of the video frame and the time distance between video frames; denoising the feature block based on the reference feature block to obtain the target feature sequence of the video frame, and generating a 3D model sequence of the target object based on the target feature sequence.

[0247] For example, an electronic device can extract features from face video frames using DINOv2 to obtain image features. Multiple noise points are randomly sampled from a standard normal distribution to obtain an initial noise sequence corresponding to the video frame. Then, a denoising network in a 3D model sequence generation model performs attention weighting on the initial noise sequence and image features to obtain a first intermediate feature sequence for the video frame corresponding to the image features. The first intermediate feature sequence includes at least one noise point. Attention weighting is then applied to the noise points in the same first intermediate feature sequence to obtain a second intermediate feature sequence for the video frame. The noise points in the second intermediate feature sequence are then divided into blocks to obtain a feature sequence set. Finally, the current video frame to which the feature block belongs is selected from the video frames, and the monocular video... The video frame in the current video frame is used as a reference video frame. Based on the frame position of the current video frame, the time distance between the current video frame and the reference video frame is calculated, and the number of preset sampling step sizes in the preset sampling step size set is obtained. Based on the time distance and the number of step sizes, the current step size index identifier corresponding to the reference video frame is determined. According to the current step size index identifier, the sampling step size of the feature sequence corresponding to the reference video frame is selected from the preset sampling step size set. Then, the block position of the feature block is identified in the current video frame, and the feature block at the block position is selected in the feature sequence corresponding to the reference video frame to obtain the reference feature block. Using the reference feature block as the sampling center, at least one feature associated with the feature block is sampled in the feature sequence corresponding to the reference video frame based on the sampling step size. Candidate reference feature blocks are selected based on the frame position of the video frame. The first video frame is selected from the monocular video. When the current video frame is not the first frame, unsampled feature blocks from the feature sequence of the first frame are added to the candidate reference feature block, resulting in a reference feature block. When the current video frame is the first frame, the candidate reference feature block is used as the reference feature block. At least one reference feature block associated with the feature block is sampled from the feature sequence corresponding to the reference video frame. Then, the current position is identified in the feature sequence to which the reference feature block belongs, and a mask sequence of the reference video frame corresponding to the feature sequence is generated based on the current position. The mask sequence is fused with the feature sequence to obtain the reference feature sequence of the reference video frame. Finally, based on the reference features... The system uses feature sequences and feature blocks to determine the attention weights of reference video frames. The attention weights indicate the degree of correlation between the feature sequences and feature blocks of the reference video frames. Based on the attention weights, the reference feature blocks are attentionally weighted to obtain weighted feature blocks. The feature blocks are then replaced with the weighted feature blocks to update the feature sequences corresponding to the feature blocks, resulting in the updated feature sequences of the video frames. The updated feature sequences are then used as the initial noise sequences of the video frames, and the system returns to perform denoising on the initial noise sequences based on image features until the denoising count reaches a preset number, resulting in the target feature sequences of the video frames. Finally, the target feature sequences are decoded by a decoder, and the system renders and generates the 3D model sequence corresponding to the monocular video of the target object.

[0248] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0249] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0250] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the three-dimensional model sequence generation methods provided in the embodiments of this application, the beneficial effects that any of the three-dimensional model sequence generation methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0251] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various alternative implementations of the generative model training and application described above.

[0252] The foregoing has provided a detailed description of a three-dimensional model sequence generation method and related equipment provided in the embodiments of this application. The related equipment may include a three-dimensional model sequence generation device, electronic equipment, computer program products, and computer-readable storage media. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for generating a three-dimensional model sequence, characterized in that, include: Obtain the monocular video corresponding to the target object, wherein the monocular video includes at least one video frame; Feature extraction is performed on the video frame to obtain the image features of the video frame, and an initial noise sequence corresponding to the three-dimensional model of the target object is generated; Based on the image features, the initial noise sequence is denoised to obtain a feature sequence set, which includes the feature sequence corresponding to the video frame, and the feature sequence includes at least one feature block; Based on the frame position of the video frame and the time distance between the video frames, at least one reference feature block associated with the feature block is selected from the feature sequence set; Based on the reference feature block, the feature block is denoised to obtain the target feature sequence of the video frame, and based on the target feature sequence, a three-dimensional model sequence of the target object is generated.

2. The method for generating a three-dimensional model sequence according to claim 1, characterized in that, The step of filtering at least one reference feature block associated with the feature block from the feature sequence set based on the frame position of the video frame and the time distance between the video frames includes: The current video frame to which the feature block belongs is selected from the video frames, and the video frames in the monocular video are used as reference video frames for the current video frame, the reference video frames including the current video frame; Based on the frame position of the video frame, the time distance between the current video frame and the reference video frame is calculated, and the sampling step size corresponding to the reference video frame is determined according to the time distance. Based on the sampling step size, at least one reference feature block associated with the feature block is sampled from the feature sequence corresponding to the reference video frame.

3. The method for generating a three-dimensional model sequence according to claim 2, characterized in that, Determining the sampling step size corresponding to the reference video frame based on the time distance includes: Obtain the number of preset sampling steps in the preset sampling step set, wherein the preset sampling step corresponds to a step index identifier; Based on the time distance and the number of steps, determine the current step size index identifier corresponding to the reference video frame; Based on the current step size index identifier, the sampling step size of the feature sequence corresponding to the reference video frame is selected from the preset sampling step size set.

4. The method for generating a three-dimensional model sequence according to claim 2, characterized in that, The step of sampling at least one reference feature block associated with the feature block from the feature sequence corresponding to the reference video frame based on the sampling step size includes: The block position of the feature block is identified in the current video frame, and the feature block at the block position is filtered out in the feature sequence corresponding to the reference video frame to obtain the reference feature block; Using the reference feature block as the sampling center, at least one candidate reference feature block associated with the feature block is sampled from the feature sequence corresponding to the reference video frame based on the sampling step size; The candidate reference feature block is updated based on the frame position of the current video frame to obtain the reference feature block.

5. The method for generating a three-dimensional model sequence according to claim 4, characterized in that, The step of updating the candidate reference feature block based on the frame position of the current video frame to obtain the reference feature block includes: Based on the frame position of the video frame, the first video frame is selected from the monocular video. When the current video frame is a video frame other than the first video frame, the unsampled feature blocks in the feature sequence of the first video frame are added to the candidate reference feature block to obtain the reference feature block; When the current video frame is the first video frame, the candidate reference feature block is used as the reference feature block.

6. The method for generating a three-dimensional model sequence according to claim 2, characterized in that, The step of denoising the feature block based on the reference feature block to obtain the target feature sequence of the video frame includes: Based on the reference feature block, the feature block is denoised to obtain the updated feature sequence of the video frame; The updated feature sequence is used as the initial noise sequence of the video frame, and the step of denoising the initial noise sequence according to the image features is returned to be executed until the number of denoising times reaches a preset number, so as to obtain the target feature sequence of the video frame.

7. The method for generating a three-dimensional model sequence according to claim 6, characterized in that, The step of denoising the feature block based on the reference feature block to obtain the updated feature sequence of the video frame includes: The current position is identified in the feature sequence to which the reference feature block belongs, and a mask sequence of the reference video frame corresponding to the feature sequence is generated based on the current position. The mask sequence and the feature sequence are fused to obtain the reference feature sequence of the reference video frame; Based on the reference feature sequence, the feature block is updated to obtain the updated feature sequence of the video frame.

8. The method for generating a three-dimensional model sequence according to claim 7, characterized in that, The step of updating the feature block based on the reference feature sequence to obtain the updated feature sequence of the video frame includes: Based on the reference feature sequence and the feature block, the attention weight of the reference video frame is determined, and the attention weight indicates the degree of correlation between the feature sequence and the feature block of the reference video frame; Based on the attention weights, the reference feature block is attention-weighted to obtain a weighted feature block; The feature block is replaced with the weighted feature block to update the feature sequence corresponding to the feature block, thereby obtaining the updated feature sequence of the video frame.

9. The method for generating a three-dimensional model sequence according to any one of claims 1 to 8, characterized in that, The step of denoising the initial noise sequence based on the image features to obtain a feature sequence set includes: The initial noise sequence is attentionally weighted with the image features by a denoising network in the 3D model sequence generation model to obtain the first intermediate feature sequence of the video frame corresponding to the image features. The first intermediate feature sequence includes at least one noise point. The noise points in the same first intermediate feature sequence are subjected to attention weighting to obtain the second intermediate feature sequence of the video frame; The noise points are divided into blocks in the second intermediate feature sequence to obtain the feature sequence set.

10. The method for generating a three-dimensional model sequence according to claim 9, characterized in that, Before the denoising network in the 3D model sequence generation model performs attention weighting on the initial noise sequence and the image features to obtain the first intermediate feature sequence of the video frame corresponding to the image features, the method further includes: Obtain the sample monocular video and sample 3D model sequence corresponding to the sample object, wherein the sample 3D model sequence includes the sample 3D model corresponding to each sample video frame in the sample monocular video; The sample 3D model is sampled using a preset 3D model sequence to obtain a set of sampling points, and based on the set of sampling points, the original feature distribution sequence of the sample 3D model corresponding to the sample video frame is generated. Based on a preset noise sequence, noise is added to the original feature distribution sequence to obtain the sample feature sequence of the sample three-dimensional model corresponding to the sample video frame; The sample feature sequence is denoised using the denoising network of the preset three-dimensional model sequence generation model to obtain the predicted feature distribution sequence; Based on the predicted feature distribution sequence and the original feature distribution sequence, the generation loss of the sample object is determined, and based on the generation loss, the denoising network is converged to obtain a three-dimensional model sequence generation model.

11. The method for generating a three-dimensional model sequence according to claim 10, characterized in that, The step of sampling the sample 3D model using a preset 3D model sequence to obtain a set of sampling points includes: A baseline sample 3D model is selected from the sample 3D model sequence, and the sample 3D models other than the baseline sample 3D model are used as deformable sample 3D models. Sampling is performed in the three-dimensional model of the reference sample to obtain at least one reference sampling point, and the sampling position corresponding to the reference sampling point is identified in the three-dimensional model of the deformed sample; Based on the sampling location, consistent sampling points corresponding to the benchmark sampling points are sampled in the three-dimensional model of the deformed sample to obtain a set of sampling points.

12. The method for generating a three-dimensional model sequence according to claim 11, characterized in that, The step of identifying the sampling position corresponding to the reference sampling point in the three-dimensional model of the deformed sample includes: Extract the index identifier and centroid coordinates of the triangular facet to which the reference sampling point belongs from the three-dimensional model of the reference sample; In the three-dimensional model of the deformed sample, identify the coordinates of the deformed vertices of the triangular facet corresponding to the index identifier; Based on the coordinates of the deformed vertex and the coordinates of the centroid, the coordinates of the consistency sampling point corresponding to the basic sampling point are determined in the three-dimensional model of the deformed sample, and the sampling position is obtained.

13. The method for generating a three-dimensional model sequence according to claim 11, characterized in that, The step of generating the original feature distribution sequence of the sample 3D model corresponding to the sample video frame based on the sampling point set includes: Project the sampling points in the sampling point set onto the watertight mesh surface corresponding to the sample 3D model to which the sampling points belong, to obtain the guide point set corresponding to the sample 3D model; At least one query point is selected from the set of guide points in the benchmark sample 3D model; The guide points corresponding to the query points are selected from the guide point set of the deformed sample 3D model to obtain the query point set; Based on the query point set and the guide point set, the original feature distribution sequence of the sample 3D model corresponding to the sample video frame is determined.

14. The method for generating a three-dimensional model sequence according to claim 13, characterized in that, The step of projecting the sampling points in the sampling point set onto the watertight mesh surface corresponding to the 3D model of the sample to which the sampling points belong, to obtain the guide point set, includes: Project the sampling points in the sampling point set onto the watertight mesh surface corresponding to the sample 3D model to which the sampling points belong, to obtain the projection point set; Based on the position of the projection points in the set of projection points, the global bounding space of the sample 3D model is determined, and any one of the projection points is within the global bounding space; Based on the size and center position of the global bounding space, the projection points are normalized to obtain a set of guiding points.

15. The method for generating a three-dimensional model sequence according to claim 14, characterized in that, The step of projecting the sampling points in the sampling point set onto the watertight mesh surface corresponding to the 3D model of the sample to which the sampling points belong, to obtain the projection point set, includes: Obtain the center point of at least one triangular facet in the watertight mesh corresponding to the three-dimensional model of the sample, and determine the spatial distance between the center point and the sampling point; Based on the spatial distance, at least one candidate point is selected from the center point, and the distance weight of the candidate point is determined according to the distance between the candidate point and the sampling point. Based on the distance weight, the coordinates of the candidate points are weighted to obtain the projected coordinates of the sampling points on the watertight grid, and the sampling points are projected onto the projected coordinates to obtain a set of projected points.

16. The method for generating a three-dimensional model sequence according to claim 14, characterized in that, Before projecting the sampling points in the sampling point set onto the watertight mesh surface corresponding to the 3D model of the sample to which the sampling points belong, to obtain the projection point set, the method further includes: Based on the model attribute information of the sample 3D model, construct the sample bounding space of the sample 3D model; At least one voxel grid is uniformly divided into the space surrounding the sample, and the distance from the center point of the voxel grid to the surface of the three-dimensional model of the sample is calculated to obtain at least one grid distance. Based on the grid distance, at least one target triangular facet corresponding to the voxel grid is generated, and the target triangular facets are spliced ​​together to obtain the watertight grid surface corresponding to the sample 3D model.

17. A three-dimensional model sequence generation device, characterized in that, include: An acquisition unit is used to acquire a monocular video corresponding to a target object, wherein the monocular video includes at least one video frame. The extraction unit is used to extract features from the video frame, obtain the image features of the video frame, and generate an initial noise sequence corresponding to the three-dimensional model of the target object; A denoising unit is configured to denoise the initial noise sequence based on the image features to obtain a feature sequence set, the feature sequence set including the feature sequence corresponding to the video frame, and the feature sequence including at least one feature block; A filtering unit is configured to filter at least one reference feature block associated with the feature block from the feature sequence set based on the frame position of the video frame and the time distance between the video frames; The generation unit is configured to denoise the reference feature block to obtain the target feature sequence of the video frame, and generate a three-dimensional model sequence of the target object based on the target feature sequence.

18. An electronic device, characterized in that, It includes a processor and a memory, the memory storing an application program, and the processor running the application program in the memory to perform the steps in the three-dimensional model sequence generation method according to any one of claims 1 to 16.

19. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps in the three-dimensional model sequence generation method according to any one of claims 1 to 16.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the three-dimensional model sequence generation method according to any one of claims 1 to 16.

Images