Method for reconstructing movable three-dimensional gaussian avatar based on autoregressive graph neural network

Abstract

The application discloses a movable three-dimensional Gaussian avatar reconstruction method based on an autoregressive graph neural network, which comprises the following steps: 1) obtaining face image data and preprocessing the same to obtain face image data without background, and dividing the face image data into source images and driving images; 2) training a constructed three-dimensional Gaussian avatar reconstruction network by using the preprocessed face image data to obtain a trained three-dimensional Gaussian avatar reconstruction network; 3) inputting a single source image as an avatar image and inputting a driving image or a video containing multiple driving images into the trained three-dimensional Gaussian avatar reconstruction network, so that a high-quality movable three-dimensional avatar can be output, and the expression and posture of the driving image can be reconstructed. The application effectively solves the problem of order-of-magnitude mismatch between the expression Gaussian and the head Gaussian, avoids excessive densification of the three-dimensional Gaussian, and realizes high-quality single-image movable three-dimensional avatar reconstruction.

Description

Technical Field

[0001] This invention relates to the technical field of 3D head reconstruction and dynamic head synthesis, and in particular to a method for reconstructing a movable 3D Gaussian head based on an autoregressive graphical neural network. Background Technology

[0002] Single-image human head reconstruction, which involves starting from a single source image and using driving video or images to animate facial expressions, is a key task in applications such as virtual digital humans and immersive online meetings. Its core objective is to synthesize a high-quality speaking head that retains the identity information of the source image while incorporating the facial expressions and postures of the driving image. The central challenge of this task lies in how to realistically reconstruct complex 3D facial geometry and appearance from a single 2D image in real time.

[0003] To address these challenges, existing methods combine the geometric representation of a 3D deformation model with a visual neural network model to achieve higher reconstruction quality and multi-view consistency, and utilize 3D Gaussian sputtering to render target facial images. However, existing 3D Gaussian sputtering-based methods still face two significant challenges: First, there is a fundamental scale mismatch between the facial priors of the 3D deformation model and the 3D Gaussian representation. Methods driven by the 3D deformation model are limited by a fixed number of vertices and sparse edge topology, which inherently restricts the scale of the reconstructed 3D Gaussian representation, making it difficult to coordinate with the dense 3D Gaussian representation required for high-fidelity head reconstruction. Some methods still fall short in matching 3D Gaussian representations for scenes of arbitrary scales. Second, multi-level 3D Gaussian generation methods suffer from uncontrollable density dilation. These methods typically employ fixed expansion rules and lack an active filtering mechanism for the generated 3D Gaussian representation, leading to significant redundancy and over-densification, while failing to allocate sufficient detail to key expression regions such as eyes and teeth. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a method for reconstructing an animated 3D Gaussian head based on an autoregressive graph neural network. This method utilizes a graph neural network to propagate information on the facial mesh topology, uses an autoregressive structure to split Gaussian layers layer by layer to capture multi-scale facial details, and introduces a soft mask network to dynamically control the Gaussian density. This effectively solves the problem of the order of magnitude mismatch between the Gaussian density of facial expressions and the Gaussian density of the head, and avoids the over-densification of 3D Gaussian density, thus achieving high-quality single-image animated 3D head reconstruction.

[0005] To achieve the above objectives, the technical solution provided by this invention is: a method for reconstructing an animated 3D Gaussian head based on an autoregressive graphical neural network, comprising the following steps:

[0006] 1) Acquire face image data and preprocess it to obtain face image data with background removed, and divide it into source image and driving image; where the source image is the face image of the person whose image needs to be reconstructed, and the driving image is the face image of the person whose expression and posture need to be tracked and transferred.

[0007] 2) The preprocessed face image data is used to train the constructed 3D Gaussian head reconstruction network. During training, a multi-factor joint loss function, including image reconstruction loss, perceptual loss, lifting loss, and split Gaussian loss, is used to calculate the loss between the network prediction result and the driving image, resulting in a trained 3D Gaussian head reconstruction network. The 3D Gaussian head reconstruction network includes a Gaussian feature adapter, an autoregressive graphical neural network, and a neural renderer. The Gaussian feature adapter is used to reconstruct the head image of the source image, and an improved double-lifting method is used to generate identity Gaussians. The specific improvement of the double-lifting method is as follows: the face features of the source image are extracted using a pre-trained visual feature extractor, and the face features are input into two sets of visions. The Transformer extracts feature planes and predicts the offsets in the forward and backward directions, thereby calculating the identity Gaussian parameters. The autoregressive graphical neural network includes a Gaussian generation module and multiple autoregressive layers to reconstruct the expression and pose of the driving image and generate expression Gaussian parameters. The Gaussian generation module processes the driving image to obtain facial mesh parameters and camera pose parameters, and generates the initial expression Gaussian for the first autoregressive layer. Each autoregressive layer contains a graphical neural network and a soft masking network. Starting from the second autoregressive layer, each autoregressive layer expands the expression Gaussian of the previous autoregressive layer to generate a new expression Gaussian. A more refined Gaussian expression is obtained, wherein the graph neural network aligns and outputs the expanded Gaussian expression using a grid topology expansion method. This grid topology expansion method copies the graph neural network structure of the previous autoregressive layer multiple times, enabling parameter propagation and training optimization between structures. The soft masking network predicts a set of masks equal to the number of Gaussian expressions in that layer, used to label the Gaussian expressions that need to be filtered. The neural renderer renders the filtered Gaussian expressions to generate prediction results. During training, the identity Gaussian obtained from the Gaussian feature adapter, all Gaussian expressions from the autoregressive graph neural network, and the masks are input into the neural renderer to obtain the prediction results.

[0008] 3) Given a single source image as the avatar, and input the driving image or a video containing multiple driving images into the trained 3D Gaussian avatar reconstruction network, a high-quality animated 3D avatar can be output, reconstructing the expression and posture of the driving image.

[0009] Further, in step 1), the preprocessing includes two steps: face detection and background removal. Face detection uses VGGHead to filter images containing faces and crops and scales the images to a uniform size. Background removal uses StyleMatte to obtain face image data with the background removed and divides it into source image and driving image.

[0010] Further, in step 2), the 3D Gaussian head reconstruction network performs the head reconstruction task by predicting multiple 3D Gaussians and rendering them into an image, wherein each 3D Gaussian... Defined by the center position μ and the covariance matrix Σ, for any point x in three-dimensional space:

[0011] ;

[0012] In the formula, express The transpose of the covariance matrix Σ is decomposed into a rotation matrix R and a scaling matrix S to ensure positive semi-qualitativeness.

[0013] ;

[0014] In the formula, and Represents the transpose of matrices R and S; each 3D Gaussian stores attributes including position, opacity, rotation, scaling, and color characteristics;

[0015] The Gaussian feature adapter utilizes an improved double-lifting method to take the source image as input and output an identity Gaussian feature. The specific improvement of the dual-lifting method is as follows: Facial features of the source image are extracted using a pre-trained DINOv2 algorithm. These facial features are then input into two sets of Vision Transformers. The intermediate layer output is processed using transposed convolution to obtain the feature plane. The offsets in the forward and backward directions are predicted respectively, thereby calculating the Gaussian position μ in three-dimensional space and generating... Attributes;

[0016] The autoregressive graphical neural network includes a Gaussian generation module and multiple autoregressive layers for generating Gaussian expressions. ;Will Defined as a sequence with a total of L+1 layers , Let represent the Gaussian expression of the i-th layer. The steps for generating all Gaussian expressions are as follows:

[0017] The facial features of the driving image and the source image are input into the Gaussian generation module. The facial mesh parameters and camera pose parameters of the driving image are calculated using a 3D deformation model estimator. Then, the obtained parameters are concatenated with the facial features, and the initial expression Gaussian is generated through the MLP to generate the 0th layer of the sequence, i.e., the initial layer. Subsequently, each autoregressive layer expands the Gaussian expression of the previous layer by a factor of t using a graph neural network and a soft masking network, predicts the Gaussian properties of the current layer, and generates new, more refined Gaussian expressions, including... Collectively referred to as split Gaussian The graph neural network requires a graph structure to guide its training, using a facial mesh as the initial graph. As the graph structure of the i-th layer of the graph neural network, it also needs to copy t copies of the initial graph; among them, Let i be the set of vertices in the i-th layer. The edge information for the i-th layer is the topologically inherited edge for the i-th layer. and the internal connection edge of the i-th layer union , Preserve the connectivity of the facial mesh within the copy:

[0018] ;

[0019] In the formula, Let the set of edges be the initial graph. Let be an edge in the initial graph, with its two vertices being... and , This indicates that the i-th layer consists of the original vertices. The m-th vertex copy generated in total One vertex copy; Indicates the initial edge The m-th edge copy derived from it;

[0020] Establish connections between vertices at the same position in each replica to ensure information propagation in the autoregressive graph neural network:

[0021] ;

[0022] In the formula, The set of vertices of the initial graph. This represents the point of the m-th replica in the i-th layer. and the point of the nth copy The edges formed by connecting them;

[0023] The autoregressive graphical neural network generates Gaussian expressions. The specific operation is as follows:

[0024] First, the i-th layer of the expression Gaussian sequence... The various attributes are obtained through the feature embedding module. Mapping to high-dimensional latent features :

[0025] ;

[0026] Will Input Graph Neural Network Add residual connections After being predicted by decoder D, the expression Gaussian of layer i+1 is obtained. Attributes:

[0027] ;

[0028] In the formula, The scaling factor is used for each attribute; the Gaussian attributes obtained here are unfiltered and have an over-dense problem, requiring the use of a designed soft masking network. Further filtering: The input consists of MLP Predict the soft mask of the (i+1)th layer. :

[0029] ;

[0030] During training, the i-th layer mask Copy t times to get Perform a logical AND operation with the (i+1)th layer soft mask to obtain the (i+1)th layer logical mask. :

[0031] ;

[0032] Quantization function Binarize the soft mask m; its expression is... :

[0033] ;

[0034] The soft masking network utilizes a stopping gradient operation. After ensuring gradient truncation via quantization and logical AND operations, the autoregressive graphical neural network training proceeds normally. Finally, the mask for layer i+1... The calculation is defined as follows:

[0035] ;

[0036] Additionally, the 0th layer mask Set all values ​​to 1, and do not calculate according to the above formula;

[0037] The neural renderer is used to render the filtered Gaussian generation prediction results, and the output is an image of the same size as the source image. The color C of each pixel in the image is calculated as follows:

[0038] ;

[0039] In the formula, N is the total number of Gaussians participating in the rendering of that pixel. The effective opacity of the k-th Gaussian after 2D projection. For the first Effective opacity of a Gaussian after two-dimensional projection It is the k-th Gaussian color feature. The first three dimensions contain RGB information, and a coarse image is rendered using RGB information. Render the final detailed image using complete color features. ;

[0040] The multivariate joint loss function during the training of the 3D Gaussian head reconstruction network consists of four parts:

[0041] Image reconstruction loss For coarse images and fine images Calculate the target image separately L1 distance:

[0042] ;

[0043] Perceived loss Through feature extraction function Measure the similarity between the generated image and the target image in the feature space:

[0044] ;

[0045] Increase losses Constrain the initial expression Gaussian The position should be as close as possible to the vertices of the tracked face mesh. :

[0046] ;

[0047] Split Gaussian loss Split Gaussian constrained graph neural network Get as close as possible to the tracked face mesh vertices :

[0048] .

[0049] Furthermore, in step 3), the specific steps for applying the trained 3D Gaussian head reconstruction network are as follows: First, prepare the source image, driving image, or video. If it is a video, it needs to be converted into multiple driving images. Then, preprocess the source image and driving image to obtain face image data with the background removed. The image size is consistent with the face image during training. Finally, input the face image data into the 3D Gaussian head reconstruction network. The output is a high-quality, animable 3D head that restores the character image of the source image and the expression and posture of the driving image.

[0050] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0051] 1. The autoregressive graphical neural network used in this invention effectively solves the order-of-magnitude mismatch between the Gaussian expression generated by the 3D deformation model and the identity Gaussian of the image reconstruction. Compared with the method of directly using the vertices of the 3D deformation model as the Gaussian expression, it can generate richer facial details.

[0052] 2. The mesh topology expansion method proposed in this invention ensures that the feature information of the facial region can be effectively propagated in each layer of the autoregressive graph neural network, providing more information for network training and reproducing more accurate facial expressions and poses.

[0053] 3. The soft masking network of this invention accelerates rendering time while ensuring the stability and efficiency of the training process by adaptively selecting Gaussians.

[0054] 4. This invention is highly configurable, and the number of Gaussian layers generated can be flexibly controlled by adjusting the number of autoregressive layers and the expansion factor to adapt to the needs of scenarios with different complexities. Attached Figure Description

[0055] Figure 1 This is an architectural diagram of the method of the present invention.

[0056] Figure 2 This is a schematic diagram of the neural network and grid topology extension of the present invention.

[0057] Figure 3 This is a schematic diagram illustrating the process of generating an animated avatar according to the present invention. Detailed Implementation

[0058] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0059] like Figures 1 to 3 As shown in the figure, this embodiment discloses a method for reconstructing a movable 3D Gaussian head based on an autoregressive graphical neural network, the specific details of which are as follows:

[0060] 1) Acquire face image data and preprocess it to obtain face image data with background removed, and divide it into source image and driving image; where the source image is the face image of the person whose image needs to be reconstructed, and the driving image is the face image of the person whose expression and posture need to be tracked and transferred.

[0061] Specifically, the preprocessing includes two steps: face detection and background removal. Face detection uses VGGHead to filter images containing faces and then crops and scales the images to a uniform size. Background removal uses StyleMatte to obtain face image data with the background removed and then divides it into source images and driving images.

[0062] 2) The preprocessed face image data is used to train the constructed 3D Gaussian head reconstruction network. During training, a multi-factor joint loss function, including image reconstruction loss, perceptual loss, lifting loss, and split Gaussian loss, is used to calculate the loss between the network prediction result and the driving image, resulting in a trained 3D Gaussian head reconstruction network. The 3D Gaussian head reconstruction network includes a Gaussian feature adapter, an autoregressive graphical neural network, and a neural renderer. The Gaussian feature adapter is used to reconstruct the head image of the source image, and an improved double-lifting method is used to generate identity Gaussians. The specific improvement of the double-lifting method is as follows: the face features of the source image are extracted using a pre-trained visual feature extractor, and the face features are input into two sets of visions. The Transformer extracts feature planes and predicts the offsets in the forward and backward directions, thereby calculating the identity Gaussian parameters. The autoregressive graphical neural network includes a Gaussian generation module and multiple autoregressive layers to reconstruct the expression and pose of the driving image and generate expression Gaussian parameters. The Gaussian generation module processes the driving image to obtain facial mesh parameters and camera pose parameters, and generates the initial expression Gaussian for the first autoregressive layer. Each autoregressive layer contains a graphical neural network and a soft masking network. Starting from the second autoregressive layer, each autoregressive layer expands the expression Gaussian of the previous autoregressive layer to generate a new expression Gaussian. A more refined Gaussian expression is generated, wherein the graph neural network aligns and outputs the expanded Gaussian expression using a grid topology expansion method. This grid topology expansion method copies the graph neural network structure of the previous autoregressive layer multiple times, enabling parameter propagation and training optimization between structures. The soft masking network predicts a set of masks equal to the number of Gaussian expressions in that layer, used to label the Gaussian expressions that need to be filtered. The neural renderer renders the filtered Gaussian expressions to generate prediction results. During training, the identity Gaussian obtained from the Gaussian feature adapter, all Gaussian expressions from the autoregressive graph neural network, and the masks are input into the neural renderer to obtain the prediction results.

[0063] Specifically, the 3D Gaussian head reconstruction network achieves the head reconstruction task by predicting multiple 3D Gaussians and rendering them into an image, wherein each 3D Gaussian... Defined by the center position μ and the covariance matrix Σ, for any point x in three-dimensional space:

[0064] ;

[0065] In the formula, express The transpose of the covariance matrix Σ is decomposed into a rotation matrix R and a scaling matrix S to ensure positive semi-qualitativeness.

[0066] ;

[0067] In the formula, and Represents the transpose of matrices R and S; each 3D Gaussian stores attributes including position, opacity, rotation, scaling, and color characteristics;

[0068] The Gaussian feature adapter utilizes an improved double-lifting method to take the source image as input and output an identity Gaussian feature. The specific improvement of the dual-lifting method is as follows: Facial features of the source image are extracted using a pre-trained DINOv2 algorithm. These facial features are then input into two sets of Vision Transformers. The intermediate layer output is processed using transposed convolution to obtain the feature plane. The offsets in the forward and backward directions are predicted respectively, thereby calculating the Gaussian position μ in three-dimensional space and generating... Attributes;

[0069] The autoregressive graphical neural network includes a Gaussian generation module and multiple autoregressive layers for generating Gaussian expressions. ;Will Defined as a sequence with a total of L+1 layers , Let represent the Gaussian expression of the i-th layer. The steps for generating all Gaussian expressions are as follows:

[0070] The facial features of the driving image and the source image are input into the Gaussian generation module. The facial mesh parameters and camera pose parameters of the driving image are calculated using a 3D deformation model estimator. Then, the obtained parameters are concatenated with the facial features, and the initial expression Gaussian is generated through the MLP to generate the 0th layer of the sequence, i.e., the initial layer. Subsequently, each autoregressive layer expands the Gaussian expression of the previous layer by a factor of t using a graph neural network and a soft masking network, predicts the Gaussian properties of the current layer, and generates new, more refined Gaussian expressions, including... Collectively referred to as split Gaussian The graph neural network requires a graph structure to guide its training, using a facial mesh as the initial graph. As the graph structure of the i-th layer of the graph neural network, it also needs to copy t copies of the initial graph; among them, Let i be the set of vertices in the i-th layer. The edge information for the i-th layer is the topologically inherited edge for the i-th layer. and the internal connection edge of the i-th layer union , Preserve the connectivity of the facial mesh within the copy:

[0071] ;

[0072] In the formula, Let the set of edges be the initial graph. Let be an edge in the initial graph, with its two vertices being... and , This indicates that the i-th layer consists of the original vertices. The m-th vertex copy generated in total One vertex copy; Indicates the initial edge The m-th edge copy derived from it;

[0073] Establish connections between vertices at the same position in each replica to ensure information propagation in the autoregressive graph neural network:

[0074] ;

[0075] In the formula, The set of vertices of the initial graph. This represents the point of the m-th replica in the i-th layer. and the point of the nth copy The edges formed by connecting them;

[0076] The autoregressive graphical neural network generates Gaussian expressions. The specific operation is as follows:

[0077] First, the i-th layer of the expression Gaussian sequence... The various attributes are obtained through the feature embedding module. Mapping to high-dimensional latent features :

[0078] ;

[0079] Will Input Graph Neural Network Add residual connections After being predicted by decoder D, the expression Gaussian of layer i+1 is obtained. Attributes:

[0080] ;

[0081] In the formula, The scaling factor is used for each attribute; the Gaussian attributes obtained here are unfiltered and have an over-dense problem, requiring the use of a designed soft masking network. Further filtering: The input consists of MLP Predict the soft mask of the (i+1)th layer. :

[0082] ;

[0083] During training, the i-th layer mask Copy t times to get Perform a logical AND operation with the (i+1)th layer soft mask to obtain the (i+1)th layer logical mask. :

[0084] ;

[0085] Quantization function Binarize the soft mask m; its expression is... :

[0086] ;

[0087] The soft masking network utilizes a stopping gradient operation. After ensuring gradient truncation via quantization and logical AND operations, the autoregressive graphical neural network training proceeds normally. Finally, the mask for layer i+1... The calculation is defined as follows:

[0088] ;

[0089] Additionally, the 0th layer mask Set all values ​​to 1, and do not calculate according to the above formula;

[0090] The neural renderer is used to render the filtered Gaussian generation prediction results, and the output is an image of the same size as the source image. The color C of each pixel in the image is calculated as follows:

[0091] ;

[0092] In the formula, N is the total number of Gaussians participating in the rendering of that pixel. The effective opacity of the k-th Gaussian after 2D projection. For the first Effective opacity of a Gaussian after two-dimensional projection It is the k-th Gaussian color feature. The first three dimensions contain RGB information, and a coarse image is rendered using RGB information. Render the final detailed image using complete color features. ;

[0093] The multivariate joint loss function during the training of the 3D Gaussian head reconstruction network consists of four parts:

[0094] Image reconstruction loss For coarse images and fine images Calculate the target image separately L1 distance:

[0095] ;

[0096] Perceived loss Through feature extraction function Measure the similarity between the generated image and the target image in the feature space:

[0097] ;

[0098] Increase losses Constrain the initial expression Gaussian The position should be as close as possible to the vertices of the tracked face mesh. :

[0099] ;

[0100] Split Gaussian loss Split Gaussian constrained graph neural network Get as close as possible to the tracked face mesh vertices :

[0101] .

[0102] 3) Given a single source image as the avatar, and inputting the driving image or a video containing multiple driving images into the trained 3D Gaussian avatar reconstruction network, a high-quality, animable 3D avatar can be output, reconstructing the expression and pose of the driving image. The specific steps are as follows:

[0103] First, prepare source images, driving images, or videos. If it is a video, it needs to be converted into multiple driving images. Then, preprocess the source images and driving images to obtain face image data with the background removed. The image size is consistent with the face image during training. Finally, input the face image data into the 3D Gaussian head reconstruction network. The output is a high-quality, animable 3D head that restores the image of the person in the source image and the expression and posture of the driving image.

[0104] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for reconstructing an animated 3D Gaussian head based on an autoregressive graphical neural network, characterized in that, Includes the following steps: 1) Acquire face image data and preprocess it to obtain face image data with background removed, and divide it into source image and driving image; where the source image is the face image of the person whose image needs to be reconstructed, and the driving image is the face image of the person whose expression and posture need to be tracked and transferred. 2) The preprocessed face image data is used to train the constructed 3D Gaussian head reconstruction network. During training, a multi-factor joint loss function, including image reconstruction loss, perceptual loss, lifting loss, and split Gaussian loss, is used to calculate the loss between the network prediction result and the driving image, resulting in a trained 3D Gaussian head reconstruction network. The 3D Gaussian head reconstruction network includes a Gaussian feature adapter, an autoregressive graphical neural network, and a neural renderer. The Gaussian feature adapter is used to reconstruct the head image of the source image, and an improved double-lifting method is used to generate identity Gaussians. The specific improvement of the double-lifting method is as follows: the face features of the source image are extracted using a pre-trained visual feature extractor, and the face features are input into two sets of visions. The Transformer extracts feature planes and predicts the offsets in the forward and backward directions, thereby calculating the identity Gaussian parameters. The autoregressive graphical neural network includes a Gaussian generation module and multiple autoregressive layers to reconstruct the expression and pose of the driving image and generate expression Gaussian parameters. The Gaussian generation module processes the driving image to obtain facial mesh parameters and camera pose parameters, and generates the initial expression Gaussian for the first autoregressive layer. Each autoregressive layer contains a graphical neural network and a soft masking network. Starting from the second autoregressive layer, each autoregressive layer expands the expression Gaussian of the previous autoregressive layer to generate a new expression Gaussian. A more refined Gaussian expression is obtained, wherein the graph neural network aligns and outputs the expanded Gaussian expression using a grid topology expansion method. This grid topology expansion method copies the graph neural network structure of the previous autoregressive layer multiple times, enabling parameter propagation and training optimization between structures. The soft masking network predicts a set of masks equal to the number of Gaussian expressions in that layer, used to label the Gaussian expressions that need to be filtered. The neural renderer renders the filtered Gaussian expressions to generate prediction results. During training, the identity Gaussian obtained from the Gaussian feature adapter, all Gaussian expressions from the autoregressive graph neural network, and the masks are input into the neural renderer to obtain the prediction results. 3) Given a single source image as the avatar, and input the driving image or a video containing multiple driving images into the trained 3D Gaussian avatar reconstruction network, a high-quality animated 3D avatar can be output, reconstructing the expression and posture of the driving image.

2. The method for reconstructing an animated 3D Gaussian head based on an autoregressive graphical neural network according to claim 1, characterized in that, In step 1), the preprocessing includes two steps: face detection and background removal. Face detection uses VGGHead to filter images containing faces and crops and scales the images to a uniform size. Background removal uses StyleMatte to obtain face image data with the background removed and divides it into source image and driving image.

3. The method for reconstructing an animated 3D Gaussian head based on an autoregressive graphical neural network according to claim 1, characterized in that, In step 2), the 3D Gaussian head reconstruction network performs the head reconstruction task by predicting multiple 3D Gaussians and rendering them into an image. Each 3D Gaussian... Defined by the center position μ and the covariance matrix Σ, for any point x in three-dimensional space: ； In the formula, express The transpose of the covariance matrix Σ is decomposed into a rotation matrix R and a scaling matrix S to ensure positive semi-qualitativeness. ； In the formula, and Represents the transpose of matrices R and S; each 3D Gaussian stores attributes including position, opacity, rotation, scaling, and color characteristics; The Gaussian feature adapter utilizes an improved double-lifting method to take the source image as input and output an identity Gaussian feature. The specific improvement of the dual-lifting method is as follows: Facial features of the source image are extracted using a pre-trained DINOv2 algorithm. These facial features are then input into two sets of Vision Transformers. The intermediate layer output is processed using transposed convolution to obtain the feature plane. The offsets in the forward and backward directions are predicted respectively, thereby calculating the Gaussian position μ in three-dimensional space and generating... Attributes; The autoregressive graphical neural network includes a Gaussian generation module and multiple autoregressive layers for generating Gaussian expressions. ;Will Defined as a sequence with a total of L+1 layers , Let represent the Gaussian expression of the i-th layer. The steps for generating all Gaussian expressions are as follows: The facial features of the driving image and the source image are input into the Gaussian generation module. The facial mesh parameters and camera pose parameters of the driving image are calculated using a 3D deformation model estimator. Then, the obtained parameters are concatenated with the facial features, and the initial expression Gaussian is generated through the MLP to generate the 0th layer of the sequence, i.e., the initial layer. Subsequently, each autoregressive layer expands the Gaussian expression of the previous layer by a factor of t using a graph neural network and a soft masking network, predicts the Gaussian properties of the current layer, and generates new, more refined Gaussian expressions, including... Collectively referred to as split Gaussian The graph neural network requires a graph structure to guide its training, using a facial mesh as the initial graph. As the graph structure of the i-th layer of the graph neural network, it also needs to copy t copies of the initial graph; among them, Let i be the set of vertices in the i-th layer. The edge information for the i-th layer is the topologically inherited edge for the i-th layer. and the internal connection edge of the i-th layer union , Preserve the connectivity of the facial mesh within the copy: ； In the formula, Let the set of edges be the initial graph. Let be an edge in the initial graph, with its two vertices being... and , This indicates that the i-th layer consists of the original vertices. The m-th vertex copy generated in total One vertex copy; Indicates the initial edge The m-th edge copy derived from it; Establish connections between vertices at the same position in each replica to ensure information propagation in the autoregressive graph neural network: ； In the formula, The set of vertices of the initial graph. This represents the point of the m-th replica in the i-th layer. and the point of the nth copy The edges formed by connecting them; The autoregressive graphical neural network generates Gaussian expressions. The specific operation is as follows: First, the i-th layer of the expression Gaussian sequence... The various attributes are obtained through the feature embedding module. Mapping to high-dimensional latent features : ； Will Input Graph Neural Network Add residual connections After being predicted by decoder D, the expression Gaussian of layer i+1 is obtained. Attributes: ； In the formula, The scaling factor is used for each attribute; the Gaussian attributes obtained here are unfiltered and have an over-dense problem, requiring the use of a designed soft masking network. Further filtering: The input consists of MLP Predict the soft mask of the (i+1)th layer. : ； During training, the i-th layer mask Copy t times to get Perform a logical AND operation with the (i+1)th layer soft mask to obtain the (i+1)th layer logical mask. : ； Quantization function Binarize the soft mask m; its expression is... : ； The soft masking network utilizes a stopping gradient operation. After ensuring gradient truncation via quantization and logical AND operations, the autoregressive graphical neural network training proceeds normally. Finally, the mask for layer i+1... The calculation is defined as follows: ； Additionally, the 0th layer mask Set all values ​​to 1, and do not calculate according to the above formula; The neural renderer is used to render the filtered Gaussian generation prediction results, and the output is an image of the same size as the source image. The color C of each pixel in the image is calculated as follows: ； In the formula, N is the total number of Gaussians participating in the rendering of that pixel. The effective opacity of the k-th Gaussian after 2D projection. For the first Effective opacity of a Gaussian after two-dimensional projection It is the k-th Gaussian color feature. The first three dimensions contain RGB information, and a coarse image is rendered using RGB information. Render the final detailed image using complete color features. ; The multivariate joint loss function during the training of the 3D Gaussian head reconstruction network consists of four parts: Image reconstruction loss For coarse images and fine images Calculate the target image separately L1 distance: ； Perceived loss Through feature extraction function Measure the similarity between the generated image and the target image in the feature space: ； Increase losses Constrain the initial expression Gaussian The position should be as close as possible to the vertices of the tracked face mesh. : ； Split Gaussian loss Split Gaussian constrained graph neural network Get as close as possible to the tracked face mesh vertices : 。 4. The method for reconstructing a movable 3D Gaussian head based on an autoregressive graphical neural network according to claim 1, characterized in that, In step 3), the specific steps for applying the trained 3D Gaussian head reconstruction network are as follows: First, prepare the source image, driving image, or video. If it is a video, it needs to be converted into multiple driving images. Then, preprocess the source image and driving image to obtain face image data with the background removed. The image size is consistent with the face image during training. Finally, input the face image data into the 3D Gaussian head reconstruction network. The output is a high-quality, animable 3D head that restores the character image of the source image and the expression and posture of the driving image.

Images