A 3D Gaussian Modeling Method and System Based on Poincaré Sphere and Three-Plane Representation

Abstract

This invention discloses a 3D Gaussian modeling method and system based on a Poincaré sphere and three-plane representation. The method includes: acquiring and preprocessing a human body image; inputting randomly generated Gaussian variables into a three-plane geometric feature branch to obtain geometric features; inputting the UV mapping obtained during the training of the 3D human body reconstruction network model into a hyperbolic U-net UV feature extraction branch to obtain UV features; fusing the geometric features and UV features and inputting them into a multi-branch decoder to obtain 3D Gaussian rendering core parameters; and synthesizing a 3D human avatar image based on the 3D Gaussian rendering core parameters. This invention can generate high-fidelity, real-time 3D human body models.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D modeling technology, and in particular to a three-dimensional Gaussian modeling method, system, storage medium, and electronic device based on Poincaré sphere and three-plane representation. Background Technology

[0002] With the widespread adoption of 3D digitization technology in entertainment and communication fields such as virtual reality (VR) interaction, augmented reality (AR) communication, digital film and television character creation, and game character modeling, high-fidelity 3D avatar modeling has become a core technological requirement to support the implementation of these scenarios. Currently, 3D avatar modeling technology is mainly divided into two major technical routes: implicit representation and explicit representation. However, both methods have significant bottlenecks, making it difficult to simultaneously meet the practical needs of complex pose adaptation, fine detail restoration, and efficient modeling.

[0003] Implicit representation methods, with Neural Radiation Field (NeRF) as their core technical framework, construct dynamic human models and map images from the current motion space to a normal space using inverse skinning techniques, thereby achieving 3D reconstruction. However, due to limitations in their technical principles, implicit methods suffer from three major drawbacks: 1. The lack of parametric constraints on human structure makes it difficult to accurately depict geometric details of the human surface (such as skin wrinkles and muscle deformation); 2. During inverse skinning, the correspondence between bones and the human surface is inaccurate, leading to the loss of dynamic details such as skin stretching and joint misalignment when the human body is in complex poses (such as crossed arms or bent legs); 3. The inherent characteristics of implicit volume representation cause a sharp drop in computational efficiency when processing high-resolution details, making it difficult to meet the requirements of real-time rendering.

[0004] Explicit representation methods, by introducing constraints on human body structure parameters, offer superior modeling quality compared to implicit methods. These methods primarily include techniques such as meshes, point clouds, volumetric primitives, and 3D Gaussian rendering (3DGS). However, each approach still has significant drawbacks: Mesh-based methods, such as existing methods that rely on fixed-topology triangular meshes for modeling, can recover some surface details, but the fixed mesh topology makes them unsuitable for adapting to drastic changes in human posture (such as significant bending or limb twisting), and require complex mesh subdivision techniques to enhance details, increasing computational overhead; Point cloud-based methods, such as existing methods that use deformable point clouds to represent the human body, require dense point cloud sampling (typically hundreds of thousands or even millions of points) to preserve fine features such as clothing wrinkles and finger joints, consuming significant storage resources and leading to low efficiency in subsequent rendering and optimization; Volumetric primitive methods, which model using structures such as voxel meshes, have a positive correlation between voxel resolution and computational overhead; high-resolution voxels result in huge memory consumption and computational latency, making it difficult to balance modeling accuracy and efficiency. A common problem with traditional explicit methods is that, whether using meshes, point clouds, or volumetric primitives, it is difficult to find a balance between "structural integrity" and "detail fidelity". 3D Gaussian rendering (3DGS), as an explicit representation technology that has emerged in recent years, has gradually become the mainstream direction for 3D human avatar modeling due to its flexible topological adaptability and excellent dynamic consistency. Although 3DGS has significant improvements over traditional explicit methods, existing solutions still have two major bottlenecks: 1. Architecture and efficiency bottlenecks: Most 3DGS-based methods adopt a two-stage framework; 2. Loss function defects: Existing methods generally rely on L1 loss or L1-like loss to control prediction accuracy. Such losses only calculate the pixel-level Euclidean distance between the predicted image and the ground truth (GT) in Euclidean space, completely ignoring the implicit relationship between the two in semantic structure and visual perception.

[0005] In summary, while existing technologies have made some progress in the field of 3D human avatar modeling, there is still a lack of a technical solution that can simultaneously achieve "complex pose adaptation (to cope with strenuous movement), fine detail restoration (to capture wrinkles and faces), and end-to-end efficient modeling (to meet real-time interaction)". This technical gap has become a key bottleneck restricting the large-scale implementation of high-fidelity 3D human avatars in VR / AR, digital entertainment and other fields. Summary of the Invention

[0006] This invention provides a three-dimensional Gaussian modeling method, system, storage medium, and electronic device based on Poincaré sphere and three-plane representation. It can solve the problems of easy distortion in complex poses, insufficient recovery of fine geometric details, and difficulty in balancing modeling efficiency and quality, and can generate high-fidelity, real-time 3D human body models.

[0007] This invention provides a three-dimensional Gaussian modeling method based on the Poincaré sphere and three-plane representation, comprising: Acquire human body images and perform preprocessing; The preprocessed human image is input into the trained 3D human reconstruction network model to obtain the reconstructed 3D human avatar image. The 3D human body reconstruction network model includes a three-plane geometric feature branch, a hyperbolic U-net UV feature extraction branch, and a multi-branch decoder. The processing steps of the 3D human body reconstruction network model include: inputting randomly generated Gaussian variables into the three-plane geometric feature branch to obtain geometric features; inputting the UV mapping obtained during the training of the 3D human body reconstruction network model into the hyperbolic U-net UV feature extraction branch to obtain UV features; fusing the geometric features and the UV features and inputting them into the multi-branch decoder to obtain 3D Gaussian rendering core parameters; and synthesizing a 3D human avatar image based on the 3D Gaussian rendering core parameters.

[0008] Furthermore, based on the aforementioned 3D Gaussian modeling method using the Poincaré sphere and three-plane representation, the randomly generated Gaussian variables are input into the three-plane geometric feature branch to obtain geometric features, including: The randomly generated Gaussian variable is input into the encoder of the three-plane geometric feature branch to obtain the initial feature tensor; The initial feature tensor is decomposed into three orthogonal 2D feature planes, XY, YZ, and XZ, through a 3D unentangled convolution operation. Any point in three-dimensional space is projected onto the XY plane, YZ plane, and XZ plane respectively. The features of each plane are extracted by linear interpolation to obtain the XY plane features, YZ plane features, and XZ plane features. The XY plane features, YZ plane features, and XZ plane features are input into a three-plane decoder, decoded, and then fused to obtain fused features. The fused features are then batch-normalized to obtain geometric features.

[0009] Furthermore, based on the aforementioned 3D Gaussian modeling method using Poincaré spheres and three-plane representation, the UV mapping obtained during the training of the 3D human reconstruction network model is input into the hyperbolic U-net UV feature extraction branch to obtain UV features, including: The UV mapping is input into the downsampling module for convolution, activation function, and max pooling operations to obtain the first feature. The first feature is mapped to the Poincaré sphere space through hyperbolic projection, and deep hyperbolic UV features are extracted through two layers of hyperbolic convolution. The deep hyperbolic UV features are projected back into Euclidean space and stitched together with the first feature to obtain stitched features. The stitched features are then passed through a convolutional layer to obtain UV features.

[0010] Furthermore, based on the above-mentioned 3D Gaussian modeling method based on Poincaré spheres and three-plane representation, the multi-branch decoder includes a pose prediction branch, a scale prediction branch, and a spherical harmonic coefficient prediction branch. The pose prediction branch, the scale prediction branch, and the spherical harmonic coefficient prediction branch have the same structure, each including three convolutional layers, and each convolutional layer includes a 1×1 convolution and a ReLU activation function.

[0011] Furthermore, based on the aforementioned 3D Gaussian modeling method using Poincaré spheres and three-plane representations, the synthesis of 3D human avatar images based on the core parameters of 3D Gaussian rendering includes: The core parameters of the 3D Gaussian rendering are input into the 3D Gaussian renderer, and a 3D human avatar image is synthesized through Gaussian sputtering.

[0012] Furthermore, based on the aforementioned 3D Gaussian modeling method using Poincaré spheres and three-plane representation, the training process of the 3D human body reconstruction network model includes: Obtain a human image dataset, which includes human images and corresponding human annotation information; The human body image is preprocessed to extract a UV mapping map from the human body annotation information; The randomly generated Gaussian variables are input into the three-plane geometric feature branch to obtain the geometric features; The UV mapping is input into the hyperbolic U-net UV feature extraction branch to obtain UV features; The geometric features and UV features are fused and input into a multi-branch decoder to obtain the 3D Gaussian rendering core parameters. Based on the 3D Gaussian rendering core parameters, a 3D human avatar image is synthesized. A total loss function is constructed based on the reconstructed 3D human avatar and the corresponding real 3D human avatar image, and the 3D human reconstruction network model is trained based on the total loss function.

[0013] Furthermore, based on the above-mentioned 3D Gaussian modeling method based on the Poincaré sphere and three-plane representation, the total loss function is:

[0014] in, For the total loss function, All are parameters. For structural similarity loss, The distance is in Euclidean form at the pixel level. For hyperbolic perception loss based on Poincaré spheres, The loss is calculated using pixel-level Euclidean distance for the face.

[0015] This invention also provides a three-dimensional Gaussian modeling system based on the Poincaré sphere and three-plane representation, comprising: The acquisition module is used to acquire human body images and perform preprocessing. The 3D human avatar image generation module is used to input the preprocessed human image into the trained 3D human reconstruction network model to obtain the reconstructed 3D human avatar image. The 3D human body reconstruction network model includes a three-plane geometric feature branch, a hyperbolic U-net UV feature extraction branch, and a multi-branch decoder. The processing steps of the 3D human body reconstruction network model include: inputting randomly generated Gaussian variables into the three-plane geometric feature branch to obtain geometric features; inputting the UV mapping obtained during the training of the 3D human body reconstruction network model into the hyperbolic U-net UV feature extraction branch to obtain UV features; fusing the geometric features and the UV features and inputting them into the multi-branch decoder to obtain 3D Gaussian rendering core parameters; and synthesizing a 3D human avatar image based on the 3D Gaussian rendering core parameters.

[0016] The present invention also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute any of the above-described three-dimensional Gaussian modeling methods based on the Poincaré sphere and three-plane representation.

[0017] The present invention also provides an electronic device, including a processor and a memory, wherein the processor is electrically connected to the memory, the memory is used to store instructions and data, and the processor is used for steps in the three-dimensional Gaussian modeling method based on the Poincaré sphere and three-plane representation described in any of the preceding claims.

[0018] The present invention provides a three-dimensional Gaussian modeling method, system, storage medium and electronic device based on Poincaré sphere and three-plane representation. The present invention, through an end-to-end 3D Gaussian rendering architecture, combined with hyperbolic space feature learning and three-plane geometric representation, can achieve high-fidelity, real-time 3D human avatar generation under monocular video input. The present invention has the following beneficial effects: (1) Spatial breakthrough of Euclidean limitation: For the first time, the hyperbolic space of the Poincaré sphere is introduced into 3D human body modeling. The hyperbolic U-net uses the "exponential capacity" and "topology preservation ability" of the hyperbolic space to accurately capture the dependency relationship of human body level details (such as finger joints, clothing folds). Compared with the Euclidean space U-net, texture stretching and distortion are reduced by 20%-30%. (2) Perceptual loss improves detail fitting: Compared with the traditional training loss which only focuses on pixel distance, the hyperbolic perceptual loss reflects the semantic consistency between the generated image and the real image more realistically through hyperbolic space distance. (3) End-to-end architecture balances efficiency and quality: End-to-end 3D Gaussian modeling is adopted, eliminating the need for phased training and meeting real-time interaction requirements; at the same time, the three-plane representation avoids the instability of directly using noisy features, and the modeling quality reaches the state-of-the-art level on the three major datasets. (4) Loss of coordination ensures global and local stability: Individually constrain facial details. Optimize semantic consistency / To ensure the overall structure, multiple loss mechanisms work together to solve the problem of "good overall fit but poor local details", achieving high-fidelity synchronization of the human body's global posture and local details. Attached Figure Description

[0019] The technical solution and other beneficial effects of the present invention will become apparent from the following detailed description of specific embodiments of the invention, in conjunction with the accompanying drawings.

[0020] Figure 1 A flowchart of a three-dimensional Gaussian modeling method based on a Poincaré sphere and three-plane representation provided in an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the structure of a three-dimensional Gaussian modeling system based on a Poincaré sphere and three-plane representation provided in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] This invention provides a method, system, storage medium, and electronic device for three-dimensional Gaussian modeling based on a Poincaré sphere and three-plane representation. The three-dimensional Gaussian modeling system based on a Poincaré sphere and three-plane representation provided by this invention can be integrated into an electronic device, such as a terminal or server. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0025] Please see Figure 1 , Figure 1 The flowchart illustrates a three-dimensional Gaussian modeling method based on a Poincaré sphere and three-plane representation, provided in an embodiment of the present invention. This method, applied in electronic devices, includes the following steps: S1: Acquire human body images and perform preprocessing.

[0026] Specifically, human images are uniformly cropped to a size of 1024×1024.

[0027] S2, input the preprocessed human image into the trained 3D human reconstruction network model to obtain the reconstructed 3D human avatar image.

[0028] The 3D human body reconstruction network model includes a three-plane geometric feature branch, a hyperbolic U-net UV feature extraction branch, and a multi-branch decoder. The processing steps of the 3D human body reconstruction network model include: inputting randomly generated Gaussian variables into the three-plane geometric feature branch to obtain geometric features; inputting the UV mapping map (U represents vertical and V represents horizontal) fitted during the training of the 3D human body reconstruction network model into the hyperbolic U-net UV feature extraction branch to obtain UV features; fusing the geometric features and UV features and inputting them into the multi-branch decoder to obtain the 3D Gaussian rendering core parameters; and synthesizing a 3D human avatar image based on the 3D Gaussian rendering core parameters.

[0029] In one embodiment, step S2, inputting randomly generated Gaussian variables into the three-plane geometric feature branch to obtain geometric features, includes: S211, input the randomly generated Gaussian variable into the encoder of the three-plane geometric feature branch to obtain the initial feature tensor.

[0030] Randomly generate Gaussian latent variables that follow a normal distribution. [Its dimensions are (3,32,128,128)], and the Gaussian latent variable is input into the encoder to obtain the initial feature tensor.

[0031] The encoder, which draws inspiration from the EG3D encoder architecture, contains four convolutional layers connected in sequence. The first convolutional layer includes a 1×1 convolution and a ReLU activation function, while the second to fourth convolutional layers include a 2×2 convolution and a ReLU activation function.

[0032] S212 decomposes the initial feature tensor into three orthogonal 2D feature planes, XY, YZ, and XZ, through 3D unentangled convolution operations [each plane tensor has dimensions of (256, 128, 128)].

[0033] S213: Project any point in the three-dimensional space onto the XY plane, YZ plane, and XZ plane respectively, and extract the features of each plane through linear interpolation to obtain the XY plane features, YZ plane features, and XZ plane features.

[0034] S214: Input the XY plane features, YZ plane features and XZ plane features into the three-plane decoder, decode and fuse them to obtain fused features, and batch normalize the fused features to obtain geometric features.

[0035] The three-plane decoder consists of three decoding layers, each convolutional layer including a transposed convolution and a ReLU activation function. The three decoding layers decode XY-plane features, YZ-plane features, and XZ-plane features respectively. The outputs of the three decoding layers are then fused to obtain the fused features. Finally, batch normalization is applied to obtain the geometric features.

[0036] In one embodiment, the hyperbolic U-net UV feature extraction branch includes a downsampling module, a hyperbolic feature processing module, and a sampling module. Step S2, which involves inputting the UV mapping obtained during the training of the 3D human reconstruction network model into the hyperbolic U-net UV feature extraction branch to obtain UV features, includes: S221, the UV mapping map is input into the downsampling module for convolution, activation function and max pooling operations to obtain the first feature.

[0037] The downsampling module comprises four sequentially connected convolutional layers. Each convolutional layer includes a 2×2 convolution, a ReLU activation function, and a 2×2 max pooling function. The UV mapping map is input into the downsampling module and processed through these four convolutional layers, increasing the number of channels from... Finally, the first feature [dimension is (512, 16, 16)] is obtained.

[0038] S222 maps the first feature to the Poincaré sphere space through hyperbolic projection, and extracts deep hyperbolic UV features through two layers of hyperbolic convolution.

[0039] First, a curvature parameter c is defined, and a conformal factor is calculated. The first feature is then mapped to the Poincaré sphere space via hyperbolic projection. Next, deep hyperbolic UV features are extracted using two layers of hyperbolic convolution (emphasizing the Möbius summation method and the hyperbolic activation function MobiusPReLU) to capture the topological dependencies at the "thigh-limbs" and "finger joints" levels, which are difficult to learn in Euclidean space.

[0040] S223 projects the deep hyperbolic UV features back into Euclidean space and concatenates them with the first feature to obtain the concatenated features. The concatenated features are then passed through a convolutional layer to obtain the UV features.

[0041] The deep hyperbolic UV features are projected back into Euclidean space, and the first feature is spliced ​​using a skip concatenation method to obtain the spliced ​​features. The spliced ​​features are then progressively restored to their spatial dimensions using 2×2 transposed convolutions (including DeConv and ReLU activation functions). Finally, the number of channels is compressed to 256 using 1×1 convolutions. After batch normalization, the output UV features have dimensions of (1, 256, 64, 64).

[0042] In one embodiment, the multi-branch decoder includes a pose prediction branch, a scale prediction branch, and a spherical harmonic coefficient prediction branch. Step S2, which fuses geometric features and UV features and inputs them into the multi-branch decoder, yields the core parameters for 3D Gaussian rendering, including: S231, the geometric features and UV features are fused to obtain the fused features.

[0043] The geometric features and UV features are added element-wise to obtain the fused features [dimensions of (1, 256, 64, 64)], thus achieving the fusion of information on geometric structure and texture details.

[0044] S232, the fused features are input into the pose prediction branch, scale prediction branch and spherical harmonic coefficient prediction branch respectively to obtain the position parameters, scale parameters and spherical harmonic coefficients.

[0045] The pose prediction branch, scale prediction branch, and spherical harmonics prediction branch have the same structure, each consisting of three convolutional layers, each including a 1×1 convolution and a ReLU activation function. The fused features are input into the pose prediction branch to obtain the spatial position parameters of the 3D Gaussian [dimensions (1, 3, 262144), where 3 represents the x / y / z 3D coordinates and 262144 is the number of Gaussian points]. The fused features are input into the scale prediction branch to obtain the scale parameters of the 3D Gaussian [dimensions (1, 1, 262144), controlling the size of the Gaussian points]. The fused features are input into the spherical harmonics prediction branch to obtain the spherical harmonics [dimensions (1, 3, 262144), used to encode the appearance information of the 3D Gaussian, such as color and lighting reflection characteristics]. Finally, the multi-branch decoder outputs the core 3D Gaussian rendering parameters: `render_params = (position, scale, SHs)` (where `position` is the spatial position parameter, `scale` is the scale parameter, and `SHs` are the spherical harmonics).

[0046] In one embodiment, the synthesis of the 3D human avatar image based on the 3D Gaussian rendering core parameters in step S2 includes: The position parameters, scale parameters, and spherical harmonic coefficients are input into the 3D Gaussian renderer, and the 3D human avatar image is synthesized through Gaussian sputtering.

[0047] The training process of the 3D human body reconstruction network model is described below: (1) Obtain the human image dataset, which includes human images and corresponding human annotation information.

[0048] (2) Preprocess the human body image and extract the UV mapping map from the human body annotation information.

[0049] In the human body annotation information, a UV map of the 3D human body surface is extracted. The UV map maps the (x,y,z) coordinates of each point on the 3D human body surface to the (u,v) coordinates in the 2D texture space, which can preserve fine geometric information such as human muscle deformation, skin wrinkles, and clothing texture, and serves as one of the core inputs for the subsequent hyperbolic U-net. During preprocessing, it is necessary to ensure that the pixels of the UV map are aligned with the corresponding human body image to avoid topological misalignment.

[0050] (3) Input the randomly generated Gaussian variables into the three-plane geometric feature branch to obtain the geometric features.

[0051] (4) Input the UV mapping map into the hyperbolic U-net UV feature extraction branch to obtain UV features.

[0052] (5) After fusing the geometric features and UV features, input them into the multi-branch decoder to obtain the 3D Gaussian rendering core parameters, and synthesize the 3D human avatar image based on the 3D Gaussian rendering core parameters.

[0053] (6) Construct a total loss function based on the reconstructed 3D human avatar and the corresponding real 3D human avatar image, and train the 3D human reconstruction network model based on the total loss function.

[0054] The total loss function is:

[0055] in, For the total loss function, All are parameters. For structural similarity loss, The distance is in Euclidean form at the pixel level. For hyperbolic perception loss based on Poincaré spheres, The loss is calculated using pixel-level Euclidean distance for the face.

[0056] The structural similarity loss is used to calculate the structural similarity between the generated image and the ground truth (GT), with a focus on constraining the consistency of the overall human structure (such as limb proportions and pose contours). The formula for structural similarity loss is:

[0057] in, Represent the predicted 3D human avatar image in the i-th local window, respectively. and real images The average pixel value, and Represent the predicted 3D human avatar image in the i-th local window, respectively. and real images The standard deviation of pixels, Represents the predicted 3D human avatar image in the i-th local window. and real images pixel covariance, , It is a small positive constant used for stable calculations.

[0058] Used to calculate the pixel-level Euclidean distance between the predicted 3D human avatar and the real image, ensuring a good fit at the basic pixel level. The formula is:

[0059] in, and These represent the predicted 3D human avatar images. and real images The pixel value at position i, where N is the total number of pixels.

[0060] Used to calculate the distance loss between the predicted 3D human avatar image and the real image in hyperbolic space. The formula is:

[0061]

[0062] in, The distance between the predicted 3D human avatar image and the real image in hyperbolic space. It is the inverse function of the hyperbolic cosine function. Representing the VGG19 network The layer's feature output. For hyperbolic projection, It is the Euclidean norm. This represents the generated 3D human avatar image. Represents the ground truth (GT) image. Used to balance the The contribution of the layer to hyperbolic perception loss.

[0063] When calculating the static perceptual loss, five layers (conv1, conv2, conv3, conv4, and conv5) from the VGG19 network are selected, and The value is This hyperbolic metric enables the network to accurately model subtle geometric changes under different poses.

[0064] The formula used to calculate the pixel difference between the face portion in a 3D human avatar image and the face portion in a real image is:

[0065] in, Let be the binary mask corresponding to position i.

[0066] Based on the method described in the above embodiments, this embodiment will further describe it from the perspective of a three-dimensional Gaussian modeling system based on Poincaré spheres and three-plane representation. This three-dimensional Gaussian modeling system based on Poincaré spheres and three-plane representation can be implemented as an independent entity or integrated into an electronic device. The electronic device can be a terminal, server, or other device. The terminal can include a tablet computer, a laptop computer, a personal computer (PC), a microprocessor box, or other devices.

[0067] Please see Figure 2 , Figure 2 This invention specifically describes a three-dimensional Gaussian modeling system based on a Poincaré sphere and three-plane representation, applied in electronic devices. This system may include: The acquisition module is used to acquire human body images and perform preprocessing. The 3D human avatar image generation module is used to input the preprocessed human image into the trained 3D human reconstruction network model to obtain the reconstructed 3D human avatar image. The 3D human body reconstruction network model includes a three-plane geometric feature branch, a hyperbolic U-net UV feature extraction branch, and a multi-branch decoder. The processing steps of the 3D human body reconstruction network model include: inputting randomly generated Gaussian variables into the three-plane geometric feature branch to obtain geometric features; inputting the UV mapping obtained during the training of the 3D human body reconstruction network model into the hyperbolic U-net UV feature extraction branch to obtain UV features; fusing the geometric features and the UV features and inputting them into the multi-branch decoder to obtain 3D Gaussian rendering core parameters; and synthesizing a 3D human avatar image based on the 3D Gaussian rendering core parameters.

[0068] In specific implementation, the above modules and / or units can be implemented as independent entities, or they can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of the above modules and / or units, please refer to the previous method embodiments. For the specific beneficial effects that can be achieved, please also refer to the beneficial effects in the previous method embodiments, which will not be repeated here.

[0069] In addition, this embodiment of the invention also provides an electronic device, which may be a computer, tablet computer, or other similar device. This electronic device can implement the steps in any embodiment of the three-dimensional Gaussian modeling method based on Poincaré spheres and three-plane representation provided in this embodiment of the invention. Therefore, it can achieve the beneficial effects achievable by any three-dimensional Gaussian modeling method based on Poincaré spheres and three-plane representation provided in this embodiment of the invention, as detailed in the preceding embodiments, and will not be repeated here.

[0070] Figure 3 A specific structural block diagram of an electronic device provided in an embodiment of the present invention is shown. This electronic device can be used to implement the three-dimensional Gaussian modeling method based on the Poincaré sphere and three-plane representation provided in the above embodiments. The electronic device 500 can be a terminal, server, or other device. The terminal can include a tablet computer, laptop computer, personal computer (PC), microprocessor box, or other devices.

[0071] The memory 520 can be used to store software programs and modules, such as the program instructions / modules corresponding to those in the above embodiments. The processor 580 executes various functional applications and data processing by running the software programs and modules stored in the memory 520. The memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 520 may further include memory remotely located relative to the processor 580, and these remote memories can be connected to the electronic device 500 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0072] The input unit 530 can be used to receive input numeric or character information, and to generate a keyboard and mouse related to user settings and function control. Display unit 540 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof. Display unit 540 may include display panel 541, which may optionally be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar forms.

[0073] Electronic device 500, through transmission module 570 (e.g., Wi-Fi module), can help users receive requests, send information, etc., providing users with wireless broadband internet access. Although transmission module 570 is shown in the figure, it is understood that it is not an essential component of electronic device 500 and can be omitted as needed without changing the essence of the invention.

[0074] The processor 580 is the control center of the electronic device 500. It connects to various parts of the phone via various interfaces and lines, and performs various functions and processes data of the electronic device 500 by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby providing overall monitoring of the electronic device. Optionally, the processor 580 may include one or more processing cores; in some embodiments, the processor 580 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 580.

[0075] Electronic device 500 also includes a power supply 590 (such as a battery) that supplies power to various components. In some embodiments, the power supply may be logically connected to processor 580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 590 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0076] Although not shown, the electronic device 500 also includes cameras (such as front-facing cameras and rear-facing cameras), Bluetooth modules, etc., which will not be described in detail here. Specifically, in this embodiment, the display unit of the electronic device is a touch screen display, and the mobile terminal also includes a memory and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors. One or more programs contain instructions for performing the following operations: Acquire human body images and perform preprocessing; The preprocessed human image is input into the trained 3D human reconstruction network model to obtain the reconstructed 3D human avatar image. The 3D human body reconstruction network model includes a three-plane geometric feature branch, a hyperbolic U-net UV feature extraction branch, and a multi-branch decoder. The processing steps of the 3D human body reconstruction network model include: inputting randomly generated Gaussian variables into the three-plane geometric feature branch to obtain geometric features; inputting the UV mapping obtained during the training of the 3D human body reconstruction network model into the hyperbolic U-net UV feature extraction branch to obtain UV features; fusing the geometric features and the UV features and inputting them into the multi-branch decoder to obtain 3D Gaussian rendering core parameters; and synthesizing a 3D human avatar image based on the 3D Gaussian rendering core parameters.

[0077] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.

[0078] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, embodiments of the present invention provide a storage medium storing multiple instructions that can be loaded by a processor to execute the steps of any embodiment of the three-dimensional Gaussian modeling method based on Poincaré spheres and three-plane representation provided by the present invention.

[0079] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0080] Since the instructions stored in the storage medium can execute the steps in any embodiment of the three-dimensional Gaussian modeling method based on Poincaré sphere and three-plane representation provided in the embodiments of the present invention, the beneficial effects that any three-dimensional Gaussian modeling method based on Poincaré sphere and three-plane representation provided in the embodiments of the present invention can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0081] The foregoing has provided a detailed description of a three-dimensional Gaussian modeling method, system, storage medium, and electronic device based on a Poincaré sphere and three-plane representation provided by embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A three-dimensional Gaussian modeling method based on Poincaré spheres and three-plane representation, characterized in that, The method includes: Acquire human body images and perform preprocessing; The preprocessed human image is input into the trained 3D human reconstruction network model to obtain the reconstructed 3D human avatar image. The 3D human body reconstruction network model includes a three-plane geometric feature branch, a hyperbolic U-net UV feature extraction branch, and a multi-branch decoder. The training process of the 3D human body reconstruction network model includes: Obtain a human image dataset, which includes human images and corresponding human annotation information; The human body image is preprocessed to extract a UV mapping map from the human body annotation information; The randomly generated Gaussian variables are input into the three-plane geometric feature branch to obtain the geometric features; The UV mapping is input into the hyperbolic U-net UV feature extraction branch to obtain UV features; The geometric features and UV features are fused and input into a multi-branch decoder to obtain the 3D Gaussian rendering core parameters. Based on the 3D Gaussian rendering core parameters, a 3D human avatar image is synthesized. A total loss function is constructed based on the reconstructed 3D human avatar and the corresponding real 3D human avatar image, and the 3D human reconstruction network model is trained based on the total loss function. The total loss function is: in, For the total loss function, All are parameters. For structural similarity loss, The distance is in Euclidean form at the pixel level. For hyperbolic perception loss based on Poincaré spheres, For pixel-level Euclidean distance loss of the face; The formula is: in, The distance between the predicted 3D human avatar image and the real image in hyperbolic space. It is the inverse function of the hyperbolic cosine function. Representing the VGG19 network The feature output of the layer, For hyperbolic projection, For the Euclidean norm, This represents the generated 3D human avatar image. Represents the truth image. Used to balance the The contribution of the layer to hyperbolic perception loss.

2. The three-dimensional Gaussian modeling method based on the Poincaré sphere and three-plane representation according to claim 1, characterized in that, The randomly generated Gaussian variables are input into the three-plane geometric feature branch to obtain the geometric features, including: The randomly generated Gaussian variable is input into the encoder of the three-plane geometric feature branch to obtain the initial feature tensor; The initial feature tensor is decomposed into three orthogonal 2D feature planes, XY, YZ, and XZ, through a 3D unentangled convolution operation. Any point in three-dimensional space is projected onto the XY plane, YZ plane, and XZ plane respectively. The features of each plane are extracted by linear interpolation to obtain the XY plane features, YZ plane features, and XZ plane features. The XY plane features, YZ plane features, and XZ plane features are input into a three-plane decoder, decoded, and then fused to obtain fused features. The fused features are then batch-normalized to obtain geometric features.

3. The three-dimensional Gaussian modeling method based on the Poincaré sphere and three-plane representation according to claim 1, characterized in that, The UV mapping obtained during the training of the 3D human reconstruction network model is input into the hyperbolic U-net UV feature extraction branch to obtain UV features, including: The UV mapping map is input into the downsampling module for convolution, activation function, and max pooling operations to obtain the first feature. The first feature is mapped to the Poincaré sphere space through hyperbolic projection, and deep hyperbolic UV features are extracted through two layers of hyperbolic convolution. The deep hyperbolic UV features are projected back into Euclidean space and stitched together with the first feature to obtain stitched features. The stitched features are then passed through a convolutional layer to obtain UV features.

4. The three-dimensional Gaussian modeling method based on the Poincaré sphere and three-plane representation according to claim 1, characterized in that, The multi-branch decoder includes a pose prediction branch, a scale prediction branch, and a spherical harmonic coefficient prediction branch. The pose prediction branch, the scale prediction branch, and the spherical harmonic coefficient prediction branch have the same structure, each including three convolutional layers, and each convolutional layer includes a 1×1 convolution and a ReLU activation function.

5. The three-dimensional Gaussian modeling method based on the Poincaré sphere and three-plane representation according to claim 4, characterized in that, 3D human avatar images are synthesized based on the core parameters of 3D Gaussian rendering, including: The core parameters of the 3D Gaussian rendering are input into the 3D Gaussian renderer, and a 3D human avatar image is synthesized through Gaussian sputtering.

6. A three-dimensional Gaussian modeling system based on a Poincaré sphere and three-plane representation, wherein the three-dimensional Gaussian modeling system based on a Poincaré sphere and three-plane representation is used to implement the three-dimensional Gaussian modeling system method based on a Poincaré sphere and three-plane representation as described in claim 1, characterized in that, include: The acquisition module is used to acquire human body images and perform preprocessing. The 3D human avatar image generation module is used to input the preprocessed human image into the trained 3D human reconstruction network model to obtain the reconstructed 3D human avatar image. The 3D human body reconstruction network model includes a three-plane geometric feature branch, a hyperbolic U-net UV feature extraction branch, and a multi-branch decoder.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to execute the three-dimensional Gaussian modeling method based on the Poincaré sphere and three-plane representation as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, It includes a processor and a memory, the processor being electrically connected to the memory, the memory being used to store instructions and data, and the processor being used to execute the steps in the three-dimensional Gaussian modeling method based on the Poincaré sphere and three-plane representation as described in any one of claims 1 to 5.

Images