Method for generating gaussian rendering model, electronic device and storage medium

Abstract

Embodiments of the present application disclose a method for generating a Gaussian rendering model. The method comprises: obtaining a human body image, extracting a plurality of vertices from the human body image, setting a Gaussian sphere at a vertex position of each vertex, removing Gaussian spheres not on the surface of the human body from the plurality of Gaussian spheres to obtain a plurality of target Gaussian spheres, generating a human body Gaussian rendering map based on preset Gaussian rendering parameters and the target Gaussian spheres, segmenting an original human body region image from the human body image, and generating the Gaussian rendering model based on the human body Gaussian rendering map and the original human body region image. The embodiments of the present application can not only remove Gaussian spheres spaced apart from the surface of the human body, so that the Gaussian rendering model is more accurate and reasonable, and the situation of artifacts in the output human body image of the Gaussian rendering model and the situation of artifacts in the output human body image of a new view angle are improved, but also remove Gaussian spheres inside the human body, so that the amount of calculation required for rendering the human body image and the memory space overhead are reduced.

Description

Technical Field

[0001] This application relates to the field of model building technology, and in particular to a method for generating Gaussian rendering models, an electronic device, and a storage medium. Background Technology

[0002] Immersive video conferencing is a new form of remote video conferencing that displays participants' heads in 3D on the meeting interface, enhancing the sense of presence and immersion. The relevant technology employs a 3D Gaussian Splatting (3DGS) rendering method to train and generate a Gaussian rendering model, which is then used to render human images. Specifically, the Gaussian rendering model uses a series of Gaussian spheres to represent the human body region to be rendered. Different Gaussian spheres have different positions, covariance matrices, colors, or transparency. By combining and rendering a large number of Gaussian spheres, the model can reconstruct complex human images. The positions and other parameters of the Gaussian spheres are typically trained using a training dataset. However, the training dataset is often limited, and the positions of the trained Gaussian spheres are not always precise or reasonable, leading to artifacts in the rendered human images. Summary of the Invention

[0003] One objective of this application is to provide a method for generating Gaussian rendering models, an electronic device, and a storage medium to solve the technical problem that Gaussian rendering models trained in related technologies easily generate human images with artifacts.

[0004] In a first aspect, embodiments of this application provide a method for generating a Gaussian rendering model, comprising: acquiring video data captured by a camera; extracting multiple frames of human body images of a target user from the video data; extracting multiple vertices from the human body images, wherein Gaussian spheres are set at the positions of the vertices; removing Gaussian spheres that are not on the surface of the target user's human body from the multiple Gaussian spheres to obtain multiple target Gaussian spheres; generating a human body Gaussian rendering map based on preset Gaussian rendering parameters and target Gaussian spheres; segmenting an original human body region image from the human body image; and generating a Gaussian rendering model based on the human body Gaussian rendering map and the original human body region image.

[0005] The embodiments of this application can not only remove Gaussian spheres that are a certain distance away from the human body surface, making the Gaussian rendering model more accurate and reasonable, and improving the artifacts in the human body images output by the Gaussian rendering model and the artifacts in the human body images output from new perspectives, but also remove Gaussian spheres inside the human body, reducing the amount of computation required to render human body images and reducing memory space overhead.

[0006] Optionally, removing Gaussian spheres that are not on the surface of the target user's human body from multiple Gaussian spheres to obtain multiple target Gaussian spheres includes: performing a depth estimation operation on the human body image to obtain a depth map; correcting the vertex positions based on the vertex positions of the vertices and preset Gaussian rendering parameters to obtain corrected vertex positions; placing the Gaussian spheres at the corrected vertex positions as candidate Gaussian spheres; the set of all candidate Gaussian spheres as a Gaussian sphere set; and removing candidate Gaussian spheres that do not meet preset rendering conditions from the Gaussian sphere set based on the depth map to obtain multiple target Gaussian spheres.

[0007] Depth maps can reliably and accurately reflect the relative distance between each Gaussian sphere and the human body surface. Based on the depth map, the embodiments of this application can reliably and accurately find Gaussian spheres that are not on the human body surface and remove such Gaussian spheres.

[0008] Optionally, the camera is configured with an image coordinate system. Based on the depth map, candidate Gaussian spheres that do not meet the preset rendering conditions are removed from the Gaussian sphere set to obtain multiple target Gaussian spheres. This includes: mapping the sphere position of the candidate Gaussian sphere to the image coordinate system to obtain the two-dimensional position of the candidate Gaussian sphere; determining the depth value corresponding to the two-dimensional position in the depth map to obtain the depth value corresponding to the candidate Gaussian sphere; and removing candidate Gaussian spheres that do not meet the preset rendering conditions from the Gaussian sphere set based on the depth value of the candidate Gaussian sphere to obtain multiple target Gaussian spheres.

[0009] Optionally, mapping the sphere position of the candidate Gaussian sphere to the image coordinate system to obtain the two-dimensional position of the candidate Gaussian sphere includes: obtaining camera extrinsic parameters and camera intrinsic parameters, and mapping the candidate Gaussian sphere to the image coordinate system based on the camera extrinsic parameters, camera intrinsic parameters and the sphere position of the candidate Gaussian sphere to obtain the two-dimensional position of the candidate Gaussian sphere.

[0010] Optionally, based on the depth value of the candidate Gaussian sphere, candidate Gaussian spheres that do not meet the preset rendering conditions are removed from the Gaussian sphere set to obtain multiple target Gaussian spheres, including: responding to the depth value of the candidate Gaussian sphere being used to indicate that the candidate Gaussian sphere is outside the human body, determining that the candidate Gaussian sphere does not meet the preset rendering conditions, and removing all candidate Gaussian spheres that do not meet the preset rendering conditions from the Gaussian sphere set to obtain multiple target Gaussian spheres.

[0011] The embodiments of this application delete candidate Gaussian spheres outside the human body, which can effectively improve the artifacts caused by such candidate Gaussian spheres, which is beneficial to obtaining high-quality human body images, and can also save computing power and memory space.

[0012] Optionally, based on the depth value of the candidate Gaussian sphere, candidate Gaussian spheres that do not meet the preset rendering conditions are removed from the Gaussian sphere set to obtain multiple target Gaussian spheres, including: responding to the depth value of the candidate Gaussian sphere being used to indicate that the candidate Gaussian sphere is inside the human body, determining that the candidate Gaussian sphere does not meet the preset rendering conditions, and removing all candidate Gaussian spheres that do not meet the preset rendering conditions from the Gaussian sphere set to obtain multiple target Gaussian spheres.

[0013] This application embodiment removes candidate Gaussian spheres inside the human body, which can save computing power and memory space.

[0014] Optionally, generating a Gaussian rendering model based on the human body Gaussian rendering map and the original human body region image includes: obtaining a human body texture rendering map corresponding to the human body image, fusing the human body texture rendering map and the human body Gaussian rendering map to obtain a rendering fusion map, and generating a Gaussian rendering model based on the rendering fusion map and the original human body region image.

[0015] Human body texture rendering maps can compensate for the deficiency of human body Gaussian rendering maps in not being able to present human body reconstruction details in non-rendered areas. Meanwhile, human body Gaussian rendering maps can provide a more accurate description of some human body reconstruction details such as clothing, which are not part of the SMPL-X template information. Therefore, human body Gaussian rendering maps and human body texture rendering maps can complement each other in terms of human body reconstruction details, making the trained Gaussian rendering model more reliable and accurate.

[0016] Optionally, obtaining the human texture rendering map corresponding to the human image includes: generating a mesh rendering map based on multiple vertices, performing texture rendering operations on the mesh rendering map based on a preset texture rendering function to obtain a texture map to be optimized, obtaining the difference information between the human image and the texture map to be optimized, optimizing the texture map to be optimized based on the difference information, and obtaining the human texture rendering map corresponding to the human image.

[0017] Optionally, fusing the human body texture rendering image and the human body Gaussian rendering image to obtain a rendering fusion image includes: masking the non-rendered areas occupied by the human body Gaussian rendering image and candidate Gaussian spheres that do not meet the preset rendering conditions on the human body Gaussian rendering image to obtain a human body mask image. The mask values ​​of the human body Gaussian rendering image are different from the mask values ​​of the non-rendered areas. The human body mask image is a binary image. Fusing the human body Gaussian rendering image and the human body mask image to obtain a first fusion image. Inverting the human body mask image to obtain an inverted mask image. Fusing the inverted mask image and the human body texture rendering image to obtain a second fusion image. Adding the first fusion image and the second fusion image to obtain the rendering fusion image.

[0018] In a second aspect, embodiments of this application provide an electronic device, including a memory and a processor. The memory is connected to the processor, and the processor is used to execute one or more computer programs stored in the memory. When the processor executes one or more computer programs, it causes the electronic device to implement the above-described method for generating a Gaussian rendering model.

[0019] In a third aspect, embodiments of this application provide a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the above-described method for generating a Gaussian rendering model. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a method for generating a Gaussian rendering model provided in an embodiment of this application;

[0022] Figure 2 A schematic diagram of the architecture of a neural network model for learning preset Gaussian rendering parameters provided in an embodiment of this application;

[0023] Figure 3 A schematic diagram showing the distribution of various candidate Gaussian spheres relative to the human body surface of a target user, provided in an embodiment of this application.

[0024] Figure 4 Another schematic diagram showing the distribution of each candidate Gaussian sphere relative to the human body surface of the target user, provided in an embodiment of this application;

[0025] Figure 5 In order to be in Figure 4 A schematic diagram of removing candidate Gaussian spheres located inside the human body;

[0026] Figure 6 A scene diagram illustrating the generation of a Gaussian rendering model provided in an embodiment of this application;

[0027] Figure 7 A schematic diagram of a Gaussian rendering model generation device provided in an embodiment of this application;

[0028] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0030] It should be noted that, unless there is a conflict, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than the module division in the device or the order in the flowchart. Moreover, the terms "first," "second," and "third" used in this application do not limit the data or execution order, but only distinguish identical or similar items with essentially the same function and effect.

[0031] In the Gaussian rendering model, a scene is represented as a collection of Gaussian spheres. Each Gaussian sphere has specific properties, such as position, size, color, and opacity. Gaussian spheres are used to approximate the geometry and appearance information of a scene. They can be considered as the "atoms" that make up the entire Gaussian rendering model. By combining and rendering a large number of Gaussian spheres, complex 3D scenes can be reconstructed. For example, when reconstructing and rendering a 3D object, multiple Gaussian spheres are placed on the object's surface, each corresponding to a local area of ​​the object's surface. By properly setting the properties of these Gaussian spheres, the shape and appearance of the object can be realistically presented.

[0032] As mentioned earlier, in the Gaussian rendering model, the position and other parameters of the Gaussian sphere are trained based on a limited dataset, which leads to some Gaussian spheres appearing in unreasonable and unscientific positions on the human body. For example, the Gaussian sphere is not placed on the surface of the human body, but appears at a certain distance from the surface. When the Gaussian rendering model needs to render and generate a human body image, the Gaussian sphere at this position will also participate in the rendering operation, which causes artifacts in the rendered human body image.

[0033] Artifacts refer to the various types of image interference and other non-random disturbances that appear on an image during image reconstruction. Essentially, they are areas in the reconstructed image that are not actually present in the object itself, but appear after reconstruction and rendering. Common manifestations of artifacts include geometric distortion, texture distortion, lighting distortion, and jagged edges. Geometric distortion refers to uneven surfaces or distorted shapes in the reconstructed image. Texture distortion refers to unnatural colors or textures in the reconstructed image. Lighting distortion refers to unrealistic lighting effects such as shadows and highlights in the reconstructed image. Jagged edges refer to jagged artifacts appearing at the edges of the reconstructed image.

[0034] In addition, the sample images constituting the dataset do not have a comprehensive viewpoint. When the Gaussian rendering model is required to render human images from new viewpoints, the Gaussian rendering model is prone to outputting human images with artifacts, which reduces the image quality of the human images.

[0035] The inventors also discovered that some Gaussian spheres appear inside the human body, and these Gaussian spheres in this location also participate in the rendering operation. Rendering the Gaussian spheres in this location is an invalid rendering operation, which wastes computing power and memory space.

[0036] The embodiments of this application can not only remove Gaussian spheres that are a certain distance away from the human body surface, making the Gaussian rendering model more accurate and reasonable, and improving the artifacts in the human body images output by the Gaussian rendering model and the artifacts in the human body images output from new perspectives, but also remove Gaussian spheres inside the human body, reducing the amount of computation required to render human body images and reducing memory space overhead.

[0037] In this embodiment, a Gaussian rendering model is jointly trained and generated based on the human body Gaussian rendering map and the human body texture rendering map. The human body Gaussian rendering map and the human body texture rendering map can compensate for each other's deficiencies in human body reconstruction details, making the trained Gaussian rendering model more reliable and accurate.

[0038] The following embodiments of this application provide a method for generating a Gaussian rendering model. Please refer to... Figure 1 The method for generating the Gaussian rendering model includes steps S11 to S18.

[0039] In this embodiment, step S11 is performed to acquire video data captured by the camera. The target user sets up the camera, controls the camera to photograph the target user, and records video data of a preset duration.

[0040] In this embodiment, step S12 involves extracting multiple frames of human images of the target user from the video data. These human images are images captured by a camera of the target user. This embodiment extracts multiple frames of human images (img1, img2, ..., img) from the video data according to a preset frame extraction frequency.N The preset frame skipping frequency can be customized by the designer according to business needs, such as a preset frame skipping frequency of 5 frames / second or 1 frame / second, etc.

[0041] In the embodiment of this application, step S13 is performed to extract multiple vertices from the human body image, and a Gaussian sphere is set at the vertex position where the vertex is located.

[0042] Vertices are points that describe changes in human posture and shape. Extracting multiple vertices from a human image involves the following steps: performing human feature extraction processing on the human image to obtain human feature parameters, and using a preset skinning blending algorithm to process the human feature parameters to obtain multiple vertices.

[0043] In some embodiments, performing human feature extraction processing on a human image to obtain human feature parameters includes: inputting the human image into a pre-trained neural network model to perform human feature extraction processing to obtain human feature parameters.

[0044] The neural network model is a model trained using a deep learning algorithm on labeled human sample images. Human feature parameters include shape feature parameters (beta), action feature parameters (pose), expression feature parameters (expr), and position feature parameters (trans). In this embodiment, the network output layer of the neural network model can be designed with four dimensions to simultaneously output shape feature parameters, action feature parameters, expression feature parameters, and position feature parameters.

[0045] Shape feature parameters describe the shape of the human body, such as height, weight, and build. Motion feature parameters primarily describe the posture of the human body, including rotational information of various joints, used to represent various human movements, such as standing, walking, and jumping. Facial expression feature parameters describe facial expressions, such as smiling, frowning, and surprise. Position parameters describe the position of the human body in three-dimensional space.

[0046] In other embodiments, the process of extracting human features from human images to obtain human feature parameters includes: obtaining a preset standard human 3D model, and performing human feature extraction on the human image based on the standard human 3D model to obtain human feature parameters.

[0047] The standard 3D human body model is the SMPL-X human body model. The SMPL-X human body model is an extension of the SMPL (Skinned Multi-Person Linear) model, which can represent rich information such as the shape, posture, facial expression, and hand posture of the human body.

[0048] The SMPL-X human body model consists of 10,475 vertices and the triangular faces connecting these vertices. Vertices are the fundamental building blocks of the SMPL-X human body model; each vertex forms the surface geometry of the human body. By adjusting the position of the vertices, the shape and posture of the human body can be changed. A large number of vertices connect to form triangular faces, collectively constituting the surface of the human body. The position and topology of the vertices determine the appearance of the human body. For example, when the human body makes different movements or postures, the vertices move and deform accordingly, thus displaying the human body shape in various postures. The SMPL-X human body model uses some low-dimensional parameters (such as shape parameters and posture parameters) to control the position of the vertices. Shape parameters can adjust the body shape, such as height, weight, etc.; posture parameters can change the posture of the human body, including the rotation and bending of various joints, which in turn drives the movement of the vertices, realizing the representation of human body movements.

[0049] The SMPL-X human body model uses a parametric approach to describe different human shapes and postures. Therefore, the SMPL-X human body model is configured with feature description parameters, which include shape description parameters, motion description parameters, facial expression description parameters, and position description parameters.

[0050] As mentioned earlier, the standard human 3D model is represented by multiple types of feature description parameters. Based on the standard human 3D model, human feature extraction processing is performed on human images to obtain human feature parameters, including: using the standard human 3D model as a reference, feature observation parameters are extracted from the human image based on a preset fitting algorithm, and the feature description parameters are optimized according to the difference between the feature description parameters and the feature observation parameters until the error between the optimized feature description parameters and the feature observation parameters is minimized. The optimized feature description parameters are human feature parameters.

[0051] Pre-defined fitting algorithms include SMPLify-X, HMR (Human Mesh Recovery), and VIBE (Video Inference for Human Body Pose and Shape Estimation). SMPLify-X is a classic optimization-based method used to extract facial and motion parameters of an SMPL-X human model from one or more human images. The HMR algorithm uses a convolutional neural network to directly map the input human image to the parameter space of the SMPL-X human model, including shape and pose parameters. The VIBE algorithm is a deep learning method specifically designed for human pose and shape estimation in video. It uses a recurrent neural network to process video sequences, predicting SMPL or SMPL-X parameters of the human body by processing frame by frame and utilizing temporal information.

[0052] The process of processing human feature parameters using a pre-defined skinning blending algorithm to obtain multiple vertices includes the following steps: Using shape feature parameters, action feature parameters, expression feature parameters, and position feature parameters as input to the pre-defined skinning blending algorithm to obtain vertices, where the expression for the vertex position is T. i =(x i ,y i ,z i ), i = 1, ..., 10475.

[0053] The Linear Blend Skinning (LBS) function is an important function in the SMPL-X human model. It is used to transfer the motion and deformation of bones to the vertices on the model's surface, realizing changes in the human body's posture and shape. Through the LBS function, abstract parameters such as posture and shape can be converted into changes in the position of vertices on the actual human body surface, thereby achieving dynamic deformation of the human body.

[0054] The Gaussian sphere is the core element of the Gaussian rendering model. Each 3D scene is represented as a set of Gaussian spheres in the Gaussian rendering model. This embodiment of the application can generate realistic human images by rendering the Gaussian spheres. The various properties of the Gaussian sphere directly affect the final rendering effect of the Gaussian rendering model. The position property determines the position of the Gaussian sphere in 3D space, which is crucial for accurately reproducing the geometric structure of the scene. The size property determines the area covered by the Gaussian sphere, affecting the level of detail in the scene. The color property gives the Gaussian sphere a specific color, reflecting the color information of the corresponding area in the scene. The opacity property controls the transparency of the Gaussian sphere, helping to represent the material properties of objects, such as transparent or translucent objects. By carefully adjusting these properties of the Gaussian sphere, the Gaussian rendering model can generate high-quality, realistic rendered images.

[0055] The Gaussian sphere is constrained by preset Gaussian rendering parameters, which include the center point offset vector Δu. i Rotation matrix R i Scale vector S i spherical harmonic function color i and opacity i .

[0056] In this embodiment, step S14 is performed to remove Gaussian spheres that are not on the surface of the target user's human body from a plurality of Gaussian spheres, thereby obtaining a plurality of target Gaussian spheres. Thus, this embodiment does not train Gaussian spheres that are outside the human body surface or inside the human body, thereby saving computing power and storage costs. Moreover, the human body image output by the Gaussian rendering model obtained after training is not prone to artifacts, thereby effectively improving the situation of existing human body images with artifacts.

[0057] The process of removing Gaussian spheres that are not on the surface of the target user's body from a plurality of Gaussian spheres to obtain a plurality of target Gaussian spheres includes steps S141 to S143.

[0058] Step S141: Perform depth estimation on the human body image to obtain a depth map.

[0059] Step S142: Correct the vertex position based on the vertex position and preset Gaussian rendering parameters to obtain the corrected vertex position.

[0060] Step S143: Based on the depth map, candidate Gaussian spheres that do not meet the preset rendering conditions are removed from the Gaussian sphere set to obtain multiple target Gaussian spheres.

[0061] In step S141, the depth map is used to represent the distance between the target user and the camera. The depth map includes multiple pixels, each pixel is configured with a depth value, and the depth value of each pixel is used to represent the distance between the pixel and the camera. In this embodiment, the depth map is obtained by performing a depth estimation operation on the human image based on a preset depth estimation algorithm. The preset depth estimation algorithm includes a monocular depth estimation algorithm. For example, the monocular depth estimation algorithm adopts the DepthAnything algorithm. The DepthAnything algorithm learns the depth information in the image through a convolutional neural network. Specifically, the input single image is used as the input of the network. Through the forward propagation of the network, various features in the image are automatically extracted, and all features are mapped to depth information. Finally, a depth map with the same size as the input image is output.

[0062] In step S142, this embodiment of the application can obtain preset Gaussian rendering parameters based on the target neural network. Specifically, this embodiment of the application determines the vertex features of the vertices, inputs the vertex features of the vertices into the neural network model, and obtains the preset Gaussian rendering parameters.

[0063] Determining the vertex features of a vertex includes the following steps: performing feature sampling operations in a preset sampling space based on the vertex position of the vertex to obtain the vertex features of the vertex. For example, in this embodiment of the application, based on the vertex position of the vertex, a random sampling algorithm, a distance-based sampling algorithm, a distribution-based sampling algorithm, or a sampling algorithm based on an adversarial network model can be used to randomly sample in the preset sampling space to obtain the vertex features of the vertex.

[0064] For example, the SMPL-X human body model is configured with a three-dimensional coordinate system {x c, y c, z c The position of the i-th vertex in the three-dimensional coordinate system is (x... i ,y i ,z iIn this embodiment of the application, the vertex position (x) of the i-th vertex is used as the basis for the calculation. i ,y i ,z i Perform feature sampling operations in the preset sampling space to obtain the vertex features of the i-th vertex.

[0065] Please see Figure 2 The neural network model is a multi-layer MLP neural network, which can automatically learn and output the center point offset vector Δu. i Rotation matrix R i Scale vector S i spherical harmonic function color i and opacity i For example, the expression for a multilayer MLP neural network is: color, o, Δu, R, S = mlp(f).

[0066] The Gaussian sphere placed at the corrected vertex position is the candidate Gaussian sphere, and the set of all candidate Gaussian spheres is the set of Gaussian spheres. In this embodiment, the center point offset vector Δu... i Adding the vertex position to the corrected vertex position T yields the corrected vertex position T. i +Δu i In this embodiment of the application, at vertex position T... i +Δu i Place a candidate Gaussian sphere on top, with the position of the sphere being the vertex position T. i +Δu i .

[0067] In this embodiment, step S143 involves removing candidate Gaussian spheres that do not meet preset rendering conditions from the Gaussian sphere set based on the depth map, resulting in multiple target Gaussian spheres. The depth map reliably and accurately reflects the relative distance between each Gaussian sphere and the human body surface. Based on the depth map, this embodiment can reliably and accurately identify Gaussian spheres not on the human body surface and remove them.

[0068] The target Gaussian sphere is the candidate Gaussian sphere remaining after removing candidate Gaussian spheres from the Gaussian sphere set that do not meet the preset rendering conditions. The preset rendering conditions include the candidate Gaussian sphere being on the surface of the human body. In some embodiments, when a candidate Gaussian sphere is outside the human body, it does not meet the preset rendering conditions. In some embodiments, when a candidate Gaussian sphere is inside the human body, it does not meet the preset rendering conditions.

[0069] In some embodiments, the camera is configured with an image coordinate system. Based on the depth map, removing candidate Gaussian spheres that do not meet the preset rendering conditions from the Gaussian sphere set to obtain multiple target Gaussian spheres includes the following steps: mapping the sphere position of the candidate Gaussian sphere to the image coordinate system to obtain the two-dimensional position of the candidate Gaussian sphere; determining the depth value corresponding to the two-dimensional position in the depth map to obtain the depth value corresponding to the candidate Gaussian sphere; and removing candidate Gaussian spheres that do not meet the preset rendering conditions from the Gaussian sphere set based on the depth value of the candidate Gaussian sphere to obtain multiple target Gaussian spheres.

[0070] Mapping the spherical position of a candidate Gaussian sphere to the image coordinate system to obtain its two-dimensional position includes the following steps: obtaining camera extrinsic and intrinsic parameters; based on the camera extrinsic and intrinsic parameters and the spherical position of the candidate Gaussian sphere, mapping the candidate Gaussian sphere to the image coordinate system to obtain its two-dimensional position. For example, in this embodiment, the two-dimensional position of the candidate Gaussian sphere is determined according to the following formula, where the camera intrinsic parameter is K, the camera extrinsic parameter is P, and the two-dimensional position of the candidate Gaussian sphere in the image coordinate system is pt. i =K·P·(Ti+Δui).

[0071] Determining the depth value corresponding to a two-dimensional position in the depth map and obtaining the depth value corresponding to a candidate Gaussian sphere includes the following steps: obtaining camera intrinsic parameters; mapping each pixel in the depth map to the image coordinate system based on the camera intrinsic parameters; obtaining the pixel position of each pixel in the depth map in the image coordinate system; determining the pixel position that is consistent with the two-dimensional position or whose error with the two-dimensional position is within a preset range as the target pixel position; and determining the depth value corresponding to the target pixel position as the depth value corresponding to the candidate Gaussian sphere.

[0072] In some embodiments, removing candidate Gaussian spheres from a Gaussian sphere set that do not meet preset rendering conditions based on their depth values ​​to obtain multiple target Gaussian spheres includes the following steps: Responding to the fact that the depth value of a candidate Gaussian sphere indicates that the candidate Gaussian sphere is outside the human body, determining that the candidate Gaussian sphere does not meet the preset rendering conditions, and removing all candidate Gaussian spheres that do not meet the preset rendering conditions from the Gaussian sphere set to obtain multiple target Gaussian spheres. See also... Figure 3 Candidate Gaussian sphere 31 is located on the surface of the target user 32's human body; therefore, candidate Gaussian sphere 31 meets the preset rendering conditions and needs to be retained. Candidate Gaussian sphere 33 is located between camera 34 and target user 32; candidate Gaussian sphere 33 does not meet the preset rendering conditions and needs to be removed. If candidate Gaussian sphere 33 is not removed, it will participate in training the Gaussian rendering model, consuming training resources. Furthermore, when the Gaussian rendering model subsequently renders and generates human body images, it will also render these candidate Gaussian spheres, which will cause artifacts in the human body images and increase the computing power and memory space required during the rendering process.

[0073] The embodiments of this application delete candidate Gaussian spheres outside the human body, which can effectively improve the artifacts caused by such candidate Gaussian spheres, which is beneficial to obtaining high-quality human body images, and can also save computing power and memory space.

[0074] The camera is configured with a camera coordinate system. The Z-axis coordinate value of the camera coordinate system is used to represent the depth value of each pixel in the depth map from the origin of the camera coordinate system. The depth value of the candidate Gaussian sphere is used to indicate that the candidate Gaussian sphere is outside the human body. Determining that the candidate Gaussian sphere does not meet the preset rendering conditions includes the following steps: Based on the camera extrinsic parameters, the sphere position of the candidate Gaussian sphere in the world coordinate system is mapped to the camera coordinate system to obtain the reference position of the candidate Gaussian sphere in the camera coordinate system. The Z-axis component of the reference position in the camera coordinate system is determined. When the Z-axis component of the reference position is less than a first preset distance threshold, the candidate Gaussian sphere is determined to not meet the preset rendering conditions. When the Z-axis component of the reference position is greater than the first preset distance threshold and less than or equal to a second preset distance threshold, the candidate Gaussian sphere is determined to meet the preset rendering conditions.

[0075] The first and second preset distance thresholds are customized by the designer based on engineering experience.

[0076] In some embodiments, the first preset distance threshold is the depth value corresponding to the candidate Gaussian sphere; in other embodiments, the first preset distance threshold is the depth value corresponding to the candidate Gaussian sphere minus a preset floating threshold, where the depth value corresponding to the candidate Gaussian sphere is D(pt). i The preset floating threshold is ε, and the first preset distance threshold is...

[0077] In some embodiments, the second preset distance threshold is the depth value corresponding to the candidate Gaussian sphere; in other embodiments, the second preset distance threshold is the depth value corresponding to the candidate Gaussian sphere plus a preset floating threshold, where the depth value corresponding to the candidate Gaussian sphere is D(pt). i The preset floating threshold is ε, and the second preset distance threshold is γ = D(pt). i )+ε.

[0078] Please continue reading. Figure 3 First preset distance threshold Second preset distance threshold γ = D(pt) i Candidate Gaussian sphere 31 is placed on the human body surface of target user 32, wherein the Z-axis component Z1 of candidate Gaussian sphere 31 in camera coordinate system x0y is greater than D(pt). i -ε but less than D(pt) i ), such candidate Gaussian spheres 31 need to be retained.

[0079] The candidate Gaussian sphere 33 is located between the camera 34 and the target user 32, wherein the Z-axis component Z2 of the candidate Gaussian sphere 33 in the camera coordinate system x0y is less than D(pt). i The expression ε indicates that the candidate Gaussian sphere 33 is far from the human body surface and moves towards the camera side, meaning the candidate Gaussian sphere 33 is outside the human body. Therefore, in this embodiment, the candidate Gaussian sphere 33 needs to be deleted. Subsequently, when this embodiment needs to render and output a human body image from a new perspective, the candidate Gaussian sphere 33 will not participate in the rendering process. Therefore, the artifact phenomenon in the human body image from the new perspective will be significantly reduced.

[0080] In some embodiments, removing candidate Gaussian spheres that do not meet preset rendering conditions from the Gaussian sphere set based on the depth value of the candidate Gaussian spheres to obtain multiple target Gaussian spheres includes the following steps: responding to the depth value of the candidate Gaussian spheres being used to indicate that the candidate Gaussian spheres are inside the human body, determining that the candidate Gaussian spheres do not meet the preset rendering conditions, and removing all candidate Gaussian spheres that do not meet the preset rendering conditions from the Gaussian sphere set to obtain multiple target Gaussian spheres.

[0081] Please see Figure 4 Candidate Gaussian sphere 41 is located on the surface of the target user 32's human body; therefore, candidate Gaussian sphere 41 meets the preset rendering conditions and needs to be retained. Candidate Gaussian sphere 42 is located inside the target user 32's human body; candidate Gaussian sphere 42 does not meet the preset rendering conditions and needs to be removed. If candidate Gaussian sphere 42 is not removed, it will participate in training the Gaussian rendering model, consuming training resources. Furthermore, when the Gaussian rendering model subsequently renders and generates a human body image, it will also render these candidate Gaussian spheres; however, rendering these candidate Gaussian spheres is an invalid operation and will also increase the computing power and memory space required during the rendering process. This embodiment of the application removes candidate Gaussian spheres inside the human body, which can save computing power and memory space.

[0082] The depth value of the candidate Gaussian sphere is used to represent the candidate Gaussian sphere within the human body. Determining whether the candidate Gaussian sphere meets the preset rendering conditions includes the following steps: Based on the camera extrinsic parameters, the sphere position of the candidate Gaussian sphere in the world coordinate system is mapped to the camera coordinate system to obtain the reference position of the candidate Gaussian sphere in the camera coordinate system. The Z-axis component of the reference position in the camera coordinate system is determined. When the Z-axis component of the reference position is greater than a second preset distance threshold, it is determined that the candidate Gaussian sphere does not meet the preset rendering conditions. When the Z-axis component of the reference position is greater than a first preset distance threshold and less than or equal to a second preset distance threshold, it is determined that the candidate Gaussian sphere meets the preset rendering conditions.

[0083] Please continue reading. Figure 4 First preset distance threshold Second preset distance threshold γ = D(pt) iCandidate Gaussian sphere 41 is located on the human body surface of target user 32, wherein the Z-axis component Z3 of candidate Gaussian sphere 41 in the camera coordinate system x0y is greater than D(pt). i -ε but less than D(pt) i ), such candidate Gaussian sphere 41 should be retained.

[0084] The Z-axis component Z4 of candidate Gaussian sphere 42 in the camera coordinate system x0y is greater than D(pt). i This indicates that candidate Gaussian sphere 42 is inside the human body, and the embodiments of this application require the removal of candidate Gaussian sphere 42.

[0085] Please see Figure 5 The surface of the target user 32's human body is covered with candidate Gaussian spheres. In this embodiment, the regions corresponding to the candidate Gaussian spheres 42 inside the human body are deleted, resulting in the removed human body region. The black elliptical regions represent the regions corresponding to the candidate Gaussian spheres 42 inside the human body. Subsequently, the candidate Gaussian spheres 42 will not participate in the training process of the Gaussian rendering model, thereby saving computing power and memory space.

[0086] In this embodiment, step S15 generates a human body Gaussian rendering image based on preset Gaussian rendering parameters and a target Gaussian sphere. This embodiment is based on the center point offset vector Δu. i Rotation matrix R i Scale vector S i spherical harmonic function color i and opacity i The target Gaussian sphere is combined and rendered to obtain a human body Gaussian rendering image.

[0087] In this embodiment of the application, step S16 is performed to segment the original human body region image from the human body image.

[0088] The original human body region image is the area occupied by the target user in the human body image. This application embodiment segments the original human body region image from the human body image based on a preset image segmentation algorithm. The preset image segmentation algorithm includes threshold-based segmentation algorithms, edge detection-based segmentation algorithms, deep learning-based segmentation algorithms, etc.

[0089] In this embodiment, step S17 is performed to generate a Gaussian rendering model based on the human body Gaussian rendering image and the original human body region image. In some embodiments, this embodiment calculates the image difference information between the human body Gaussian rendering image and the original human body region image, and optimizes the image difference information based on the gradient descent optimization algorithm to obtain the Gaussian rendering model.

[0090] In other embodiments, generating a Gaussian rendering model based on a human body Gaussian rendering map and an original human body region image includes the following steps: obtaining a human body texture rendering map corresponding to the human body image, fusing the human body texture rendering map with the human body Gaussian rendering map to obtain a rendering fusion map, and generating a Gaussian rendering model based on the rendering fusion map and the original human body region image.

[0091] Candidate Gaussian spheres that do not meet the preset rendering conditions will not participate in the rendering of the image region on the human body Gaussian rendering map. In other words, the image region occupied by candidate Gaussian spheres that do not meet the preset rendering conditions is a non-rendered area on the human body Gaussian rendering map. This manifests as a loss of human body reconstruction details in the non-rendered area. The human body texture rendering map can describe the overall human body reconstruction details, including the human body reconstruction details corresponding to the non-rendered areas in the human body Gaussian rendering map. Therefore, the human body texture rendering map can compensate for the deficiency of the human body Gaussian rendering map in not presenting human body reconstruction details in non-rendered areas. Furthermore, the human body Gaussian rendering map can provide a more accurate description of some human body reconstruction details, such as clothing, which are not part of the SMPL-X template information. Therefore, the human body Gaussian rendering map and the human body texture rendering map can complement each other in terms of human body reconstruction details, making the trained Gaussian rendering model more reliable and accurate.

[0092] Obtaining a human texture rendering map corresponding to a human image includes the following steps: generating a mesh rendering map based on multiple vertices, performing texture rendering operations on the mesh rendering map based on a preset texture rendering function to obtain a texture map to be optimized, obtaining the difference information between the human image and the texture map to be optimized, optimizing the texture map to be optimized based on the difference information, and obtaining a human texture rendering map corresponding to the human image.

[0093] Adjacent vertices are connected to form triangular patches, and these triangular patches can constitute a mesh rendering map. In this embodiment, a preset texture rendering function f(S,t) is used to perform texture rendering on the mesh rendering map to obtain a texture map to be optimized, where f is the preset texture rendering function, S is the mesh rendering map, and t is the texture map to be optimized. In this embodiment, the difference information between the human image and the texture map to be optimized is calculated according to the following formula: L1 = |img - f(S,t)|, where L1 is the difference information and img is the human image.

[0094] In this embodiment, the difference information is optimized using the gradient descent algorithm. For example... Thus, a human body image (img) is obtained. t Corresponding human body texture rendering image R t Subsequently, as the embodiments of this application continuously extract new vertices from the human body image and continuously update the image (img) based on the new vertices... t Corresponding human body texture rendering image R tThis human body texture rendering map can describe the details of human body reconstruction as a whole, including the human body reconstruction details corresponding to non-rendered areas in the human body Gaussian rendering map, which can help prepare for the generation of more accurate and reliable Gaussian rendering models.

[0095] The process of fusing the human body texture rendering map with the human body Gaussian rendering map to obtain the rendering fusion map includes steps S181 to S185.

[0096] In the embodiment of this application, step S181 is performed to mask the non-rendered areas occupied by the human body Gaussian rendering image and the candidate Gaussian spheres that do not meet the preset rendering conditions on the human body Gaussian rendering image to obtain a human body mask image.

[0097] The human body mask image is a binary image. The mask values ​​of the human body Gaussian rendering image are different from the mask values ​​of the non-rendered areas. It can be understood that the mask values ​​of the human body Gaussian rendering image and the non-rendered areas can be customized by the designer based on engineering experience. For example, the mask value of the human body Gaussian rendering image is 1, and the mask value of the non-rendered areas is 0.

[0098] In this embodiment of the application, the pixel value of the pixel in the human body Gaussian rendering image is set to 1, that is, the mask value of the human body Gaussian rendering image is 1, and the pixel value of the pixel in the non-rendering area is set to 0, that is, the mask value of the non-rendering area is 0.

[0099] In this embodiment of the application, step S182 is performed to fuse the human body Gaussian rendering image and the human body mask image to obtain a first fused image. For example, in this embodiment of the application, the human body Gaussian rendering image and the human body mask image are fused according to the following formula: Q = G·A, where Q is the first fused image, G is the human body Gaussian rendering image, and A is the human body mask image.

[0100] Since candidate Gaussian spheres that do not meet the preset rendering conditions occupy non-rendered areas on the human body Gaussian rendering map, they do not participate in the Gaussian rendering operation. Therefore, the areas in the human body mask map A corresponding to the human body Gaussian rendering map are high-quality, high-confidence Gaussian rendering areas, while the areas in the human body mask map A corresponding to non-rendered areas are low-quality, low-confidence areas. Furthermore, the mask value of the areas in the human body mask map A corresponding to the human body Gaussian rendering map is 1, and the mask value of the areas in the human body mask map A corresponding to non-rendered areas is 0. Therefore, during the G·A calculation process, for pixels at the same pixel location, the pixel (x) in the human body Gaussian rendering map G... i ,y i The pixel value M of the human body mask image A is compared with the pixel value (x) of the human body mask image A. i ,y i Multiply the pixel value of (x) by 1, and the pixel value of (x) is obtained. i ,y i The pixel value of ) remains M, and the pixel value of the non-rendered area (x) is M. j,y j The pixel value N of the human body mask image A is compared with the pixel value (x) of the human body mask image A. j ,y j Multiply the pixel value of (x) by 0, and the pixel value of (x) is equal to the pixel value of (x). j ,y j The pixel value of ) is 0. In this way, the first fused image Q can retain the high-quality and high-confidence Gaussian rendering area, but it lacks the human body reconstruction details corresponding to the non-rendered area.

[0101] In this embodiment of the application, step S183 is performed to invert the human body mask image to obtain an inverted mask image.

[0102] For example, in this embodiment of the application, the human body mask image is inverted according to the following formula: Y = 1 - A, where Y is the inverted mask image. Therefore, in the human body mask image, when the pixel value of a pixel is 1, after the inversion operation, the pixel value of that pixel is 0, and when the pixel value of a pixel is 0, after the inversion operation, the pixel value of that pixel is 1. Therefore, the pixel values ​​of pixels at the same pixel position in the inverted mask image and the pixel values ​​in the human body mask image are mutually exclusive.

[0103] In this embodiment, step S184 is performed to fuse the inverse mask image and the human body texture rendering image to obtain a second fused image. For example, in this embodiment, the inverse mask image and the human body texture rendering image are fused according to the following formula: E = (1-A)·R = Y·R, where E is the second fused image.

[0104] In the inverse mask Y, the mask value of the region corresponding to the human body Gaussian rendering map is 0, and the mask value of the region corresponding to the non-rendered region is 1.

[0105] During the calculation of (1-A)·RG·A, for pixels at the same pixel location, pixels in the non-rendered area (x j ,y j The pixel value N and the pixel points (x) of the inverse mask Y j ,y j Multiply the pixel value of (x) by 1, and the pixel value of (x) is obtained. j ,y j The pixel value of ) is still N, and the pixel value of the human body Gaussian rendering image G is (x i ,y i The pixel value M of the inverse mask image Y and the pixel value (x) i ,y i Multiply the pixel value of (x) by 0, and the pixel value of (x) is equal to the pixel value of (x). i ,y i The pixel value of ) is 0, so that the second fused image E can retain the human body reconstruction details corresponding to the non-rendered area.

[0106] In this embodiment, step S185 involves adding the first fused image and the second fused image to obtain a rendered fused image. For example, this embodiment adds the first fused image and the second fused image according to the following formula: Render = Q + E, where Render is the rendered fused image. As can be seen from this formula, the first fused image Q carries a high-quality, high-confidence Gaussian rendering region, and the second fused image E carries the human body reconstruction details corresponding to the non-rendered regions. After adding the first fused image Q and the second fused image E, the rendered fused image Render not only has a high-quality, high-confidence Gaussian rendering region, but also has the human body reconstruction details corresponding to the non-rendered regions. Thus, this embodiment not only achieves the goals of removing artifacts and deleting Gaussian spheres inside the human body, but also compensates for the lost human body reconstruction details in the human body Gaussian rendering image by using the human body texture rendering image, making the rendered fused image Render more accurate and reliable.

[0107] In this embodiment, the image difference between the rendered fusion image and the original human body region image is obtained, and the image difference is optimized based on the gradient descent algorithm to obtain the optimized Gaussian rendering parameters. A Gaussian rendering model is generated based on the optimized Gaussian rendering parameters.

[0108] For example, in embodiments of this application, image differences are calculated according to the following formula:

[0109] L = |Render-GT|

[0110] L represents the image difference, and GT represents the original human body region image.

[0111] The image differences were optimized using the gradient descent algorithm, resulting in the following optimized Gaussian rendering parameters:

[0112]

[0113] At this point, the Gaussian rendering model has been trained and generated in this embodiment of the application.

[0114] To illustrate in detail the method for generating Gaussian rendering models provided in the embodiments of this application, the embodiments of this application are combined with... Figure 6 This will be explained in detail, and the specific process is as follows:

[0115] This application embodiment sets up four processing modules, namely image preprocessing module a, Gaussian rendering module b, texture rendering module c, and fine rendering module d.

[0116] ① Image preprocessing module a includes the following processes:

[0117] a1, obtain human body image 61.

[0118] a2, vertices are determined based on SMPL-X human body template 62.

[0119] a3, generates a mesh rendering graph S based on vertices.

[0120] a4, based on a preset depth estimation algorithm, performs depth estimation on the human body image to obtain a depth map D.

[0121] ② The Gaussian rendering module b includes the following processes:

[0122] b1. Generate preset Gaussian rendering parameters based on the vertex position of the i-th vertex. The preset Gaussian rendering parameters include the center point offset vector Δu. i Rotation matrix R i Scale vector S i spherical harmonic function color i and opacity i .

[0123] b2. Combine the depth map D and the preset Gaussian rendering parameters to generate a human body Gaussian rendering map G. In the human body Gaussian rendering map G, the black area is a Gaussian sphere that meets the preset rendering conditions, and the white area is a Gaussian sphere that does not meet the preset rendering conditions. The Gaussian spheres that do not meet the preset rendering conditions have been removed from the human body Gaussian rendering map G.

[0124] b3. Generate a human body mask A based on the human body Gaussian rendering image G, where the human body mask A is a binary image, and in the human body mask A, the pixel value of the white area is 1, and the pixel value of the black area is 0.

[0125] ③The texture rendering module c includes the following processes:

[0126] c1. Generate a mesh rendering map based on multiple vertices.

[0127] c2. Perform texture rendering operations on the mesh rendering map based on the preset texture rendering function to obtain the texture map to be optimized.

[0128] c3. Obtain the difference information between the human body image and the texture image to be optimized.

[0129] c4. Optimize the texture map to be optimized based on the difference information to obtain the human body texture rendering map R.

[0130] ④ The refined rendering module d includes the following processes:

[0131] d1. Generate the rendering blending map Render according to the formula Render=G·A+(1-A)·R.

[0132] d2. Segment the original human body region image GT from the human body image.

[0133] d3. Generate a Gaussian rendering model based on the rendering fusion graph Render and the original human body region image GT.

[0134] In summary, the embodiments of this application not only remove Gaussian spheres spaced a certain distance from the human body surface, making the Gaussian rendering model more accurate and reasonable, and improving the artifacts in the human body images output by the Gaussian rendering model, as well as the artifacts appearing when outputting human body images from new perspectives, but also remove Gaussian spheres inside the human body, reducing the computational load required for rendering human body images and lowering memory space consumption. Furthermore, the embodiments of this application, based on the human body Gaussian rendering map, also combine a human body texture rendering map to jointly train and generate a Gaussian rendering model. The human body Gaussian rendering map and the human body texture rendering map can mutually compensate for each other's deficiencies in human body reconstruction details, making the trained Gaussian rendering model more reliable and accurate.

[0135] It should be noted that in the above embodiments, there is no necessarily a certain order between the steps. Those skilled in the art can understand from the description of the embodiments of this application that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0136] As another aspect of the embodiments of this application, this application provides an apparatus for generating a Gaussian rendering model. The apparatus for generating the Gaussian rendering model can be a software module, which includes several instructions stored in a memory. A processor can access the memory, call the instructions, and execute them to complete the Gaussian rendering model generation method described in the above embodiments.

[0137] In some implementations, the Gaussian rendering model generation apparatus can also be constructed from hardware devices. For example, the Gaussian rendering model generation apparatus can be constructed from one or more chips, which can work in coordination to complete the Gaussian rendering model generation method described in the various implementations above. As another example, the Gaussian rendering model generation apparatus can also be constructed from various logic devices, such as general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, ARM (Acorn RISC Machine) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination of these components.

[0138] Please see Figure 7 The Gaussian rendering model generation device 700 includes a video recording module 71, an image extraction module 72, a vertex extraction module 73, a Gaussian sphere filtering module 74, a Gaussian rendering module 75, a human body extraction module 76, and a model generation module 77.

[0139] The video recording module 71 acquires video data captured by the camera. The image extraction module 72 extracts multiple frames of human images of the target user from the video data. The vertex extraction module 73 extracts multiple vertices from the human images, with Gaussian spheres set at the vertex positions. The Gaussian sphere filtering module 74 removes Gaussian spheres not on the target user's human body surface from the multiple Gaussian spheres, obtaining multiple target Gaussian spheres. The Gaussian rendering module 75 generates a human body Gaussian rendering map based on preset Gaussian rendering parameters and the target Gaussian spheres. The human body extraction module 76 segments the original human body region image from the human body image. The model generation module 77 generates a Gaussian rendering model based on the human body Gaussian rendering map and the original human body region image.

[0140] This application not only removes Gaussian spheres spaced a certain distance from the human body surface, making the Gaussian rendering model more accurate and reasonable, and improving the artifacts in the human body images output by the Gaussian rendering model, as well as the artifacts appearing when outputting human body images from new perspectives, but also removes Gaussian spheres inside the human body, reducing the computational load required for rendering human body images and lowering memory space overhead. Furthermore, this application's embodiments, based on the human body Gaussian rendering map, also combine it with the human body texture rendering map to jointly train and generate the Gaussian rendering model. The human body Gaussian rendering map and the human body texture rendering map can compensate for each other's deficiencies in human body reconstruction details, making the trained Gaussian rendering model more reliable and accurate.

[0141] In some embodiments, the Gaussian sphere filtering module 74 is specifically used to: perform depth estimation on the human image to obtain a depth map; correct the vertex position of the vertex based on the vertex position of the vertex and preset Gaussian rendering parameters to obtain the corrected vertex position; place the Gaussian sphere at the corrected vertex position as a candidate Gaussian sphere; the set of all candidate Gaussian spheres is a Gaussian sphere set; and based on the depth map, remove candidate Gaussian spheres that do not meet the preset rendering conditions from the Gaussian sphere set to obtain multiple target Gaussian spheres.

[0142] In some embodiments, the camera is configured with an image coordinate system, and the Gaussian sphere filtering module 74 is further specifically used to: map the sphere position of the candidate Gaussian sphere to the image coordinate system to obtain the two-dimensional position of the candidate Gaussian sphere, determine the depth value corresponding to the two-dimensional position in the depth map to obtain the depth value corresponding to the candidate Gaussian sphere, and remove candidate Gaussian spheres that do not meet the preset rendering conditions from the Gaussian sphere set based on the depth value of the candidate Gaussian sphere to obtain multiple target Gaussian spheres.

[0143] In some embodiments, the Gaussian sphere filtering module 74 is further specifically used to: obtain camera extrinsic parameters and camera intrinsic parameters, and based on the camera extrinsic parameters, camera intrinsic parameters and the sphere position of the candidate Gaussian sphere, map the candidate Gaussian sphere to the image coordinate system to obtain the two-dimensional position of the candidate Gaussian sphere.

[0144] In some embodiments, the constituent elements of the position non-compliance condition include the candidate Gaussian sphere being outside the human body. The Gaussian sphere filtering module 74 is further specifically used to: respond to the depth value of the candidate Gaussian sphere indicating that the candidate Gaussian sphere is outside the human body, determine that the candidate Gaussian sphere does not meet the preset rendering condition, and remove all candidate Gaussian spheres that do not meet the preset rendering condition from the Gaussian sphere set to obtain multiple target Gaussian spheres.

[0145] In some embodiments, the constituent elements of the position non-compliance condition include the candidate Gaussian sphere being inside the human body. The Gaussian sphere filtering module 74 is further specifically used to: respond to the depth value of the candidate Gaussian sphere used to indicate that the candidate Gaussian sphere is inside the human body, determine that the candidate Gaussian sphere does not meet the preset rendering conditions, and remove all candidate Gaussian spheres that do not meet the preset rendering conditions from the Gaussian sphere set to obtain multiple target Gaussian spheres.

[0146] In some embodiments, the model generation module 77 is specifically used to: obtain a human texture rendering map corresponding to a human image, fuse the human texture rendering map with a human Gaussian rendering map to obtain a rendering fusion map, and generate a Gaussian rendering model based on the rendering fusion map and the original human region image.

[0147] In some embodiments, the model generation module 77 is further specifically used to: generate a mesh rendering map based on multiple vertices, perform texture rendering operation on the mesh rendering map based on a preset texture rendering function to obtain a texture map to be optimized, obtain the difference information between the human body image and the texture map to be optimized, optimize the texture map to be optimized based on the difference information, and obtain a human body texture rendering map corresponding to the human body image.

[0148] In some embodiments, the model generation module 77 is further specifically used for: performing masking processing on the human body Gaussian rendering image and the non-rendering area occupied by candidate Gaussian spheres that do not meet the preset rendering conditions on the human body Gaussian rendering image to obtain a human body mask image. The mask value of the human body Gaussian rendering image is different from the mask value of the non-rendering area. The human body mask image is a binary image. The human body Gaussian rendering image and the human body mask image are fused to obtain a first fused image. The human body mask image is inverted to obtain an inverted mask image. The inverted mask image is fused with the human body texture rendering image to obtain a second fused image. The first fused image and the second fused image are added to obtain a rendering fused image.

[0149] It should be noted that the above-described Gaussian rendering model generation apparatus can execute the Gaussian rendering model generation method provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in the embodiments of the Gaussian rendering model generation apparatus can be found in the Gaussian rendering model generation method provided in the embodiments of this application.

[0150] See Figure 8, Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 800 includes one or more processors 81 and a memory 82. The memory 82 is connected to one or more processors 81, for example, via a bus.

[0151] Processor 81 is configured to support the electronic device in performing the corresponding functions in the methods described in the above method embodiments. Processor 81 may be a central processing unit (CPU), a network processor (NP), a hardware chip, or any combination thereof. The aforementioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The aforementioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0152] Memory 82 is used to store program code, etc. Memory may include volatile memory (VM), such as random access memory (RAM); memory may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); memory may also include combinations of the above types of memory.

[0153] The memory 82 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the Gaussian rendering model generation method in the embodiments of this application. The processor executes the various functional applications and data processing of the Gaussian rendering model generation method and the Gaussian rendering model generation apparatus by running the non-volatile software programs, instructions, and modules stored in the memory, thereby realizing the functions of each module or unit of the Gaussian rendering model generation method and the Gaussian rendering model generation apparatus provided in the above method embodiments.

[0154] The memory 82 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function. The data storage area may store data created based on the use of the Gaussian rendering model generation apparatus. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the Gaussian rendering model generation apparatus via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0155] The one or more modules are stored in the memory. When executed by the one or more processors, they perform the method for generating the Gaussian rendering model in any of the above method embodiments. For example, they perform the method steps described in the above method embodiments to realize the functions of the modules described in the above device embodiments.

[0156] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by an electronic device, cause the electronic device to perform the method described in the foregoing embodiments.

[0157] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0158] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A method for generating a Gaussian rendering model, characterized in that, include: Acquire video data captured by the camera; Extract multiple frames of human images of the target user from the video data; Multiple vertices are extracted from the human body image, and a Gaussian sphere is set at the vertex position where the vertex is located; By removing Gaussian spheres that are not on the human body surface of the target user from the plurality of Gaussian spheres, a plurality of target Gaussian spheres are obtained; A human body Gaussian rendering image is generated based on preset Gaussian rendering parameters and the target Gaussian sphere. The original human body region image is segmented from the human body image; A Gaussian rendering model is generated based on the human body Gaussian rendering image and the original human body region image.

2. The generation method according to claim 1, characterized in that, The step of removing Gaussian spheres that are not on the surface of the target user's body from a plurality of Gaussian spheres to obtain a plurality of target Gaussian spheres includes: Perform depth estimation on the human body image to obtain a depth map; The vertex position of the vertex is corrected based on the vertex position and preset Gaussian rendering parameters to obtain the corrected vertex position. The Gaussian sphere placed on the corrected vertex position is the candidate Gaussian sphere, and the set of all the candidate Gaussian spheres is the Gaussian sphere set. Based on the depth map, candidate Gaussian spheres that do not meet the preset rendering conditions are removed from the Gaussian sphere set to obtain multiple target Gaussian spheres.

3. The generation method according to claim 2, characterized in that, The camera is configured with an image coordinate system. Based on the depth map, candidate Gaussian spheres that do not meet the preset rendering conditions are removed from the Gaussian sphere set to obtain multiple target Gaussian spheres, including: The position of the candidate Gaussian sphere is mapped to the image coordinate system to obtain the two-dimensional position of the candidate Gaussian sphere; Determine the depth value corresponding to the two-dimensional position in the depth map to obtain the depth value corresponding to the candidate Gaussian sphere; Based on the depth value of the candidate Gaussian spheres, candidate Gaussian spheres that do not meet the preset rendering conditions are removed from the Gaussian sphere set to obtain multiple target Gaussian spheres.

4. The generation method according to claim 3, characterized in that, The step of mapping the sphere position of the candidate Gaussian sphere to the image coordinate system to obtain the two-dimensional position of the candidate Gaussian sphere includes: Obtain camera extrinsic and intrinsic parameters; Based on the camera extrinsic parameters, the camera intrinsic parameters, and the sphere position of the candidate Gaussian sphere, the candidate Gaussian sphere is mapped to the image coordinate system to obtain the two-dimensional position of the candidate Gaussian sphere.

5. The generation method according to claim 3, characterized in that, The process of removing candidate Gaussian spheres from the Gaussian sphere set based on their depth values, which does not meet the preset rendering conditions, yields multiple target Gaussian spheres, including: The depth value of the candidate Gaussian sphere is used to indicate that the candidate Gaussian sphere is outside the human body, and it is determined that the candidate Gaussian sphere does not meet the preset rendering conditions; All candidate Gaussian spheres that do not meet the preset rendering conditions are removed from the Gaussian sphere set to obtain multiple target Gaussian spheres.

6. The generation method according to claim 3, characterized in that, The process of removing candidate Gaussian spheres from the Gaussian sphere set based on their depth values, which does not meet the preset rendering conditions, yields multiple target Gaussian spheres, including: The depth value of the candidate Gaussian sphere is used to indicate that the candidate Gaussian sphere is inside the human body, and it is determined that the candidate Gaussian sphere does not meet the preset rendering conditions; All candidate Gaussian spheres that do not meet the preset rendering conditions are removed from the Gaussian sphere set to obtain multiple target Gaussian spheres.

7. The generation method according to any one of claims 1 to 6, characterized in that, The step of generating a Gaussian rendering model based on the human body Gaussian rendering image and the original human body region image includes: Obtain the human texture rendering image corresponding to the human body image; The human body texture rendering map and the human body Gaussian rendering map are fused together to obtain a rendering fusion map; A Gaussian rendering model is generated based on the rendered fusion map and the original human body region image.

8. The generation method according to claim 7, characterized in that, The step of obtaining the human texture rendering map corresponding to the human body image includes: A mesh rendering map is generated based on multiple vertices; The mesh rendering image is subjected to texture rendering operation based on a preset texture rendering function to obtain a texture image to be optimized. Obtain the difference information between the human body image and the texture map to be optimized; Based on the difference information, the texture map to be optimized is optimized to obtain a human texture rendering map corresponding to the human body image.

9. The generation method according to claim 8, characterized in that, The step of fusing the human body texture rendering map with the human body Gaussian rendering map to obtain the rendering fusion map includes: The human body Gaussian rendering image and the non-rendered area occupied by the candidate Gaussian spheres that do not meet the preset rendering conditions are masked to obtain a human body mask image. The mask value of the human body Gaussian rendering image is different from the mask value of the non-rendered area. The human body mask image is a binary image. The human body Gaussian rendering image and the human body mask image are fused together to obtain the first fused image; Invert the human body mask image to obtain the reverse mask image; The inverse mask image is fused with the human body texture rendering image to obtain a second fused image; The first fused image and the second fused image are added together to obtain the rendered fused image.

10. An electronic device, characterized in that, The device includes a memory and a processor, the memory being connected to the processor, the processor being configured to execute one or more computer programs stored in the memory, the processor causing the electronic device to implement the method for generating a Gaussian rendering model as described in any one of claims 1-9 when executing the one or more computer programs.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method for generating a Gaussian rendering model as described in any one of claims 1-9.

Images