Method and device for reconstructing three-dimensional object

Abstract

The invention discloses a three-dimensional object reconstruction method and device, and relates to the technical field of image processing. The method comprises the steps of performing head reconstruction and body reconstruction based on an object historical video of a to-be-constructed object, and aligning a reconstructed head mesh model and a reconstructed body mesh model to obtain an initial three-dimensional object model; performing three-dimensional coordinate point sampling on the initial three-dimensional object model, and performing neighborhood construction on each three-dimensional sampling point in an obtained three-dimensional sampling point set to obtain a neighborhood construction result; and performing Gaussian training based on the three-dimensional sampling point set and a neighborhood construction result to obtain a target three-dimensional object model corresponding to the to-be-constructed object. According to the method and the device, the problem of poor modeling quality of a reconstruction method of a three-dimensional object in the related technology is solved, and the effect of improving the modeling quality of reconstruction of the three-dimensional object is further achieved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method and apparatus for reconstructing three-dimensional objects. Background Technology

[0002] With the development of computer vision technology, 3D object reconstruction technology has been widely used in recent years. Compared with 2D objects for simple applications, 3D objects can play a more complex role in application scenarios, and in many scenarios, 3D objects also have better visual effects.

[0003] However, in related technologies, the reconstruction of 3D objects is quite difficult. Traditional 3D reconstruction methods mainly rely on manual modeling or 3D reconstruction techniques based on multi-view images. The cost of 3D object reconstruction is usually high, and the accuracy, driving effect, and visual effect of the obtained 3D object are difficult to guarantee. Therefore, it is evident that the 3D object reconstruction methods in related technologies suffer from poor modeling quality. Summary of the Invention

[0004] This application provides a method and apparatus for reconstructing three-dimensional objects, so as to at least solve the problem of poor modeling quality in the reconstruction methods of three-dimensional objects in related technologies.

[0005] According to one embodiment of this application, a method for reconstructing a three-dimensional object is provided, comprising: performing head reconstruction and body reconstruction based on the object's historical video, and aligning the reconstructed head mesh model and body mesh model to obtain an initial three-dimensional object model, wherein the initial three-dimensional object model is a mesh model; sampling three-dimensional coordinate points on the initial three-dimensional object model, and constructing a neighborhood for each three-dimensional sampling point in the obtained three-dimensional sampling point set to obtain a neighborhood construction result, wherein the neighborhood construction result is used to indicate the neighborhood relationship of each three-dimensional sampling point; performing Gaussian training based on the three-dimensional sampling point set and the neighborhood construction result to obtain a target three-dimensional object model corresponding to the object to be constructed, wherein the target three-dimensional object model is an object model represented by a set of Gaussian spheres, each Gaussian sphere in the Gaussian sphere set is generated based on a three-dimensional sampling point in the three-dimensional sampling point set, and the neighborhood relationship of each Gaussian sphere matches the neighborhood relationship of the three-dimensional sampling point corresponding to each Gaussian sphere.

[0006] According to another aspect of the present invention, a three-dimensional object reconstruction apparatus is also provided, comprising: a first execution unit, configured to perform head reconstruction and body reconstruction based on an object history video of the object to be constructed, and align the reconstructed head mesh model and body mesh model to obtain an initial three-dimensional object model, wherein the initial three-dimensional object model is a mesh model; a second execution unit, configured to sample three-dimensional coordinate points of the initial three-dimensional object model, and perform neighborhood construction on each three-dimensional sampling point in the obtained three-dimensional sampling point set to obtain a neighborhood construction result, wherein the neighborhood construction result is used to indicate the neighborhood relationship of each three-dimensional sampling point; and a Gaussian training unit, configured to perform Gaussian training based on the three-dimensional sampling point set and the neighborhood construction result to obtain a target three-dimensional object model corresponding to the object to be constructed, wherein the target three-dimensional object model is an object model represented by a set of Gaussian spheres, each Gaussian sphere in the set of Gaussian spheres is generated based on a three-dimensional sampling point in the set of three-dimensional sampling points, and the neighborhood relationship of each Gaussian sphere matches the neighborhood relationship of the three-dimensional sampling point corresponding to each Gaussian sphere.

[0007] In an exemplary embodiment, the first execution unit includes: a positioning module for locating a bridging region between the head mesh model and the body mesh model; and a first execution module for performing an alignment operation on the other mesh model of the head mesh model and the body mesh model based on the bridging region and using one of the mesh models as a reference to obtain the initial three-dimensional object model, wherein the alignment operation includes at least one of the following: translation operation and rotation operation.

[0008] In an exemplary embodiment, the second execution unit includes: a first mapping module, configured to map the mesh vertices of an initial three-dimensional object model from three-dimensional space to UV space to obtain a set of two-dimensional vertices to be sampled; a sampling module, configured to uniformly sample the set of two-dimensional vertices to be sampled to obtain a set of two-dimensional orthogonal coordinate points; and a second mapping module, configured to map the two-dimensional orthogonal coordinate points in the set of two-dimensional orthogonal coordinate points from UV space back to three-dimensional space to obtain the set of three-dimensional sampling points.

[0009] In an exemplary embodiment, the apparatus further includes: a third execution unit, configured to perform a local density evaluation operation on each of the three-dimensional sampling points after sampling the three-dimensional coordinate points of the initial three-dimensional object model, to obtain a local density evaluation result for each of the three-dimensional sampling points, wherein the local density evaluation operation is used to evaluate the density of three-dimensional sampling points within a specified range with each of the three-dimensional sampling points as reference points; a marking unit, configured to mark a region to be densified based on the local density evaluation result of each of the three-dimensional sampling points and a density threshold corresponding to the object part to which each of the three-dimensional sampling points belongs; and a fourth execution unit, configured to perform a densification operation on the region to be densified and add the three-dimensional sampling points added by the densification region to the set of three-dimensional sampling points to obtain an updated set of three-dimensional sampling points, wherein the densification operation is used to increase the density of the three-dimensional sampling points within the region to be densified.

[0010] In an exemplary embodiment, the second execution unit includes: a second execution module, configured to perform the following neighborhood construction operation with each three-dimensional sampling point as the current three-dimensional sampling point, to obtain the neighborhood construction result: determining all mesh faces where the current three-dimensional sampling point is located, to obtain a current mesh face set; traversing each current mesh face in the current mesh face set to search for adjacent faces of each current mesh face; and establishing a neighborhood relationship between the current three-dimensional sampling point and the three-dimensional sampling points contained in the adjacent faces of each current mesh face.

[0011] In an exemplary embodiment, the Gaussian training unit includes: a third execution module, configured to generate Gaussian spheres using each three-dimensional sampling point in the set of three-dimensional sampling points as the center point of a Gaussian sphere, and configure the neighborhood relationship of the Gaussian sphere corresponding to each three-dimensional sampling point based on the neighborhood relationship of each three-dimensional sampling point, to obtain the set of Gaussian spheres; and a fourth execution module, configured to perform Gaussian attribute training and Gaussian deformation model training on the set of Gaussian spheres respectively, and fuse the basic Gaussian attributes of each Gaussian sphere obtained by training with the Gaussian attribute deformation of each Gaussian sphere to obtain the target three-dimensional object model, wherein the object to be constructed includes multiple object parts, the Gaussian deformation model training is performed on the Gaussian sphere set part by part based on the object parts to which the Gaussian spheres in the set of Gaussian spheres belong, and the set of control parameters used in the Gaussian deformation model training includes the part parameters of each object part among the multiple object parts, the set of control parameters being preset or predicted based on the historical video of the object.

[0012] In one exemplary embodiment, the apparatus further includes: a conversion unit, configured to, after Gaussian training based on the three-dimensional sampling point set and the neighborhood construction result, convert the acquired object control signal into an initial parameter value sequence corresponding to the control parameter set in response to the acquired object control signal, wherein the initial parameter value sequence is used to record the changes in the parameter values ​​of the control parameters in the control parameter set over time; an adjustment unit, configured to adjust the initial parameter values ​​in the initial parameter value sequence according to specified constraints to obtain a target parameter value sequence, wherein the specified constraints include at least one of the following: constraints on the parameter values ​​of the control parameters in the control parameter set; association conditions between the parameter values ​​of at least two control parameters that have an association relationship in the control parameter set; and a driving unit, configured to drive the target three-dimensional object model using the target parameter value sequence to cause the target three-dimensional object model to perform an action matching the object control signal.

[0013] In an exemplary embodiment, the apparatus further includes: a generation unit, configured to generate a combination of lens parameters for multiple viewpoint lenses via a lens parameter generator, after Gaussian training based on the three-dimensional sampling point set and the neighborhood construction result, when an object image corresponding to the target three-dimensional object model is to be output, wherein the lens parameter combination includes the lens parameters of each of the multiple viewpoint lenses; and a fifth execution unit, configured to perform the following image generation operations on each viewpoint lens as a current viewpoint lens to obtain an object image corresponding to each viewpoint lens, wherein, in the lens parameter combination, the lens parameters of the current viewpoint lens are current lens parameters, and the two-dimensional plane of the current viewpoint lens is a current two-dimensional plane: determining the projected area of ​​each Gaussian sphere on the current two-dimensional plane according to the current lens parameters, and hiding the Gaussian sphere. In the sphere set, Gaussian spheres whose projected area on the current two-dimensional plane is less than a specified area threshold are obtained to form a hidden Gaussian sphere set, wherein the projected area corresponding to each Gaussian sphere is the projected area of ​​each Gaussian sphere projected onto the specified plane; according to the neighborhood relationship of the hidden Gaussian spheres in the hidden Gaussian sphere set, the hidden Gaussian spheres in the hidden Gaussian sphere set are merged, and the Gaussian spheres in the merged Gaussian sphere set whose corresponding projected area is greater than or equal to the specified area threshold are added to the Gaussian sphere set to be rendered, wherein the Gaussian sphere set to be rendered also includes the unhidden Gaussian spheres in the Gaussian sphere set; a rendering operation is performed on each Gaussian sphere in the Gaussian sphere set to be rendered, and the rendered object image corresponding to the current viewpoint is output; an output unit is used to output multi-view images, wherein the multi-view images include the object image corresponding to each viewpoint.

[0014] In an exemplary embodiment, the fifth execution unit includes: an acquisition module, configured to acquire the Gaussian sphere projection of each Gaussian sphere to be rendered, wherein the Gaussian sphere projection of each Gaussian sphere to be rendered is obtained by projecting each Gaussian sphere to be rendered from three-dimensional space onto the current two-dimensional plane according to the current lens parameters; a first determination module, configured to determine the Gaussian sphere projection covering each pixel on the current two-dimensional plane among the Gaussian sphere projections of the Gaussian spheres to be rendered in the set of Gaussian spheres to be rendered, thereby obtaining a set of Gaussian sphere projections corresponding to each pixel; and a second determination module, configured to determine the Gaussian sphere projection of each pixel corresponding to each Gaussian sphere. The probability density value of the sphere projection on each pixel is determined as the weight of each Gaussian sphere projection corresponding to each pixel; the fifth execution module is used to mix the colors of each Gaussian sphere projection corresponding to each pixel according to the weight of each Gaussian sphere projection corresponding to each pixel, and determine the mixed color as the color of each pixel, wherein the color of each Gaussian sphere projection corresponding to each pixel is the color of the Gaussian sphere to be rendered to which each Gaussian sphere projection corresponding to each pixel belongs; the rendering module is used to render the object image corresponding to the current viewpoint lens according to the color of each pixel.

[0015] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer program, which is configured to execute the above-described method for reconstructing a three-dimensional object at runtime.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the reconstruction method of the three-dimensional object through the computer program.

[0017] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the above-described method for reconstructing a three-dimensional object.

[0018] In this embodiment, head and body reconstruction are performed based on the historical video of the object to be constructed, and the reconstructed head and body mesh models are aligned to obtain an initial 3D object model, wherein the initial 3D object model is a mesh model; 3D coordinate points are sampled on the initial 3D object model, and a neighborhood is constructed for each 3D sampling point in the obtained 3D sampling point set to obtain a neighborhood construction result, wherein the neighborhood construction result is used to indicate the neighborhood relationship of each 3D sampling point; Gaussian training is performed based on the 3D sampling point set and the neighborhood construction result to obtain the target 3D object model corresponding to the object to be constructed, wherein the target 3D object model... It is an object model represented by a set of Gaussian spheres. Each Gaussian sphere in the set is generated based on a 3D sampling point in the set of 3D sampling points. The neighborhood relationship of each Gaussian sphere is matched with the neighborhood relationship of the corresponding 3D sampling point. Since the target 3D object model has undergone detailed sampling, it has high precision. Moreover, the neighborhood relationship between sampling points is taken into account during Gaussian training, which can achieve better visual effects. It can maintain better coherence and naturalness in dynamic scenes and has high modeling quality. It can solve the problem of poor modeling quality in the reconstruction methods of 3D objects in related technologies and improve the modeling quality of 3D object reconstruction. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the hardware environment for a method for reconstructing a three-dimensional object according to an embodiment of this application;

[0022] Figure 2 This is a flowchart illustrating an optional method for reconstructing a three-dimensional object according to an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of an optional method for reconstructing a three-dimensional object according to an embodiment of this application;

[0024] Figure 4 This is a schematic diagram of the structure of a three-dimensional object reconstruction system according to an embodiment of this application;

[0025] Figure 5This is a schematic diagram of another optional method for reconstructing a three-dimensional object according to an embodiment of this application;

[0026] Figure 6 This is a structural block diagram of an optional three-dimensional object reconstruction apparatus according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0030] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0031] The methods and embodiments provided in this application can be executed on a computer terminal or similar computing device. Taking running on a computer terminal as an example, Figure 1 This is a hardware structure block diagram of a computer terminal for a method of reconstructing a three-dimensional object according to an embodiment of this application. For example... Figure 1 As shown, a computer terminal may include one or more ( Figure 1Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a CPU or FPGA) and a memory 104 for storing data are also shown. The computer terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0032] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the three-dimensional object reconstruction method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0033] The computer terminal uses a wireless network provided by a communications provider. In one example, transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0034] This embodiment provides a method for reconstructing a three-dimensional object, which is applied to the aforementioned computer terminal. Figure 2 This is a flowchart of a method for reconstructing a three-dimensional object according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0035] Step S202: Reconstruct the head and body based on the historical video of the object to be constructed, and align the reconstructed head mesh model and body mesh model to obtain an initial three-dimensional object model, wherein the initial three-dimensional object model is a mesh model;

[0036] Step S204: Sample three-dimensional coordinate points of the initial three-dimensional object model, and construct a neighborhood for each three-dimensional sampling point in the obtained three-dimensional sampling point set to obtain a neighborhood construction result, wherein the neighborhood construction result is used to indicate the neighborhood relationship of each three-dimensional sampling point;

[0037] Step S206: Gaussian training is performed based on the set of three-dimensional sampling points and the neighborhood construction results to obtain the target three-dimensional object model corresponding to the object to be constructed. The target three-dimensional object model is an object model represented by a set of Gaussian spheres. Each Gaussian sphere in the set of Gaussian spheres is generated based on a three-dimensional sampling point in the set of three-dimensional sampling points. The neighborhood relationship of each Gaussian sphere is matched with the neighborhood relationship of the three-dimensional sampling point corresponding to each Gaussian sphere.

[0038] The 3D object reconstruction method in this embodiment can be applied to the field of image processing technology, specifically to scenarios where a corresponding 3D object model is constructed based on historical video of the object to be built. A 3D object is a fundamental element used in computer graphics and 3D modeling to create and represent entities in the real world or other fictional environments. It exists in a 3D coordinate system and has three dimensions: length, width, and height, enabling it to exhibit a sense of depth and solidity. 3D objects have wide applications in fields such as game development, film production, virtual reality, and augmented reality. 3D objects are typically in the form of 3D object models, which can be driven as needed to perform specified actions.

[0039] With the development of computer vision technology, 3D object reconstruction technology has been widely used in recent years. However, among related technologies, the reconstruction of 3D objects is relatively difficult. Traditional 3D reconstruction methods mainly rely on manual modeling or multi-view image-based 3D reconstruction techniques. Manual modeling-based 3D reconstruction methods are costly, and the production of high-precision, hyper-realistic 3D object models requires a long cycle and high cost, while the quality of manual modeling is also difficult to guarantee. Multi-view image-based 3D reconstruction achieves high-precision static scene reconstruction through Gaussian sputtering, but the accuracy of 3D object reconstruction in dynamic scenes is relatively low, and the visual effect of the reconstructed 3D object model is poor.

[0040] This shows that the methods for reconstructing 3D objects in related technologies suffer from poor modeling quality.

[0041] To at least partially solve the above-mentioned technical problems, in this embodiment, an initial three-dimensional object model is obtained by performing preliminary modeling based on the object's historical video. Then, the initial three-dimensional object model is further sampled, its neighborhood is constructed, and Gaussian training is performed to obtain a high-precision and high-quality target three-dimensional object model. Since the target three-dimensional object model has undergone detailed sampling and has high precision, and the neighborhood relationship between sampling points is considered during Gaussian training, better visual effects can be obtained, resulting in higher modeling quality. Therefore, the problem of poor modeling quality mentioned above can be solved.

[0042] In this embodiment, head and body reconstruction are performed based on the historical video of the object to be constructed. The reconstructed head and body mesh models are then aligned to obtain an initial 3D object model, which is a mesh model. Here, the historical video includes the dynamic changes in the appearance and shape of the object to be constructed at different times. The historical video can be analyzed based on relevant algorithms to locate the head and body of the object to be constructed in the historical video. Then, key feature points of the head and body are extracted to achieve the initial construction of the head and body mesh models. By aligning and fusing these mesh models, an initial 3D object model can be obtained, which can retain the basic shape and dynamic features of the object to be constructed.

[0043] Optionally, based on facial mesh models such as the Facial Landmark Animation Model (FLAME) or the Three-Dimensional Morphable Model (3DMM), a 3D reconstruction algorithm based on monocular video can be used to achieve consistent 3D reconstruction of facial features across multiple video frames of the historical video of the object to be constructed, while simultaneously tracking 3D motion details such as head posture, expression, lip shape, and gaze direction for each video frame. Alternatively, based on human mesh models such as the Skinned Multi-Person Linear Model (SMPL), a 3D reconstruction algorithm based on monocular video can be used to achieve consistent 3D reconstruction of human body features across multiple video frames of the historical video of the object to be constructed, while simultaneously tracking 3D motion details such as body posture, hand posture, and leg posture for each video frame.

[0044] The initial 3D object model is sampled using 3D coordinate points. For each 3D sample point in the resulting set, a neighborhood construction is performed, yielding a neighborhood construction result that indicates the neighborhood relationships of each 3D sample point. After initial modeling, the mesh model can be sampled using 3D coordinate points to convert the initial 3D object model into a point cloud representation, preparing for subsequent Gaussian sphere representation. Optionally, the sampling points can be selected using a uniform distribution, or the surface details of the model can be considered, increasing the sampling density in areas with more detail to ensure reconstruction accuracy and detail richness. Neighborhood construction determines the relationships between sampling points. The neighborhood construction result indicates the neighborhood relationships of each 3D sample point and is crucial for controlling the coordinated deformation between Gaussian spheres, helping to maintain a natural transition between adjacent sampling points during dynamic changes in the model.

[0045] Gaussian training is performed based on the 3D sampling point set and neighborhood construction results to obtain the target 3D object model corresponding to the object to be constructed. The target 3D object model is an object model represented by a set of Gaussian spheres. Each Gaussian sphere in the set is generated based on a 3D sampling point from the 3D sampling point set, and the neighborhood relationship of each Gaussian sphere matches the neighborhood relationship of the corresponding 3D sampling point. The final step is to further model and train the sampling points using the Gaussian sphere set representation. Gaussian spheres, centered on their location, possess attributes such as size, orientation, and color. They can flexibly describe and reconstruct the surface of complex objects, offering advantages such as flexible attributes and easy control, and are widely used in the representation and rendering of 3D scenes or objects.

[0046] In this embodiment, the target 3D object model is an object model represented by a set of Gaussian spheres. During Gaussian training, the attributes of each Gaussian sphere are matched with the neighborhood relationship of the sampling points. Through learning and optimization, the target 3D object model can not only maintain high accuracy, but also reflect the subtle effects of dynamic changes.

[0047] For example, taking a dancer as the object to be constructed, the target 3D object model is the dancer model. When driving the dancer model to perform complex rotational movements, the Gaussian sphere set can accurately track muscle stretching and limb swing by adjusting their respective positions, sizes, and orientations. The trained model can maintain the continuity and naturalness of the dancer model's movements even in fast-paced dynamic scenes, presenting a realistic visual effect.

[0048] Through the embodiments provided in this application, head and body reconstruction are performed based on the historical video of the object to be constructed, and the reconstructed head mesh model and body mesh model are aligned to obtain an initial 3D object model, wherein the initial 3D object model is a mesh model; 3D coordinate points are sampled on the initial 3D object model, and a neighborhood is constructed for each 3D sampling point in the obtained 3D sampling point set to obtain a neighborhood construction result, wherein the neighborhood construction result is used to indicate the neighborhood relationship of each 3D sampling point; Gaussian training is performed based on the 3D sampling point set and the neighborhood construction result to obtain the target 3D object model corresponding to the object to be constructed, wherein the target 3D object model... The model is an object model represented by a set of Gaussian spheres. Each Gaussian sphere in the set is generated based on a 3D sampling point in the set of 3D sampling points. The neighborhood relationship of each Gaussian sphere is matched with the neighborhood relationship of the corresponding 3D sampling point. Since the target 3D object model has undergone detailed sampling, it has high precision. Moreover, the neighborhood relationship between sampling points is taken into account during Gaussian training, which can achieve better visual effects. It can maintain better coherence and naturalness in dynamic scenes and has high modeling quality. It can solve the problem of poor modeling quality in the reconstruction methods of 3D objects in related technologies and improve the modeling quality of 3D object reconstruction.

[0049] In an exemplary embodiment, aligning the reconstructed head mesh model and body mesh model to obtain an initial three-dimensional object model includes: locating a bridging region between the head mesh model and the body mesh model; and, based on the bridging region, performing an alignment operation on the other mesh model of the head mesh model and the body mesh model, using one of the mesh models as a reference, to obtain the initial three-dimensional object model, wherein the alignment operation includes at least one of the following: translation operation and rotation operation.

[0050] To improve the quality of head and body reconstruction, different suitable algorithms can be used for head and body reconstruction, and the reconstructions can be performed separately. To ensure model consistency, the reconstructed head and body mesh models can be aligned after each reconstruction.

[0051] In this embodiment, the bridging region between the head mesh model and the body mesh model is first located. Here, the boundary between the head mesh model and the body mesh model can be identified and defined as the bridging region. The bridging region can be the part connecting the head and torso, such as the neck and shoulders. The bridging region can be automatically detected by analyzing the dynamic changes of objects in the video.

[0052] After locating the bridging region, an alignment operation can be performed on the other mesh model (either the head mesh model or the body mesh model) based on this region, using one of the mesh models as a reference. This yields an initial 3D object model. The alignment operation includes at least one of the following: translation or rotation. Specifically, the other mesh model can be translated and rotated using either the head mesh model or the body mesh model as a reference to ensure accurate connection between the body mesh model and the head model, eliminating visual inconsistencies.

[0053] When performing alignment operations, if the relative positions of the head and body models in space do not match, the position of the body model can be adjusted by translation to ensure that the head model and body model are precisely aligned in the vertical and horizontal directions. When it is determined that the connection direction of the head and body models in the bridging area is inconsistent, the head model or body model can be rotated to adjust the orientation of the head or the posture of the body to ensure that the head and body are aligned in the three-dimensional space, thus achieving natural fusion alignment.

[0054] This embodiment improves the visual coordination and overall consistency of the model by performing positioning and alignment operations on the bridging region, thereby enhancing the visual effect of the reconstructed model.

[0055] In an exemplary embodiment, sampling three-dimensional coordinate points of an initial three-dimensional object model includes: mapping the mesh vertices of the initial three-dimensional object model from three-dimensional space to UV space to obtain a set of two-dimensional vertices to be sampled; uniformly sampling the set of two-dimensional vertices to be sampled to obtain a set of two-dimensional orthogonal coordinate points; and mapping the two-dimensional orthogonal coordinate points in the set of two-dimensional orthogonal coordinate points back from UV space to three-dimensional space to obtain a set of three-dimensional sampling points.

[0056] The initial 3D object model obtained in the foregoing embodiments is a mesh model, which has low accuracy and is not suitable for generating Gaussian sphere sets. To optimize the initial 3D object model, 3D coordinate points can be sampled from the initial 3D object model to obtain a more efficient representation of the 3D model, namely, a point cloud model represented by 3D coordinate points.

[0057] In this embodiment, to improve the uniform distribution of sampling points and increase sampling efficiency, UV space can be used to generate sampling points. The mesh vertices of the initial 3D object model are mapped from 3D space to UV space to obtain a set of 2D vertices to be sampled. Here, the mesh vertices in the initial 3D object model are transformed from the complex 3D space to the relatively simple UV space. UV space is a two-dimensional coordinate system (U-axis is horizontal, V-axis is vertical). Mapping to UV space unfolds the surface of the 3D model into a 2D plane, facilitating sampling operations. The set of 2D vertices to be sampled retains the structural information of the model surface but eliminates the complexity of 3D space, thus improving sampling efficiency.

[0058] Uniform sampling is performed on the set of 2D vertices to be sampled to obtain a set of 2D orthogonal coordinate points. Uniform sampling ensures that sufficient sampling points are obtained even in areas with rich model surface detail, while avoiding oversampling in flat regions. This helps reduce the computational burden in subsequent processing, improves rendering performance, and yields a uniformly and efficiently distributed set of 2D orthogonal coordinate points. Optionally, during the sampling process, valid regions can be filtered based on surface attributes to avoid invalid sampling of invalid regions, further improving sampling efficiency.

[0059] Finally, the two-dimensional orthogonal coordinate points in the set of two-dimensional orthogonal coordinate points are mapped back from UV space to three-dimensional space to obtain a set of three-dimensional sampling points. Here, the mapping relationship of mapping the mesh vertices of the initial three-dimensional object model from three-dimensional space to UV space can be reversed to map the set of two-dimensional orthogonal coordinate points back to three-dimensional space.

[0060] In this embodiment, by uniformly sampling in the UV space and then mapping the sampling points back to the 3D space, sampling in the UV space can fully sample 3D surface areas with large variations, avoid oversampling flat 3D surface areas, improve rendering performance, and map back to the 3D space to obtain a point cloud model composed of 3D sampling points, which is convenient for subsequent model processing.

[0061] In an exemplary embodiment, after sampling three-dimensional coordinate points on the initial three-dimensional object model, the method further includes: performing a local density evaluation operation on each three-dimensional sampling point to obtain a local density evaluation result for each three-dimensional sampling point, wherein the local density evaluation operation is used to evaluate the density of three-dimensional sampling points within a specified range with each three-dimensional sampling point as a reference point; marking the region to be densified based on the local density evaluation result of each three-dimensional sampling point and the density threshold corresponding to the object part to which each three-dimensional sampling point belongs; performing a densification operation on the region to be densified, and adding the three-dimensional sampling points added by the densification region to the set of three-dimensional sampling points to obtain an updated set of three-dimensional sampling points, wherein the densification operation is used to increase the density of the three-dimensional sampling points in the region to be densified.

[0062] In 3D reconstruction, to ensure sufficient preservation of model surface details while avoiding oversampling of non-critical areas, it is necessary to reasonably control the density of sampling points. However, sampling methods in related technologies often struggle to balance detail and efficiency, easily resulting in detail loss in fine areas of the model and resource waste in flat areas. To address these issues, this embodiment employs an adaptive density enhancement strategy based on local density evaluation, ensuring both efficiency and accuracy in model reconstruction.

[0063] In this embodiment, a local density evaluation operation is performed on each 3D sampling point to obtain the local density evaluation result for each 3D sampling point. The local density evaluation operation is used to evaluate the density of 3D sampling points within a specified range with each 3D sampling point as a reference point. Here, the local density of a point is evaluated by calculating the number of other sampling points within a specified range centered on each sampling point. The specified range can be set based on the features and resolution requirements of the model surface to ensure that the evaluation process accurately reflects the neighborhood characteristics of the sampling point.

[0064] Based on the local density evaluation results of each 3D sampling point and the density threshold corresponding to the object part to which each 3D sampling point belongs, the area to be densified is marked. Here, the area that needs high-density sampling (i.e., the area to be densified) can be determined and marked based on the density threshold corresponding to the object part to which each 3D sampling point belongs. Different density thresholds can be set for areas corresponding to different object parts. For example, facial feature areas such as eyes, nose, and mouth, as well as body detail areas such as fingers and toes, can be set with higher density thresholds to retain more details; while flat areas such as arms and legs can be set with lower thresholds to reduce unnecessary sampling points and improve efficiency.

[0065] Optionally, while marking the area to be densified, the densification weight can be determined based on the difference between the local density assessment result of each 3D sampling point and the density threshold corresponding to the object part. For example, if the difference is large, a larger densification weight can be set to focus on densifying the area.

[0066] A densification operation is performed on the region to be densified, and the additional 3D sampling points from the densified region are added to the 3D sampling point set, resulting in an updated 3D sampling point set. The densification operation increases the density of 3D sampling points within the region to be densified. Specifically, the densification operation is performed on all marked regions to be densified, adding additional sampling points to these regions to increase their sampling point density. Then, the sampling operation described in the previous embodiment is performed again based on the added sampling points, and the resulting 3D sampling points are added to the 3D sampling point set, resulting in an updated 3D sampling point set. Optionally, additional sampling points can be added to the densified region using algorithms such as rule-based face segmentation algorithms, interpolation-based algorithms, or machine learning-based optimization algorithms.

[0067] In this embodiment, by performing a local density assessment operation on each sampling point, the region that needs to be densified is identified based on the assessment results and the density threshold of the location, and sampling points are added in a targeted manner. This can form a set of three-dimensional sampling points that can both preserve details and save resources, thereby improving the model details while ensuring reconstruction efficiency.

[0068] In an exemplary embodiment, neighborhood construction is performed on each three-dimensional sampling point in the obtained three-dimensional sampling point set to obtain a neighborhood construction result. This includes: performing the following neighborhood construction operation on each three-dimensional sampling point as the current three-dimensional sampling point to obtain a neighborhood construction result: determining all mesh faces where the current three-dimensional sampling point is located to obtain a current mesh face set; traversing each current mesh face in the current mesh face set to search for the adjacent faces of each current mesh face; and establishing the neighborhood relationship between the current three-dimensional sampling point and the three-dimensional sampling points contained in the adjacent faces of each current mesh face.

[0069] To ensure the continuity and rationality of the deformation of the 3D model during dynamic driving, in this embodiment, a neighborhood construction operation can be performed with each 3D sampling point as the current 3D sampling point to obtain the neighborhood construction result, so as to build the neighborhood relationship between sampling points, so as to control the degree of deformation through the neighborhood and achieve a more reasonable local deformation connection.

[0070] In this embodiment, the neighborhood construction operation includes: determining all grid faces where the current 3D sampling point is located to obtain the current grid face set; traversing each current grid face in the current grid face set to search for the neighboring faces of each current grid face; and establishing the neighborhood relationship between the current 3D sampling point and the 3D sampling points contained in the neighboring faces of each current grid face.

[0071] For each sampling point in the set of 3D sampling points, determine all the mesh faces it occupies. Alternatively, this can be achieved by finding the position of each sampling point within the 3D model mesh, ensuring that each sampling point is associated with the structural features of the model.

[0072] Iterate through the set of mesh faces where each sampling point is located, and for each mesh face, search for all its adjacent mesh faces. The definition of adjacent mesh faces can be based on vertex sharing between faces, and can be determined by checking whether two mesh faces have at least one common vertex.

[0073] After determining the adjacent mesh surfaces, the relationship between the current 3D sampling point and other sampling points contained in the adjacent mesh surfaces can be further defined as a neighborhood relationship. By recording the neighboring sampling points of each sampling point, the neighborhood set of each sampling point can be obtained, i.e., the neighborhood construction result.

[0074] In this embodiment, by constructing the neighborhood relationship of each sampling point, a topology-based (rather than distance-based) neighborhood relationship of sampling points is realized, which can enhance the structural coherence and deformation rationality of the 3D reconstruction model.

[0075] In an exemplary embodiment, Gaussian training is performed based on a set of three-dimensional sampling points and the neighborhood construction results to obtain a target three-dimensional object model. This includes: generating Gaussian spheres using each three-dimensional sampling point in the set as the center point of a Gaussian sphere, and configuring the neighborhood relationship of the Gaussian sphere corresponding to each three-dimensional sampling point based on the neighborhood relationship of each three-dimensional sampling point to obtain a set of Gaussian spheres; performing Gaussian attribute training and Gaussian deformation model training on the Gaussian sphere set respectively, and fusing the basic Gaussian attributes of each trained Gaussian sphere with the Gaussian attribute deformation of each Gaussian sphere to obtain the target three-dimensional object model. The object to be constructed includes multiple object parts. The Gaussian deformation model training is performed on the Gaussian sphere set based on the object parts to which the Gaussian spheres belong in the Gaussian sphere set, and the control parameter set used for the Gaussian deformation model training includes the part parameters of each object part among the multiple object parts. The control parameter set is preset or predicted based on the object's historical video.

[0076] To ensure the continuity and rationality of the deformation and achieve high-precision representation of the model, in this embodiment, a set of Gaussian spheres can be generated based on a set of three-dimensional sampling points, and the neighborhood relationships between the Gaussian spheres can be configured using a neighborhood construction structure.

[0077] In this embodiment, Gaussian spheres are generated by using each 3D sampling point in the set of 3D sampling points as the center point of the Gaussian sphere. The neighborhood relationships of the Gaussian spheres corresponding to each 3D sampling point are configured based on their neighborhood relationships to obtain a set of Gaussian spheres. Optionally, the model surfaces can be divided into different categories (e.g., eyes, mouth, head, neck, torso, etc.) according to different parts of the human body. Based on the category of the surface to which the sampling point belongs, the generated Gaussian spheres are divided into different regions. The basic attributes of each generated Gaussian sphere may include its center position (i.e., the position of the corresponding 3D sampling point), the part it belongs to, its size, rotation, color, and opacity.

[0078] After generating Gaussian spheres, the neighborhood relationships of Gaussian spheres corresponding to each 3D sampling point can be configured. Based on the neighborhood construction results obtained in the previous embodiments, the neighborhood relationships of each Gaussian sphere are configured to obtain a set of Gaussian spheres. This allows the use of neighborhood sampling point information of each sampling point to ensure the correlation between Gaussian spheres and the continuity of deformation, providing topological constraints for subsequent deformation training.

[0079] After configuring neighborhood relationships, Gaussian attribute training and Gaussian deformation model training can be performed separately on the Gaussian sphere set. The basic Gaussian attributes of each trained Gaussian sphere are then fused with the Gaussian attribute deformations of each Gaussian sphere to obtain the target 3D object model. The object to be constructed includes multiple object parts. The Gaussian deformation model training is performed on the Gaussian sphere set based on the object parts to which the Gaussian spheres belong, segmented by part. The set of control parameters used in the Gaussian deformation model training includes the part parameters of each of the multiple object parts. The set of control parameters is either pre-set or predicted based on historical video of the object. Optionally, the basic attributes of each Gaussian sphere can include the object part to which the Gaussian sphere belongs. During Gaussian training, the object part can be directly read from the basic attributes of the Gaussian sphere, improving training efficiency.

[0080] Here, Gaussian attribute training can be performed on each Gaussian sphere in the Gaussian sphere set, based on its center point position and the object part it belongs to. The basic output of the training can include properties such as size, rotation, color, and opacity. Optionally, the basic properties of the Gaussian spheres other than the center position and the object part can be initialized first. During initialization, the degrees of freedom of parameters can be reduced by setting the isotropy of the Gaussian spheres, and the upper limit of the basic Gaussian model can be increased by setting the uniformity and differences of properties among the various Gaussian spheres. Furthermore, by setting corresponding optimization algorithms and hyperparameters, the aforementioned basic properties can be learned through backpropagation. The Gaussian attribute training process aims to optimize the basic Gaussian properties so that the Gaussian spheres can accurately represent the surface features of the 3D model.

[0081] Gaussian deformation model training is performed part-by-part based on the object's location associated with the Gaussian sphere. For example, a facial Gaussian sphere can use a set of facial-specific parameters, while a torso Gaussian sphere can use a different set of parameters. This part-by-part training strategy ensures that the model's deformation effects are more natural and personalized across different parts, improving the training's relevance and effectiveness, while also simplifying the training process and increasing efficiency. Here, different part parameters collectively constitute a set of control parameters, which can be pre-defined or predicted based on historical video of the object. Optionally, the control parameter set can include position encoding and control encoding. For sampled point data, a learnable three-plane encoder can be used to map the three-dimensional coordinates of the sampled points to obtain compact and efficient position encoding; for controllable parameter data, a fixed encoder is used to obtain control encoding. By combining the position encoding and control encoding, part parameters for different parts can be obtained, forming the control parameter set.

[0082] After inputting the corresponding part parameters, the property deformation of each Gaussian sphere can be obtained by outputting deformation models for different regions (different deformation models are used for different parts). The deformation models for different regions can be learned through backpropagation. Optionally, the deformation model can be trained using machine learning algorithms (such as deep learning networks), enabling the model to learn how to adjust the properties of the Gaussian sphere according to specific control parameters, thereby achieving the deformation of the target 3D object model under different control parameters. This allows the property deformation of each Gaussian sphere to change according to different character poses.

[0083] After completing the Gaussian property training and Gaussian deformation model training, the basic Gaussian properties of each Gaussian sphere and the Gaussian property deformation of each Gaussian sphere can be fused to obtain the target 3D object model. The target 3D object model obtained after fusion can more finely depict different actions and facial expressions.

[0084] For example, such as Figure 3As shown, firstly, controllable parameters of the whole body and coordinates of Gaussian sampling points are determined. Based on the coordinates of Gaussian sampling points, basic Gaussian properties are trained to obtain the basic properties of the Gaussian sphere. Based on the controllable parameters of the whole body and the coordinates of Gaussian sampling points, Gaussian spheres are divided according to body parts and the data is reorganized. Body part parameters are obtained through a learnable position encoder and a control encoder. A regional Gaussian deformation model is trained to obtain the attribute deformation of the Gaussian sphere. By fusing the basic properties and attribute deformation of the Gaussian sphere, the Gaussian sphere properties of each Gaussian sphere can be obtained, forming a target 3D object model.

[0085] In this embodiment, by generating a set of Gaussian spheres, configuring the neighborhood relationships of Gaussian spheres, and training Gaussian attributes and Gaussian deformation models, a target 3D object model with high-precision dynamic deformation capabilities can be constructed, improving the quality of the target 3D object model. In addition, the method of combining basic Gaussian and Gaussian model deformation can reduce the difficulty of driving learning, while retaining the personalized deformation effects of specific characters. Establishing a regional deformation mechanism and introducing the correlation between parameters and Gaussian spheres can accelerate the modeling process.

[0086] In an exemplary embodiment, after Gaussian training based on the three-dimensional sampling point set and the neighborhood construction result, the method further includes: in response to the acquired object control signal, converting the object control signal into an initial parameter value sequence corresponding to the control parameter set, wherein the initial parameter value sequence is used to record the changes in the parameter values ​​of the control parameters in the control parameter set over time; adjusting the initial parameter values ​​in the initial parameter value sequence according to specified constraints to obtain a target parameter value sequence, wherein the specified constraints include at least one of the following: constraints on the parameter values ​​of the control parameters in the control parameter set; association conditions between the parameter values ​​of at least two control parameters that have an association relationship in the control parameter set; and using the target parameter value sequence to drive the target three-dimensional object model so that the target three-dimensional object model performs an action matching the object control signal.

[0087] After obtaining the target 3D object model, it can be driven to move as required and change the facial expressions, actions, and other features of the 3D object model.

[0088] In this embodiment, in response to the acquired object control signal, the object control signal is converted into an initial parameter value sequence corresponding to the control parameter set. The initial parameter value sequence records the changes in the parameter values ​​of the control parameters in the control parameter set over time. Here, the object control signal can be input text or audio. Optionally, a 3D face animation algorithm can be used to generate face mesh model parameters that are consistent with and aligned with the input text or audio. Simultaneously, a 3D body animation algorithm can be used to generate limb mesh model parameters that conform to the semantics of the input text or audio. This allows the model to be driven based on the generated parameters. Here, the generated parameters are the initial parameter value sequence, used to record the changes in the parameter values ​​of the control parameters in the control parameter set over time. That is, the initial parameter value sequence can be used to drive a series of dynamic changes in the target 3D object model.

[0089] The initial parameter values ​​in the initial parameter value sequence are adjusted according to specified constraints to obtain the target parameter value sequence. These specified constraints include at least one of the following: constraints on the parameter values ​​of control parameters in the control parameter set; and association conditions between the parameter values ​​of at least two related control parameters in the control parameter set. Here, to achieve a smooth and reasonable human-driven effect, the initial parameter value sequence can be checked to ensure it conforms to preset specified constraints, such as the value range and rate of change of control parameters, avoiding unreasonable or out-of-range driving instructions. By limiting the dynamic range and trend of parameter values, the stability and safety of model driving can be ensured. Optionally, for related control parameters, such as facial expressions and head posture, or limb movements and overall balance, adjustments can be made according to preset association conditions to ensure coordination between parameters of different parts when the model performs complex movements, achieving reasonable related movements between limbs, avoiding unnatural distortions or contradictory movement performances, and improving the naturalness and credibility of model driving. Optionally, to avoid instability in the parameters generated by the algorithm, a temporal filtering algorithm can be used to achieve more continuous and smooth parameter changes. Here, the constraints on the parameter values ​​of the control parameters in the control parameter set may include range restrictions, rate of change restrictions, logical restrictions, etc., and the association conditions between the parameter values ​​of at least two control parameters that are related in the control parameter set may include physical constraints, kinematic models, etc.

[0090] After obtaining the optimized sequence of target parameter values, the target 3D object model can be driven by the target parameter value sequence to perform actions that match the object control signals. Optionally, the driving process may include: using a blend shape to obtain the offsets of the face controllable parameters and limb controllable parameters relative to the template mesh model; using a sparse matrix to calculate the joint positions from the mesh vertices; and using a hybrid skinning algorithm (such as Linear Blend Skinning (LBS)) to transform each vertex in the model mesh surface using the weighted influence of its adjacent bones to obtain a full-body human mesh for each target parameter value in the target parameter value sequence, thereby driving the model.

[0091] This embodiment, through the dynamic driving of a 3D object model based on control signals, not only improves the interactivity and expressiveness of the model, but also ensures the naturalness and rationality of the driving process.

[0092] In an exemplary embodiment, after Gaussian training based on the 3D sampling point set and neighborhood construction results, the method further includes: when the object image corresponding to the target 3D object model is to be output, generating a combination of lens parameters for multiple viewpoint lenses through a lens parameter generator, wherein the lens parameter combination includes the lens parameters of each viewpoint lens among the multiple viewpoint lenses; performing the following image generation operation on each viewpoint lens as the current viewpoint lens to obtain the object image corresponding to each viewpoint lens, wherein, in the lens parameter combination, the lens parameters of the current viewpoint lens are the current lens parameters, and the 2D plane of the current viewpoint lens is the current 2D plane; determining the projected area of ​​each Gaussian sphere on the current 2D plane according to the current lens parameters, and hiding it in the Gaussian sphere set, in the current... Gaussian spheres with projected areas less than a specified area threshold on the two-dimensional plane are used to obtain a set of hidden Gaussian spheres. The projected area of ​​each Gaussian sphere is the projected area of ​​the Gaussian sphere onto the specified plane. Hidden Gaussian spheres in the set are merged according to their neighborhood relationships. Gaussian spheres with projected areas greater than or equal to the specified area threshold are added to the set of Gaussian spheres to be rendered. The set of Gaussian spheres to be rendered also includes the unhidden Gaussian spheres in the set. Rendering is performed on each Gaussian sphere in the set of Gaussian spheres to be rendered, and the rendered object image corresponding to the current viewpoint is output. Multi-view images are output, including the object image corresponding to each viewpoint.

[0093] After obtaining the target 3D object model or driving the target 3D object model accordingly, it is necessary to render it as a video for output. However, the rendering methods in related technologies suffer from poor visual effects or low rendering efficiency. For example, due to the smooth property of the Gaussian distribution itself, a large number of Gaussian spheres are needed to cover the rendering in order to restore realistic details. Although the visual effect is good, it also brings the problem of low rendering efficiency. In order to solve the above problems, in this embodiment, some Gaussian spheres can be hidden and merged, and the remaining Gaussian spheres can be rendered, thereby balancing visual effects and rendering efficiency.

[0094] In this embodiment, when an object image corresponding to the target 3D object model is to be output, a lens parameter generator generates a combination of lens parameters for multiple viewpoints. The lens parameter combination includes the lens parameters of each viewpoint. Here, the combination of lens parameters for multiple viewpoints corresponds to multiple viewpoints, and the rendering result based on the lens parameter combination can include multi-view video. By setting the lens parameter combination for multiple viewpoints, synchronous rendering output from multiple perspectives can be achieved, improving the comprehensiveness of the output video.

[0095] Optionally, the lens parameters can include two different reference frames: a global Gaussian reference frame and a lens reference frame. A coordinate system transformation can be performed using fixed lens intrinsic parameters and lens extrinsic parameter distributions provided by training data to obtain the lens's global position. A custom lens parameter generator can be set to automatically generate continuous parameters that conform to the changing rules, allowing for intuitive adjustment of the lens position in the global coordinate system.

[0096] Each viewpoint is used as the current viewpoint and an image generation operation is performed to obtain the object image corresponding to each viewpoint. In the lens parameter combination, the lens parameters of the current viewpoint are the current lens parameters, and the two-dimensional plane of the current viewpoint is the current two-dimensional plane.

[0097] The image generation operation performed for each viewpoint includes: determining the projected area of ​​each Gaussian sphere on the current 2D plane based on the current camera parameters; and hiding Gaussian spheres in the Gaussian sphere set whose projected area on the current 2D plane is less than a specified area threshold, resulting in a hidden Gaussian sphere set. The projected area of ​​each Gaussian sphere is the projected area of ​​that Gaussian sphere onto the specified plane. Here, a resolution adjustment strategy can be used for optimization. The size of the Gaussian sphere projected onto the plane is calculated based on the camera parameters. If the projected size is less than the specified area threshold (i.e., its impact is very small), the Gaussian sphere can be hidden and not directly rendered. Optionally, the specified area threshold can be one pixel. Simultaneously, the hidden Gaussian sphere data can be saved to obtain the hidden Gaussian sphere set.

[0098] Based on the neighborhood relationships of hidden Gaussian spheres in the hidden Gaussian sphere set, the hidden Gaussian spheres in the hidden Gaussian sphere set are merged. Gaussian spheres with a projected area greater than or equal to a specified area threshold from the merged set are added to the set of Gaussian spheres to be rendered. This set also includes unhidden Gaussian spheres from the original set. Rendering is then performed on each Gaussian sphere in the set to be rendered, and the rendered image of the object corresponding to the current viewpoint is output. Here, merging multiple hidden Gaussian spheres into a single, larger Gaussian sphere based on neighborhood relationships and adding it to the set of Gaussian spheres to be rendered effectively reduces the number of Gaussian spheres that need to be rendered while ensuring the exposure of rendering details. This improves rendering efficiency and saves computational resources while maintaining visual quality.

[0099] Finally, the rendering operation is performed on each Gaussian sphere in the set of Gaussian spheres to be rendered, and the rendered object image corresponding to the current viewpoint is output; and multi-view images are also output, where the multi-view images include the object image corresponding to each viewpoint. The multi-view images can include images of the target 3D model viewed from different angles, which can adapt to the needs of multi-view display devices such as 3D displays, virtual reality (VR) / augmented reality (AR) headsets, or facilitate the selection of the most suitable camera viewpoint.

[0100] In this embodiment, by dynamically adjusting the number of Gaussian spheres according to lens parameters and rendering parameters, rendering efficiency can be improved while ensuring visual effects, enabling the target 3D model to be driven and output video in a smooth process. In addition, by setting multiple lens parameters and outputting multi-view images, continuous rendering output of multi-view images of the target 3D model in the same time and space can be achieved.

[0101] In an exemplary embodiment, performing a rendering operation on each Gaussian sphere in the set of Gaussian spheres to be rendered includes: obtaining the Gaussian sphere projection of each Gaussian sphere to be rendered, wherein the Gaussian sphere projection of each Gaussian sphere to be rendered is obtained by projecting each Gaussian sphere to be rendered from three-dimensional space onto a current two-dimensional plane according to the current camera parameters; determining the Gaussian sphere projections of the Gaussian spheres to be rendered in the set of Gaussian spheres to be rendered that cover each pixel on the current two-dimensional plane, thereby obtaining a set of Gaussian sphere projections corresponding to each pixel; determining the probability density value of each Gaussian sphere projection corresponding to each pixel on each pixel as the weight of each Gaussian sphere projection corresponding to each pixel; mixing the colors of each Gaussian sphere projection corresponding to each pixel according to the weight of each Gaussian sphere projection corresponding to each pixel, and determining the mixed color as the color of each pixel, wherein the color of each Gaussian sphere projection corresponding to each pixel is the color of the Gaussian sphere to be rendered to which each Gaussian sphere projection corresponding to each pixel belongs; and rendering the object image corresponding to the current viewpoint camera according to the color of each pixel.

[0102] To render a target 3D model into a corresponding 2D image under given lens parameters, a rasterization algorithm can be used, based on Gaussian sphere properties and lens parameters.

[0103] In this embodiment, the Gaussian sphere projection for each Gaussian sphere to be rendered is first obtained. This projection is obtained by projecting each Gaussian sphere from three-dimensional space onto a current two-dimensional plane based on the current camera parameters. Here, Gaussian sphere attributes may include position, size, rotation, and color, while camera parameters may include position, viewpoint, and projection matrix. The projection matrix is ​​used to convert the position of the Gaussian sphere from three-dimensional to two-dimensional. Then, combined with the camera parameters, the covariance matrix of the Gaussian distribution (representing the rotation and scaling of the Gaussian sphere) is projected from three-dimensional space onto the two-dimensional plane to obtain the Gaussian sphere projection for each Gaussian sphere to be rendered. Since the distance and angle between each Gaussian sphere and the camera are different, the covariance matrix can represent the scaling and rotation of the Gaussian sphere based on the distance and angle relationship between the Gaussian sphere and the camera, converting the Gaussian sphere into a specified size and shape in two-dimensional space.

[0104] The process involves determining the Gaussian sphere projections within the set of Gaussian spheres to be rendered, and then covering each pixel on the current 2D plane with the Gaussian sphere projections. This yields a set of Gaussian sphere projections for each pixel. After projecting the Gaussian spheres onto the 2D plane, the Gaussian sphere projections covering each pixel on the current 2D plane can be determined. Then, the probability density value of each Gaussian sphere projection corresponding to each pixel can be used to determine the weight of each Gaussian sphere projection corresponding to each pixel. In other words, the Gaussian distribution covered by each pixel and its weights are calculated on the 2D plane. The probability density value reflects the magnitude of the Gaussian sphere projection's contribution to the pixel's color and can be calculated based on the center position, size, and rotation of the Gaussian sphere, as well as the relative position of the pixel to the center of the Gaussian sphere.

[0105] According to the weight of each Gaussian sphere projection corresponding to each pixel, the colors of each Gaussian sphere projection corresponding to each pixel are mixed, and the mixed color is determined as the color of each pixel. Here, the color of each Gaussian sphere projection corresponding to each pixel is the color of the Gaussian sphere to be rendered to which each Gaussian sphere projection belongs. By mixing the colors of these Gaussian sphere projections according to their weights, the final color of the pixel can be obtained. Optionally, weighted averaging or other mixing algorithms can be used to ensure smooth color transitions and natural visual effects.

[0106] Finally, the object image corresponding to the current viewpoint is rendered according to the color of each pixel. Applying the final color of each pixel to the current 2D plane completes the rendering of the 3D object image under the current viewpoint. Compared with traditional computationally intensive rendering processes (such as volume rendering), the rasterization rendering in this embodiment only needs to handle the projection and blending of the Gaussian sphere within the view frustum, which avoids the calculation of per-ray sampling and integration, and enables real-time and efficient rendering.

[0107] This embodiment utilizes multi-view rendering based on Gaussian sphere projection to not only ensure rendering efficiency but also achieve high-quality output of object images of the target 3D model from different perspectives.

[0108] The reconstruction method of a three-dimensional object in this application embodiment will be explained below with reference to an optional example. In this optional example, the object to be reconstructed by the three-dimensional object reconstruction method is a 3D digital human, and the reconstruction method is performed by a three-dimensional object reconstruction system. Figure 4 This is a schematic diagram of the structure of the 3D object reconstruction system in this optional example, such as... Figure 4 As shown, the reconstruction system for this 3D object may include A. a human figure tracking module, B. a sampling module, C. a Gaussian training module, D. a parameter-driven module, and E. a Gaussian rendering module.

[0109] The A-1 Human Tracking Module, consisting of the A-1 Head Tracking Module, A-2 Body Tracking Module, A-3 Head-Body Alignment Module, and A-4 Parameter Statistics Module, receives raw video, outputs controllable full-body parameters to the C-1 Gaussian Training Module, and outputs a mesh to the sampling module for processing the input raw video, achieving relatively accurate head and body tracking as well as parameter prediction and statistics. The A-4 Parameter Statistics Module is used to predict the mesh model parameters and their statistical results corresponding to the output tracking results. Based on a parameterized 3D deformable mesh model, face, human pose, and shape estimation algorithms (such as 3D Human Pose Estimation (3D HPE)) are used to predict controllable full-body parameters, including head shape parameters, head pose parameters, expression parameters, body shape parameters, limb pose parameters, and hidden camera parameters. Parameter statistics are performed simultaneously with the output, facilitating use by the parameter optimization module.

[0110] B. Sampling Module, consisting of B-1. Mesh Sampling Module, B-2. Adaptive Sampling Module, and B-3. Neighborhood Construction Module, can receive meshes from A. Human Image Tracking Module and D. Parameter Driven Module, and output Gaussian coordinates to the trained model and C. Gaussian Training Module. It can be used to achieve uniform, adaptive density sampling of 3D coordinate points of the mesh model for the generation and deformation constraints of Gaussian spheres.

[0111] The C. Gaussian Training Module, composed of C-1. Local Organization Module, C-2. Encoder Module, C-3. Basic Gaussian Module, C-4. Gaussian Deformation Module, and C-5. Loss Calculation Module, receives controllable full-body parameters from the A. Human Image Tracking Module, Gaussian coordinates from the B. Sampling Module, and images from the E. Gaussian Rendering Module. It outputs images to the E. Gaussian Rendering Module, enabling training of Gaussian attributes and deformation models based on preprocessing results, forming a complete, clear, and realistic Gaussian digital human model. The C-5. Loss Calculation Module is used to render predicted images using a Gaussian sphere, calculate image loss and other constraint losses, and perform gradient backpropagation. First, it receives the output from the Gaussian Rendering Module, obtaining rendered images and real images of different human body parts. Then, it compares the differences between the predicted rendered images and real images of different body parts, calculating various image losses. Furthermore, to constrain the deformation of Gaussian sphere attributes within a reasonable range, it uses the neighborhood relationships established by the B-3. Neighborhood Construction Module to compare the difference between each Gaussian sphere attribute and its neighborhood average, calculating the Laplacian loss. The weights of each loss are balanced to form an overall loss for learning and iteration.

[0112] The D. Parameter-driven module, consisting of the D-1. Parameter Generation module, the D-2. Parameter Optimization module, and the D-3. Driving Mesh module, can receive text / audio and output a mesh to the B. Sampling module. It outputs controllable full-body parameters to the trained model and can be used to generate reasonable dynamic human body mesh sequences and their parameter sequences from input text / audio signals. This is used to drive the Gaussian digital face and body during inference, i.e., to drive the trained model.

[0113] The E. Gaussian Rendering module, composed of the E-1. Lens Controller module, E-2. Level of Detail module, E-3. Sputtering Rendering module, and E-4. Image Post-Processing module, receives Gaussian attributes from the trained model and from the C. Gaussian Training module, outputting multi-view images. It can be used to render all input Gaussian spheres into 2D images of the desired viewpoints under specific performance requirements, and perform image processing for training and output. The E-4. Image Post-Processing module performs necessary processing on the sputtered rendered images. During training, to enable the C-5. Loss Calculation module to calculate the training loss, the real images used for training are combined with the pre-processed image masks to obtain real images of different body parts. Simultaneously, Gaussian masks are created according to the data positions of different human body parts, and all Gaussian spheres are combined with the Gaussian masks and input into the E-3. Sputtering Rendering module to render the predicted images for each body part. During inference, this module is also responsible for outputting the rendered images according to the specified arrangement and format, obtaining multi-view image frames.

[0114] For example, such as Figure 5 As shown, the E. Gaussian rendering module's workflow includes configuring the lens and parameters using the lens controller; optimizing the number of Gaussian spheres based on Gaussian sphere properties using a level of detail mechanism; performing sputtering rendering with the help of a rasterization algorithm to achieve real-time inference speed; performing necessary post-processing on the image for training and inference output; and finally outputting multi-view images.

[0115] Through this optional example, a high-quality 3D model can be reconstructed by the combined processing of multiple modules. The mechanism of aligning the 3D deformable mesh model and sampling to generate a unified set of Gaussian spheres can generate Gaussian spheres for the head and body and establish matching neighborhood relationships, ensuring a complete and rich driving expression of the 3D model. The introduction of Laplacian loss can effectively constrain the deformation differences between adjacent Gaussian spheres, conforming to the objective laws of deformation, and reducing the learning difficulty of a high-degree-of-freedom system where all Gaussian spheres have their own different deformation mappings.

[0116] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0118] According to another aspect of the embodiments of this application, a three-dimensional object reconstruction apparatus is also provided. This apparatus can be used to implement the three-dimensional object reconstruction method provided in the above embodiments, and details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0119] Figure 6 This is a structural block diagram of an optional three-dimensional object reconstruction apparatus according to an embodiment of this application, such as... Figure 6 As shown, the apparatus for reconstructing the three-dimensional object includes:

[0120] The first execution unit 602 is used to perform head reconstruction and body reconstruction based on the object history video of the object to be constructed, and to align the reconstructed head mesh model and body mesh model to obtain an initial three-dimensional object model, wherein the initial three-dimensional object model is a mesh model.

[0121] The second execution unit 604 is used to sample three-dimensional coordinate points of the initial three-dimensional object model, and to construct a neighborhood for each three-dimensional sampling point in the obtained three-dimensional sampling point set, thereby obtaining a neighborhood construction result. The neighborhood construction result is used to indicate the neighborhood relationship of each three-dimensional sampling point.

[0122] The Gaussian training unit 606 is used to perform Gaussian training based on the set of three-dimensional sampling points and the neighborhood construction results to obtain the target three-dimensional object model corresponding to the object to be constructed. The target three-dimensional object model is an object model represented by a set of Gaussian spheres. Each Gaussian sphere in the set of Gaussian spheres is generated based on a three-dimensional sampling point in the set of three-dimensional sampling points. The neighborhood relationship of each Gaussian sphere is matched with the neighborhood relationship of the three-dimensional sampling point corresponding to each Gaussian sphere.

[0123] It should be noted that the first execution unit 602 in this embodiment can be used to execute the above step S202, the second execution unit 604 in this embodiment can be used to execute the above step S204, and the Gaussian training unit 606 in this embodiment can be used to execute the above step S206.

[0124] Through the embodiments provided in this application, head and body reconstruction are performed based on the historical video of the object to be constructed, and the reconstructed head mesh model and body mesh model are aligned to obtain an initial 3D object model, wherein the initial 3D object model is a mesh model; 3D coordinate points are sampled on the initial 3D object model, and a neighborhood is constructed for each 3D sampling point in the obtained 3D sampling point set to obtain a neighborhood construction result, wherein the neighborhood construction result is used to indicate the neighborhood relationship of each 3D sampling point; Gaussian training is performed based on the 3D sampling point set and the neighborhood construction result to obtain the target 3D object model corresponding to the object to be constructed, wherein the target 3D object model... The model is an object model represented by a set of Gaussian spheres. Each Gaussian sphere in the set is generated based on a 3D sampling point in the set of 3D sampling points. The neighborhood relationship of each Gaussian sphere is matched with the neighborhood relationship of the corresponding 3D sampling point. Since the target 3D object model has undergone detailed sampling, it has high precision. Moreover, the neighborhood relationship between sampling points is taken into account during Gaussian training, which can achieve better visual effects. It can maintain better coherence and naturalness in dynamic scenes and has high modeling quality. It can solve the problem of poor modeling quality in the reconstruction methods of 3D objects in related technologies and improve the modeling quality of 3D object reconstruction.

[0125] In an exemplary embodiment, the first execution unit includes: a positioning module for locating a bridging region between a head mesh model and a body mesh model; and a first execution module for performing an alignment operation on the other mesh model of the head mesh model and the body mesh model based on the bridging region and using one of the mesh models as a reference to obtain an initial three-dimensional object model, wherein the alignment operation includes at least one of the following: translation operation and rotation operation.

[0126] In an exemplary embodiment, the second execution unit includes: a first mapping module, configured to map the mesh vertices of the initial three-dimensional object model from three-dimensional space to UV space to obtain a set of two-dimensional vertices to be sampled; a sampling module, configured to uniformly sample the set of two-dimensional vertices to be sampled to obtain a set of two-dimensional orthogonal coordinate points; and a second mapping module, configured to map the two-dimensional orthogonal coordinate points in the set of two-dimensional orthogonal coordinate points from UV space back to three-dimensional space to obtain a set of three-dimensional sampling points.

[0127] In an exemplary embodiment, the apparatus further includes: a third execution unit, configured to perform a local density evaluation operation on each three-dimensional sampling point after sampling three-dimensional coordinate points on an initial three-dimensional object model, to obtain a local density evaluation result for each three-dimensional sampling point, wherein the local density evaluation operation is used to evaluate the density of three-dimensional sampling points within a specified range with each three-dimensional sampling point as a reference point; a marking unit, configured to mark the region to be densified based on the local density evaluation result of each three-dimensional sampling point and the density threshold corresponding to the object part to which each three-dimensional sampling point belongs; and a fourth execution unit, configured to perform a densification operation on the region to be densified and add the three-dimensional sampling points added by the densification region to the set of three-dimensional sampling points to obtain an updated set of three-dimensional sampling points, wherein the densification operation is used to increase the density of the three-dimensional sampling points in the region to be densified.

[0128] In an exemplary embodiment, the second execution unit includes: a second execution module, configured to perform the following neighborhood construction operation for each three-dimensional sampling point as the current three-dimensional sampling point, to obtain a neighborhood construction result: determine all mesh faces where the current three-dimensional sampling point is located, to obtain a current mesh face set; traverse each current mesh face in the current mesh face set to search for the neighboring faces of each current mesh face; and establish the neighborhood relationship between the current three-dimensional sampling point and the three-dimensional sampling points contained in the neighboring faces of each current mesh face.

[0129] In an exemplary embodiment, the Gaussian training unit includes: a third execution module, configured to generate Gaussian spheres using each three-dimensional sampling point in the set of three-dimensional sampling points as the center point of a Gaussian sphere, and configure the neighborhood relationship of the Gaussian sphere corresponding to each three-dimensional sampling point based on the neighborhood relationship of each three-dimensional sampling point, to obtain a set of Gaussian spheres; and a fourth execution module, configured to perform Gaussian attribute training and Gaussian deformation model training on the set of Gaussian spheres respectively, and fuse the basic Gaussian attributes of each trained Gaussian sphere with the Gaussian attribute deformation of each Gaussian sphere to obtain a target three-dimensional object model, wherein the object to be constructed includes multiple object parts, the Gaussian deformation model training is performed on the Gaussian sphere set based on the object parts to which the Gaussian spheres belong in the set of Gaussian spheres, and the set of control parameters used in the Gaussian deformation model training includes the part parameters of each object part among the multiple object parts, and the set of control parameters is preset or predicted based on the object's historical video.

[0130] In one exemplary embodiment, the apparatus further includes: a conversion unit, configured to, after Gaussian training based on a set of three-dimensional sampling points and a neighborhood construction result, convert the object control signal into an initial parameter value sequence corresponding to a set of control parameters in response to the acquired object control signal, wherein the initial parameter value sequence is used to record the changes in the parameter values ​​of the control parameters in the set of control parameters over time; an adjustment unit, configured to adjust the initial parameter values ​​in the initial parameter value sequence according to specified constraints to obtain a target parameter value sequence, wherein the specified constraints include at least one of the following: constraints on the parameter values ​​of the control parameters in the set of control parameters; association conditions between the parameter values ​​of at least two control parameters in the set of control parameters that have an association relationship; and a driving unit, configured to drive a target three-dimensional object model using the target parameter value sequence to cause the target three-dimensional object model to perform an action matching the object control signal.

[0131] In one exemplary embodiment, the apparatus further includes: a generation unit, configured to generate a combination of lens parameters for multiple viewpoint lenses via a lens parameter generator, after Gaussian training based on a set of three-dimensional sampling points and neighborhood construction results, wherein the lens parameter combination includes the lens parameters of each viewpoint lens among the multiple viewpoint lenses; and a fifth execution unit, configured to perform the following image generation operations on each viewpoint lens as the current viewpoint lens to obtain the object image corresponding to each viewpoint lens, wherein the lens parameters of the current viewpoint lens in the lens parameter combination are the current lens parameters, and the two-dimensional plane of the current viewpoint lens is the current two-dimensional plane: determining the projected area of ​​each Gaussian sphere on the current two-dimensional plane based on the current lens parameters, and hiding the Gaussian sphere set. A set of hidden Gaussian spheres is obtained by identifying Gaussian spheres whose projected area on the current two-dimensional plane is less than a specified area threshold. The projected area of ​​each Gaussian sphere is the area projected onto the specified plane. Hidden Gaussian spheres in the set are merged according to their neighborhood relationships. Gaussian spheres with projected areas greater than or equal to the specified area threshold are added to a set of Gaussian spheres to be rendered. This set also includes unhidden Gaussian spheres from the original set. Rendering is performed on each Gaussian sphere in the set, and the rendered object image corresponding to the current viewpoint is output. An output unit is used to output multi-view images, including object images corresponding to each viewpoint.

[0132] In one exemplary embodiment, the fifth execution unit includes: an acquisition module, configured to acquire the Gaussian sphere projection of each Gaussian sphere to be rendered, wherein the Gaussian sphere projection of each Gaussian sphere to be rendered is obtained by projecting each Gaussian sphere to be rendered from three-dimensional space onto a current two-dimensional plane according to the current lens parameters; a first determination module, configured to determine the Gaussian sphere projection covering each pixel on the current two-dimensional plane among the Gaussian sphere projections of the Gaussian spheres to be rendered in the set of Gaussian spheres to be rendered, thereby obtaining a set of Gaussian sphere projections corresponding to each pixel; a second determination module, configured to determine the probability density value of each Gaussian sphere projection corresponding to each pixel on each pixel as the weight of each Gaussian sphere projection corresponding to each pixel; a fifth execution module, configured to mix the colors of each Gaussian sphere projection corresponding to each pixel according to the weight of each Gaussian sphere projection corresponding to each pixel, and determine the mixed color as the color of each pixel, wherein the color of each Gaussian sphere projection corresponding to each pixel is the color of the Gaussian sphere to be rendered to which each Gaussian sphere projection corresponding to each pixel belongs; and a rendering module, configured to render the object image corresponding to the current viewpoint lens according to the color of each pixel.

[0133] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0134] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0135] S1. Based on the historical video of the object to be constructed, perform head reconstruction and body reconstruction, and align the reconstructed head mesh model and body mesh model to obtain the initial three-dimensional object model, where the initial three-dimensional object model is a mesh model;

[0136] S2, sample three-dimensional coordinate points of the initial three-dimensional object model, and construct a neighborhood for each three-dimensional sampling point in the obtained three-dimensional sampling point set to obtain the neighborhood construction result, wherein the neighborhood construction result is used to indicate the neighborhood relationship of each three-dimensional sampling point;

[0137] S3. Gaussian training is performed based on the set of 3D sampling points and the neighborhood construction results to obtain the target 3D object model corresponding to the object to be constructed. The target 3D object model is an object model represented by a set of Gaussian spheres. Each Gaussian sphere in the set of Gaussian spheres is generated based on a 3D sampling point in the set of 3D sampling points. The neighborhood relationship of each Gaussian sphere is matched with the neighborhood relationship of the 3D sampling point corresponding to each Gaussian sphere.

[0138] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0139] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0140] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0141] S1. Based on the historical video of the object to be constructed, perform head reconstruction and body reconstruction, and align the reconstructed head mesh model and body mesh model to obtain the initial three-dimensional object model, where the initial three-dimensional object model is a mesh model;

[0142] S2, sample three-dimensional coordinate points of the initial three-dimensional object model, and construct a neighborhood for each three-dimensional sampling point in the obtained three-dimensional sampling point set to obtain the neighborhood construction result, wherein the neighborhood construction result is used to indicate the neighborhood relationship of each three-dimensional sampling point;

[0143] S3. Gaussian training is performed based on the set of 3D sampling points and the neighborhood construction results to obtain the target 3D object model corresponding to the object to be constructed. The target 3D object model is an object model represented by a set of Gaussian spheres. Each Gaussian sphere in the set of Gaussian spheres is generated based on a 3D sampling point in the set of 3D sampling points. The neighborhood relationship of each Gaussian sphere is matched with the neighborhood relationship of the 3D sampling point corresponding to each Gaussian sphere.

[0144] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0145] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0146] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0147] Embodiments of this application also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0148] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0149] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0150] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for reconstructing a three-dimensional object, characterized in that, include: Based on the historical video of the object to be constructed, head and body reconstruction are performed, and the reconstructed head mesh model and body mesh model are aligned to obtain an initial three-dimensional object model, wherein the initial three-dimensional object model is a mesh model; The initial three-dimensional object model is sampled for three-dimensional coordinate points, and a neighborhood is constructed for each three-dimensional sample point in the obtained three-dimensional sample point set to obtain a neighborhood construction result, wherein the neighborhood construction result is used to indicate the neighborhood relationship of each three-dimensional sample point; Gaussian training is performed based on the set of three-dimensional sampling points and the neighborhood construction results to obtain the target three-dimensional object model corresponding to the object to be constructed. The target three-dimensional object model is an object model represented by a set of Gaussian spheres. Each Gaussian sphere in the set of Gaussian spheres is generated based on a three-dimensional sampling point in the set of three-dimensional sampling points. The neighborhood relationship of each Gaussian sphere is matched with the neighborhood relationship of the three-dimensional sampling point corresponding to each Gaussian sphere.

2. The method for reconstructing a three-dimensional object according to claim 1, characterized in that, The process of aligning the reconstructed head mesh model and body mesh model to obtain the initial 3D object model includes: Locate the bridging region between the head mesh model and the body mesh model; Based on the bridging region, taking one of the head mesh model and the body mesh model as a reference, an alignment operation is performed on the other mesh model of the head mesh model and the body mesh model to obtain the initial three-dimensional object model, wherein the alignment operation includes at least one of the following: translation operation and rotation operation.

3. The method for reconstructing a three-dimensional object according to claim 1, characterized in that, The step of sampling three-dimensional coordinate points of the initial three-dimensional object model includes: The mesh vertices of the initial 3D object model are mapped from 3D space to UV space to obtain a set of 2D vertices to be sampled; Uniform sampling is performed on the set of two-dimensional vertices to be sampled to obtain a set of two-dimensional orthogonal coordinate points; The two-dimensional orthogonal coordinate points in the set of two-dimensional orthogonal coordinate points are mapped from UV space back to three-dimensional space to obtain the set of three-dimensional sampling points.

4. The method for reconstructing a three-dimensional object according to claim 1, characterized in that, After sampling the three-dimensional coordinate points of the initial three-dimensional object model, the method further includes: A local density evaluation operation is performed on each of the three-dimensional sampling points to obtain a local density evaluation result for each three-dimensional sampling point. The local density evaluation operation is used to evaluate the density of three-dimensional sampling points within a specified range with each three-dimensional sampling point as a reference point. Based on the local density evaluation results of each three-dimensional sampling point and the density threshold corresponding to the object part to which each three-dimensional sampling point belongs, the area to be densified is marked; A densification operation is performed on the region to be densified, and the three-dimensional sampling points added by the densified region are added to the set of three-dimensional sampling points to obtain an updated set of three-dimensional sampling points, wherein the densification operation is used to increase the density of three-dimensional sampling points in the region to be densified.

5. The method for reconstructing a three-dimensional object according to claim 1, characterized in that, The process of constructing a neighborhood for each 3D sampling point in the obtained 3D sampling point set, to obtain the neighborhood construction result, includes: The following neighborhood construction operation is performed on each of the three-dimensional sampling points as the current three-dimensional sampling point to obtain the neighborhood construction result: Determine all mesh surfaces where the current 3D sampling point is located to obtain the current set of mesh surfaces; Iterate through each current mesh face in the current mesh face set to search for the neighboring faces of each current mesh face; The neighborhood relationship between the current 3D sampling point and the 3D sampling points contained in the adjacent faces of each current mesh face.

6. The method for reconstructing a three-dimensional object according to claim 1, characterized in that, The process of performing Gaussian training based on the set of three-dimensional sampling points and the neighborhood construction results to obtain the target three-dimensional object model includes: Gaussian spheres are generated by using each three-dimensional sampling point in the set of three-dimensional sampling points as the center point of the Gaussian sphere, and the neighborhood relationship of the Gaussian sphere corresponding to each three-dimensional sampling point is configured based on the neighborhood relationship of each three-dimensional sampling point to obtain the set of Gaussian spheres; Gaussian property training and Gaussian deformation model training are performed on the Gaussian sphere set respectively. The basic Gaussian properties of each Gaussian sphere obtained from the training are fused with the Gaussian property deformation of each Gaussian sphere to obtain the target 3D object model. The object to be constructed includes multiple object parts. The Gaussian deformation model training is performed on the Gaussian sphere set based on the object parts to which the Gaussian spheres belong. The set of control parameters used in the Gaussian deformation model training includes the part parameters of each of the multiple object parts. The set of control parameters is preset or predicted based on the historical video of the object.

7. The method for reconstructing a three-dimensional object according to claim 6, characterized in that, After performing Gaussian training based on the set of three-dimensional sampling points and the neighborhood construction results, the method further includes: In response to the acquired object control signal, the object control signal is converted into an initial parameter value sequence corresponding to the control parameter set, wherein the initial parameter value sequence is used to record the changes in the parameter values ​​of the control parameters in the control parameter set over time; The initial parameter values ​​in the initial parameter value sequence are adjusted according to specified constraints to obtain the target parameter value sequence, wherein the specified constraints include at least one of the following: constraints on the parameter values ​​of the control parameters in the control parameter set; and association conditions between the parameter values ​​of at least two control parameters that have an association relationship in the control parameter set. The target 3D object model is driven by the sequence of target parameter values ​​so that the target 3D object model performs actions that match the object control signals.

8. The method for reconstructing a three-dimensional object according to any one of claims 1 to 7, characterized in that, After performing Gaussian training based on the set of three-dimensional sampling points and the neighborhood construction results, the method further includes: When an object image corresponding to the target 3D object model is to be output, a combination of lens parameters for multiple perspective lenses is generated by a lens parameter generator, wherein the combination of lens parameters includes the lens parameters of each of the multiple perspective lenses. Each of the aforementioned viewpoints is treated as the current viewpoint, and the following image generation operations are performed to obtain the object image corresponding to each viewpoint. In the lens parameter combination, the lens parameters of the current viewpoint are defined as the current lens parameters, and the two-dimensional plane of the current viewpoint is defined as the current two-dimensional plane. Based on the current lens parameters, the projected area of ​​each Gaussian sphere on the current two-dimensional plane is determined, and Gaussian spheres in the Gaussian sphere set whose projected area on the current two-dimensional plane is less than a specified area threshold are hidden, resulting in a hidden Gaussian sphere set. Each Gaussian sphere corresponds to... The projected area is the projected area of ​​each Gaussian sphere projected onto the specified plane; according to the neighborhood relationship of the hidden Gaussian spheres in the set of hidden Gaussian spheres, the hidden Gaussian spheres in the set of hidden Gaussian spheres are merged, and the Gaussian spheres in the merged Gaussian spheres whose corresponding projected area is greater than or equal to the specified area threshold are added to the set of Gaussian spheres to be rendered, wherein the set of Gaussian spheres to be rendered also includes the unhidden Gaussian spheres in the set of Gaussian spheres; a rendering operation is performed on each Gaussian sphere to be rendered in the set of Gaussian spheres to be rendered, and the rendered object image corresponding to the current viewpoint is output; Output multi-view images, wherein the multi-view images include object images corresponding to each viewpoint lens.

9. The method for reconstructing a three-dimensional object according to claim 8, characterized in that, The step of performing a rendering operation on each Gaussian sphere in the set of Gaussian spheres to be rendered includes: Obtain the Gaussian sphere projection of each Gaussian sphere to be rendered, wherein the Gaussian sphere projection of each Gaussian sphere to be rendered is obtained by projecting each Gaussian sphere to be rendered from three-dimensional space onto the current two-dimensional plane according to the current camera parameters; Determine the Gaussian sphere projections of the Gaussian spheres to be rendered in the set of Gaussian spheres to be rendered, and cover the Gaussian sphere projections of each pixel on the current two-dimensional plane to obtain the set of Gaussian sphere projections corresponding to each pixel. The probability density value of each Gaussian sphere corresponding to each pixel is projected onto each pixel, and is determined as the weight of each Gaussian sphere projection corresponding to each pixel. According to the weight of each Gaussian sphere projection corresponding to each pixel, the colors of each Gaussian sphere projection corresponding to each pixel are mixed, and the mixed color is determined as the color of each pixel. The color of each Gaussian sphere projection corresponding to each pixel is the color of the Gaussian sphere to be rendered to which each Gaussian sphere projection corresponding to each pixel belongs. The image of the object corresponding to the current viewpoint is rendered according to the color of each pixel.

10. A device for reconstructing a three-dimensional object, characterized in that, include: The first execution unit is used to perform head reconstruction and body reconstruction based on the object history video of the object to be constructed, and to align the reconstructed head mesh model and body mesh model to obtain an initial three-dimensional object model, wherein the initial three-dimensional object model is a mesh model; The second execution unit is used to sample three-dimensional coordinate points of the initial three-dimensional object model, and to construct a neighborhood for each three-dimensional sampling point in the obtained three-dimensional sampling point set, thereby obtaining a neighborhood construction result, wherein the neighborhood construction result is used to indicate the neighborhood relationship of each three-dimensional sampling point; A Gaussian training unit is used to perform Gaussian training based on the set of three-dimensional sampling points and the neighborhood construction results to obtain a target three-dimensional object model corresponding to the object to be constructed. The target three-dimensional object model is an object model represented by a set of Gaussian spheres. Each Gaussian sphere in the set of Gaussian spheres is generated based on a three-dimensional sampling point in the set of three-dimensional sampling points. The neighborhood relationship of each Gaussian sphere is matched with the neighborhood relationship of the three-dimensional sampling point corresponding to each Gaussian sphere.

Images