Three-dimensional object reconstruction and rendering method, electronic equipment and storage medium

Abstract

The invention provides a three-dimensional object reconstruction and rendering method, electronic equipment and a storage medium, which can be applied to the technical field of computer vision. The method comprises the following steps: generating a rough tetrahedral mesh of a target three-dimensional object based on a multi-view image set of the target three-dimensional object; hash mapping, implicit recursion subdivision, density prediction and color prediction are carried out on the rough tetrahedral mesh in a multi-stage training mode so as to complete reconstruction of a target three-dimensional object, and a trained Hash feature vector, a density prediction neural network and a color prediction neural network are obtained; performing physical simulation on the rough tetrahedral mesh of the target three-dimensional object to obtain a position sequence of vertexes in the rough tetrahedral mesh within a preset time; and based on the position sequence of the vertexes of each frame in a preset time, performing animation rendering through spatial mapping operation and by using the trained Hash feature vector, the density prediction neural network and the color prediction neural network, and obtaining an animation sequence of the target three-dimensional object.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a reconstruction and rendering method of a three-dimensional object, an electronic device and a storage medium. Background Art

[0002] Synthesis of new perspectives of 3D objects is an important research direction in the field of computer vision technology. This technology is used to generate new perspectives from existing perspectives to achieve a comprehensive understanding and accurate modeling of 3D objects. The application of new perspective synthesis is of great significance in fields such as autonomous driving, virtual reality, and augmented reality. Since the advent of Neural Radiance Fields (NeRF), research on new perspective synthesis has made significant progress in the fields of computer vision and graphics. In a large number of subsequent studies, feature grid-based methods have performed well in rendering quality and speed. Despite the impressive results, NeRF, as an implicit continuous representation, does not directly provide an explicit control mechanism. Due to the above limitations, NeRF faces major challenges in object deformation and manipulation, especially in scenarios that require users to make precise adjustments and manipulations. Although a variety of technical solutions have been provided in the prior art to address the limitations of NeRF, the prior art solutions still have technical problems such as time-consuming calculations, high storage costs, and poor reconstruction and rendering quality. Summary of the invention

[0003] In view of the above problems, the present invention provides a method for reconstructing and rendering a three-dimensional object, an electronic device and a storage medium.

[0004] According to a first aspect of the present invention, a method for reconstructing and rendering a three-dimensional object is provided, the method comprising:

[0005] Generate a tetrahedral mesh filling a unit cube space, and iteratively perform hash mapping, implicit recursive subdivision, density prediction, and color prediction on the tetrahedral mesh filling the unit cube space in a first training phase using a multi-view image set of a target three-dimensional object;

[0006] Using the training results of the first training phase to update the occupancy grid, and using the updated occupancy grid to prune the tetrahedral grid filling the unit cube space to obtain a rough tetrahedral grid of the target three-dimensional object;

[0007] The second training phase is to iterate the coarse tetrahedral mesh using multi-view image sets for hash mapping, implicit recursive subdivision, density prediction, and color prediction.

[0008] Performing physical simulation on the rough tetrahedral mesh of the target three-dimensional object to obtain a position sequence of vertices in the rough tetrahedral mesh within a preset time;

[0009] Based on the position sequence of vertices in each frame of the rough tetrahedral mesh within a preset time, the animation sequence of the target three-dimensional object is obtained by performing a spatial mapping operation and animating the multi-level tetrahedral mesh obtained in the second training stage.

[0010] According to an embodiment of the present invention, the above-mentioned generating a tetrahedral mesh filling a unit cube space, and iteratively performing hash mapping, implicit recursive subdivision, density prediction, and color prediction of the tetrahedral mesh filling the unit cube space in a first training phase using a multi-view image set of a target three-dimensional object includes:

[0011] A tetrahedral mesh completely filling the unit cube space is generated by using a predefined tetrahedral mesh generation algorithm, and images in the multi-view image set of the target three-dimensional object are preprocessed to obtain a preprocessed multi-view image set;

[0012] Using the intersection calculation result of the camera ray and the tetrahedral mesh filling the unit cube space and the position information of the generated sampling point, the centroid coordinates of the sampling point in the tetrahedral mesh filling the unit cube space are calculated;

[0013] Mapping positions of all vertices in a tetrahedral mesh filling a unit cube space to a hash feature vector using a hash function, wherein the hash feature vector is stored in a hash table corresponding to the hash function;

[0014] Based on the initial subdivision threshold, an initial implicit recursive subdivision operation is performed on the tetrahedral mesh filling the unit cube space to obtain the initial subdivided tetrahedral mesh, and the positions of the vertices of all levels in the initial subdivided tetrahedral mesh are mapped to the hash feature vector;

[0015] Based on the barycentric coordinates of the sampling points in the tetrahedral mesh after initial subdivision, the feature vectors of different levels of the sampling points in the tetrahedral mesh after initial subdivision are obtained by barycentric interpolation, and the feature vectors of different levels of the sampling points in the tetrahedral mesh after initial subdivision are spliced ​​to obtain multi-level initial feature vectors;

[0016] The density prediction neural network and the color prediction neural network are used to perform density prediction and color prediction on the multi-level initial feature vectors respectively, to obtain the initial density value and initial color value of the sampling point, and based on the initial density value and initial color value, the initial color value of the light at the sampling point is obtained using a predefined volume rendering formula;

[0017] Input the pixel color values ​​of the preprocessed multi-view image set and the initial color values ​​of the light at the sampling point into a predefined loss function for processing to obtain an initial loss value;

[0018] Based on the initial loss value, the hash feature vector, the density prediction neural network and the color prediction neural network are initially iteratively updated to obtain a preliminarily trained hash feature vector, a preliminarily trained density prediction neural network and a preliminarily trained color prediction neural network.

[0019] According to an embodiment of the present invention, the calculation of the centroid coordinates of the sampling point in the tetrahedral mesh filling the unit cube space using the intersection calculation result of the camera ray and the tetrahedral mesh filling the unit cube space and the position information of the generated sampling point includes:

[0020] Execute an intersection calculation operation between the camera ray and the tetrahedral mesh filling the unit cube space to obtain an intersection calculation result;

[0021] Starting from the camera origin, move along the direction of the camera ray to generate sampling points;

[0022] Based on the intersection calculation results, determine in which tetrahedron of the tetrahedron grid filling the unit cube space the sampling point is located;

[0023] The centroid coordinates of the sampling point are calculated according to the vertex position of the tetrahedron where the sampling point is located and the spatial position of the sampling point.

[0024] According to an embodiment of the present invention, based on the barycentric coordinates of the sampling points in the tetrahedral mesh after the initial subdivision, the feature vectors of different levels of the sampling points in the tetrahedral mesh after the initial subdivision are obtained by barycentric interpolation, and the feature vectors of different levels of the sampling points in the tetrahedral mesh after the initial subdivision are spliced ​​to obtain the multi-level initial feature vectors, including:

[0025] By identifying the coordinates of four vertices of the sampling point in the tetrahedron mesh filling the unit cubic space, a hash feature vector corresponding to the four vertices in the tetrahedron mesh filling the unit cubic space is obtained;

[0026] The centroid coordinates of the sampling point in the tetrahedron mesh filling the unit cube space are calculated based on the hash feature vectors corresponding to the four vertices in the tetrahedron mesh filling the unit cube space, and the feature vector of the sampling point on the rough tetrahedron mesh is calculated by the feature vector weighting method;

[0027] Determine a target sub-tetrahedral mesh where the sampling point is located based on the barycentric coordinates of the sampling point in the tetrahedral mesh that fills the unit cube space, wherein the target sub-tetrahedral mesh is located in the tetrahedral mesh after the initial subdivision;

[0028] Based on the coordinates of the four vertices in the target sub-tetrahedral mesh and the corresponding hash eigenvectors, the centroid coordinates of the sampling point in the target sub-tetrahedral mesh are obtained, and the eigenvector of the sampling point on the target sub-tetrahedral mesh is calculated by eigenvector weighting;

[0029] The number of iterations is determined based on the initial subdivision threshold, and the barycentric coordinate calculation operation and the eigenvector calculation operation are iteratively performed in a barycentric interpolation manner to obtain the eigenvectors of different levels of the sampling points in the tetrahedral mesh after the initial subdivision;

[0030] The feature vectors of different levels in the tetrahedral mesh of the sampling point after initial subdivision are concatenated to obtain the multi-level initial feature vectors of the sampling point.

[0031] According to an embodiment of the present invention, the above-mentioned initial implicit recursive subdivision operation is performed on the tetrahedral mesh filling the unit cube space based on the initial subdivision threshold to obtain the tetrahedral mesh after initial subdivision, which includes:

[0032] A subdivision process is completed by connecting the midpoints of all edges in the tetrahedral mesh that fills the unit cube space, and the same subdivision process is performed on the obtained sub-tetrahedrons until the number of subdivision processes meets the requirements of the initial subdivision threshold, thereby obtaining the tetrahedral mesh after initial subdivision;

[0033] The coordinates of the sampling point in the tetrahedron mesh filling the unit cube space are weightedly summed by using the coordinates of the four vertices in the tetrahedron mesh filling the unit cube space to obtain the coordinates of the sampling point in the tetrahedron mesh filling the unit cube space;

[0034] Based on the coordinates of the sampling point in the tetrahedral mesh filling the unit cube space, the centroid coordinates of the sampling point in the tetrahedral mesh after initial subdivision are judged to obtain the target sub-tetrahedral mesh where the sampling point is located.

[0035] According to an embodiment of the present invention, the above-mentioned updating the occupied grid using the training results of the first training stage, and using the updated occupied grid to delete the tetrahedral grid filling the unit cube space to obtain the rough tetrahedral grid of the target three-dimensional object includes:

[0036] The density of the occupied grid is updated using the initially trained hash feature vector and the initially trained density prediction neural network obtained in the first training phase;

[0037] Based on the preset density threshold, the tetrahedral mesh filling the unit cube space is deleted using the density-updated occupied mesh to obtain a coarse tetrahedral mesh of the target three-dimensional object.

[0038] According to an embodiment of the present invention, the above-mentioned iterative hash mapping, implicit recursive subdivision, density prediction and color prediction of the second training stage on the coarse tetrahedral mesh using the multi-view image set includes:

[0039] Preprocessing the images in the multi-view image set to obtain a preprocessed multi-view image set, and using a hash function to map the positions of vertices of all levels in the rough tetrahedral mesh to the initially trained hash feature vectors;

[0040] Based on the intermediate subdivision threshold, the rough tetrahedral mesh is implicitly recursively subdivided to obtain an intermediate multi-level tetrahedral mesh, and the positions of the vertices of all levels in the intermediate multi-level tetrahedral mesh are mapped to the initially trained hash feature vector;

[0041] The feature vectors of different levels of the sampling points in the middle multi-level tetrahedral grid are obtained by means of barycentric interpolation, and the feature vectors of different levels of the sampling points in the middle multi-level tetrahedral grid are spliced ​​to obtain the multi-level intermediate feature vectors;

[0042] Using the preliminarily trained density prediction neural network and the preliminarily trained color prediction neural network, density prediction and color prediction are performed on the multi-level intermediate feature vectors respectively to obtain the intermediate density value and the intermediate color value of the sampling point;

[0043] Based on the intermediate density value and the intermediate color value, the intermediate color value of the light at the sampling point is obtained using a predefined volume rendering formula, and the pixel color values ​​of the images in the multi-view image set and the intermediate color values ​​of the light at the sampling point are processed using a predefined loss function to obtain an intermediate loss value;

[0044] Based on the intermediate loss value, the initially trained hash feature vector, the initially trained density prediction neural network and the initially trained color prediction neural network are updated iteratively to obtain the trained hash feature vector, the trained density prediction neural network and the trained color prediction neural network.

[0045] According to an embodiment of the present invention, the above-mentioned animation sequence of the target three-dimensional object is obtained by performing a spatial mapping operation and performing animation rendering on the multi-level tetrahedral mesh obtained in the second training stage based on the position sequence of vertices in each frame in the rough tetrahedral mesh within a preset time, including:

[0046] Based on the position sequence of vertices in each frame within a preset time, the direction of the camera ray and the position of the sampling point are mapped from the deformation space to the standard space;

[0047] Based on the position sequence of vertices in each frame within a preset time, hash mapping is performed on each frame image of the target three-dimensional object using the trained hash feature vector, wherein the trained hash feature vector is obtained through the second training stage;

[0048] Based on the position sequence of vertices in each frame within a preset time, density prediction is performed on each frame image of the target three-dimensional object using a trained density prediction neural network, wherein the trained density prediction neural network is obtained through a second training stage;

[0049] Based on the position sequence of vertices in each frame within a preset time, the trained color prediction neural network is used to predict the color of each frame of the target three-dimensional object, wherein the trained color prediction neural network is obtained through a second training stage;

[0050] Through hash mapping operations, density prediction operations and color prediction operations, animation rendering of multi-level tetrahedral meshes is completed within a preset time to obtain an animation sequence of the target three-dimensional object.

[0051] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0052] The fourth aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0053] The above-mentioned three-dimensional object reconstruction and rendering method provided by the present invention can not only be used for static objects, but also can effectively process the rendering of deformed objects, ensure the accuracy of color and density, and break the limitations of the prior art in processing deformed objects; at the same time, the present invention significantly reduces the use of GPU memory by using occupied grids and dynamic subdivision methods, improves the training and reasoning efficiency of the model, and is suitable for processing practical application scenarios of complex geometric bodies; in addition, the two-stage training process proposed by the present invention first quickly generates a coarse tetrahedral grid, and then performs refined training, which can improve the accuracy and performance of the model while ensuring the training speed; finally, the present invention dynamically generates tetrahedral grids of different levels through an implicit recursive subdivision process, without the need to store all high-resolution grids, thereby saving a lot of storage space and computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0055] Figure 1is a flow chart of a method for reconstructing and rendering a three-dimensional object according to an embodiment of the present invention;

[0056] Figure 2 is a schematic diagram of a multi-resolution representation method based on an implicit recursive subdivision grid;

[0057] Figure 3 is a schematic diagram of a subdivision mode of a tetrahedral mesh according to an embodiment of the present invention;

[0058] Figure 4 is a flow chart of an algorithm for calculating the sampling points falling into a sub-tetrahedral grid according to the present invention;

[0059] Figure 5 is a schematic diagram of sampling point mapping when a target three-dimensional object is deformed according to an embodiment of the present invention;

[0060] Figure 6 is a block diagram of an electronic device suitable for implementing a method for reconstructing and rendering a three-dimensional object according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0062] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0063] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0064] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0065] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0066] In order to solve the technical problems of Neural Radiance Fields (NeRF) in the field of new perspective synthesis technology, relevant researchers use explicit geometric agents to achieve deformation effects. For example, some research works generate a closed triangular mesh surrounding the target object, and then use the triangular mesh to deform the object; other research works use tetrahedral meshes as geometric agents, and control the deformation of objects by manipulating the shape of these meshes based on the characteristic mesh method. However, these methods face two major challenges: first, they need to apply the Marching Cube algorithm and tetrahedralization process after training the implicit representation, which is time-consuming and computationally demanding. Second, for objects with complex topological structures or thin structures, this process is particularly challenging. Usually, there are two situations: one is to generate too many small tetrahedrons in order to accurately fit the complex details of the object, which significantly increases the storage overhead; the other is to limit the number of tetrahedrons, resulting in poor quality of the generated tetrahedrons, affecting the effect of the deformation task.

[0067] In order to solve the technical problems faced by the prior art, the present invention provides a neural radiation field representation method, which is based on tetrahedral mesh representation, combined with multi-resolution feature encoding technology, and utilizes multi-perspective picture sets and corresponding camera pose information. It can generate high-quality rendering results from the input data set and can achieve efficient deformation of complex objects, thereby applying this method to large-scale three-dimensional modeling and animation production needs in the fields of movie special effects, virtual reality, etc.

[0068] Figure 1 is a flowchart of a method for reconstructing and rendering a three-dimensional object according to an embodiment of the present invention.

[0069] like Figure 1 As shown, the above-mentioned three-dimensional object reconstruction and rendering method includes operations S110 to S150.

[0070] In operation S110, a tetrahedral mesh filling a unit cube space is generated, and a multi-view image set of a target three-dimensional object is used to iteratively perform hash mapping, implicit recursive subdivision, density prediction, and color prediction of a first training phase on the tetrahedral mesh filling the unit cube space.

[0071] In the process of generating a tetrahedral mesh filling a unit cube space of a target three-dimensional object, optionally, a QuarTet algorithm is used to generate a tetrahedral mesh filling a unit cube space, and the unit cube is completely filled with the tetrahedral mesh.

[0072] In operation S120, the occupancy grid is updated using the training result of the first training stage, and the tetrahedral grid filling the unit cube space is deleted using the updated occupancy grid to obtain a coarse tetrahedral grid of the target three-dimensional object.

[0073] In operation S130 , the coarse tetrahedral mesh is iteratively subjected to a second training phase of hash mapping, implicit recursive subdivision, density prediction, and color prediction using a multi-view image set.

[0074] The present invention adopts a two-stage training mode to reconstruct the target three-dimensional object. Those skilled in the art can set other multi-stage training modes according to actual needs.

[0075] The present invention provides a new implicit recursive subdivision method to subdivide tetrahedral meshes, and realizes multi-resolution feature encoding by refining the initial coarse tetrahedral mesh, while avoiding the large amount of memory required for explicit storage of high-resolution meshes. And by introducing an iterative algorithm to calculate the barycentric coordinates, the algorithm uses the known barycentric coordinates to recursively determine the coordinates at a finer level, so as to efficiently perform feature encoding in the recursively subdivided tetrahedral hierarchy.

[0076] The present invention adopts a two-stage training strategy. The first stage generates a regular tetrahedral mesh (i.e., a coarse tetrahedral mesh). The second stage improves the visual fidelity of the model by increasing the subdivision level. The present invention also expands the application scope of NeRF, supports physically based simulation and binding animation, and allows users to control the deformation of objects by manipulating the bone structure while maintaining the realism of the rendering results.

[0077] In operation S140, physical simulation is performed on the rough tetrahedral mesh of the target three-dimensional object to obtain a position sequence of vertices in the rough tetrahedral mesh within a preset time.

[0078] During the physical simulation, optionally, an XPBD (Extended Position-Based Dynamics) algorithm is used to perform the physical simulation.

[0079] In operation S150, based on the position sequence of vertices in each frame in the rough tetrahedral mesh within a preset time, an animation sequence of the target three-dimensional object is obtained by performing a spatial mapping operation and performing animation rendering on the multi-level tetrahedral mesh obtained in the second training stage.

[0080] The above-mentioned three-dimensional object reconstruction and rendering method provided by the present invention can not only be used for static objects, but also can effectively process the rendering of deformed objects, ensure the accuracy of color and density, and break the limitations of the prior art in processing deformed objects; at the same time, the present invention significantly reduces the use of GPU memory by using occupied grids and dynamic subdivision methods, improves the training and reasoning efficiency of the model, and is suitable for processing practical application scenarios of complex geometric bodies; in addition, the two-stage training process proposed by the present invention first quickly generates a coarse tetrahedral grid, and then performs refined training, which can improve the accuracy and performance of the model while ensuring the training speed; finally, the present invention dynamically generates tetrahedral grids of different levels through an implicit recursive subdivision process, without the need to store all high-resolution grids, thereby saving a lot of storage space and computing resources.

[0081] The following is combined with specific embodiments and the attached Figures 2~5 , the multi-stage training process (ie, operation S110 and operation S130) involved in the present invention is further described in detail.

[0082] Figure 2 Schematic diagram of a multi-resolution representation method based on implicit recursive subdivision grid.

[0083] Figure 3 is a schematic diagram of a subdivision mode of a tetrahedral mesh according to an embodiment of the present invention.

[0084] In the first training phase, the QuarTet algorithm is used to generate an initial tetrahedralization of the space, resulting in a tetrahedral mesh that fills the unit cube space. The occupancy grid is used to determine whether the tetrahedrons within the grid are retained. At the end of this phase, a rough tetrahedral mesh is obtained that roughly encloses the object. This has the advantage of being able to quickly build an initial structure that encloses complex geometry while minimizing memory consumption by retaining only the tetrahedrons that are actually occupied.

[0085] According to an embodiment of the present invention, the above-mentioned generation of a tetrahedral mesh filling a unit cube space, and using a multi-view image set of a target three-dimensional object to iteratively perform hash mapping, implicit recursive subdivision, density prediction, and color prediction on the tetrahedral mesh filling the unit cube space in a first training phase include: using a predefined tetrahedral mesh generation algorithm to generate a tetrahedral mesh that completely fills the unit cube space, and preprocessing the images in the multi-view image set of the target three-dimensional object to obtain a preprocessed multi-view image set; using the intersection calculation results of the camera ray and the tetrahedral mesh filling the unit cube space and the position information of the generated sampling points, calculating the centroid coordinates of the sampling points in the tetrahedral mesh filling the unit cube space; using a hash function to map the positions of all vertices in the tetrahedral mesh filling the unit cube space to a hash feature vector, wherein the hash feature vector is stored in a hash table corresponding to the hash function; based on an initial subdivision threshold, performing an initial implicit recursive subdivision operation on the tetrahedral mesh filling the unit cube space to obtain an initially subdivided tetrahedral mesh, and dividing the initial subdivided tetrahedral mesh into the initial implicit recursive subdivision operation. The positions of vertices at all levels are mapped to hash feature vectors; based on the centroid coordinates of the sampling points in the tetrahedral mesh after initial subdivision, the feature vectors of different levels of the sampling points in the tetrahedral mesh after initial subdivision are obtained by centroid interpolation, and the feature vectors of different levels of the sampling points in the tetrahedral mesh after initial subdivision are spliced ​​to obtain multi-level initial feature vectors; the density prediction neural network and the color prediction neural network are used to perform density prediction and color prediction on the multi-level initial feature vectors respectively to obtain the initial density value and initial color value of the sampling point, and based on the initial density value and initial color value, the initial color value of the light where the sampling point is located is obtained by using a predefined volume rendering formula; the pixel color values ​​of the images in the preprocessed multi-view image set and the initial color values ​​of the light where the sampling point is located are input into a predefined loss function for processing to obtain an initial loss value; based on the initial loss value, the hash feature vector, the density prediction neural network and the color prediction neural network are initially iteratively updated to obtain a preliminarily trained hash feature vector, a preliminarily trained density prediction neural network and a preliminarily trained color prediction neural network.

[0086] In the first training stage described above, the three-dimensional object corresponding to the preprocessed image is completely wrapped by a tetrahedral mesh that fills the unit cube space.

[0087] In the hash mapping process of the first training stage, the method provided by the present invention innovatively applies multi-resolution hash coding to multi-level tetrahedral meshes. For each level of the tetrahedral mesh, its vertices correspond to entries in its respective hash tables. The present invention uses a spatial hash function as shown in formula (1) to convert the vertex positions Mapped to feature vectors stored in a hash table:

[0088] (1),

[0089] Among them, the symbol Represents a bitwise exclusive OR operation, represents a unique large prime number associated with each dimension, is the size of the hash table.

[0090] In the process of obtaining the complete multi-level initial feature vector of the sampling point in the first training stage, unlike the previous feature grid method using regular grids, the method of the present invention uses tetrahedral grids, so it is necessary to implement centroid interpolation in these grids to perform feature interpolation. The present invention uses the centroid coordinates of each sampling point to calculate the weighted sum of the features of the tetrahedron vertices containing the point. This process effectively derives the features of the sampling point at each level. Subsequently, the features of these tetrahedral grids from different levels are spliced ​​together to form a complete feature vector of the sampling point.

[0091] like Figure 2 As shown, in the method of the present invention, an intuitive method is adopted to generate tetrahedral grids of different levels from coarse grids, and dynamically infer these high-resolution grids without storing them, wherein Sampling Point Feature represents sampling point feature, Density MLP represents density multi-level perceptron, Color MLP represents color multi-level perceptron, Hierarchical Barycentric Inference represents multi-level barycentric coordinate inference, and SH Encoding represents SH encoding.

[0092] Given a coarse tetrahedral mesh, a recursive subdivision process is performed to generate tetrahedral meshes of increasing resolution at multiple levels.

[0093] like Figure 3 As shown, starting from a coarse tetrahedral mesh, each tetrahedron is subdivided into eight smaller tetrahedrons by connecting the midpoints of its six edges. It should be noted that there are multiple ways to connect the midpoints of the edges. Here, the present invention selects a specific subdivision mode and uniformly applies it to all tetrahedrons in the mesh. Those skilled in the art may also select other subdivision modes according to actual needs. This subdivision process is applied recursively, and the hierarchical Each tetrahedron is converted into a layer The eight smaller tetrahedra of the , thereby generating a progressively finer tetrahedral mesh at each level.

[0094] For multi-resolution tetrahedral representation, storing a high-resolution mesh is not necessary, as the essential information for feature encoding can be obtained by focusing on two key aspects. First, identifying the coordinates of the four vertices of the tetrahedron at the sampling point enables access to the features of these vertices. Second, computing the barycentric coordinates of the sampling point inside the tetrahedron is essential for implementing barycentric interpolation. Once these two key aspects are addressed, the feature encoding process can be successfully completed.

[0095] Based on the need for feature encoding through vertex identification and barycentric calculation, the present invention proposes an iterative method that simplifies the process of recursively subdividing the tetrahedron hierarchy. This method allows the present invention to use the barycentric coordinates of the midpoints of the parent tetrahedron to determine the Which of the eight subtetrahedrons in the layer contains a particular sampling point. In addition, the sampling point is in The barycentric coordinates of a layer can be obtained by The coordinates of the layer are computed explicitly using analytical expressions.

[0096] According to an embodiment of the present invention, the above-mentioned hash mapping, implicit recursive subdivision, density prediction and color prediction of the second training stage of the iterative rough tetrahedral mesh using a multi-view image set include: preprocessing the images in the multi-view image set to obtain a preprocessed multi-view image set, and using a hash function to map the positions of vertices of all levels in the rough tetrahedral mesh to the initially trained hash feature vector; based on an intermediate subdivision threshold, implicitly recursively subdividing the rough tetrahedral mesh to obtain an intermediate multi-level tetrahedral mesh, and mapping the positions of vertices of all levels in the intermediate multi-level tetrahedral mesh to the initially trained hash feature vector; obtaining feature vectors of sampling points at different levels in the intermediate multi-level tetrahedral mesh by means of barycentric interpolation, and splicing the feature vectors of sampling points at different levels in the intermediate multi-level tetrahedral mesh Then, a multi-level intermediate feature vector is obtained; a preliminarily trained density prediction neural network and a preliminarily trained color prediction neural network are used to perform density prediction and color prediction on the multi-level intermediate feature vectors respectively, and an intermediate density value and an intermediate color value of the sampling point are obtained; based on the intermediate density value and the intermediate color value, a predefined volume rendering formula is used to obtain the intermediate color value of the light where the sampling point is located, and a predefined loss function is used to process the pixel color values ​​of the images in the multi-view image set and the intermediate color value of the light where the sampling point is located, and an intermediate loss value is obtained; based on the intermediate loss value, the preliminarily trained hash feature vector, the preliminarily trained density prediction neural network and the preliminarily trained color prediction neural network are intermediately iteratively updated to obtain a trained hash feature vector, a trained density prediction neural network and a trained color prediction neural network.

[0097] The technical solution of the second training stage involved in the present invention is further described in detail below through specific embodiments.

[0098] In the second stage of training, by increasing the number of subdivision levels to improve the training of tetrahedral representation. This stage focuses on refining the tetrahedral representation obtained from the first stage, aiming to achieve a more accurate and visually appealing neural representation. Although this refinement process requires more computational resources, since it is based on an already roughly formed structure, it can focus on optimizing areas that require detailed representation.

[0099] In view of the fact that the method provided by the present invention adopts a tetrahedral mesh as a geometric proxy, it naturally inherits the application scenarios of the tetrahedron, the most important of which is the ability to deform objects. In order to render the deformed object, the method provided by the present invention first intersects the camera ray with the coarse deformed tetrahedral mesh to obtain sampling points in the deformed space. In order to ensure the accuracy of color and density, the sampling points and ray directions need to be mapped to the canonical space, that is, the space where the training model is located. Since the geometric proxy used for deformation and the tetrahedral mesh used for encoding are the same, there is no need to explicitly calculate the position of the sampling points in the canonical space. Instead, the centroid coordinates of the sampling points in the deformed space are calculated. This process is equivalent to transforming a series of sampling points along the ray from the deformed space to the canonical space.

[0100] According to an embodiment of the present invention, the above-mentioned calculation of the centroid coordinates of the sampling point in the tetrahedron mesh filling the unit cube space using the intersection calculation results of the camera ray and the tetrahedron mesh filling the unit cube space and the position information of the generated sampling point includes: performing an intersection calculation operation between the camera ray and the tetrahedron mesh filling the unit cube space to obtain an intersection calculation result; starting from the camera origin, moving along the direction of the camera ray to generate the sampling point; based on the intersection calculation result, determining in which tetrahedron of the tetrahedron mesh filling the unit cube space the sampling point is located; and calculating the centroid coordinates of the sampling point according to the vertex position of the tetrahedron where the sampling point is located and the spatial position of the sampling point.

[0101] The following is a brief description of the calculation process of the centroid coordinates of the above sampling points through a specific embodiment: perform the intersection calculation of the ray-tetrahedron mesh. Then, starting from the camera origin, proceed along the direction of the camera ray to generate a series of sampling points. Using the results of the ray-mesh intersection calculation and the positions of the sampling points, it is possible to determine in which tetrahedron of the coarse mesh each point is located, and calculate the corresponding centroid coordinates.

[0102] According to an embodiment of the present invention, the above-mentioned method of obtaining feature vectors of different levels of the sampling point in the tetrahedral mesh after initial subdivision by means of barycentric interpolation based on the centroid coordinates of the sampling point in the tetrahedral mesh after initial subdivision, and splicing the feature vectors of different levels of the sampling point in the tetrahedral mesh after initial subdivision to obtain the multi-level initial feature vector includes: obtaining the hash feature vectors corresponding to the four vertices in the tetrahedral mesh filling the unit cube space by identifying the coordinates of the four vertices of the sampling point in the tetrahedral mesh filling the unit cube space; calculating the centroid coordinates of the sampling point in the tetrahedral mesh filling the unit cube space based on the hash feature vectors corresponding to the four vertices in the tetrahedral mesh filling the unit cube space, and calculating the feature vector of the sampling point on the rough tetrahedral mesh by means of feature vector weighting; based on The barycentric coordinates of the sampling point in the tetrahedral mesh filling the unit cube space determine the target sub-tetrahedral mesh where the sampling point is located, wherein the target sub-tetrahedral mesh is located in the tetrahedral mesh after initial subdivision; based on the coordinates of the four vertices in the target sub-tetrahedral mesh and the corresponding hash feature vectors, the barycentric coordinates of the sampling point in the target sub-tetrahedral mesh are obtained, and the feature vector of the sampling point on the target sub-tetrahedral mesh is calculated by feature vector weighting; the number of iterations is determined based on the initial subdivision threshold, and the barycentric coordinate calculation operation and the feature vector calculation operation are iteratively performed in a barycentric interpolation manner to obtain feature vectors of different levels of the sampling point in the tetrahedral mesh after initial subdivision; the feature vectors of different levels of the sampling point in the tetrahedral mesh after initial subdivision are vector spliced ​​to obtain multi-level initial feature vectors of the sampling point.

[0103] According to an embodiment of the present invention, the above-mentioned initial implicit recursive subdivision operation is performed on the tetrahedral mesh filling the unit cube space based on the initial subdivision threshold to obtain the tetrahedral mesh after initial subdivision, including: completing a subdivision process by connecting the midpoints of all edges in the tetrahedral mesh filling the unit cube space, and performing the same subdivision process on the obtained sub-tetrahedron until the number of subdivision processes meets the requirement of the initial subdivision threshold to obtain the tetrahedral mesh after initial subdivision; using the coordinates of the four vertices in the tetrahedral mesh filling the unit cube space to perform weighted summation of the centroid coordinates of the sampling point in the tetrahedral mesh filling the unit cube space to obtain the coordinates of the sampling point in the tetrahedral mesh filling the unit cube space; judging the centroid coordinates of the sampling point in the tetrahedral mesh after initial subdivision based on the coordinates of the sampling point in the tetrahedral mesh filling the unit cube space to obtain the target sub-tetrahedral mesh where the sampling point is located.

[0104] The following is a specific embodiment and combined with the attached Figure 4 The process of obtaining the initial multi-level feature vector of the sampling point and the process of implicit recursive subdivision of the present invention are further described in detail.

[0105] Figure 4 is a flow chart of an algorithm for calculating the sampling points falling into the sub-tetrahedral grid according to the present invention.

[0106] After obtaining the barycentric coordinates of the sampling points at the coarsest level, the focus of the present invention shifts to identifying which eight sub-tetrahedrons each sampling point falls into. Without loss of generality, consider An arbitrary tetrahedron in the layer, consisting of four vertices For a given sampling point located within the tetrahedron , assuming that The barycentric coordinates of the layer are known and are expressed as ,but The coordinates can be expressed as shown in formula (2):

[0107] (2).

[0108] By connecting these subdivided original tetrahedra, eight smaller tetrahedra can be formed. The present invention determines the sampling point by a set of criteria based on the barycentric coordinates. In which subtetrahedron does it fall? Figure 4 The algorithm shown, if Any value in , such as , exceeds 0.5, the sampling point is located at one of the four corners of the original tetrahedron; otherwise, if all values ​​are less than 0.5, it is located at one of the four central subtetrahedrons. If any value of is equal to 0.5, the sampling point is theoretically located on the shared face of two adjacent sub-tetrahedrons. In the method of the present invention, such a point can be assigned to any adjacent tetrahedron without affecting the result. If the sampling point is not located in the four corners of the original tetrahedron, the present invention checks and The range of the sub-tetrahedron can be further narrowed by whether the sum of these sums exceeds 0.5. According to whether these sums exceed 0.5, the present invention can accurately determine the sub-tetrahedron. After determining the sub-tetrahedron containing the sampling point, the present invention can access the positions of the sub-tetrahedron vertices. This enables the present invention to The layer retrieves the features of these vertices. Recall that The barycentric coordinates of the sampling points required for performing barycentric interpolation on the layer. The barycentric coordinates in a particular subtetrahedron are For each of the eight subconfigurations subdivided from the parent tetrahedron, a different Coefficient Matrix ,in , so that .

[0109] Since the subdivision pattern of the present invention is predetermined and uniformly applied to all tetrahedrons, each coefficient matrix are constant and specific to their configuration. These constant matrices accurately determine the coordinates of points in each successive level of the grid. Therefore, using the appropriate , can be obtained from The barycentric coordinates of the layer are calculated The barycentric coordinates of the layer ensure accurate and efficient interpolation of all levels in the grid hierarchy.

[0110] According to an embodiment of the present invention, the above-mentioned updating of the occupied grid using the training results of the first training stage, and using the updated occupied grid to delete the tetrahedral grid filling the unit cube space to obtain a rough tetrahedral mesh of the target three-dimensional object includes: updating the density of the occupied grid using the preliminary trained hash feature vector and the preliminary trained density prediction neural network obtained in the first training stage; based on a preset density threshold, using the occupied grid after the density update to delete the tetrahedral mesh filling the unit cube space to obtain a rough tetrahedral mesh of the target three-dimensional object.

[0111] Pruning unnecessary meshes by density of occupied mesh: In the first stage, the initial tetrahedralization of the space is generated using the QuarTet algorithm, resulting in a tetrahedral mesh that fills the unit cube space. The occupied mesh is used to determine whether the tetrahedrons within the mesh are retained. At the end of this stage, a rough tetrahedral mesh is obtained that roughly encloses the object. This has the advantage of being able to quickly build an initial structure that encloses complex geometry while minimizing memory consumption by retaining only the tetrahedrons that are actually occupied.

[0112] According to an embodiment of the present invention, the above-mentioned method of obtaining the animation sequence of the target three-dimensional object by performing a spatial mapping operation and performing animation rendering on the multi-level tetrahedral mesh obtained in the second training stage based on the position sequence of vertices in each frame of the rough tetrahedral mesh within the preset time includes: mapping the direction of the camera ray and the position of the sampling point from the deformation space to the standard space based on the position sequence of vertices in each frame within the preset time; performing hash mapping on each frame image of the target three-dimensional object using a trained hash feature vector based on the position sequence of vertices in each frame within the preset time, wherein the trained hash feature vector is obtained through the second training stage; performing density prediction on each frame image of the target three-dimensional object using a trained density prediction neural network based on the position sequence of vertices in each frame within the preset time, wherein the trained density prediction neural network is obtained through the second training stage; performing color prediction on each frame image of the target three-dimensional object using a trained color prediction neural network based on the position sequence of vertices in each frame within the preset time, wherein the trained color prediction neural network is obtained through the second training stage; completing animation rendering of the multi-level tetrahedral mesh within the preset time through the hash mapping operation, the density prediction operation and the color prediction operation to obtain the animation sequence of the target three-dimensional object.

[0113] The method of the present invention provides a neural radiation field representation method, which can generate high-quality rendering results from an input multi-view image data set and realize efficient manipulation and deformation of complex objects and scenes. The method achieves high-precision rendering effects by representing the scene to be processed as a multi-resolution tetrahedral mesh, and using recursive subdivision of tetrahedrons and feature encoding technology. First, an initial coarse tetrahedral mesh is constructed, and subdivided by connecting the midpoints of the edges of the tetrahedron to form a tetrahedral mesh structure of multiple levels. Then, a hash coding strategy is adopted to map the features of the tetrahedron vertices into a hash table, and the features of each sampling point are extracted using the barycentric interpolation method, and a complete feature representation is formed by connecting feature vectors of different levels. The barycentric coordinates of the sampling points in the subdivided tetrahedron are calculated by an iterative algorithm to achieve efficient feature interpolation. In addition, the method combines a two-stage training strategy, identifies and retains the key tetrahedrons of the target object in the first stage, and improves visual fidelity by subdivision in the second stage. Through the predefined tetrahedral subdivision mode, a high-resolution mesh is implicitly and dynamically generated, greatly reducing storage requirements while maintaining fine detail representation. The present invention provides efficient deformation capability for three-dimensional objects and scenes while ensuring high-quality rendering, and is suitable for three-dimensional modeling and animation production in the fields of movie special effects, virtual reality, etc.

[0114] The following is a specific embodiment and combined with the attached Figure 5 The reconstruction and rendering method of the above three-dimensional object provided by the present invention is further described in detail.

[0115] Figure 5 4 is a schematic diagram of sampling point mapping when a target three-dimensional object is deformed according to an embodiment of the present invention.

[0116] In the field of three-dimensional reconstruction and rendering, although traditional feature encoding methods based on regular grids are widely used, they still have significant limitations when dealing with complex geometries and dynamic scenes. On the one hand, these methods are difficult to accurately capture and restore subtle geometric details. On the other hand, they usually require a large amount of storage space to save high-resolution mesh data, resulting in excessive consumption of computing resources. In addition, traditional methods also show shortcomings when creating geometric proxies. It takes a long time to establish a tetrahedral mesh to fill the object based on the surface triangular mesh of the object. To this end, an embodiment of the present invention provides a tetrahedral mesh feature interpolation method based on multi-resolution hash coding, which achieves high-quality feature encoding and dynamic deformation of objects by dynamically generating high-resolution tetrahedral meshes and efficient center of gravity interpolation.

[0117] The three-dimensional object reconstruction and rendering method provided by the present invention mainly includes steps 1-10.

[0118] Step 1: Set the total number of tetrahedral mesh levels to For each level of the tetrahedral mesh, its vertices correspond to entries in their respective hash tables, and the vertex positions are converted to Mapped to feature vectors stored in a hash table:

[0119] (1)

[0120] Step 2: Perform ray-tetrahedron mesh intersection calculations, starting from the camera origin and moving along the camera ray direction to generate a series of sampling points. Using the results of the ray-mesh intersection calculation and the positions of the sampling points, determine which tetrahedron of the coarse mesh each point is located in, and calculate the corresponding centroid coordinates.

[0121] Step 3: Generate tetrahedral meshes at different levels from the initial mesh and dynamically infer these high-resolution meshes, i.e., perform a recursive subdivision process to generate tetrahedral meshes of increasing resolution at multiple levels. Starting with a coarse tetrahedral mesh, each tetrahedron is subdivided into eight smaller tetrahedrons by connecting the midpoints of its six edges. A specific subdivision mode is selected and applied uniformly to all tetrahedrons in the mesh.

[0122] Step 4: Identify the coordinates of the four vertices of the tetrahedron at the sampling point to access the features of these vertices and calculate the barycentric coordinates of the sampling point inside the tetrahedron to achieve barycentric interpolation.

[0123] Step 5: Use the barycentric coordinates of the parent tetrahedron's midpoint to determine Which of the eight subtetrahedrons in the layer contains a particular sampling point. The coordinates of the layer, calculated explicitly using analytical expressions The barycentric coordinates of the layer.

[0124] Step 6: Repeat steps 4 and 5 for a total of times, for At each level, the barycentric coordinates of the sampling point are used to calculate the weighted sum of the features of the tetrahedron vertices containing the point, and the features of the tetrahedral meshes from different levels are spliced ​​together to form a complete feature vector of the sampling point.

[0125] Step 7: Input the complete feature vector into the density prediction neural network and the color prediction neural network to obtain the density and color of the sampling point, and then obtain the color value corresponding to the light through the volume rendering formula, as shown in formula (3):

[0126] (3),

[0127] in, .

[0128] Step 8: Implement a two-stage training strategy. The present invention uses the QuarTet algorithm to generate the initial tetrahedronization of the space, and obtains a tetrahedral mesh that fills the unit cube space. In the first stage, the total number of levels of the tetrahedral mesh is Set it to a smaller number (such as 1 or 2), and execute steps 1-7 to prune unnecessary meshes by occupying the density of the mesh. Keep the mesh obtained in the first stage, set the total number of levels to a larger number (such as 5 or 6), and execute steps 1-7 again to obtain a more refined three-dimensional representation.

[0129] Step 9: Use the XPBD algorithm to perform physical simulation on the coarse grid obtained in the first stage of training to obtain the time The position sequence of all mesh vertices in .

[0130] Step 10: Perform animation rendering based on the position sequence. For each frame in the vertex position sequence, map the sampling points and ray directions from the deformation space to the standard space, such as Figure 5 As shown, from Deformed Space to Canonical Space, and then perform steps 1-7 for rendering to obtain an animation sequence.

[0131] Figure 6 is a block diagram of an electronic device suitable for implementing a method for reconstructing and rendering a three-dimensional object according to an embodiment of the present invention.

[0132] like Figure 6As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0133] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0134] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the input / output (I / O) interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0135] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0136] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0137] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0138] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0139] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for reconstructing and rendering a three-dimensional object, characterized in that: The method comprises: Generate a tetrahedral mesh filling a unit cube space, and iteratively perform hash mapping, implicit recursive subdivision, density prediction, and color prediction of a first training phase on the tetrahedral mesh filling the unit cube space using a multi-view image set of a target three-dimensional object; Using the training results of the first training stage to update the occupied grid, and using the updated occupied grid to delete the tetrahedral grid filling the unit cube space to obtain a rough tetrahedral grid of the target three-dimensional object; Iteratively performing hash mapping, implicit recursive subdivision, density prediction, and color prediction of a second training phase on the rough tetrahedral mesh using the multi-view image set; Performing physical simulation on the rough tetrahedral mesh of the target three-dimensional object to obtain a position sequence of vertices in the rough tetrahedral mesh within a preset time; Based on the position sequence of vertices in each frame of the rough tetrahedral mesh within the preset time, the animation sequence of the target three-dimensional object is obtained by performing a spatial mapping operation and performing animation rendering on the multi-level tetrahedral mesh obtained in the second training stage.

2. The method according to claim 1, characterized in that Generate a tetrahedral mesh filling a unit cube space, and iteratively perform hash mapping, implicit recursive subdivision, density prediction, and color prediction of a first training phase on the tetrahedral mesh filling the unit cube space using a multi-view image set of a target three-dimensional object, including: Generate a tetrahedral mesh that completely fills the unit cube space using a predefined tetrahedral mesh generation algorithm, and preprocess the images in the multi-view image set of the target three-dimensional object to obtain a preprocessed multi-view image set; Calculate the centroid coordinates of the sampling point in the tetrahedral mesh of the unit cube space using the intersection calculation result of the camera ray and the tetrahedral mesh of the unit cube space and the position information of the generated sampling point; Mapping positions of all vertices in the tetrahedral mesh filling the unit cube space to a hash feature vector using a hash function, wherein the hash feature vector is stored in a hash table corresponding to the hash function; Based on the initial subdivision threshold, performing an initial implicit recursive subdivision operation on the tetrahedral mesh filling the unit cube space to obtain an initially subdivided tetrahedral mesh, and mapping the positions of vertices of all levels in the initially subdivided tetrahedral mesh to the hash feature vector; Based on the barycentric coordinates of the sampling point in the tetrahedral mesh after the initial subdivision, obtaining feature vectors of different levels of the sampling point in the tetrahedral mesh after the initial subdivision by barycentric interpolation, and splicing the feature vectors of different levels of the sampling point in the tetrahedral mesh after the initial subdivision to obtain a multi-level initial feature vector; Using a density prediction neural network and a color prediction neural network to perform density prediction and color prediction on the multi-level initial feature vectors respectively, to obtain an initial density value and an initial color value of the sampling point, and based on the initial density value and the initial color value, using a predefined volume rendering formula to obtain an initial color value of the light at the sampling point; Inputting the pixel color values ​​of the images in the preprocessed multi-view image set and the initial color value of the light at the sampling point into a predefined loss function for processing to obtain an initial loss value; Based on the initial loss value, the hash feature vector, the density prediction neural network and the color prediction neural network are initially iteratively updated to obtain a preliminarily trained hash feature vector, a preliminarily trained density prediction neural network and a preliminarily trained color prediction neural network.

3. The method according to claim 2, characterized in that Calculating the centroid coordinates of the sampling point in the tetrahedron mesh of the filled unit cube space by using the intersection calculation result of the camera ray and the tetrahedron mesh of the filled unit cube space and the position information of the generated sampling point includes: Execute an intersection calculation operation between the camera ray and the tetrahedral mesh filling the unit cube space to obtain an intersection calculation result; Starting from the camera origin, proceed along the direction of the camera ray to generate sampling points; Based on the intersection calculation result, determining in which tetrahedron of the tetrahedron grid filling the unit cube space the sampling point is located; The centroid coordinates of the sampling point are calculated according to the vertex positions of the tetrahedron where the sampling point is located and the spatial position of the sampling point.

4. The method according to claim 2, characterized in that: Based on the barycentric coordinates of the sampling point in the tetrahedral mesh after the initial subdivision, the feature vectors of different levels of the sampling point in the tetrahedral mesh after the initial subdivision are obtained by barycentric interpolation, and the feature vectors of different levels of the sampling point in the tetrahedral mesh after the initial subdivision are spliced ​​to obtain the multi-level initial feature vectors, including: Acquire hash feature vectors corresponding to the four vertices in the tetrahedron mesh filling the unit cube space by identifying the coordinates of the four vertices of the sampling point in the tetrahedron mesh filling the unit cube space; Calculate the centroid coordinates of the sampling point in the tetrahedron mesh filling the unit cube space based on the hash feature vectors corresponding to the four vertices in the tetrahedron mesh filling the unit cube space, and calculate the feature vector of the sampling point on the rough tetrahedron mesh by feature vector weighting; Determine a target sub-tetrahedral mesh where the sampling point is located based on the barycentric coordinates of the sampling point in the tetrahedral mesh of the filled unit cube space, wherein the target sub-tetrahedral mesh is located in the initially subdivided tetrahedral mesh; Based on the coordinates of the four vertices in the target sub-tetrahedron mesh and the corresponding hash feature vectors, the centroid coordinates of the sampling point in the target sub-tetrahedron mesh are obtained, and the feature vector of the sampling point on the target sub-tetrahedron mesh is calculated by feature vector weighting; Determine the number of iterations based on the initial subdivision threshold, iteratively perform barycentric coordinate calculation operations and feature vector calculation operations in a barycentric interpolation manner, and obtain feature vectors of different levels of the sampling point in the tetrahedral mesh after the initial subdivision; Performing a vector concatenation operation on the feature vectors of different levels of the sampling point in the tetrahedral mesh after the initial subdivision, to obtain a multi-level initial feature vector of the sampling point.

5. The method according to claim 2, characterized in that: Based on the initial subdivision threshold, an initial implicit recursive subdivision operation is performed on the tetrahedral mesh filling the unit cube space, and the tetrahedral mesh after the initial subdivision is obtained, including: A subdivision process is completed by connecting the midpoints of all edges in the tetrahedral mesh filling the unit cube space, and the same subdivision process is performed on the obtained sub-tetrahedrons until the number of subdivision processes meets the requirement of the initial subdivision threshold, thereby obtaining the tetrahedral mesh after the initial subdivision; Using the coordinates of four vertices in the tetrahedron mesh of the filled unit cube space, weighted summation is performed on the centroid coordinates of the sampling point in the tetrahedron mesh of the filled unit cube space to obtain the coordinates of the sampling point in the tetrahedron mesh of the filled unit cube space; The centroid coordinates of the sampling point in the tetrahedral mesh after the initial subdivision are judged based on the coordinates of the sampling point in the tetrahedral mesh of the filled unit cube space to obtain the target sub-tetrahedral mesh where the sampling point is located.

6. The method according to claim 1, characterized in that Using the training results of the first training stage to update the occupied grid, and using the updated occupied grid to delete the tetrahedral grid filling the unit cube space to obtain the rough tetrahedral grid of the target three-dimensional object includes: Using the preliminarily trained hash feature vector and the preliminarily trained density prediction neural network obtained in the first training stage to update the density of the occupied grid; Based on a preset density threshold, the tetrahedral mesh filling the unit cubic space is deleted using the occupied mesh after density update to obtain a coarse tetrahedral mesh of the target three-dimensional object.

7. The method according to claim 1, characterized in that Iteratively performing hash mapping, implicit recursive subdivision, density prediction, and color prediction of a second training phase on the coarse tetrahedral mesh using the multi-view image set includes: Preprocessing the images in the multi-view image set to obtain a preprocessed multi-view image set, and mapping the positions of vertices of all levels in the rough tetrahedral mesh to the preliminarily trained hash feature vector using the hash function; Based on an intermediate subdivision threshold, the rough tetrahedral mesh is implicitly recursively subdivided to obtain an intermediate multi-level tetrahedral mesh, and the positions of vertices of all levels in the intermediate multi-level tetrahedral mesh are mapped to the initially trained hash feature vector; Obtaining feature vectors of different levels of the sampling point in the intermediate multi-level tetrahedral mesh by means of barycentric interpolation, and concatenating feature vectors of different levels of the sampling point in the intermediate multi-level tetrahedral mesh to obtain a multi-level intermediate feature vector; Using the preliminarily trained density prediction neural network and the preliminarily trained color prediction neural network to respectively perform density prediction and color prediction on the multi-level intermediate feature vectors to obtain intermediate density values ​​and intermediate color values ​​of the sampling points; Based on the intermediate density value and the intermediate color value, a predefined volume rendering formula is used to obtain an intermediate color value of the light where the sampling point is located, and a predefined loss function is used to process the pixel color values ​​of the images in the multi-view image set and the intermediate color value of the light where the sampling point is located to obtain an intermediate loss value; Based on the intermediate loss value, the initially trained hash feature vector, the initially trained density prediction neural network and the initially trained color prediction neural network are intermediately iterated and updated to obtain the trained hash feature vector, the trained density prediction neural network and the trained color prediction neural network.

8. The method according to claim 1, characterized in that Based on the position sequence of vertices in each frame of the rough tetrahedral mesh within the preset time, by performing a spatial mapping operation and performing animation rendering on the multi-level tetrahedral mesh obtained in the second training stage, the animation sequence of the target three-dimensional object is obtained, including: Based on the position sequence of vertices in each frame within the preset time, mapping the direction of the camera ray and the position of the sampling point from the deformation space to the standard space; Based on the position sequence of vertices in each frame within the preset time, hash mapping is performed on each frame image of the target three-dimensional object using the trained hash feature vector, wherein the trained hash feature vector is obtained through the second training stage; Based on the position sequence of vertices in each frame within the preset time, using the trained density prediction neural network to perform density prediction on each frame image of the target three-dimensional object, wherein the trained density prediction neural network is obtained through the second training stage; Based on the position sequence of vertices in each frame within the preset time, using the trained color prediction neural network to perform color prediction on each frame image of the target three-dimensional object, wherein the trained color prediction neural network is obtained through the second training stage; The animation rendering of the multi-level tetrahedral mesh within the preset time is completed through hash mapping operation, density prediction operation and color prediction operation to obtain the animation sequence of the target three-dimensional object.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A computer program or instruction is stored thereon, wherein the computer program or instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images